Data Platform Engineer

About the Role

You’ll build, operate, and improve the “engine room” of our data platform—reusable ingestion frameworks, reliability systems, and pipelines across structured, semi-structured, and unstructured sources. You’ll own ETL/ELT workflows (Fivetran and AWS Glue) and develop stable, observable, and cost-efficient pipelines that power analytics and AI. Your work prevents incidents before they happen through great design, guardrails, and monitoring—translating technical reliability into a seamless experience for downstream customers.

Who you are

• Pragmatic builder who writes clear SQL/Python and leaves systems more reliable than you found them.
• Infrastructure-minded engineer comfortable with Python, IaC, orchestration, and Snowflake administration.
• Customer-centric and fundamentals-first; you translate reliability into a delightful data consumer experience.
• Velocity-oriented: you deliver “good today” increments, measure impact, and iterate toward excellence.
• Owner mindset: you proactively drive outcomes, communicate trade-offs, and follow through on commitments.
• Intellectually honest: you share clear, candid updates and invite feedback to improve systems.
• Security-first with sound judgment around PII/PHI, least privilege, and secret management.
• Collaborative partner who can explain technical topics to both engineers and non-technical stakeholders.
• Naturally curious and thrive in ambiguity, seeking to solve business problems with pragmatic solutions.
• A self-starter who takes ownership of outcomes and iterates quickly to add value fast.
• Always balancing excellence with velocity—knowing when good enough today beats perfect next week.

What you will do

• Develop reusable ingestion frameworks (Python/Airflow/AWS Glue) for APIs and unstructured sources beyond Fivetran, handling various data formats (JSON, Parquet, etc.).
• Own the end-to-end Medallion (bronze/silver/gold) architecture for core domains, ensuring robust lineage and metadata across diverse data sources.
• Implement data observability (native tests, alerts, lineage hooks); lead incident management and root-cause analysis (RCA) for data.
• Help standardize reusable “paved-road” patterns (e.g. CI templates, ingestion operators) to improve developer productivity.
• Prepare datasets for AI/LLM use cases (feature stores, embeddings/RAG prep).

Must Haves

• 3–5+ years of data engineering with strong Python and SQL; hands-on Spark/PySpark (ideally via AWS Glue).
• Deep experience in AWS (S3, IAM, Lambda, CloudWatch) running secure, observable data workloads.
• Proficiency operating Snowflake (warehouse sizing, RBAC, resource monitors, clustering/partitioning).
• Proven governance/security patterns: masking policies, row-level security, and auditability.
• Orchestration experience (Airflow/MWAA) and event/file/API ingestion beyond managed connectors.
• CI/CD for data with GitHub Actions; test/promotion workflows; secrets and PII handling.
• Solid grasp of Medallion architecture, dimensional modeling (star schema), and data quality frameworks.
• Ownership of incident management and RCA with measurable reduction in MTTR.

Nice to Have

• Familiarity with BI tools (Sigma, Looker, Tableau) for downstream troubleshooting and enablement.
• Experience with iPaaS/automation (e.g., Workato) and reverse ETL patterns.
• Data observability tools (e.g., Monte Carlo) and open standards like OpenLineage.
• IaC for data infrastructure (Terraform) and environment provisioning.
• Experience with Parquet/S3/Iceberg lakehouse patterns and event/data contracts.
• Fivetran administration and ELT operations.
• Experience contributing to paved-road standards (templates, operators, codegen).
• Exposure to feature stores or embeddings/RAG pipelines supporting AI products.