Position Summary:
The Lead Data Engineer is responsible for the technical direction, reliability, and scalability of Penn Foster Group’s data platform. This role provides hands-on technical leadership across data ingestion, transformation, and analytics enablement, while setting engineering standards and guiding the work of other data engineers.
The Lead Data Engineer partners closely with BI, product, platform, and MLOps teams to ensure data systems are trusted, well-governed, and aligned to business outcomes. This role operates as a player-coach, combining deep hands-on technical execution with technical leadership, mentoring, and architectural ownership in a highly visible and strategic data organization.
Essential Job Functions:
- Lead and coordinate the day-to-day technical work of data engineers, providing direction, prioritization, and technical guidance.
- Set clear technical expectations and hold the team accountable to engineering standards and delivery commitments.
- Own the technical architecture and engineering standards for the data platform, with a primary focus on Databricks in Azure.
- Lead the design, development, and operation of scalable, reliable data pipelines using SQL, Python, Spark, and Databricks.
- Establish and enforce best practices for data modeling, data quality, testing, performance optimization, and observability.
- Provide hands-on technical leadership through code reviews, design reviews, and mentoring of data engineers.
- Guide and influence the work of other data engineers, setting clear technical direction and expectations.
- Partner with BI and analytics teams to enable trusted datasets, performant semantic models, and governed self-service analytics.
- Collaborate with platform, DevOps, and MLOps teams on infrastructure, security, deployment patterns, and operational readiness.
- Balance hands-on delivery with technical roadmap planning and architectural decision-making.
- Act as the escalation point for complex data issues, production incidents, and cross-team technical challenges.
- Contribute to the evolution of the data operating model, including governance, access control, and self-service enablement
Knowledge, Skills, Abilities:
- 7+ years of experience in data engineering or platform engineering.
- 3+ years of experience leading or mentoring data engineers in a technical lead, player-coach, or team lead capacity.
- Demonstrated ability to guide and influence engineers without formal people-management authority.
- Deep hands-on expertise with Databricks, including Spark (PySpark/Scala), Delta Lake, Unity Catalog, cluster policies, jobs, notebooks, and performance optimization.
- Strong expertise with Azure, including ADLS Gen2, Databricks on Azure, networking, identity and permissions (AAD/RBAC), and integration with other Azure data services.
- Advanced proficiency in SQL and Python, with experience building reusable libraries and frameworks for common data engineering patterns.
- Strong experience designing and operating production-grade data pipelines at scale.
- Proficiency in data modeling for warehouse and lakehouse architectures, including star and snowflake schemas, slowly changing dimensions, snapshot versus event-based models, and medallion patterns.
- Experience with orchestration and scheduling, including Databricks Jobs or similar workflow tools, and implementing robust retry mechanisms, alerting, and SLAs.
- Solid understanding of data governance concepts, including access control, PII handling, compliance, data quality checks, and data lineage.
- Experience with CI/CD practices, version control, and infrastructure-as-code concepts.
- Familiarity with cloud platforms, with Azure strongly preferred.
- Ability to lead through influence, setting technical direction without formal people-management authority.
- Strong communication skills, with the ability to work effectively with both technical and business partners.
- Experience supporting BI tools (Tableau, Power BI) and semantic layers is a plus.
- MLOps or advanced analytics platform experience is a plus.years of experience in data engineering or platform engineering.
About Us: At Penn Foster Group, we are transforming online learning to help learners by bringing together Penn Foster, CareerStep, Ashworth College, James Madison High School, the New York Institute of Photography, the New York Institute of Art and Design, and other education platforms. Together, we create an accelerated path to greater economic mobility through real-world skills and knowledge that enable learners to achieve long-term success in the workplaces of the future. Our history dates back to 1890 when our founder, Thomas Foster, pioneered distance education by offering training by mail for coal miners to get the necessary skills for safer jobs. Today, with the partners who use our education and training programs, we continue that mission of providing accessible training and education for in-demand skills and are building a workforce that’s prepared for the future job market.
Equal Employment Opportunity: We strive toward Diversity, Equity, and Inclusion at Penn Foster Group by intentionally building diverse teams – in identities, lived experiences, and ideas to create a culture where people feel connected to each other and have a sense of belonging. We value diversity, equity, and inclusion because it is the foundation that enables us to achieve what we set out to do as an organization – from maximizing the number of learners who can reach their goals while giving them the kinds of experiences we want them to have, to becoming the type of company we want to work in.
What We Offer: We offer a robust benefits package that includes medical, dental, vision, flexible spending, generous paid time off, sponsored volunteer opportunities, a 401K with a company match, and free access to our online programs.
United States
