Senior Director of Data Engineering

Senior Director of Data Engineering

Overview

At Penn Foster Group, our purpose is to open doors to brighter futures by helping learners gain the skills they need to grow their careers. We are seeking a highly experienced and dynamic Senior Director of Data Engineering to lead and manage our data engineering team. The ideal candidate will have over 10 years of experience in the field, with a minimum of 5 years as a data architect. This role requires a deep understanding of Databricks, Azure, AI/MLops, strong leadership skills, and proficiency in managing teams of data analysts and data engineers. This leader will also oversee data governance, data quality, observability, and cost optimization across the data platform to ensure reliability and scalability.

Key Responsibilities

• Lead, mentor, and develop a team of approximately 15 professionals, including 4 direct reports comprised of high-quality analysts, data engineering, and Tableau/Power BI developers.
• Oversee the design and implementation of data architecture strategies.
• Partner with engineering teams to incorporate AI/ML models into our core products.
• Ensure the effective operationalization of AI/ML models.
• Oversee data quality frameworks and enforce medallion architecture principles.
• Manage onshore and offshore staffing models to optimize productivity and efficiency.
• Implement and maintain event streaming architectures, preferably in Azure.
• Run and support agile teams, ensuring adherence to agile methodologies and best practices.
• Set and carry out engineering standards, including coding practices, data management, and security protocols, ensuring compliance with regulatory requirements.
• Introduce and scale data observability practices to monitor pipeline health, lineage, and performance.
• Confirmed proficiency in Python, PySpark, and SQL, with a deep understanding of data processing frameworks and large-scale distributed systems.
• Collaborate with senior leadership to align data strategies and team priorities with business objectives.
• Ensure strong data governance policies and compliance across the data ecosystem.
• Manage and optimize the data platform budget, including Databricks and Azure compute/storage costs.

Qualifications

• 10 plus years of experience in data engineering or related fields, with a minimum of 5 years as a data architect.
• 6-10 years of experience running agile teams.
• BA/BS in Computer Science, Math, Physics, or other technical fields
• Proven experience with Databricks, preferably on Azure managing its implementation and cost.
• Familiarity with SAP Business Objects is a plus
• Extensive experience managing teams of data engineers, data analysts and Tableau/Power BI engineers.
• Strong leadership skills with the ability to mentor and develop team members.
• Experience with onshore and offshore staffing models.
• Proven experience managing and selecting vendors
• Proficiency in designing and implementing event streaming architectures, especially in Azure.
• Experience in operationalizing machine learning models, including live and batch inferences with integrations into production systems.

Preferred Skills

• Excellent communication and interpersonal skills.
• Ability to work in a fast-paced and dynamic environment.
• Strong problem-solving and analytical skills.
• Experience with other data platforms and tools is a plus.

Location:

Virtual, with ability to travel regularly to meet with customers, partners, and internal teams. Teams are primarily virtual, with hubs in Chandler, AZ; Boston, MA; and Scranton, PA.

Equal Employment Opportunity:

At Penn Foster Group we are proud to be an Equal Employment Opportunity employer. We are committed to creating a work environment that embraces and celebrates diversity. We do not discriminate based on race, color, religion, sex (including pregnancy and gender identity), national origin, political affiliation, sexual orientation, marital status, disability, genetic information, age, membership in an employee organization, retaliation, parental status, military service, or other status protected under federal, state, or local law.

What We Offer:

We offer a competitive base salary and incentive compensation, plus a robust benefits package that includes medical, dental, vision, flexible spending, generous paid time off, sponsored volunteer opportunities, a 401K with a company match, plus free access to all our online programs.