Principal Data Engineer

Job Summary:

Responsibilities:

• Lead and contribute to architecting and building a modern data stack that is scalable, maintainable, and highly-performant. This includes building frameworks, data pipelines, and other data infrastructure using a variety of raw data sources.
• Collaborate with data source providers both internal and external to set rules, processes, and checks that ensure data availability and integrity.
• Monitor and maintain the data pipelines and ETL processes to proactively remediate issues and preserve data availability.
• Support the migration from legacy orchestration systems and ETL, including migrating pipelines reading from internal databases and APIs, remapping data sources, and translating transformation logic.
• Ensure proper governance practices and effective documentation are implemented throughout the ETL migration process.
• Communicate with team members and convey results efficiently and clearly.

In the first week, expect to:

• Meet mentors that will help you during your onboarding month.
• Meet your team, managers, and other key stakeholders like Business Analysts.
• Start participating in company-wide training sessions.
• Set up your local environment and become familiar with our tech stack.

In the first month, expect to:

• Have a comfortable understanding of our data stack and systems.
• Have a clear understanding of the team’s data strategy and direction for the infrastructure.
• Develop an understanding of Toptal’s business and offerings.
• Develop an understanding of Toptal’s different processes and team structure, and identify and meet key technical and business stakeholders.
• Start contributing to legacy cleanup and migration efforts.

In the first three months, expect to:

• Have strong knowledge of Toptal’s business.
• Actively supply daily support to users, development, migration, and maintenance work.
• Take part in the on-going monitoring and maintenance of data pipelines and processes.
• Be ready to propose and implement improvements to Toptal’s processes and codebase.
• Deliver value in a regular cadence.
• Get comfortable in your daily work within your team.

In the first six months, expect to:

• Own technical initiatives on our team.
• Drive improvements to the codebase and processes.
• Contribute to planning and executing long-term initiatives inside your team.
• Be able to not only solve complex problems, but also consider multiple solutions, weigh them and decide on the best course of action.
• Exercising discretion and independent judgment, proactively identify technical debt and product areas that require attention or improvements and suggest improvements in our technology stack.

In the first year, expect to:

• Have a detailed understanding of Toptal’s business, collaboration rituals, processes, performance, and future work.
• Determine what your career path looks like at Toptal.
• Mentor Toptal’s new team members.

Qualifications and Job Requirements:

• Bachelor’s degree is required.
• 10+ years of experience working with data infrastructure–architecture, cloud data warehouses, data modeling, ETL tools and processes, and data ingestion techniques.
• Extensive experience working with Python, Pandas, and SQL
• Experience with Google Cloud Platform (including Google Cloud Storage and BigQuery), object-oriented programming, CI/CD, and ETL technologies such as Airflow, Luigi, Dagster, and CDC is preferred
• Outstanding English written and verbal communication skills.
• Be excited about collaborating daily with your team and other groups while working via a distributed model.
• Be eager to help your teammates, share your knowledge with them, and learn from them.
• Be open to receiving constructive feedback.
• Ability to work in a fast-paced, rapidly growing company and handle a wide variety of challenges, deadlines, and a diverse array of contacts.
• You must be a world-class individual contributor to thrive at Toptal. You will not be here just to tell other people what to do.