Principal Data Engineer - Databricks

Team Sparq is committed to creating high-quality tech careers while helping clients accelerate their digital transformation journey. We are committed to being an inclusive workplace, maintaining a culture of equitable, diverse employment and advancement company-wide.

Why you will enjoy Mondays again:

• Opportunity to collaborate with a diverse group of colleagues in a fun, creative environment
• Progressive career journey and opportunity for advancement
• Continuous development through training, mentorship and certification programs
• Exposure to modern technologies across various industries in an agile environment
• Remote work

A Day in the Life:

• Design and implement scalable, high-performance data pipelines using tools such as Databricks, Snowflake, DBT, and the Microsoft Data Stack (Azure SQL Database, Azure Data Factory, Azure Synapse Analytics)
• Develop and maintain robust ETL processes using Matillion
• Orchestrate and automate workflows using Apache Airflow
• Integrate and manage data flows with Apache NiFi
• Leverage Databricks for data engineering tasks, advanced data processing, and large-scale data transformations
• Collaborate with cross-functional teams to gather requirements, define project scope, and develop data models
• Provide technical leadership and mentorship to junior data engineers, guiding them in best practices and coding standards
• Conduct code reviews to ensure quality, performance, and adherence to best practices
• Serve as a subject matter expert, staying abreast of industry trends and advancements in data engineering
• Troubleshoot and optimize existing data pipelines and processes to improve performance and reliability
• Create and maintain technical documentation, including architecture diagrams, design documents, and implementation guides

What it takes:

• Hands-on experience with Databricks, Snowflake, Matillion, Apache Airflow, and Apache NiFi
• Experience with DBT and the Microsoft Data Stack (Azure SQL Database, Azure Data Factory, Azure Synapse Analytics)
• Proficiency in SQL and Python for data manipulation and transformation
• Experience building scalable data pipelines and performing large-scale data processing using Databricks
• Familiarity with cloud platforms such as AWS, GCP, or Azure
• Strong problem-solving skills and the ability to troubleshoot and optimize data pipelines and processes
• Excellent communication and interpersonal skills, with the ability to effectively collaborate with both technical and non-technical stakeholders
• Prior experience in a leadership or mentorship role is highly desirable
• Demonstrated ability to thrive in a fast-paced, dynamic environment and manage multiple priorities effectively

Equal Employment Opportunity Policy: Sparq is proud to offer equal employment opportunity without regard to age, color, disability, gender, gender identity, genetic information, marital status, military status, national origin, race, religion, sexual orientation, veteran status, or any other legally protected characteristic.