Data Engineer – Azure Databricks & Fabric

About the team:

Capco’s Data Team helps our clients transform every aspect of their business. We are highly skilled at formulating data strategy, defining business and technology initiatives across the data management lifecycle, and aligning multi-year strategic roadmaps with client’s business goals. As digital technologies advance and regulations tighten, today’s consumers – and, therefore, today’s businesses – are becoming more aware of the importance of good quality data. We work to establish holistic ways to effectively manage data through the modern data supply chain and facilitate consumption through analytics, modelling, AI, machine learning, dashboarding, and reporting.

About the Job:

The Data Engineer will serve as the lead technical specialist for designing and implementing data science and advanced analytics capabilities on Microsoft Azure Fabric and Databricks.This role focuses on data processing, identity resolution, entity linking, and data warehouse development that enable organizations to unify fragmented data across multiple systems into a trusted, governed, and analytics-ready model.The ideal candidate combines deep hands-on expertise in Databricks engineering, data modeling, and applied data science, with the ability to build scalable, production-grade data solutions in collaboration with business, engineering, and analytics teams.

What You’ll Get to Do:

Data Platform Warehouse Development

• Design and develop data lakehouse and warehouse structures within Azure Databricks and Fabric environments.

• Build ETL and ELT pipelines to extract, cleanse, normalize, and enrich data from CRM, ERP, LMS, and financial systems .

• Develop reusable data transformation and validation frameworks leveraging PySpark, SQL, and Delta Live Tables.

• Support the operationalization of the central data warehouse using Azure SQL and Fabric Data Warehouse.

Identity Resolution Data Linking

• Implement entity resolution models to unify customer, member, or participant records across systems using deterministic and probabilistic matching techniques.

• Design and deploy matching algorithms utilizing Databricks MLflow, PySpark, and Azure Machine Learning for cross-system deduplication and linkage.

• Collaborate with architects to define unique identifiers, external keys, and golden record frameworks for enterprise data integration.

• Monitor and continuously refine data matching accuracy, precision, and recall metrics.

Data Processing Automation

• Develop and schedule data ingestion pipelines in Azure Fabric and Databricks for recurring Excel, CSV, and structured PDF sources using Power Automate, Form Recognizer, and Fabric Dataflows.

• Apply data quality and validation rules to flag incomplete, inconsistent, or stale records.

• Build and automate data lineage, change tracking (CDC), and error-handling workflows.

• Support performance tuning and scalability for high-volume processing environments.

Analytics Modeling Support

• Provide curated and feature-engineered datasets for Power BI dashboards and machine learning use cases.

• Partner with data analysts to define KPIs and enable cross-system reporting and predictive insights.

• Develop scripts and notebooks to support exploratory data analysis (EDA) and visualization in Databricks.

What You’ll Bring with You:

• BA in Data Science, Computer Science, Applied Mathematics, or related discipline.

• 5+ years of experience in data engineering and applied data science on Azure platforms.

• 3+ years building and managing pipelines in Azure Databricks (PySpark, Delta Lake, MLflow).

• 2+ years hands-on experience with Microsoft Fabric (Data Factory, Dataflow Gen2, Data Warehouse).

• Power BI integration and data modeling

• Entity resolution and master data management (MDM) methods

• Statistical modeling, clustering, and record linkage algorithms

• Data governance, lineage tracking, and compliance (PII, HIPAA, etc)

• Proven track record implementing identity resolution and entity linking frameworks.

• Strong background in SQL, Python, and large-scale data processing for analytics.

Preferred Certifications:

• Microsoft Certified: Fabric Analytics Engineer Associate

• Microsoft Certified: Azure Data Scientist Associate

• Databricks Certified Machine Learning Professional

• Azure Data Engineer Associate

Why Capco?

A career at Capco is a chance to help reshape the competitive landscape in financial services. We launch new banks, transform existing ones, and help our clients navigate complex change. As consultants, we work on the front-end business design all the way through to technology implementation.

We are the largest Financial Services focused consultancy in the world, serving everyone from global banks to emerging FinTechs, from strategy through digital transformation, design, business consulting, data and analytics, cyber, cloud, technology architecture, and engineering.

Capco is a young and growing firm. We maintain an entrepreneurial spirit and growth mindset, and have minimal bureaucracy. We have no internal silos that get in the way of your career opportunities or ability to focus on our clients and make a difference to the business. We offer the opportunity for everyone to learn rapidly, take on tough challenges, and get promoted quickly. We take pride in our creative, collaborative, diverse, and inclusive culture, where everyone can #BYAW.

We offer highly competitive benefits, including medical, dental and vision insurance, a 401(k) plan, tuition reimbursement, and a work culture focused on innovation and creation of lasting value for our clients and employees.

Ready to take the Next Step?

If this sounds like you, we would love to hear from you. This is an opportunity to make a difference and contribute to a highly successful company with a significant growth trajectory.