Lead Data Engineer

Title:Lead Data Engineer (6 month contract)

Location: Remote in the UK

Division: AI Practice

Reports to: Chief Architect - Data and Insights

Clearance:Active Security Clearance

ABOUT US

Agilisys is at the forefront of digital transformation and innovation in the public services sector. With over two decades of experience, we have established ourselves as a trusted partner for governments, local authorities, and organizations nationwide. Our mission is to empower our clients to deliver exceptional public services by harnessing the full potential of technology and data.

  

OUR VALUES

Partnership: we become one team and family with organisations, helping them to navigate change and stay agile.

Integrity: our people really care, going beyond the brief to make change happen for organisations and citizens.

Innovation: we bring together the right technologies and services to design solutions that work.

Passion: we are passionate about - and dedicated to - public services and improving people’s lives.

THE ROLE

We are seeking a Lead Data Engineerwith active Security Clearance to be the technical cornerstone of our data engineering team, delivering a high-profile programme of work for a major Public Sector client. This is a hands-on, deeply technical role where you’ll lead from the front — setting the technical direction, shaping architectural decisions, and building modern data solutions using Microsoft Fabric and the Azure data platform.

You’ll play a key role in defining how we deliver data engineering at Agilisys — not just for this engagement, but as part of the ongoing evolution of our Data AI Practice. Working alongside architects, analysts, and delivery teams, you’ll help us build scalable, secure, and future-ready data platforms and help define best practices for modern data engineering across the organisation.

Key responsibilities

You will be responsible for shaping our data architecture and advising on industry best practices, while actively building and optimising our clients’ data platforms. Your mission is to mentor a talented team, champion a culture of automation and DevOps, and work collaboratively to design and deliver secure, scalable data solutions that meet our mission-critical needs.

• Data Platform Strategy and Technical Leadership

• Lead the design and evolution of scalable, modern data platforms using Microsoft Fabric, Azure Synapse Analytics, and Azure Data Factory, with the ability to bridge legacy systems and cloud-native architectures.

• Provide strategic technical direction, shaping engineering standards, reusable patterns, and architectural decisions that support performance, integrity, and long-term platform sustainability across hybrid environments.

• Work across modern cloud-native data platforms, with a strong understanding of distributed data processing, storage, and compute services within the Azure ecosystem.

• Engineering Delivery Development Best Practice

• Lead the hands-on development of end-to-end data solutions using Microsoft Fabric components such as Lakehouses, Warehouses, Notebooks, and Dataflows.

• Design scalable data models and ingestion frameworks, supporting both real-time and batch workloads using best-practice techniques (e.g. 3NF, Kimball).

• Enable data consumption through well-structured APIs, semantic models, and data products, with a strong understanding of Power BI integration within Fabric.

• Drive engineering excellence through robust development practices including version control, automated testing, peer reviews, and documentation.

• Optimise performance and cost-efficiency, tuning queries, storage formats (e.g. Delta, Parquet), and compute resources across Fabric and Azure workloads.

• Expertise in ETL/ELT and Data Processing

We're looking for an expert in building dynamic and efficient data pipelines. This includes demonstrable experience in:

• Designing and implementing scalable ETL/ELT pipelines for both batch and real-time data processing and orchestrating complex data workflows using tools like Azure Data Factory, Fabric Data Pipelines, or equivalent.

• Apply foundational data engineering principles to design scalable and maintainable solutions, including schema design, indexing, and query optimisation, while implementing effective strategies for evolving data sources and historical tracking through CDC and SCD techniques

• Writing highly optimised transformation logic using Python, SQL, and Spark for large-scale data workloads.

• Platform Automation DevOps

• Lead CI/CD implementation for Microsoft Fabric components, integrating Git for version control and automating multi-environment deployments using Fabric Deployment Pipelines.

• Embed automated testing across pipelines and transformations, including unit tests, data quality checks, and schema validation.

• Manage secure configuration using Fabric Environments and Azure Key Vault for secrets and environment-specific parameters.

• Data Security, Governance Observability

• Ensure data solutions are secure by design, implementing appropriate access controls, encryption, and authentication mechanisms across all layers of the data platform.

• Embed governance and compliance principles into engineering workflows, aligning with public sector standards and regulations (e.g. GDPR, UK DPA).

• Leverage Microsoft Purview (and Fabric-native governance features) to manage metadata, enforce data classification, and support data discovery across the platform.

• Establish observability and data quality practices, including data profiling, validation, lineage,and monitoring of pipeline health and freshness, while promoting stewardship across teams.

Opportunity

This role is an excellent platform for experienced professionals to take their career to the next level. You will be operating within our Data AI Practise and join a company committed to delivering excellence and innovation within the Data, AI and Cloud space with a focus on the Microsoft Cloud stack (specifically Fabric). This will give you a great opportunity to grow your skills in these core technology areas within an exciting and high-profile customer environment.

 

ABOUT YOU

The ideal candidate will have a track-record in delivering results while embracing change and uncertainty. Excellent stakeholder management experience is essential to being successful in this role.

The Ideal Candidate

• Has 3+ years' experience as a Senior or Lead Data Engineer, delivering large-scale production-grade data platforms and with active Security Clearance.

• Leads by example—writing code, reviewing PRs, and mentoring engineers.

• Comfortable in fast-paced environments with shifting priorities.

• Experienced with large-scale data systems across cloud and on-prem setups.

• Has worked in public sector or regulated environments and understands compliance needs.

• Comfortable working in an augmented team environment

• Thorough understanding of secure engineering and design with experience delivering Security by Design

• Excellent stakeholder management.

• Comprehensive understanding of Fabric and a keen follower of its evolving roadmap

• Passion for innovation and a commitment to shaping the future of public services.

WHAT WE CAN OFFER YOU:

This role will offer exposure to the right mix of challenges, within a culture that promotes continuous learning and development.

Benefits include:

• Enhanced Pension Scheme

• Life Assurance

• Access to exclusive discounts and offers through the company’s “Perks at Work” scheme

• 25 days annual leave (with the option to buy more)

PROCESS

Simply submit your CV.

By submitting your CV, you understand that we have a legitimate interest to use your personal data for the purposes of assessing your eligibility for this role. This means that we may use your personal data to contact you to discuss your CV or arrange an interview or transfer your CV to the hiring manager(s) of the role you have applied for. You can ask us at any time to remove your CV from our database by emailing talentacquisition@agilisys.co.uk – but please note that this means we will no longer consider you for the role you have applied for.

We have a rigorous recruitment process, which we use for all our roles to ensure we attract the very best talent.

Individuals seeking employment at Agilisys must note that we see diversity as something that creates a better workplace and delivers better outcomes. As such, we are keen to maximise the diversity of our workforce and actively encourage applications from all. We encourage diversity through perspective, background, identity, and thought whilst also fostering an environment where everyone can express themselves regardless of your race, religion, sex, gender, colour, national origin, disability, or any other applicable legally protected characteristic. We are committed to continuing to nurture an inclusive environment and building a diverse workforce.