Open to opportunities

neha sharma

@nehasharma9

Message

Senior Data Engineer with expertise in Big Data technologies.

United Kingdom

Message

What I'm looking for

I am looking for a challenging role in a collaborative environment that fosters innovation and growth.

I am a Senior Data Engineer with over 8 years of experience specializing in Big Data technologies such as Spark, Databricks, and Hadoop. My expertise lies in designing and developing data ingestion, curation, and consumption layers in data lakes. I have a proven track record of optimizing data pipelines and delivering insights that drive business decisions.

Throughout my career, I have successfully developed over 25 data pipelines using Spark, Scala, Python, and Hive in various projects, including HCSC. I have led significant initiatives such as migrating legacy data lakes to Azure Cloud and implementing robust ETL pipelines using Azure Data Factory. My technical leadership has resulted in improved data quality and performance, achieving a 25% reduction in execution time for critical reports.

I am passionate about mentoring junior team members and continuously seeking innovative solutions to enhance data processing efficiency. My goal is to leverage my skills in a dynamic environment where I can contribute to impactful data-driven projects.

Experience

Work history, roles, and key accomplishments

Current

Azure Data Engineer

Current

Accenture

Mar 2024 - Present (1 year 4 months)

Developed data ingestion pipelines from various source systems to target systems using Databricks medallion architecture. Led the design for the migration of Java/Gemfire Application to Databricks/Azure, handling 20M records of historical data and 10M+ daily delta data.

Databricks Pyspark Azure Azure SQL ADLS Data Ingestion Data Migration Data Pipelines

Data Engineer

Accenture

Jun 2021 - Mar 2024 (2 years 9 months)

Developed multiple zones – Raw/Curated/Gold and Consumption in the datalake where multiple analytical reports are generated. Enhanced the Generic Spark Scala program to process complex nested Claims JSON Files.

Spark Scala Hive Talend Data Pipelines Data Quality Performance Optimization JSON

Data Engineer

Accenture

Jan 2023 - Mar 2024 (1 year 2 months)

Led the end-to-end development migrating On-Prem Cloudera Data Lake to Azure Cloud. Migrated data ingestion pipelines to a Streaming Framework leveraging Confluent Kafka and Spark Streaming.

Cloudera Azure Cloud Confluent Databricks Delta Lake Azure Synapse ETL

Hadoop Developer

Sensiple Software Solutions

Apr 2017 - May 2018 (1 year 1 month)

Created and worked with Sqoop jobs with incremental load. Implemented Partitioning, Dynamic partitioning and Bucketing in Hive using internal and external table for more efficient data.

Sqoop Hive Partitioning Bucketing Zena Software Testing

Web Developer

Red Ballons Advertiser

Aug 2013 - Aug 2016 (3 years)

Developed and deployed new features to facilitate related procedures and tools. Provided immediate fixation on issues and actively involved in development activities.

Web Development

Education

Degrees, certifications, and relevant coursework

Unknown University

Bachelor of Technology, Computer Science and Engineering

Completed a Bachelor of Technology in Computer Science and Engineering. This program provided a strong foundation in computer science principles and practices.