Open to opportunities

043 Eshan Shrivastava

@eshanshrivastava

Senior Data Engineer Skills : Python , SQL, PySpark,Apache Spark, Databricks, Airflow,AWS Glue, Lambda, Redshift, Athena, S3, EMR, Kinesis,CloudWatchD

India

Message

What I'm looking for

I’m looking to build and optimize secure, production-grade data pipelines and lakehouse platforms—owning architecture, performance tuning, and data governance end-to-end using Python/PySpark on AWS and Databricks.

Data engineering professional with over 5 years of experience in creating and optimizing large-scale data pipelines. Key achievements include achieving an 85% reduction in Spark job runtime and a 35% reduction in monthly pipeline failure incidents. Languages: Python, SQL, PySpark Frameworks/Tools: Apache Spark, Databricks, Airflow AWS Services: Glue, Lambda, Redshift, Athena, S3, EMR, Kinesis,CloudWatch Databases: DynamoDB (NoSQL), Postgres SQL Versioning: Git CI/CD: Jenkins, UCD Gen AI Tools - Github Co-Pilot

Experience

Work history, roles, and key accomplishments

Data Engineering Consultant

Deloitte USI

Aug 2024 - Jun 2026 (1 year 10 months)

Engineered a batch ELT pipeline using Medallion architecture to ingest heterogeneous sources into a data warehouse, enabling customer analytics via S3, AWS Lambda, Airflow, Athena, and Redshift. Reduced Spark job runtime by 85% through refactoring, partitioning changes, and Spark AQE, and improved Delta Lake query latency by 20% using Z-ORDER, compaction, and liquid clustering.

Python Pyspark AWS Lambda Apache Airflow Amazon S3 Amazon Athena Amazon Redshift Delta Lake Data Governance Medallion Architecture

System Engineer (Data)

Tata Consultancy Services

Jan 2021 - Apr 2024 (3 years 3 months)

Automated data pipeline execution with Amazon S3 event notifications and AWS Lambda, reducing monthly failure incidents by 35%. Built AWS Glue ETL pipelines to convert CSV data into standardized Parquet for faster Spark processing, and improved operational reliability by scheduling/orchestrating pipelines in Airflow and tracking failures with root-cause analysis.