Devashish User

I’m a Big Data Engineering professional with 6+ years of expertise in designing and delivering scalable ETL/ELT pipelines for high-volume data processing environments. I focus on building robust data architectures for efficient ingestion, transformation, and delivery across multiple systems, while maintaining data reliability, consistency, and availability for analytics and reporting.

In my recent role, I engineered a data validation framework using Python and Cerberus, and architected scalable pipelines with AWS Glue, PySpark, Lambda, and DynamoDB to support both real-time and batch processing. I’ve led end-to-end data reconciliation across AWS S3 and relational/non-relational databases (PostgreSQL, MySQL, DynamoDB), built serverless APIs with API Gateway and Lambda, and improved refresh performance by building dbt models and orchestrating them with MWAA. Earlier, I delivered distributed ETL for large HR datasets using Apache Spark (PySpark) and Hive, orchestrated pipelines with Apache Airflow, and implemented governance through an automated HDFS cleanup framework—while also integrating partner APIs for advertising campaign data ingestion.