Akash Soni

Designed and implemented a real-time CDC streaming pipeline using AWS MSK (Kafka) and AWS Glue to capture PostgreSQL changes into S3 for near real-time transaction analytics. Built ACID-compliant data lake upserts with Apache Hudi, added schema evolution with AWS Glue Schema Registry, and improved reliability through Airflow automation plus CloudWatch/Datadog monitoring.

Owned end-to-end AWS data pipeline architecture for large-scale BI workloads using AWS Glue ETL with Snowflake and Amazon Athena. Reduced end-to-end job processing time by ~30% via PySpark optimization and incremental/dynamic partitioning, and cut overall cloud costs by ~25% through S3 storage optimization, Snowflake warehouse right-sizing, and query tuning.

Migrated aircraft maintenance data from on-premise Hadoop to AWS S3 to improve scalability and reduce infrastructure overhead. Built PySpark transformation pipelines, deployed automated ETL CI/CD, and scheduled/monitored production workflows with Control-M.

Built AWS Glue-based ETL pipelines for healthcare analytics across structured and semi-structured datasets. Migrated Hadoop datasets to Snowflake using AWS Lambda and Glue, and automated workflows with Control-M and CI/CD while designing data models for downstream analytics.

Developed data pipelines for truck tracking and logistics analytics by integrating multiple sources into Hadoop and MySQL. Applied PySpark for real-time fleet data processing, built SQL/Python-based driver performance and incentive models, and optimized data warehouse query performance to improve report generation efficiency.

Earned a B.Tech in Computer Science & Engineering from Greater Noida Institute of Technology (AKTU).