Looking for a job

Shubham Paliwal

@shubhamp2494

I am a Senior Cloud Data Engineer with 10 years of expertise building serverless AWS platforms, event-driven pipelines, and scalable data lakes.

India

Message

What I'm looking for

I seek a Senior Cloud Data Engineer role focused on scalable, event-driven platforms and serverless data lakes. I aim to use my PySpark, AWS, and Databricks expertise to transform massive data streams into governed analytics assets. I want to drive infrastructure automation, enforce strict data quality, and mentor agile engineering teams.

I am a Senior Cloud Data Engineer with a decade of expertise building scalable, event-driven platforms for Fortune 500 enterprises. Specializing in AWS, Databricks, and PySpark, I transform massive, complex data streams into governed analytics assets.

Key Career Highlights:

Scale: Engineered an event-driven Kinesis backend for McKinsey processing 15,000 events/sec and an AWS platform for Delta Air Lines handling 500 GB/day.
Governance: Built a Glue Data Quality framework reducing anomalies by 85%, and implemented Lake Formation compliance for Ameriprise.
Automation: Cut manual infrastructure provisioning times by 70% using Terraform and AWS CDK.

Currently a Senior Technical Consultant at Xebia, I hold AWS and Databricks certifications and lead agile pods to architect highly available, high-performance data ecosystems.

Experience

Work history, roles, and key accomplishments

Current

Senior Technical Consultant

Current

Xebia IT Architects Pvt Ltd

Oct 2020 - Present (5 years 10 months)

Lead data architecture and pipeline engineering for Fortune 500 enterprise clients, specializing in AWS/Azure migrations, PySpark distributed processing, and automated data quality frameworks. Provision cloud infrastructure and enforce strict IAM role governance utilizing AWS CDK and Terraform, standardizing deployments and reducing infrastructure provisioning time by 60%.

AWS Azure PySpark Terraform AWS CDK IAM Data Architecture Data Quality

Senior Python Developer

PureSoftware Pvt. Ltd.

Apr 2020 - Oct 2020 (6 months)

Architected an organizational hierarchy data backend using Python and MySQL, optimizing relational schemas to manage complex structures for 40,000+ employees. Engineered automated Python batch ETL pipelines for HR data, improving downstream report query performance by 40%.

Python MySQL ETL Data Modeling Batch Processing

Senior Software Engineer

Virtual Employee Pvt. Ltd.

Sep 2018 - Apr 2020 (1 year 7 months)

Engineered scalable REST APIs and PostgreSQL data ingestion pipelines for the healthcare domain, translating intricate clinical requirements into robust integrations. Optimized complex SQL queries and data models, contributing to a 20% increase in overall platform responsiveness and faster patient analytics.

Python PostgreSQL REST APIs SQL Data Modeling

Software Developer

Skysoft IT Services Pvt. Ltd.

Aug 2016 - Aug 2018 (2 years)

Developed custom Python data extraction pipelines capturing high-volume pixel tracker events to optimize email marketing campaign analytics. Automated daily reporting dashboards by integrating SQL data extracts with business intelligence tools, saving 10 hours of manual data aggregation weekly.

Python SQL ETL Data Extraction Business Intelligence