Chethan Kumar N
@chethankumarn
Senior Data Engineer building scalable lakehouse ETL on Databricks with PySpark.
IndiaWhat I'm looking for
I’m a Senior Data Engineer with 5+ years of experience designing and delivering scalable, distributed data platforms across the Pharmaceutical, Clinical Trials, RWE, and Commercial domains. I’m an expert in Databricks, PySpark, SQL, Delta Lake, and AWS cloud services, with a proven track record of processing 500GB+ datasets and billions of rows.
In my recent work, I reduced ETL pipeline runtimes by ~50% (from 2 hours to 35 minutes) through Spark optimization like partitioning, caching, broadcast joins, and skew handling. I build production-grade pipelines using Medallion Architecture, SCD Type 2, and Star/Snowflake data modeling, supported by data validation frameworks, event-driven ingestion with AWS Lambda, and reliable orchestration via Apache Airflow and Databricks Jobs.
Experience
Work history, roles, and key accomplishments
Senior Data Engineer
Mu Sigma
Nov 2022 - Present (3 years 6 months)
Built end-to-end ETL pipelines in Databricks and PySpark for clinical trial, workforce, and RWE analytics, processing 500GB+ datasets (billions of rows). Reduced ETL runtime from 2 hours to 35 minutes (~50%) through Spark optimization and implemented Delta Lake SCD Type 2 and Medallion Architecture with reliable orchestration and data validation.
Supported enterprise-scale data migration and validation by performing SQL-based reconciliation, data quality checks, and discrepancy analysis across large datasets. Gained hands-on experience with ETL workflows and CI/CD processes to support reliable data engineering delivery.
Education
Degrees, certifications, and relevant coursework
Visvesvaraya Technological University (VTU)
Bachelor of Engineering, Computer Science
2016 - 2020
Grade: CGPA: 7.5
Completed a Bachelor of Engineering in Computer Science at Visvesvaraya Technological University (2016–2020) with a CGPA of 7.5.
Tech stack
Software and tools used professionally
Availability
Location
IndiaAuthorized to work in
Job categories
Interested in hiring Chethan?
You can contact Chethan and 90k+ other talented remote workers on Himalayas.
Message ChethanFind your dream job
Sign up now and join over 100,000 remote workers who receive personalized job alerts, curated job matches, and more for free!
