Gatik Misra
@gatikmisra
Data Engineer building real-time Azure Databricks pipelines with Apache Spark & Kafka.
IndiaWhat I'm looking for
I’m a Data Engineer with 3.5+ years building and optimizing large-scale data pipelines on Azure Databricks and Apache Spark in a global IT consulting delivery environment. I focus on ETL/ELT development, real-time streaming, and data lakehouse design to turn raw events into reliable, analysis-ready data.
In my current role, I developed scalable pipelines using PySpark and Python on Azure Databricks and Azure Data Factory, processing 1TB+ of structured and semi-structured data monthly. I built near real-time ingestion using Apache Kafka / Azure Event Hub and Databricks Structured Streaming, reducing data latency from 30 minutes to under 5 minutes.
I also migrated legacy batch workflows to a cloud-native, event-driven Data Lakehouse architecture using Delta Lake and the Medallion Architecture, cutting end-to-end latency by 83%. I optimize for outcomes—improving SLA reliability from 94% to 99.7%, reducing processing time by 40%, and enforcing PySpark-based data quality and validation frameworks that reduced data errors by 90%.
Experience
Work history, roles, and key accomplishments
Data Engineer
Accenture Technologies
Nov 2022 - Present (3 years 8 months)
Built and maintained scalable ETL/ELT data pipelines on Azure Databricks and Azure Data Factory using PySpark and Python, processing 1TB+ of data monthly. Delivered near real-time ingestion and migrated to an event-driven Delta Lake data lakehouse, reducing latency by 83% and improving SLA reliability from 94% to 99.7%.
Data Analyst Intern
Boston Scientific
Feb 2022 - Sep 2022 (7 months)
Performed SQL-based data extraction, cleansing, and transformation to create analysis-ready datasets with consistent metadata for reporting. Built automated SQL/Excel workflows and Power BI dashboards, reducing manual effort and improving data accuracy.
Real-Time Data Engineering Pipeline
Uber
Built an end-to-end real-time pipeline integrating web events and batch data using Azure services and Databricks Structured Streaming. Implemented a Medallion (Bronze/Silver/Gold) architecture with a Gold-layer star schema to achieve sub-minute latency using Delta Lake ACID guarantees.
Education
Degrees, certifications, and relevant coursework
Sikkim Manipal Institute of Technology
Bachelor of Technology, Computer Science Engineering
2018 - 2022
Grade: CGPA: 9.40/10.0
Completed a B.Tech in Computer Science Engineering at Sikkim Manipal Institute of Technology from 2018–2022, achieving a CGPA of 9.40/10.0.
Tech stack
Software and tools used professionally
Availability
Location
IndiaAuthorized to work in
Job categories
Skills
Interested in hiring Gatik?
You can contact Gatik and 90k+ other talented remote workers on Himalayas.
Message GatikGet matched with your dream remote job
Sign up now and join over 250,000+ remote workers who receive personalized job alerts, curated job matches, and more for free!
