Open to opportunities

pavani dachepalli

@pavanidachepalli

Message

Senior Data Engineer specializing in scalable Spark pipelines and cloud data platforms.

India

Message

What I'm looking for

I seek cloud-native data engineering roles focused on scalable ETL, real-time systems, and team collaboration, with opportunities for technical leadership and innovation.

I am a Senior Data Engineer with extensive experience developing scalable data pipelines and analytics solutions using Apache Spark and cloud platforms like AWS and GCP.

I have architected and migrated legacy data systems to modern cloud environments, built robust ETL pipelines with PySpark on EMR, implemented real-time ingestion with Kafka and Kinesis, and automated orchestration using Airflow and Oozie.

I deliver high-performance data infrastructure, integrate BI solutions such as Tableau and Power BI, implement data quality and CI/CD practices, and aim to drive innovation and operational efficiency in data engineering roles.

Experience

Work history, roles, and key accomplishments

Current

Senior Data Engineer

Current

Wawa, Inc.

Dec 2024 - Present (8 months)

Developed scalable PySpark ETL pipelines on Amazon EMR to ingest and transform data from MySQL, PostgreSQL, MongoDB, SFTP and APIs; built real-time ingestion with Kafka and Kinesis and implemented Lambda-based event-driven processing to enable timely analytics.

Pyspark Apache Spark Amazon EMR Apache Airflow Kafka Amazon Kinesis AWS Lambda Databricks Hive Parquet

Senior Data Engineer

HSBC

Apr 2020 - Aug 2023 (3 years 4 months)

Built and deployed scalable PySpark and Spark SQL ETL on cloud platforms and migrated legacy on-premise data warehouse to Azure SQL Data Warehouse; designed real-time streaming pipelines using Kafka, Spark Streaming and Azure Event Hub for near-real-time analytics.

Pyspark Apache Spark Azure Data Factory Apache Kafka Hive HBase Power BI Great Expectations

Data Engineer

Agilon Health

Jan 2017 - Mar 2020 (3 years 2 months)

Designed and implemented ETL pipelines on GCP using Cloud Dataflow, Data Fusion and BigQuery to enable scalable ingestion and analytics; migrated on-premises ETL to GCP and built Spark Streaming pipelines from Kafka for real-time processing.