Open to opportunities

PAVANI VALASAPALLI

@pavanivalasapalli

Message

Data Engineer with expertise in scalable data pipeline design.

United States

Message

What I'm looking for

I seek a role that fosters innovation, collaboration, and continuous learning in data engineering.

I am a Data Engineer with over 4 years of experience in designing and building scalable data pipelines across AWS and Azure. My expertise lies in developing high-performance ETL/ELT workflows that significantly reduce data processing time and enhance query speed in platforms like Snowflake and Redshift. I have successfully managed petabyte-scale datasets, ensuring excellent pipeline uptime while optimizing storage costs.

Throughout my career, I have leveraged technologies such as Apache Spark, Kafka, Airflow, and Terraform to support both real-time and batch data processing for analytics and business intelligence. My role involves collaborating with cross-functional teams to transform complex data challenges into scalable solutions that drive impactful business outcomes. I am passionate about implementing secure, cloud-optimized data architectures while maintaining compliance with industry regulations.

Experience

Work history, roles, and key accomplishments

Current

Data Engineer

Current

Progressive Corporation

Feb 2025 - Present (1 year 4 months)

Constructed Azure-based data pipelines, processing TBs of client data from on-premise & cloud sources, improving data accessibility for analytics teams. Engineered ML feature stores, streamlining data retrieval and model training workflows across teams.

Azure Data Pipelines ETL Spark Delta Lake Data Validation Anomaly Detection RBAC Encryption Cloud Native Architecture Machine Learning

Data Engineer

Aspire Technolab

Mar 2021 - Apr 2023 (2 years 1 month)

Implemented Azure Data Factory (ADF) extensively for ingesting data from different source systems like relational and semi-structured data to meet business functional requirements. Designed and developed batch processing and real-time solutions using PySpark, Kafka, and Databricks.

Azure Data Factory Pyspark Kafka Databricks Parquet Delta Lake Azure DevOps Jenkins Git Jira Azure Data Lake Store

Data Engineer

Sage Softtech

Jun 2019 - Mar 2021 (1 year 9 months)

Created pipelines, data flows, and complex data transformations and manipulations using Azure Data Factory (ADF) and PySpark with Databricks. Developed Spark applications using PySpark and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats.

Azure Data Factory Pyspark Databricks Hive Azure SQL SQL Agent Jobs Git Jira