Looking for a job

Yagnika Kancherla

@yagnikakancherla

Data Engineer at the State of Ohio building SparkSQL pipelines processing 10 million+ records daily and improving data quality by 40%.

United States

Message

At the State of Ohio, I design SparkSQL pipelines in Python and maintain Apache Spark workflows that process 10 million+ records daily for operational and risk analytics.

I developed dimensional data models and medallion architectures, creating 15+ fact and dimension tables for datamarts. I also implemented Databricks ETL workflows with a cloud data lake, reducing refresh latency by 30% for compliance-driven reporting.

I orchestrated Airflow DAGs for data quality monitoring using Monte Carlo methodologies, reducing data errors by 40%. I optimized SQL and SparkSQL queries and enhanced lineage instrumentation and testing frameworks.

Previously, at Sagarsoft Inc. for client JPMorgan Chase & Co., I developed Spark and Python pipelines on Databricks and modeled reporting marts. Earlier, at Aditya Birla Capital Limited, I worked on SparkSQL pipelines, cloud data lake migrations, and Kafka and Kinesis streaming data for operational dashboards.

Experience

Work history, roles, and key accomplishments

Current

AI/Data Engineer

Current

State of Ohio

Jun 2024 - Present (2 years 3 months)

Designed over 25 scalable SparkSQL data pipelines using Python, developed dimensional data models and medallion architectures, and built production pipelines handling daily ingestion of 10 million+ records. Orchestrated Airflow DAGs for data quality monitoring, reducing data errors by 40%.

SparkSQL Python Apache Spark Databricks Airflow Data Lake Schema Design Dimensional Data Models Medallion Data Architecture Incremental Modeling Data Quality Management Monte Carlo Data Lineage Testing SQL

Data Analytics Engineer

Sagarsoft Inc.

Apr 2023 - May 2024 (1 year 1 month)

Constructed complex SparkSQL queries and optimized SQL scripts, engineered production Spark and Python data pipelines, and modeled dimensional schemas. Scheduled and monitored Airflow workflows, ensuring 99.8% on-time job completion.

SparkSQL SQL Python Apache Spark Databricks Data Lake Dimensional Data Models Star Schemas Dimensional Modeling (Star And Snowflake Schemas)Airflow Data Quality Monte Carlo Data Marts

Data Analyst

Aditya Birla Capital Limited

Oct 2020 - Jul 2022 (1 year 9 months)

Extracted and optimized over 100 complex SQL queries using Python scripts, designed dimensional data models, and maintained production data pipelines with SparkSQL transformations. Implemented Spark jobs in Scala within Databricks and integrated streaming data sources.