Looking for a job

Madhan Lingareddy

@madhanreddy575

Message

Data Engineer with 8 years of experience building on Low Latency Data Platforms for real-time analytics.

India

Message

What I'm looking for

I want to help set up a Data platform that scales well in terms of handling volumes and variety while not compromising on cost and performance.

Love to bring enterprises the ability to make confident, analytical and data driven decisions on the go without a “lag”.

Technically strong in Python, Apache Spark (PySpark) and SQL. Experience across Payments, Finance, Retail and Construction domains. Worked on cloud platforms like AWS, Databricks and GCP. Built near-real time data apps and data lakes using Spark Structured Streaming and kafka. Responsive expert experienced in monitoring database performance, troubleshooting issues and optimizing database environment. Possesses strong analytical skills, excellent problem-solving abilities, and deep understanding of database technologies and systems. Equally confident working independently and collaboratively as needed and utilizing excellent communication skills.

Experience

Work history, roles, and key accomplishments

Current

Senior Data Engineer

Current

Procore Technologies

Jun 2024 - Present (2 years 1 month)

Designed and developed low-latency streaming data pipelines using Apache Kafka and Databricks Delta Lake to enable near real-time analytics. Re-architected ETL workflows, built a metadata-driven data quality framework, and delivered real-time warehousing solutions with Feldera.

Apache Kafka Databricks Delta Lake Apache Spark AWS SQL Lakehouse Databricks Delta Lake Feldera Data Modeling ETL Data Quality Framework Metadata Driven Framework

Senior Data Engineer

Pasarpolis

Apr 2022 - May 2024 (2 years 1 month)

Designed and implemented cloud-native ETL architecture on Google Cloud Platform using Cloud Composer (Airflow) with production-grade ingestion, transformation, and reporting workflows. Built streaming pipelines and dimensional warehouse models for SLA monitoring, customer lifetime value, and operational KPIs.

Orchestration Google Cloud Platform Cloud Composer Apache Airflow ETL Python BigQuery Google Datastream Google BigQuery Tableau AWS monitoring Streaming data pipelines Data Archival Monitoring

Senior Data Engineer

ADF Data Science

Apr 2021 - Mar 2022 (11 months)

Built scalable AWS data pipelines for enterprise data warehousing using AWS Glue, PySpark, Amazon S3, Redshift, and AWS DMS. Implemented CDC-based historical reconstruction, schema evolution detection with Pentaho ETL, data governance with Amundsen, monitoring with CloudWatch and Slack, and automated validation utilities from MySQL binary logs.

BI Developer / Data Engineer

Wipro Technologies

Feb 2018 - Mar 2021 (3 years 1 month)

Worked as a BI Developer and Data Engineer translating business requirements into analytics solutions, including enterprise data models and ETL pipelines using Incorta. Developed Python-based data enrichment integrating REST APIs and Pandas, optimized workloads with multithreading and multiprocessing, and automated dashboard generation and delivery via Incorta REST APIs.