Open to opportunities

Shivani Singh

@shivanisingh8

Message

I am a Lead Data Engineer who builds scalable data platforms and delivers business-impacting data products.

United States

Message

What I'm looking for

I seek senior data engineering roles where I can lead cloud-native ETL and ML pipelines, mentor teams, and build scalable, reliable data platforms that deliver measurable business impact.

I am a Lead Data Engineer who takes pride in solving real-world business problems using a robust data toolkit and computing. I have designed and developed data-engineered products, built data warehouses at scale, and delivered pipelines for high-volume production environments.

I have led cross-functional workshops and PoC development, including travel to Japan to collaborate on a hybrid collections model with PayPay, and I have published research on preprocessing techniques and MITRE ATT&CK analyses. I have guided undergraduates and supported postgraduates while contributing domain expertise in healthcare, finance, and logistics.

I work across cloud platforms (AWS, GCP, Azure) and modern data stacks to implement ETL, streaming, OLAP, and ML-supporting pipelines, and I focus on reliable, performant solutions while mentoring teams and aligning technical work to business outcomes.

Experience

Work history, roles, and key accomplishments

Current

Lead Data Engineer

Current

Paytm

Dec 2024 - Present (9 months)

Led cross-functional PoC work with PayPay and traveled to Japan to develop a hybrid collections model, improving model performance and reliability across batch and streaming pipelines. Built scalable pipelines using PySpark on AWS and GCP services to evaluate and deploy production models.

Pyspark BigQuery Pub Sub Cloud Composer GCS)Data Modeling

Graduate Research Assistant

University of West Florida

Aug 2023 - Dec 2024 (1 year 4 months)

Published research on performance of data preprocessing techniques on CPUs vs GPUs and comparative analyses using MITRE ATT&CK; guided undergraduates and assisted postgraduates on data-driven projects. Produced reproducible experimental results and technical documentation for academic publication.

Research data preprocessing BigQuery SQL Data Warehousing Scientific Writing Mentoring

Senior Data Engineer

Paytm

May 2021 - Aug 2023 (2 years 3 months)

Delivered multiple high-performance ETL pipelines and OLAP solutions for products serving 125M+ users, ingesting data from SQL, S3, and MongoDB into Apache Iceberg to support loan eligibility decisions. Integrated Spark-based pipelines with CI/CD and scheduling tools to improve data reliability and throughput.

Apache Spark Kafka Apache Iceberg Airflow Python Cassandra SQL Jenkins

Data Engineer - Software Engineer

ACS Solutions (Innova Solutions)

Jan 2019 - May 2021 (2 years 4 months)

Implemented end-to-end ETL pipelines using Azure Data Factory, Databricks, Azure Data Lake, SSIS and IBM DataStage and developed PowerBI dashboards for healthcare analytics. Automated deployments and maintained CI/CD practices to ensure data delivery and quality.