Open to opportunities

diwas rai

@diwasrai

Message

Senior Data Engineer specializing in cloud data platforms, Lakehouse architectures, and AI-ready pipelines.

United States

Message

What I'm looking for

I seek challenging roles building scalable, secure cloud data platforms and Lakehouse architectures with strong emphasis on governance, AI/ML enablement, and team mentorship.

I am a results-driven Senior Data Engineer with 5+ years building and scaling cloud data platforms, enterprise data warehouses, and AI-ready ecosystems across healthcare, finance, insurance, and retail. I design batch, real-time, and event-driven pipelines on Azure, AWS, and GCP using Databricks, Snowflake, Delta Lake, and Lakehouse architectures while enforcing data governance and compliance (HIPAA, SOX).

I have delivered large-scale projects—integrating 50M+ records, processing 10TB+ daily transactional data, and enabling predictive analytics—while optimizing performance, automating CI/CD, and mentoring junior engineers. I bring hands-on expertise in Python, PySpark, SQL, ML/LLM integrations, RAG pipelines, Infrastructure-as-Code, and cost/governance-minded cloud architecture.

Experience

Work history, roles, and key accomplishments

Current

Senior Data Engineer

Current

State Farm Insurance

Jan 2023 - Present (3 years 2 months)

Led a data modernization project building Azure Data Factory pipelines to integrate 50M+ records from 12+ sources, reducing report generation time by 40% and enabling Snowflake and Databricks solutions that processed 10TB+ daily for predictive analytics.

Azure Data Factory Databricks Snowflake Azure Stream Analytics Python Pyspark SQL Power BI

Data Engineer

TIAA

May 2020 - Dec 2022 (2 years 7 months)

Designed and developed end-to-end Azure Data Factory and Databricks pipelines, improving ETL automation by 40%, boosting Snowflake query performance by 25%, and migrating on-prem systems to Azure reducing costs by 20%.

Azure Data Factory Databricks Snowflake Python Pyspark Data Lake ETL

ETL Developer

Pfizer

Aug 2019 - Apr 2020 (8 months)

Developed Informatica and SSIS ETL processes integrating 20M+ records into the data warehouse, reducing errors by 35% and improving ETL processing times by 30% through SQL and PL/SQL optimizations.

Informatica SSIS SQL Server Oracle Pl SQL Data Quality