Vivek Gupta

Developed Spark ingestion pipelines on Databricks integrating 5+ heterogeneous sources, boosting throughput by 40% for real-time delivery. Built Delta Lake medallion architecture with incremental MERGE processing 1 TB+/day, reducing runtime by 35% and improving reliability with governed production-ready data.

Designed and developed Azure Data Factory ETL/ELT pipelines with CI/CD integration, achieving 99.9% reliability on regulated enterprise workloads. Implemented data governance (PII masking, lineage tracking, RBAC, audit logging) and built document intelligence workflows for automated extraction, classification, and compliance analysis.

Built NLP feature engineering pipelines on large-scale behavioural datasets using Python and SQL, improving search relevance by 85%. Deployed XGBoost models into production inference pipelines using SageMaker, reducing customer attrition by 20%.

Designed a cloud-native lakehouse integrating 20+ structured and unstructured sources with IAM multi-tenant controls across Azure and GCP. Built Kafka + Spark Streaming pipelines processing 1M+ events/min at sub-second latency and automated validation/reconciliation to reduce data quality incidents by 50%.

Developed Python + SQL ETL frameworks processing 100M+ clinical records with 90%+ validation accuracy and complete reconciliation audit trails. Built RESTful APIs integrating PostgreSQL and graph databases for federated data access, cutting latency by 30%, and designed data models for complex entity relationships supporting analytics and regulatory reporting.

Built SQL reporting pipelines and automated ETL workflows, improving operational efficiency by 25%. Developed Python backend services for data processing, query optimization, and database performance tuning.

Activities and societies: Relevant coursework: Big Data Analytics, Machine Learning, Cloud Computing, NLP.

Completed an M.Tech in Computer Engineering, focusing on Big Data Analytics, Machine Learning, Cloud Computing, and NLP.

Activities and societies: Relevant coursework: Data Structures, Databases, Data Mining, Information Retrieval.

Completed a B.E. in Computer Engineering with coursework spanning data structures, databases, data mining, and information retrieval.