Looking for a job

Vedansh Porwal

@vedanshporwal

Message

I’m a senior Big Data engineer specializing in PySpark ETL and reliable distributed data pipelines.

India

Message

What I'm looking for

I’m looking for a role where I can lead scalable Spark/NiFi/Airflow ETL and data validation for compliance-critical, multi-country environments, collaborate cross-functionally, and mentor others while building reliable distributed pipelines.

I’m a results-driven Senior Big Data Engineer with 4.5+ years of experience (4 years full-time + 6 months internship) designing, developing, and optimizing large-scale ETL pipelines, data architectures, and distributed data processing solutions. I bring deep expertise in Apache Spark (PySpark), Apache NiFi, Hadoop, Hive, Trino/Presto, Apache Airflow, and Apache Kafka.

In my current role, I lead the design and delivery of scalable big data solutions for compliance and business-critical use cases across 14+ operational countries. I drive architectural decisions for ETL pipelines and data validation frameworks built on PySpark and Apache Airflow, and I mentor junior engineers through code reviews to improve delivery velocity. I’ve been recognized with a “Going Extra Mile Award.”

Previously, I delivered ABACUS to process KYC data from 14 countries, building PySpark-based validation pipelines and anomaly detection to support fraud detection and regulatory audit readiness. Earlier, I built FINREP for standardized financial reporting, automating extraction with Apache NiFi using custom Java-based processors and record readers, and developing 50+ ETL pipelines with Java-based logic. As a software development intern, I designed and tested 100+ NiFi ETL pipelines and supported production reliability as a SPOC for SRE, earning additional awards for collaboration.

Experience

Work history, roles, and key accomplishments

Current

Senior Software Engineer – Big Data

Current

Airtel Africa Digital Labs

Dec 2025 - Present (7 months)

Leading design and delivery of scalable big data solutions for compliance and business-critical use cases across 14+ African operational countries. Driving ETL pipeline and data validation architecture using PySpark and Apache Airflow, and mentoring junior engineers through code reviews.

PySpark Airflow ETL Data Validation Mentor Code Review

Software Engineer II – Big Data

Airtel Africa Digital Labs

Jun 2024 - Nov 2025 (1 year 5 months)

Developed and optimized PySpark-based data validation pipelines for KYC data from 14 operational countries, supporting compliance and regulatory requirements. Built anomaly detection to flag fraudulent/non-compliant records and created Trino/Presto SQL views for compliance reporting.

PySpark Data Validation KYC Anomaly Detection Trino SQL Fraud Detection

Software Engineer I – Big Data

Airtel Africa Digital Labs

Jun 2022 - May 2024 (1 year 11 months)

Designed and implemented an end-to-end Financial Reporting system deployed across 14 operational countries to standardize financial data extraction and reporting. Automated extraction using Apache NiFi with custom Java-based processors and record readers, built 50+ NiFi/Java-based ETL pipelines, and loaded results into PostgreSQL for downstream analysis.

Apache NiFi Java Processors Record Readers ETL Java Expression Language PostgreSQL Data Extraction Financial Reporting

Software Development Intern – Big Data

Airtel Africa Digital Labs

Jan 2022 - Jun 2022 (5 months)

Designed, developed, and tested 100+ Apache NiFi ETL pipelines handling terabytes of data with Oracle/PostgreSQL, SFTP, and HDFS integrations. Optimized query performance via dataset partitioning and supported SRE by diagnosing and resolving production issues to maintain uptime and data accuracy.