Open to opportunities

Padmanjali V R

@padmanjalivr

Message

Experienced Big Data engineer specializing in PySpark, Databricks, and scalable data pipelines.

India

Message

What I'm looking for

I seek hands-on data engineering roles building scalable PySpark/Databricks pipelines, with a collaborative team, emphasis on data quality, performance optimization, and opportunities to mentor others.

I am a Big Data professional with over six years of experience building and optimizing data engineering solutions using PySpark and Databricks.

I have migrated Oracle pipelines to Databricks, processed terabytes of data from S3 datalakes, and implemented production-grade workflows emphasizing performance, partitioning, compression, and cost efficiency.

I have strong experience with Hive (managed/external/partitioned tables), SQL, shell scripting, and automating data refreshes and deployments, and I have trained and led junior team members to ensure successful project delivery.

I focus on data quality, lineage, and operational reliability while collaborating with stakeholders to translate requirements into robust, maintainable data pipelines.

Experience

Work history, roles, and key accomplishments

Current

IT Analyst

Current

Tata Consultancy Services

Jul 2021 - Present (4 years 6 months)

Led migration of Oracle-based pipelines to Databricks using PySpark, improving data processing scalability and cost-efficiency; developed, tested, and deployed pipelines to higher environments with no discrepancies. Performed monthly data validation, introduced new product features into analytics workflows, and trained junior team members to maintain high data quality.

Pyspark Databricks AWS S3 SQL Data Pipeline Development Spark Optimization Data Validation

Data Engineer

Atos Syntel

Oct 2018 - Jul 2021 (2 years 9 months)

Managed Hive data models and automated data refreshes using Hive and shell scripts, deployed production scripts via scheduler, and implemented monitoring and email alerts to improve data reliability and reduce downtime. Optimized query performance using partitioning, bucketing, and query optimization techniques.

Hive SQL Shell Scripting Data Modeling job scheduling Query Optimization