Looking for a job

Sai Nagendra Pavan Kumar Pilli

@sainagendrapavankuma

Data Engineer at Ascendion who improved healthcare data pipeline performance by 30% using Azure Databricks and PySpark.

At Ascendion, I designed and maintained healthcare data pipelines with Azure Data Factory, Azure Databricks, PySpark, and Python. I improved end-to-end processing performance by 30% through Spark transformation and workflow optimization.

I built lakehouse solutions in Microsoft Fabric, transforming healthcare data through Bronze, Silver, and Gold layers and connecting curated data to Fabric Data Warehouse and Power BI Direct Lake models.

Across my roles at Ascendion, Capgemini, Data Core Technologies (TCS), and Y Creators Media LLP, I developed batch and streaming data pipelines for healthcare, IoT, and retail use cases. My work included CDC-based loading, governance controls, and near real-time reporting.

Experience

Work history, roles, and key accomplishments

Sr Software Engineer

Ascendion

Aug 2025 - May 2026 (9 months)

Designed and maintained scalable ETL/ELT pipelines using Azure Data Factory, Azure Databricks, PySpark, and Python for healthcare data. Built Delta Live Tables frameworks with CDC-based incremental loading and implemented Auto Loader and Delta Lake ingestion patterns.

Sr Software Engineer I

Capgemini

Jan 2025 - Jul 2025 (6 months)

Developed and managed ETL pipelines using Azure Data Factory, Azure Databricks, and Python. Implemented Unity Catalog for data governance and created streaming and batch ingestion pipelines using Auto Loader and Delta Lake.

Azure Data Factory Azure Databricks Python Unity Catalog Auto Loader Delta Lake Azure Stream Analytics Azure Event Hubs Azure Logic Apps Azure SQL Database Data Governance RBAC Streaming ETL

Software Engineer

Data Core Technologies (TCS)

Jul 2022 - Nov 2023 (1 year 4 months)

Automated orchestration and monitoring workflows using ADF and Azure Logic Apps. Developed real-time streaming ingestion with Azure Event Hubs and Spark Structured Streaming, and supported live data feeds into Azure Synapse Analytics.

Azure Data Factory Azure Logic Apps Azure Event Hubs Spark Structured Streaming Azure Databricks ADLS Gen2 Delta Lake Azure Synapse Data Warehouse Star Schemas Dimensional Modeling Data Quality Monitoring

Software Engineer

Y Creators Media LLP

Oct 2019 - Jun 2022 (2 years 8 months)

Monitored and optimized data pipeline performance using Azure Data Factory and Databricks metrics. Collaborated with data scientists and analysts to deliver insights, and implemented data governance policies in streaming environments.