# Abhijit Patra

Abhijit Patra is a remote worker on Himalayas, Lead Data Engineer at PayNearby, based in India.

Source: https://himalayas.app/@abhijitpatra1

```text
Lead Data Engineer at PayNearby who cut storage costs by 88% with a 32TB Spark and Iceberg Lakehouse migration.
```

| Detail | Value |
| --- | --- |
| Availability | Open to opportunities |
| Location | India |
| Authorized to work in | India |
| Salary expectations | 4981k+ INR |
| Job categories | Lead Data Engineer, Data Engineer, Backend Engineer |
| Skills | Data Engineering, ETL Architectures, Real Time Data Pipelines, Data Warehouse, Data Pipeline Monitoring, GCP, AWS, BigQuery, Apache Spark, Apache Iceberg, PySpark, Kafka, Debezium, Airflow, DBT |
| LinkedIn | https://linkedin.com/in/de-abhijit-patra |

## About

```text
At PayNearby, I executed a 32TB log migration to Apache Spark and Iceberg Lakehouse, cutting storage costs by 88%. I also innovated a multilingual AI financial counselor using Gemini Live API, achieving sub-500ms response time.

I maintain an almost real-time micro-batch pipeline syncing data to BigQuery with Kafka, Debezium, and Airflow. I also built and maintain a centralized Elasticsearch stack and alerting system for applications, system, and Kubernetes metrics.

Previously at Affine Analytics, I used AWS Lambda to process 2,000 files daily, increasing processing efficiency by 80%. At Iqvia, I built an AWS data warehouse using Oracle and Star Schema models, and deployed Airflow DAGs for job scheduling.
```

## Experience

### Lead Data Engineer (current)

PayNearby. Jul 2024 - Present

```text
Leading and managing a team of 5 data engineers, exemplifying leadership and strategic oversight. Innovated a multilingual AI financial counselor and executed a 32TB log migration to Apache Spark and Iceberg Lakehouse.
```

Skills: Gemini-Live-API, Apache-Spark, Apache-Iceberg, Elasticsearch, Kafka, Debezium, Airflow, BigQuery, Team-Leadership, AI, Real-Time-Pipeline

### Senior Data Engineer

PayNearby. Jun 2022 - Jul 2024

```text
Developed and automated BigQuery platform for data reconciliation using DBT, FastAPI, MongoDB, and Celery. Engineered a real-time heartbeat tool for pipeline health monitoring and alerting.
```

Skills: BigQuery, DBT, fastAPI, MongoDB, Celery, Real-Time-Pipeline, Data-Reconciliation, Alerting, Checksum

### Senior Data Engineer

Affine Analytics. Aug 2021 - Jun 2022

```text
Increased processing efficiency by 80% by processing 2000 files daily using AWS Lambda. Designed scalable Oracle database schemas and incremental ETL pipelines to optimize data loading.
```

Skills: AWS-Lambda, Oracle, ETL, Data-Pipelines, Mentorship, Documentation

### Backend Engineer

[Iqvia](https://himalayas.app/companies/iqvia). May 2018 - Jul 2021

```text
Built a ground-up Data Warehouse on AWS using Oracle and Star Schema models, managing Slowly Changing Dimensions. Deployed Airflow DAGs for job scheduling and managed database schemas/versioning using Liquibase.
```

Skills: AWS, Oracle, Star-Schemas, SCD, Airflow, Liquibase, Python, Docker, DBFit, S3, AWS-SDK-(Boto3), Fernet

## Education

### Python Institute

PCAP-Python3 Certified Associate Programmer, Python Programming

```text
Earned the PCAP-Python3 Certified Associate Programmer certification from the Python Institute.
```

### Python Institute

PCEP-Python3 Certified Entry-Level Programmer, Python Programming

```text
Earned the PCEP-Python3 Certified Entry-Level Programmer certification from the Python Institute.
```

## Tech stack

- Apache Spark
- Amazon S3
- PySpark
- Debezium
- dbt
- Liquibase
- MongoDB
- Oracle
- Python
- Kafka
- Celery
- FastAPI
- Elasticsearch
- AWS Lambda
- Docker
- Airflow
- Amazon Web Services (AWS)
- Amazon EMR
- SQL
- Google Cloud Run
- Apache Iceberg

To contact Abhijit, message them from this profile with a Himalayas employer account: [Hire on Himalayas](https://himalayas.app/hire).

## People also viewed

- [Arun K C](https://himalayas.app/@arunkc1): Open Source Contributor, naas.ai
- [Gunj Desai](https://himalayas.app/@gunjdesai): Engineering Manager, HackerRank
- [Saumya Ranjan](https://himalayas.app/@saumyaranjan1): Data Engineer, Axis Bank
- [Manjot Singh](https://himalayas.app/@manjotsingh7): Senior Data Engineer, Aptus Data Labs
- [ashish kumar](https://himalayas.app/@ashishkumar39): Data Engineer, Workday
- [Madhan Lingareddy](https://himalayas.app/@madhanreddy575): Senior Data Engineer, Procore Technologies
- [Harish Rajoori](https://himalayas.app/@harishrajoori): Staff Data Engineer, Warner Bros. Discovery
- [Wameen Ahmad](https://himalayas.app/@wameenahmad): Data Engineer Lead, Cisco
- [Aditi Vaidya](https://himalayas.app/@aditivaidya): Senior Data Engineer, Konverge.AI
- [mallikarjuna reddy guruguntla](https://himalayas.app/@mallikarjunareddygur): Lead Data Engineer, Dodge Construction Network
