rajin User
@rajinuser
Senior Data Engineer specializing in distributed data infrastructure, PySpark ETL, and near real-time CDC pipelines.
IndiaWhat I'm looking for
I’m a Senior Data Engineer with over 8 years of experience in data pipeline development and cloud environments, focused on building scalable, reliable distributed data infrastructure. I’m expert in PySpark and SQL, and I consistently optimize performance and delivery speed through automation and data-driven decision-making.
My key achievements include processing over 40 TB of records through scalable PySpark ETL pipelines and improving query performance by 20% using Delta Lake optimization strategies like partitioning and incremental loading. I’ve also reduced ingestion latency from 1 hour to 5 minutes by implementing near real-time CDC pipelines with Debezium, Kafka Connect, and Azure Event Hubs.
In recent roles, I designed and optimized PySpark ETL pipelines for enterprise supply chain analytics, migrated SAP HANA datasets to Azure Data Lake and Delta Lake, and monitored structured and unstructured datasets with Databricks and PySpark. Earlier, I built automated extraction and reporting pipelines using Python, SQL, Selenium, and REST APIs, reducing manual reporting effort by 8hrs, and I supported production reliability for multiple client-facing systems with AWS-hosted applications.
Experience
Work history, roles, and key accomplishments
Designed and optimized scalable PySpark ETL pipelines processing 40+ TB of records for enterprise supply chain analytics workloads. Migrated SAP HANA datasets to Azure Data Lake/Delta Lake and implemented Delta Lake optimizations (partitioning and incremental loading) to improve query performance by 20%.
Data Engineer
Linqux
May 2021 - Oct 2023 (2 years 5 months)
Developed Azure Data Factory pipelines migrating 5 TB of on-premise enterprise data into Azure. Built near real-time CDC pipelines using Debezium, Kafka Connect, and Azure Event Hubs to reduce ingestion latency from 1 hour to 5 minutes, and supported pipelines using Databricks/PySpark, Airflow, Terraform, and GitHub.
Software Engineer II
Shopalyst Technologies
Jun 2018 - May 2021 (2 years 11 months)
Developed automated extraction and reporting pipelines using Python, SQL, Selenium, and REST APIs, reducing manual reporting effort by 8 hours. Built backend APIs using Node.js supporting 4+ client integrations and analytics workflows.
Education
Degrees, certifications, and relevant coursework
Cochin University of Science and Technology (CUSAT)
Bachelor of Technology
2012 - 2016
Earned a Bachelor of Technology degree from CUSAT from 2012 to 2016.
Tech stack
Software and tools used professionally
Availability
Location
IndiaAuthorized to work in
Job categories
Skills
Interested in hiring rajin?
You can contact rajin and 90k+ other talented remote workers on Himalayas.
Message rajinGet matched with your dream remote job
Sign up now and join over 250,000+ remote workers who receive personalized job alerts, curated job matches, and more for free!
