Looking for a job

Arnab Chowdhury

@arnabchowdhury1

Message

Results-driven Lead Data Engineer with 8+ years of experience.

India

Message

What I'm looking for

I am looking for a challenging role in a dynamic environment that fosters innovation and growth, where I can leverage my data engineering skills to drive impactful solutions.

I am a results-driven Lead Data Engineer with over 8 years of experience in designing scalable ETL pipelines and optimizing cloud-based data warehouses. My expertise lies in Python, Java, and Scala, and I have a proven track record of automating workflows to reduce costs by up to 70%. I excel in translating complex business needs into technical solutions across various industries, including finance, logistics, and SaaS.

In my current role at Tavant, I have automated ETL pipelines using Python and Snowflake, significantly reducing manual intervention and storage costs. My experience also includes developing AWS Glue workflows and enhancing database query performance. I am passionate about leveraging my skills in Agile methodologies and CI/CD pipelines to drive efficiency and innovation in data management.

Find your dream job

Sign up now and join over 100,000 remote workers who receive personalized job alerts, curated job matches, and more for free!

Experience

Work history, roles, and key accomplishments

Current

Lead Data Engineer

Current

Tavant

Feb 2023 - Present (2 years 9 months)

Automated ETL pipelines using Python and Snowflake, reducing manual intervention by 90%. Developed AWS Glue workflows to load data into Snowflake, cutting storage costs by 70%. Created Information Marts for comprehensive reports on waste, ecology, environment, and safety.

Snowflake AWS Glue AWS S3 AWS Lambda Python Alteryx Docker

Senior Data Engineer

Tavant

Feb 2022 - Feb 2023 (1 year)

Designed and implemented ETL workflows using Python and Snowflake, orchestrating jobs with Airflow. Improved transaction reconciliation efficiency through targeted query enhancements. Developed jobs to deliver information to customers via SFTP and AWS S3, ensuring secure data distribution.

Snowflake Airflow Python GitHub

Senior Data Engineer

Tavant

Sep 2021 - Feb 2022 (5 months)

Designed and implemented end-to-end ETL workflows using Apache Spark, optimizing data extraction, transformation, and loading processes. Identified and implemented enhancements to improve system performance and efficiency. Utilized HiveQL within Hue and crafted complex queries using Presto to efficiently retrieve data for report generation.

Apache Spark Scala GitHub Presto

Software Development Specialist

NTT Data

Feb 2021 - Sep 2021 (7 months)

Streamlined data retrieval from multiple sources, generating daily and monthly reports on historical trades and disputes. Enhanced performance by upgrading object-based storage systems and migrating codebases to Java 8. Designed RESTful solutions with cross-team alignment and interfaced with business teams to address their needs effectively.

SVN Spring Spring Boot OBS

System Engineer

TCS

Nov 2016 - Feb 2021 (4 years 3 months)

Built Spring Boot APIs with Java and JUnit tests, ensuring robust and reliable functionality. Collaborated with Business Analysts to refine project requirements and developed comprehensive design documentation with diagrams and data mappings. Executed Proof of Concepts (POCs) using Redis caching, centralized logging/error handling, and connection pooling.

TFS Spring Spring Boot SQL