Skip to main content
PP
Open to opportunities

Piyush Patel

@piyushpatel7

Professional Data and AI consultant

Canada
Message

What I'm looking for

Data Engineer, Fullstack Engineer, Terraform Automation, Staff Data Engineer, Senior Data Engineer

I'm a Software Engineer with over 12 years' experience including 5 years with technologies like Apache Spark, Kafka, Streamsets, Spark Streaming, Apache Beam (Dataflow), MapReduce, Hive, Pig, Sqoop, Impala, HBase, Oozie, Apache Flume, Tez, Impala, Hue, Azure Data Factory, AWS Glue, Redshift, Terraform, Databricks, etc. I was also working for application development team with various technologies like Java, Scala. Play framework, Akka, Spring, JavaScript, Python, Django, SQL, NoSQL and many more. I've also got three major cloud providers experience and latest Kubernetes and containerization experience under my belt. I have worked with latest automation build tools like Terraform, Ansible for infrastructure deployment in cloud. I'd love to combine my diverse skillsets and keep exploring new exciting technologies and challenges.

Big Data technologies: Hadoop, HDFS, Hive, Confluent, AWS, Shell Scripting, Pig, Sqoop, Scala, Spark, Flink, Beam, Oozie, Flume, Kafka, Streamsets, NiFi, AWS Glue, Azure Data Factory, Redshift, Azure Synapse, Databricks, Zeppelin, Blob storage, Google DataFlow, Dataproc, Presto, Iceberg
AI Stack: LangGraph, OpenAI, LangSmith, Google Gemini, Claude, Cursor, Copilot, Spec Driven Development
Programming Languages: Java, Scala, JavaScript, Python, R, Bash Scripting, YAML, XML, JSON
Web Technologies: HTML, CSS, Bootstrap, AngularJS, SQL, Angular
Databases: MySQL, MongoDB, HBase, Oracle SQL Server, Hive, Impala, Google Big Query
Cloud Platforms: Google Cloud Platform (GCP), AWS, Microsoft Azure
Frameworks: ExpressJS, Spring, Play Framework, Hibernate, Spring Boot, Angular, Mongoose
Build tools: Maven, NPM, Grunt, Gulp, SBT, Jenkins
IDEs and Editors: Eclipse, IntelliJ IDEA, Atom, Sublime Text, Visual Studio Code
Testing Frameworks: JUnit, Jasmine, Spark testing base, Scala Tests
Additional Tools: Git, Bitbucket, Bamboo, Gitlab, JIRA, Confluence

Experience

Work history, roles, and key accomplishments

Warner Bros. Games logoWG
Current

Senior Big Data & AI Consultant

Feb 2025 - Present (1 year 6 months)

- Supported feature development and bug fixes for promotions, personalization API and data services written in Python, FastAPI 
- Driving AI driven development efforts including MCP, Spec-driven development, etc. 
- Created internal knowledge base powered by AWS Bedrock, Qdrant and LangGraph
- Set up internal Eval pipeline using LangSmith to enhance observability and performance tuning of our AI s

Bell logoBE

Sr. Big Data Hadoop Developer

Nov 2018 - Jul 2021 (2 years 8 months)

- Developed terraform configuration files to spin up GCP environment for data pipelines migration
- Implement highly efficient Stream processing pipeline with Apache Kafka and Spark Streaming
- Helped team migrate large PB scale cluster to GCP cloud storage using distcp
- Developed Apache Beam code to port existing spark code to GCP Dataflow in Java/Scala language.
- Optimize spark and streaming j

Rogers Communications logoRC

Big Data Developer

Jun 2018 - Nov 2018 (5 months)

- Developed Hive scripts for data warehousing and data analysis
- Developed shell scripts to automate the data ingestion process into Hive warehouse
- Worked with Apache NiFi to develop automated data flow pipelines to ingest data into Hadoop ecosystem
- Fixed the features for data ingestion as required by the client.
- Developed Software solutions for data ingestion along with data modelling for

Zone·tv™ logoZO

Java/Scala Developer at Zone·tv™

Sep 2017 - Jun 2018 (9 months)

- Supporting product development features for company's television platform.
- Worked on Angular 4 , TypeScript for features and bug fixes
- Worked on NodeJS application for new API endpoint implementations
- Developed API endpoints in Java Spring Boot application for one of the REST services
- Integrated Loggly web service for Front end issues monitoring using Angular Global exception handler
- W

Meditab Software, Inc. logoMI

Java Hadoop/Spark Developer

Oct 2015 - Sep 2017 (1 year 11 months)

- Developed ETL jobs using Scala and Java as a programming language with Spark computation engine
- Used RDDs to transform and store data
- Used DataFrames for interactive queries
- Created ETL jobs in Talend to load data warehouse
- Gained domain knowledge of financial, retail and relation between certain traits
- Interacted with onshore team on a daily basis for the project
- Loaded data from te

ENVALE Systems Pvt. Ltd logoEL

Big Data Engineer

Jan 2015 - Oct 2015 (9 months)

- Sqoop data from RDBMS like oracle 12c server to HDFS system
- Transform the data based on the client requirements
- Partiticipated in design of workflows for Big Data ETL project
- Prepare documents with instructions for cluster deployment for the admin team
- Test the product on pre-production cluster with desired outputs
- Write Hive and Pig scripts to extract, transform and load data into/fr

GeekMind Solutions logoGS

Software Developer

Jul 2013 - Dec 2014 (1 year 5 months)

- Developed backend software solutions using JavaScript, NodeJS, ExpressJS, Java, Spring, Hibernate technologies
- Worked on front end interactive applications using AngularJS and/or jQuery libraries.
- Developed REST APIs for seamless integration with Angular client
- Developed schemas for MongoDB database using Mongoose
- Performed manual testing while developing application
- Worked on Grunt sc

EITL logoEI

Java Programmer

Oct 2009 - Jul 2013 (3 years 9 months)

Analyzed and prepared total system architecture for different web projects.
Worked on various projects with Java web frameworks like Struts and Servlets
Developed applications using Servlets, JSP and Spring framework
Explained and prepare curriculum of Core Java and Advanced Java training courses.
Trained university students with the core concepts of C and Java languages.
Conducted labs and hands

Education

Degrees, certifications, and relevant coursework

SU

Sardar Patel University

Bachelor of Engineering (B.E.), Electrical

- Graduated with Honors from Sardar Patel University in Gujarat and scored excellent grades with good control over theory as well as practical aspects of engineering.

- Received great admirations from my teacher and fellow students for excellent record during the span of four years.

- Successfully completed a project on Power Electronics Motion control for DC motor using SCR technology with self

SU

Sardar Patel University

Bachelor of Engineering - BE, Electrical and Electronics Engineering

- ranked number one in my batch of Electrical engineering graduates.

CO

Coursera

Google Cloud Certificates, Computer Engineering

2019 - 2020

Some of the courses and topics include:
- Essential Cloud infrastructure
- Elastic Cloud: Scaling and Automation with Deployment manager
- Reliable Infrastructure: Design and Process
- App Deployment, Debugging and Performance on GCP
- Building Batch Data pipeline with Dataproc and Dataflow
- Building Resilient Streaming Analytics Systems on GCP (PubSub, Dataproc, Data Studio)
- Modernizing Data w

HS

Harvard Extension School

Data Science Specialization, Data Science

2018 - 2019

This program covers many of the concepts of Data Science in the real world using R programming language. It covers machine learning and different algorithms, Data visualization, etc.

ED

edX

Certificate, Data Analytics

2018 - 2018

SE

Sikkim Manipal University - Distance Education

Certificate in Business Analytics, IT

2016 - 2017

CO

Coursera

Full Stack Web Development Specialization, Web Development

2014 - 2015

FR

freeCodeCamp

Full Stack Web Development Certification, Computer Science

2014 - 2015

On the way to acquire my first certification in FreeCodeCamp which is "Front End Developer certification". During this certification, I solved many Design challenges and developed Design and layout projects using HTML, CSS, Bootstrap. Towards the later phase of the certification, I added interactivity in the design and solved many algorithm challenges to develop concrete understanding of Algorithm

UD

Udacity

Full Stack and Cloud Engineer Nanodegree, Information Technology

2014 - 2015

- Built an interactive resume with the help of jQuery.
- Built a profile website with Bootstrap framework.
- Learn the concepts of Responsive web design with image optimization
- Optimized the workflow with the help of Grunt and Gulp

DA

D. N. High School, Anand

High School, Science

2000 - 2002

Graduated with high grades and went on to study at the Univerisity.

UD

Udacity

Cloud DevOps Nanodegree, Computer Science

2019 - 2020

In this nanodegree program, I worked to deploy large scale websites, static website hosting using Amazon S3, event driven application deployment using Amazon Lambda

Sardar Patel University logoSU

Sardar Patel University

Bachelor of Engineering, Electrical

Graduated with Honors from Sardar Patel University in Gujarat, scoring excellent grades with strong command over theory and practical aspects of engineering. Completed a project on Power Electronics Motion control for DC motor using SCR technology with self-designed control circuitry and boards.

Sardar Patel University logoSU

Sardar Patel University

Bachelor of Engineering, Electrical and Electronics Engineering

Ranked number one in the batch of Electrical engineering graduates.

Interested in hiring Piyush?

You can contact Piyush and 90k+ other talented remote workers on Himalayas.

Message Piyush

People also viewed

View all talent

Get matched with your dream remote job

Sign up now and join over 250,000+ remote workers who receive personalized job alerts, curated job matches, and more for free!

Sign up
Himalayas profile for an example user named Frankie Sullivan