Data Engineer, Fullstack Engineer, Terraform Automation, Staff Data Engineer, Senior Data Engineer
Piyush Patel
@piyushpatel7
Professional Data and AI consultant
What I'm looking for
I'm a Software Engineer with over 12 years' experience including 5 years with technologies like Apache Spark, Kafka, Streamsets, Spark Streaming, Apache Beam (Dataflow), MapReduce, Hive, Pig, Sqoop, Impala, HBase, Oozie, Apache Flume, Tez, Impala, Hue, Azure Data Factory, AWS Glue, Redshift, Terraform, Databricks, etc. I was also working for application development team with various technologies like Java, Scala. Play framework, Akka, Spring, JavaScript, Python, Django, SQL, NoSQL and many more. I've also got three major cloud providers experience and latest Kubernetes and containerization experience under my belt. I have worked with latest automation build tools like Terraform, Ansible for infrastructure deployment in cloud. I'd love to combine my diverse skillsets and keep exploring new exciting technologies and challenges.
Big Data technologies: Hadoop, HDFS, Hive, Confluent, AWS, Shell Scripting, Pig, Sqoop, Scala, Spark, Flink, Beam, Oozie, Flume, Kafka, Streamsets, NiFi, AWS Glue, Azure Data Factory, Redshift, Azure Synapse, Databricks, Zeppelin, Blob storage, Google DataFlow, Dataproc, Presto, Iceberg
AI Stack: LangGraph, OpenAI, LangSmith, Google Gemini, Claude, Cursor, Copilot, Spec Driven Development
Programming Languages: Java, Scala, JavaScript, Python, R, Bash Scripting, YAML, XML, JSON
Web Technologies: HTML, CSS, Bootstrap, AngularJS, SQL, Angular
Databases: MySQL, MongoDB, HBase, Oracle SQL Server, Hive, Impala, Google Big Query
Cloud Platforms: Google Cloud Platform (GCP), AWS, Microsoft Azure
Frameworks: ExpressJS, Spring, Play Framework, Hibernate, Spring Boot, Angular, Mongoose
Build tools: Maven, NPM, Grunt, Gulp, SBT, Jenkins
IDEs and Editors: Eclipse, IntelliJ IDEA, Atom, Sublime Text, Visual Studio Code
Testing Frameworks: JUnit, Jasmine, Spark testing base, Scala Tests
Additional Tools: Git, Bitbucket, Bamboo, Gitlab, JIRA, Confluence
Experience
Work history, roles, and key accomplishments
- Supported feature development and bug fixes for promotions, personalization API and data services written in Python, FastAPI
- Driving AI driven development efforts including MCP, Spec-driven development, etc.
- Created internal knowledge base powered by AWS Bedrock, Qdrant and LangGraph
- Set up internal Eval pipeline using LangSmith to enhance observability and performance tuning of our AI s
Worked with several medium to enterprise scale clients to deliver high value growth oriented data products.
These products helped them make better marketing, financial and infrastructure decisions while saving costs and increasing revenues.
- Developed terraform configuration files to spin up GCP environment for data pipelines migration
- Implement highly efficient Stream processing pipeline with Apache Kafka and Spark Streaming
- Helped team migrate large PB scale cluster to GCP cloud storage using distcp
- Developed Apache Beam code to port existing spark code to GCP Dataflow in Java/Scala language.
- Optimize spark and streaming j
- Developed Hive scripts for data warehousing and data analysis
- Developed shell scripts to automate the data ingestion process into Hive warehouse
- Worked with Apache NiFi to develop automated data flow pipelines to ingest data into Hadoop ecosystem
- Fixed the features for data ingestion as required by the client.
- Developed Software solutions for data ingestion along with data modelling for
- Supporting product development features for company's television platform.
- Worked on Angular 4 , TypeScript for features and bug fixes
- Worked on NodeJS application for new API endpoint implementations
- Developed API endpoints in Java Spring Boot application for one of the REST services
- Integrated Loggly web service for Front end issues monitoring using Angular Global exception handler
- W
- Developed ETL jobs using Scala and Java as a programming language with Spark computation engine
- Used RDDs to transform and store data
- Used DataFrames for interactive queries
- Created ETL jobs in Talend to load data warehouse
- Gained domain knowledge of financial, retail and relation between certain traits
- Interacted with onshore team on a daily basis for the project
- Loaded data from te
- Sqoop data from RDBMS like oracle 12c server to HDFS system
- Transform the data based on the client requirements
- Partiticipated in design of workflows for Big Data ETL project
- Prepare documents with instructions for cluster deployment for the admin team
- Test the product on pre-production cluster with desired outputs
- Write Hive and Pig scripts to extract, transform and load data into/fr
- Developed backend software solutions using JavaScript, NodeJS, ExpressJS, Java, Spring, Hibernate technologies
- Worked on front end interactive applications using AngularJS and/or jQuery libraries.
- Developed REST APIs for seamless integration with Angular client
- Developed schemas for MongoDB database using Mongoose
- Performed manual testing while developing application
- Worked on Grunt sc
Analyzed and prepared total system architecture for different web projects.
Worked on various projects with Java web frameworks like Struts and Servlets
Developed applications using Servlets, JSP and Spring framework
Explained and prepare curriculum of Core Java and Advanced Java training courses.
Trained university students with the core concepts of C and Java languages.
Conducted labs and hands
Education
Degrees, certifications, and relevant coursework
Sardar Patel University
Bachelor of Engineering (B.E.), Electrical
- Graduated with Honors from Sardar Patel University in Gujarat and scored excellent grades with good control over theory as well as practical aspects of engineering.
- Received great admirations from my teacher and fellow students for excellent record during the span of four years.
- Successfully completed a project on Power Electronics Motion control for DC motor using SCR technology with self
Sardar Patel University
Bachelor of Engineering - BE, Electrical and Electronics Engineering
- ranked number one in my batch of Electrical engineering graduates.
Coursera
Google Cloud Certificates, Computer Engineering
2019 - 2020
Some of the courses and topics include:
- Essential Cloud infrastructure
- Elastic Cloud: Scaling and Automation with Deployment manager
- Reliable Infrastructure: Design and Process
- App Deployment, Debugging and Performance on GCP
- Building Batch Data pipeline with Dataproc and Dataflow
- Building Resilient Streaming Analytics Systems on GCP (PubSub, Dataproc, Data Studio)
- Modernizing Data w
Harvard Extension School
Data Science Specialization, Data Science
2018 - 2019
This program covers many of the concepts of Data Science in the real world using R programming language. It covers machine learning and different algorithms, Data visualization, etc.
edX
Certificate, Data Analytics
2018 - 2018
Sikkim Manipal University - Distance Education
Certificate in Business Analytics, IT
2016 - 2017
Coursera
Full Stack Web Development Specialization, Web Development
2014 - 2015
freeCodeCamp
Full Stack Web Development Certification, Computer Science
2014 - 2015
On the way to acquire my first certification in FreeCodeCamp which is "Front End Developer certification". During this certification, I solved many Design challenges and developed Design and layout projects using HTML, CSS, Bootstrap. Towards the later phase of the certification, I added interactivity in the design and solved many algorithm challenges to develop concrete understanding of Algorithm
Udacity
Full Stack and Cloud Engineer Nanodegree, Information Technology
2014 - 2015
- Built an interactive resume with the help of jQuery.
- Built a profile website with Bootstrap framework.
- Learn the concepts of Responsive web design with image optimization
- Optimized the workflow with the help of Grunt and Gulp
D. N. High School, Anand
High School, Science
2000 - 2002
Graduated with high grades and went on to study at the Univerisity.
Udacity
Cloud DevOps Nanodegree, Computer Science
2019 - 2020
In this nanodegree program, I worked to deploy large scale websites, static website hosting using Amazon S3, event driven application deployment using Amazon Lambda
Sardar Patel University
Bachelor of Engineering, Electrical
Graduated with Honors from Sardar Patel University in Gujarat, scoring excellent grades with strong command over theory and practical aspects of engineering. Completed a project on Power Electronics Motion control for DC motor using SCR technology with self-designed control circuitry and boards.
Sardar Patel University
Bachelor of Engineering, Electrical and Electronics Engineering
Ranked number one in the batch of Electrical engineering graduates.
Tech stack
Software and tools used professionally
AWS Shell
Amazon Redshift
Snowflake
Azure Synapse
Apache Spark
Presto
AWS Glue
Talend
Data Studio
Microsoft Azure
Google Cloud Platform
Amazon S3
Google Cloud Storage
GitHub
GitLab
Bitbucket
Kubernetes
Jenkins
GitHub Actions
Bamboo
PySpark
dbt
Sqoop
MySQL
PostgreSQL
MongoDB
Cassandra
Hadoop
HBase
Django
Spring Boot
Spring Framework
Spring MVC
Yarn
Databricks
Terraform
Visual Studio
IntelliJ IDEA
Jira
jQuery
AngularJS
JavaScript
Python
HTML5
Java
JSON
Scala
XML
Apache Flume
Kafka
Apache NiFi
ExpressJS
FastAPI
Linux
Confluence
Google Cloud Dataflow
Ansible
AWS Lambda
Kafka Streams
JUnit
Visual Studio Code
Sublime Text
Apache Beam
Google BigQuery
Amazon Web Services (AWS)
SQL
npm
Loggly
Akka
Elastic Cloud
Qdrant
Port
Cursor
Trino
Terragrunt
GitHub Copilot
Bash
LangGraph
LangSmith
Claude Code
Factory
Beam
Availability
Location
Authorized to work in
Salary expectations
Social media
Job categories
Skills
Interested in hiring Piyush?
You can contact Piyush and 90k+ other talented remote workers on Himalayas.
Message PiyushGet matched with your dream remote job
Sign up now and join over 250,000+ remote workers who receive personalized job alerts, curated job matches, and more for free!
