I am a Senior Bigdata Engineer with over 10 years of experience in the financial domain. Throughout my career, I have developed expertise in the areas of Data warehouse and Bigdata, with a focus on Apache Spark with Python, ETL, SQL (Oracle), and Shell Script. I have a strong background in the Agile methodology of development and DevOps tools, which has allowed me to effectively lead teams and deliver high-quality solutions.
At Wells Fargo International Solutions Private LTD, I led a team in architecting an end-to-end compliance monitoring system by harmonizing requirements from global stakeholders. I developed an operational configuration-driven ETL pipeline to consume data from multiple sources and populate the Data warehouse using the Bigdata (Pyspark) framework. By optimizing and automating complex logic, I was able to save approximately 1800 hours per year. I also played a key role in data modeling and improving the performance of legacy code.
During my tenure at JPMorgan Services India Pvt. Ltd., I collaborated with product managers to design and implement a compliance monitoring and alert model for international customers. I developed data ingestion pipelines and transformation processes using Apache Spark, Python, AWS S3, AWS RDS, and Kubernetes. I also led the design and development of a high-performance ETL environment and implemented data validation and cleaning frameworks. Additionally, I played a crucial role in transitioning to Agile methodologies and DevOps tools.
