Ayush Patwa

I’m a Senior Data Engineer focused on building high-concurrency Big Data and real-time streaming pipelines. I design ingestion and metadata extraction systems that can scale for massive file workloads and keep data workflows reliable under heavy operational pressure.

In my current role at v4c.ai, I leverage Databricks and Autoloader to enable seamless streaming and metadata extraction for large S3-hosted genomic files (CRAM/FASTQ). I’ve built a resilient ingestion framework with automated retry mechanisms for job failures and delivered Databricks AI/BI dashboards for real-time monitoring of pipeline health and file extraction status.

I also drive performance benchmarking and optimization across classic and serverless compute clusters. By tuning execution paths and reducing inefficiencies in long-running jobs, I’ve cut job run times from 30 hours to 4 hours while minimizing data extraction latency and resource overhead for large-scale genomic workflows.

Previously as a Consultant at Polestar Analytics, I spearheaded a Qlik-to-Fabric migration and redesigned scalable PySpark + Microsoft Fabric ELT pipelines to improve stakeholder-ready reporting. I’ve delivered medallion architectures, implemented row-level and column-level security, optimized Snowflake query runtime by 50%, improved data integrity with conformance checks, and established CI/CD automation to keep deployments dependable.