Fazal User

I’m a Big Data Engineer with 1 year of experience building and supporting scalable data platforms in enterprise environments. I design, develop, and maintain AWS-based data pipelines to process large-scale datasets for analytics and reporting.

In my recent role at Nityo Infotech Services (Client: Infosys), I built an AWS pipeline that reads from S3 and Snowflake, transforms data with Spark and PySpark, and delivers results to S3 and Redshift. I automated ingestion, transformation, and orchestration using Airflow so teams could rely on clean, accurate, ready-to-use data.

I optimize batch processing for performance, scalability, and reliability by improving Spark jobs through partitioning, caching, and minimizing shuffles. I also monitor daily executions, troubleshoot failures via logs and DAGs, and add data quality checks to ensure integrity before loading.