At DataMatter Technologies, I implemented an asynchronous pipeline orchestration engine using Celery and Redis, with real-time status updates and recovery for in-flight work.
I built 25+ production REST APIs with FastAPI for multi-tenant authentication, RBAC, job management, and data ingestion. I also developed APIs for filtering and transforming data.
I owned the Delta Lake file versioning module, which snapshots dataset states on modification and enforces a zero-overwrite policy across migrations.
I built chunk-based streaming for file uploads and downloads, processing 1–2GB files without memory overflow using PostgreSQL bulk-insert strategies. I’ve also worked with AWS S3 for file storage and EC2 for deployment.
