aum khant
@aumkhant
Emerging Lead building scalable LLM infrastructure and responsible AI guardrails at State Street.
IndiaWhat I'm looking for
I’m an Emerging Lead at State Street, focused on building reliable, scalable infrastructure for self-hosted LLMs and strengthening responsible AI controls. I’ve designed systems that reduce memory leaks, improve availability, and keep guardrails effective against malicious prompt patterns.
Recently, I built an automated gunicorn worker life-cycle manager that monitored RSS memory and gracefully restarted workers past configurable thresholds, improving Guardrail service availability to 99.9% uptime. I also optimized guardrail memory usage by eliminating unnecessary FastEmbed model loading across workers, reducing memory footprint by 40%+.
I lead request-driven auto-scaling for self-hosted LLMs using vLLM, delivering 2× faster responses and a 30% GPU cost reduction. Before that, I quantified and fine-tuned LLMs using QAT/AWQ/GPTQ and PEFT (LoRA), built secure AKS–Databricks integration for PromptFlow and evaluation workflows, and implemented guardrail/jailbreak detection microservices that prevented 95%+ malicious prompts.
Experience
Work history, roles, and key accomplishments
Designed and implemented an automated Gunicorn worker life-cycle manager to monitor RSS memory and gracefully restart workers exceeding thresholds, mitigating memory leaks and improving service availability to 99.9%. Built vLLM-based request-driven auto-scaling for self-hosted LLMs and reduced memory by eliminating unnecessary FastEmbed model loading across workers.
Quantized fine-tuned LLMs using QAT, AWQ, and GPTQ to improve inference speed by 1.5× and reduce memory requirements. Built integrations and orchestration for AKS–Databricks workloads, developed PEFT/LoRA data curation and fine-tuning, and implemented guardrail/jailbreak detection microservices for a Responsible AI Gateway.
Built scalable FastAPI services for the Responsible AI Gateway, enabling secure RBAC-controlled access to LLM services for 100+ teams. Engineered Helm-based deployment pipelines with Azure DevOps for AKS microservices, and contributed to an AI-enabled MRM platform for automated model documentation evaluation.
Education
Degrees, certifications, and relevant coursework
Institute of Technology, Nirma University
Bachelor of Technology (B.Tech), Computer Science Engineering
2019 - 2023
Grade: CGPA: 8.1
B.Tech in Computer Science Engineering at Institute of Technology, Nirma University, completed in 2023.
Tech stack
Software and tools used professionally
Availability
Location
IndiaAuthorized to work in
Portfolio
github.com/aumJob categories
Interested in hiring aum?
You can contact aum and 90k+ other talented remote workers on Himalayas.
Message aumGet matched with your dream remote job
Sign up now and join over 250,000+ remote workers who receive personalized job alerts, curated job matches, and more for free!
