Skip to main content
Prashant JhaPJ
Looking for a job

Prashant Jha

@prashantjha

LLM Engineer building production RAG systems and evaluation pipelines to ship reliable AI.

India
Message

What I'm looking for

I’m looking for roles where I can build production RAG and LLM evaluation systems—turning messy user data into reliable, testable AI. I want to ship end-to-end pipelines, collaborate with product teams, and keep improving quality through rigorous metrics and iteration.

I’m an LLM Engineer and AI Engineer with 5 years of experience, transitioning from data science roots into full-stack LLM engineering. I focus on production-ready LLM pipelines—especially Retrieval Augmented Generation (RAG) and generative AI systems that behave consistently in the real world.

In my current work, I built an end-to-end AI job aggregation pipeline collecting 150+ daily remote AI/ML job postings with multi-region and multi-query coverage. I implemented a two-stage semantic filtering approach using embeddings and structured evaluation (OpenAI GPT models), reducing raw listings to ~30 highly relevant results.

I also designed evaluation-focused systems—like golden dataset generation for Copilot features and a document-grounding pipeline that creates reduced summaries and query-specific evidence views from enterprise DOCX/PPTX/PDF content. I’ve delivered multimodal extraction for visually rich documents, improved throughput with multithreaded batch execution and caching, and earned recognition such as the Winner of Smart India Hackathon 2020, a Stroke of Excellence Award 2023, and a published research paper on automated traffic management.

Experience

Work history, roles, and key accomplishments

Microsoft logoMI
Current

AI Job Discovery Engineer

Mar 2026 - Present (3 months)

Built an end-to-end job aggregation pipeline (Microsoft via Volga Partners) to collect 150+ daily AI/ML job postings across RemoteOK, We Work Remotely, and Google Jobs. Implemented two-stage semantic filtering with text-embedding-3-small to reduce 150+ raw listings to ~30 highly relevant results.

Microsoft logoMI

LLM Evaluation Data Scientist

Jun 2021 - Apr 2024 (2 years 10 months)

Designed and built golden datasets to evaluate Microsoft Copilot features across multimodal scenarios (image upload, image generation, file handling, and multi-turn queries). Created LLM-driven evaluation pipelines with human-in-the-loop validation to ensure quality, correctness, and scenario coverage.

Education

Degrees, certifications, and relevant coursework

JC

Jabalpur Engineering College

Bachelor of Engineering, Computer Science

2016 - 2020

Earned a Bachelor of Engineering in Computer Science at Jabalpur Engineering College from 2016 to 2020.

Find your dream job

Sign up now and join over 250,000+ remote workers who receive personalized job alerts, curated job matches, and more for free!

Sign up
Himalayas profile for an example user named Frankie Sullivan