norah liu
@norahliu
Principal Product Manager building agent evaluation frameworks that turn failures into iteration.
United StatesWhat I'm looking for
I build evaluation frameworks that help agents improve through iteration instead of churn. I bring 14 years of software product management experience from Microsoft and Alibaba Cloud, and I now lead the evaluation charter for Microsoft Agent Builder.
My work extends an eval discipline proven on generated apps to agents that act across connectors, MCP servers, and WorkIQ. I focus on attribution across multi-component failure modes and design customer-facing evaluation experiences so agent makers can configure their own evals.
In Power Apps Copilot, I owned AI evaluation end to end, directing vendor teams through rubric authoring, human grading, grader calibration, and quality-score aggregation. I also closed the loop between offline evals and production telemetry in Python, driving targeted reassessment of eval design and quality thresholds based on conversation-length behavior.
I lead with a clear thesis: agents rarely succeed on the first build. I’ve turned evaluation tooling into self-serve capabilities, mined real customer failure signals (like thumbs-down feedback), and scaled responsible AI quality norms without relying on direct management authority.
Experience
Work history, roles, and key accomplishments
Leads the evaluation charter for Microsoft Agent Builder, extending evaluation from generated apps to agents that act across connectors, MCP servers, and WorkIQ. Building a customer-facing, self-serve evaluation experience for agent makers to support iteration based on intent-to-behavior gaps.
Owned end-to-end AI evaluation for the Power Apps Copilot portfolio, directing vendor teams for rubric authoring, human grading, calibration, and quality-score aggregation. Built a closed loop between offline evaluation and production telemetry in Python and used real customer feedback to drive roadmap changes.
Introduced model-evaluation metrics and usage insights for the Power Platform to give makers visibility into model performance. Partnered with data scientists to translate evaluation outputs (e.g., confusion matrices) into a single quality score and shipped models into Power Apps and Power Automate.
Owned OneBox, the pipeline ingesting public sources into Microsoft’s Satori knowledge graph, including source onboarding, fact extraction, and conflation of contradictory partial records. Worked at large scale to unify entities powering Bing and Office.
Founding Product Manager, Commerce
Aliyun / Alibaba Cloud
Jan 2015 - Jan 2016 (1 year)
Built an international commerce platform from a blank page to first paying customers in about five months. Integrated PayPal and Bank of America, handled OneSource tax compliance, and rebuilt fraud detection after a launch-day attack.
Shipped V1 of an internal content authoring tool used by 100 tech writers. Supported development of the tool used by the DevDiv content authoring team.
Worked on fraud-signal exploration and false-positive trade-off analysis alongside data scientists. Helped maintain chargebacks below 2% across millions of daily transactions.
Education
Degrees, certifications, and relevant coursework
Johns Hopkins University, Whiting School of Engineering
Master of Science in Data Science, Data Science
2021 - 2025
M.S. in Data Science covering machine learning, data visualization, and neural networks from 2021 to 2025.
Nanjing University
Bachelor of Engineering, Computer Software Engineering
2008 - 2012
Grade: GPA: 4.3/5.0
B.E. in Computer Software Engineering from 2008 to 2012.
Availability
Location
United StatesAuthorized to work in
Job categories
Skills
Interested in hiring norah?
You can contact norah and 90k+ other talented remote workers on Himalayas.
Message norahGet matched with your dream remote job
Sign up now and join over 250,000+ remote workers who receive personalized job alerts, curated job matches, and more for free!
