How Amazon Turns Real Failures Into Better AI Models

How does Amazon build its agentic AI? Michael Giannangeli, Head of Product for Amazon Nova and Agentic AI, breaks down evals, RL gyms, and model routing. He also explains why the bottleneck in software has shifted away from engineering hours and what takes its place.In this video, we cover:The eval lifecycle: building from real failure modes, saturation, and why 100% means deleteRL gyms: training models on real environments like migrations, DevOps, and pen testingModel routing, cost-per-token trade-offs, and why routing isn't solvedThe agent stack of an Amazon product lead: Claude Code, Codex, and KiroAutonomous migrations, trust, and how much human-in-the-loop survivesFor engineers and product people building with AI agents who want to see how a frontier lab actually closes its feedback loops.Recorded at the AI4 conference 2026. Timestamps:00:00:00 - Intro00:00:36 - The Agents an Amazon Product Lead Uses Daily00:03:36 - Why Nobody's Heard of Amazon Nova00:04:55 - Model Costs and the Routing Problem00:08:10 - Why Building Good Evals Is So Hard00:10:05 - When Evals Saturate and Get Deleted00:12:17 - Turning Real Failure Modes Into Hundreds of Evals00:15:26 - Improving Models Without Training on Customer Data00:18:26 - If Everyone Uses Agents, You Need Agents00:20:22 - The Bottleneck Is No Longer Engineering Hours00:23:20 - Ship Fast to Validate the Right Thing00:26:44 - Staying at the Frontier Amid Constant Noise00:29:37 - Spend 10-20% of Your Time Experimenting00:32:54 - RL Gyms: How Models Learn From Failure00:37:09 - Will Migrations Become Fully Autonomous?Guest: Michael Giannangeli - Head of Product, Agentic AI & Amazon Nova at Amazon#AmazonNova #AgenticAI #AIEngineering