Turning Fails into Features - Zapier's Hard-Won Eval Lessons

Discover how to transform AI agent failures into systematic improvements through this 16-minute conference talk from the AI Engineer World's Fair. Learn Zapier's proven methodology for building evaluation systems that convert user frustrations into actionable development insights, creating a continuous data flywheel that strengthens AI agents over time. Explore practical strategies for constructing effective data flywheels, identifying implicit user feedback signals, and developing robust evaluation frameworks. Understand how to prioritize metrics that drive meaningful performance improvements while avoiding common evaluation pitfalls that can secretly undermine your AI systems. Gain insights from Zapier's AI Tech Lead and Staff Software Engineer who share hard-won lessons from building intelligent automation systems that serve millions of users, including their work on Zapier Agents, MCP, and AI-powered workflow builders.