Building CI/CD for Non-Deterministic AI Agents at Scale

Learn to build reliable CI/CD pipelines for non-deterministic AI agent systems in this 50-minute conference talk. Discover why traditional deployment patterns fail when dealing with AI agents that produce different outputs from identical inputs, and explore practical solutions for testing, deploying, and monitoring these unpredictable systems. Understand the unique failure modes of AI agents including inconsistency issues, environment-specific problems, and challenges arising from model changes. Master the concept of evaluations (evals) as a critical tool for measuring and improving AI agent performance, and learn how to integrate these evaluations directly into your CI/CD pipelines to quantify the impact of every code change on agent quality. Gain insights into building robust deployment processes that account for the inherent non-deterministic nature of AI systems while maintaining reliability and measurable improvements at scale.