Testing AI Agents - A Practical Framework for Reliability and Performance

Learn to build robust testing frameworks for AI agents in production through this 25-minute conference talk from MLOps World GenAI Summit 2025. Discover practical strategies for ensuring reliability, safety, and consistency in AI agents powered by large language models as they become critical components of production systems. Explore iterative regression testing fundamentals, including how to design, execute, and refine tests that detect failures and performance drifts as agents evolve over time. Examine a concrete case study showcasing real-world deployment experience, covering unit tests for tools, adversarial testing for robustness, and ethical testing for bias and compliance. Understand the automated testing pipelines developed at PagerDuty for test execution, scoring, and benchmarking that enable faster iteration and continuous improvement. Master techniques for testing correctness, robustness, and ethical alignment while learning why conventional testing methods fail for agentic systems and what replaces them. Gain insights from deploying reliable AI agents at scale and implementing comprehensive testing strategies that address the unique challenges of AI agent reliability and performance in production environments.