Fuzzing in the GenAI Era - AI System Evaluation and Quality Assurance

Explore a comprehensive 19-minute conference talk that redefines AI evaluation through the lens of fuzzing methodology, moving beyond traditional static dataset approaches to dynamic stress testing of AI systems. Learn how to identify and address the "last mile problem" in AI applications, where systems appear to work well in controlled environments but fail when exposed to real-world user interactions and edge cases. Discover the brittleness inherent in GenAI applications through concrete examples of chatbot failures and understand why standard evaluation methods fall short in capturing these vulnerabilities. Master the concept of "Haizing" - a systematic approach to simulating unexpected user inputs at scale to uncover corner cases before deployment. Examine two critical components of robust AI evaluation: developing quality metrics that accurately capture human judgment criteria and generating diverse, representative stimuli that can expose potential system failures. Understand how to scale evaluation processes using AI agents as judges, balancing accuracy with latency considerations, and explore reinforcement learning techniques for tuning evaluation systems. Differentiate between fuzzing and adversarial testing approaches in AI contexts, and see how simulation can be framed as prompt optimization. Analyze real-world case studies including implementations at major European and Fortune 500 banks, demonstrating practical applications of these evaluation methodologies for voice agents and AI applications in financial services.