Coursera Spring Sale
40% Off Coursera Plus Annual!
Grab it
Explore a comprehensive 45-minute conference talk that addresses the unique testing challenges posed by AI applications and provides practical solutions for maintaining quality assurance in unpredictable AI environments. Learn how AI applications fundamentally disrupt traditional QA practices through non-determinism, latency issues, rate limits, and schema violations that make assertion tests unreliable and cause continuous integration failures. Discover the additional complexity layers introduced by security and privacy requirements, including data anonymization, prompt sanitization, and leak prevention that extend beyond standard security scanning approaches. Understand how real-world user behaviors—including jailbreak attempts, informal queries, and varying RAG patterns between staging and production environments—create testing scenarios that traditional methods cannot adequately address. Examine how agent workflows compound these challenges through multi-step tool orchestration involving retries, approvals, and inter-agent coordination that exceed the scope of conventional unit or integration testing. Master a layered testing strategy that combines fast mock-based tests using mokksy.dev, behavioral tests for validating tool calls and security measures, and live-model sanity checks that leverage LLMs as evaluation and assertion mechanisms through promptfoo. See practical implementations demonstrated using the JetBrains Koog agentic framework to understand how these testing patterns work in real-world scenarios. Gain actionable techniques and a clear roadmap for navigating AI testing pitfalls while maintaining the optimal balance between development velocity and quality assurance in inherently unpredictable AI-driven environments.
