Evaluating AI Search - A Practical Framework for Augmented AI Systems

Learn to evaluate AI-powered search systems through a comprehensive framework presented in this 21-minute conference talk from the AI Engineer World's Fair. Discover practical evaluation methods for Retrieval-Augmented Generation (RAG), Search-Augmented Generation (SAG), and custom AI agents across three critical dimensions: source relevance to queries, answer completeness, and faithful use of sources in generated responses. Explore real-world lessons from search companies and examine early benchmark findings comparing popular augmented AI systems, focusing on understanding where different approaches excel or fail rather than simple rankings. Gain insights into how evaluation tradeoffs inform product decisions and learn to build trusted, high-quality AI search experiences with measurable performance indicators. Benefit from expertise shared by Julia Neagu (CEO of Quotient AI and former Director of Data for GitHub Copilot), Deanna Emery (Founding AI Researcher at Quotient AI), and Maitar Asher (Head of Engineering at Tavily), who bring extensive experience in AI evaluation, language model assessment, and search infrastructure development.