Coursera Flash Sale
40% Off Coursera Plus for 3 Months!
Grab it
Learn about a revolutionary approach to search system evaluation through persona-based testing in this 40-minute conference talk. Discover how to overcome the notorious complexity and instability of search evaluation by leveraging user modeling inspired by the "LLM as a judge" paradigm to automate query and result-set generation. Explore how this groundbreaking method dynamically creates realistic query-result pairs by simulating diverse user personas designed to evaluate different modalities including image, video, and audio content within catalogs under varying prompts. Understand the key innovations of this system including dynamic test coverage that adapts to catalog changes without manual intervention, multi-modality testing that mirrors modern search use case diversity, and a fully automated pipeline that eliminates manually written test cases while reducing overhead and accelerating iteration cycles. Gain insights into the methodology behind dynamic user modeling, examine real-world results from system implementation, and acquire actionable strategies for transforming search evaluation approaches to achieve new levels of stability and precision in search algorithms.
