Beyond LLM-as-Judge Towards LLM Relevance Engineering

Explore advanced search relevance engineering techniques in this 42-minute conference talk that moves beyond basic LLM-as-judge approaches to implement sophisticated ensemble judging systems. Learn how to deploy multiple specialized LLM judges, each focused on different search strategies, and compare their performance against human ground truth to identify the most effective relevance signals. Discover how to leverage judge agreement patterns to guide feature development - from recognizing when product name matching achieves 80% human agreement to implementing query expansion features that boost agreement to 90%. Master the implementation of an LLM-in-the-loop search relevance process that treats individual LLM judges as modular components while using traditional machine learning as connective tissue to understand feature inter-relationships. Gain insights into how different judges complement each other, such as when query expansion becomes most valuable in the absence of strong product name matches, and how this knowledge can inform targeted improvements like description-only searching in specific scenarios.