Build a Learning Habit
Download Class Central's free printable study calendar
Download for Free
Learn how to build fair and reproducible benchmarks for comparing search engine performance in this 20-minute conference talk from Vector Space Day 2025. Discover the common pitfalls and misleading practices in search engine benchmarking, where different hardware configurations, datasets, and optimal parameters create unfair comparisons between competing engines. Explore the challenges of creating truly comparable benchmarks even when controlling for variables, as demonstrated through real-world examples from the industry. Understand the development process behind building a reproducible benchmarking harness and public leaderboard system designed to enable fair performance comparisons across modern search engines. Gain insights into the technical considerations and methodological approaches needed to establish trustworthy benchmarking standards in the search engine ecosystem.
