Understanding LLM Benchmark Quality - Who Watches the Watchmen?

This talk examines how LLM benchmarks measure model quality, including robustness to variations in human language and the limits of benchmark scores for specific applications. It surveys popular benchmarks and the dimensions of performance they assess.