LLM Metrics - The Hard Truths Nobody Tells You About Production AI

Explore the harsh realities of evaluating Large Language Models in production environments through this 29-minute conference talk that goes beyond misleading single metrics. Discover critical lessons learned from real-world LLM platform development, including why semantic similarity and "LLM as a judge" approaches often fall short in practice. Learn why treating models as observable systems is essential and understand how to build metrics that directly address user issues while delivering measurable business value. Master the crawl, walk, run methodology for developing LLM metric maturity and avoid common pitfalls like dashboard overload that plague many AI implementations. Gain practical insights into the unexpected challenges that arise when deploying LLMs to production and develop strategies for building robust evaluation frameworks that actually work in real-world scenarios.