Judging LLMs - Evaluating and Improving AI Engineering Products

Learn how to effectively evaluate and improve Large Language Model (LLM) applications in this conference talk from the AI Engineer World's Fair. Discover practical strategies for iterating on AI products, implementing robust evaluation frameworks, and overcoming the common challenge of building LLM-powered applications without proper assessment mechanisms. Explore real-world approaches to judging LLM performance, understanding evaluation metrics, and establishing feedback loops that enable continuous improvement of AI systems. Gain insights from an AI evangelist with extensive full-stack engineering experience on how to move beyond basic LLM implementation to create measurable, improvable AI products that can be systematically enhanced over time.