How to Improve AI Apps with Automated Evals

Learn how to scale up the evaluation of AI applications through automated evaluation techniques in this comprehensive tutorial. Explore the challenges of evaluating open-ended LLM tasks that typically require human assessment and discover practical solutions using automated evals. Master the typical LLM workflow and understand common problems that arise when building AI applications. Dive deep into two distinct types of automated evaluations and their applications in real-world scenarios. Follow along with a detailed case study featuring an eval-driven LinkedIn Ghostwriter project that demonstrates the complete process from identifying failure modes to creating LLM judges. Gain hands-on experience with curating user inputs, generating content, applying evaluations, and refining results based on feedback. Access example code and references to implement these techniques in your own AI projects, and see a live demonstration of the automated evaluation system in action.