LLM Evaluation Techniques in Practice

Learn how to benchmark large language models using multiple-choice QA, summarization, and scoring techniques like fuzzy matching, ROUGE, and semantic similarity. Compare GPT models across tasks and dive into internal evaluation with log probabilities and perplexity.

This course explores benchmarking for open-ended generation tasks like summarization. You'll experiment with different prompting styles, compare models like GPT-3.5 and GPT-4, and evaluate results using both fuzzy string similarity and semantic similarity via embeddings.

In this course, you'll explore how to evaluate the fluency and likelihood of LLM outputs using internal scoring signals like log probabilities and perplexity. You'll work with OpenAI's completion models to analyze how models "think" under the hood. This course builds naturally on the first two by focusing on model-internal evaluation instead of external references.