Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

CodeSignal

Scoring LLM Outputs with Logprobs and Perplexity

via CodeSignal

Go to class Write review

Overview

In this course, you'll explore how to evaluate the fluency and likelihood of LLM outputs using internal scoring signals like log probabilities and perplexity. You'll work with OpenAI's completion models to analyze how models "think" under the hood. This course builds naturally on the first two by focusing on model-internal evaluation instead of external references.

Syllabus

  • Unit 1: Extracting Log Probabilities for Tokens
    • Fixing Token Probability Display Code
    • Making Token Probabilities Dynamic
    • Filtering Tokens by Probability Threshold
  • Unit 2: Comparing Sentence Likelihoods Using Log Probabilities
    • Extracting Log Probabilities from Responses
    • Comparing Sentences with Log Probabilities
    • Finding the Most Plausible Sentence
  • Unit 3: Calculating Perplexity in Language Models
    • Implementing the Perplexity Formula
    • Applying Perplexity to Real Sentences
    • Flexible Token Generation for Perplexity Analysis
    • Error Handling for Perplexity Calculations
  • Unit 4: Model Fluency Comparison in Language Models
    • Extracting Token Text from API Responses
    • Calculating Perplexity for Model Comparison
    • Evaluating Multiple Sentences for Fluency

Reviews

Start your review of Scoring LLM Outputs with Logprobs and Perplexity

Go to class

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Sign up for free
Someone learning on their laptop while sitting on the floor.