Overview

Make AI agents you can trust by measuring what they do and watching how they behave in production. You'll enforce structured outputs with schemas so an agent's responses stay predictable and machine-readable. Then you'll build automated evaluation frameworks that score an agent's task completion, quality, and reliability. You'll apply these skills to real document-extraction systems: a validated, routed insurance policy pipeline and a resilient mortgage extraction system that holds up under messy input. You'll also fuse disagreeing evidence sources into a single supply-chain risk assessment. A final project brings evaluation and observability together.

Syllabus

Introduction to Agent Evaluation and Observability with Claude

Get oriented before you start: see what the course covers, who it's for, and the outcomes you'll reach, plus the prerequisites and resources you'll use along the way.

Introduction to Structured Outputs

Learn how to use schemas to enforce structured outputs in Claude Agent SDK, transforming unpredictable text into reliable, validated data for robust applications.

Implement Structured Outputs

Learn to implement structured outputs in Claude Agent SDK by defining schemas with Zod, enabling reliable, validated data extraction for robust applications.

Build a Resilient Mortgage Document Extraction System

In this lesson, you'll build a production-shaped document extraction pipeline with the Anthropic API, one technique at a time. You'll design a JSON Schema that refuses to fabricate, orchestrate a two-pass classify-then-extract flow with forced and `any` tool choice, write an extractor prompt that stops the model from inventing values, and add a validator that catches arithmetic the model would otherwise silently "correct." Along the way you'll build the intuition for when each pattern is the right tool — and when it isn't.

Agent Evaluation

Agent Evaluation guides assessing an agent’s task completion, quality, tool use, and system metrics using response, step, or trajectory strategies to ensure reliable and efficient operations.

Evaluate Agents Built with Claude Agent SDK

Learn to build automated evaluation frameworks for Claude Agent SDK agents, using agent traces, evaluators, test cases, and schema validation to ensure reliable performance.

Build a Validated, Routed Insurance Policy Extraction Pipeline

In this lesson, you'll build a production-shaped document extraction pipeline with the Anthropic API, one piece at a time. You'll learn how to retry the mistakes a model can fix and escalate the ones it can't, batch a monthly workload for a cost discount, add an independent reviewer for a second opinion, and route each policy to the right human queue.

Investigate Supply Chain Risk with Multi-Source Synthesis

Build a Python system that fuses four disagreeing evidence sources into one honest, provenance-preserving briefing that annotates conflicts, escalates on explicit criteria, and keeps running when a source fails.

Evaluation and Observability Project

Operate three document and risk systems you built: run them, break them on purpose, and defend their design with your own evidence. No new code to write.