Analyze Agent Performance: Build and Test

Overview

Analyze Agent Performance: Build and Test is an intermediate course for data analysts, ML engineers, and developers tasked with optimizing AI systems. In a world where agentic AI is increasingly common, it is not enough to build an agent—you must prove its effectiveness. This course equips you with the data-driven skills to measure, monitor, and improve AI agents built with frameworks like LangChain, Autogen, and CrewAI. You will learn to transform raw, noisy logs into actionable KPIs by applying data aggregation techniques with SQL and dbt. Through hands-on labs, you will design and execute controlled A/B experiments, comparing agent versions to identify meaningful improvements. You will master core statistical methods, including the Chi-square test, to determine whether your results are statistically significant or just random chance. You will be able to move beyond correlation to causation, making objective, evidence-based recommendations on deploying agent enhancements.

Syllabus

Data Transformation for Business Intelligence

This module establishes the foundation for effective AI agent performance analysis. Learners will move beyond raw system logs to create structured, high-level metrics suitable for business intelligence and monitoring. The module focuses on applying data aggregation techniques with SQL and dbt to transform operational data into meaningful key performance indicators (KPIs) like conversation counts and latency.

Statistical Significance in Agent Experiments

Module Description: This module equips learners with the skills to scientifically prove the effectiveness of changes to their AI agents. Learners will move from correlation to causation by designing and analyzing controlled A/B experiments. The module provides hands-on experience with statistical hypothesis testing, focusing on the Chi-square test to determine if observed performance improvements are statistically significant.