Learn to distinguish between predicting recorded outcomes and grouping unfamiliar patterns. You’ll practice assessing if data reflects your real-world cases and treating evaluation as a business decision. Master the "Fresh-Case Check" and the "Two-Mistake Trade-off" to weigh missed cases against false alarms, ensuring AI performance aligns with your operational tolerance for risk.
Overview
Syllabus
- Unit 1: Choosing Defensible Inputs
- Understanding Inputs in Everyday Work
- Screening a Proposed Input List
- Trimming the Input List
- Unit 2: Spotting Leakage and Proxies
- Spotting Hindsight Data in Predictions
- Sorting a Field List: Usable, Hindsight, Proxy
- Catching the Too Good Number
- Unit 3: Reading Confidence Scores
- What Confidence Scores Really Mean
- Judging Confidence Scores in Refunds
- Briefing a Team on Confidence Bands
- Unit 4: Questioning Correlation Before Policy
- Sorting Claims About a Finding
- Quick Check: Pattern, Cause, and Next Evidence
- Defending the Claim Before the Board