The Challenge of Valid Evaluations in Predictive Algorithm Deployment

Learn about the fundamental challenges of evaluating predictive algorithms in high-stakes social systems through this 33-minute conference talk by Amanda Coston from UC Berkeley at the Simons Institute. Explore how standard evaluation practices can misrepresent model performance and disparities across groups when dealing with biased, incomplete, or noisy data. Discover how the deployment of models can alter observed outcomes, creating additional evaluation complexities. Examine common threats to valid evaluation including measurement error, selection bias, and distribution shift. Understand principled evaluation methods that enable accurate performance assessment under these challenging conditions when appropriate requirements are satisfied. Gain insights into bridging the gap between prediction and intervention problems in social systems, with practical approaches for conducting reliable evaluations in real-world domains where algorithmic decisions have significant societal impact.