Fault Tolerant Machine Learning Operations

Explore the challenges and solutions for running machine learning models in production in this keynote address from Code BEAM America 2025 by Chelsea Troy. Delve into critical questions about building resilient ML systems when using third-party compute resources, implementing consumer feedback and telemetry as core components, safely training on production data without introducing risks, and evaluating non-deterministic systems effectively. The presentation offers practical solutions applicable to any machine learning stack, featuring helpful diagrams and potential code examples while maintaining language-agnostic principles. Learn strategies for creating fault-tolerant machine learning operations that can withstand the unique challenges of ML in production environments.