Minder - Faulty Machine Detection for Large-scale Distributed Model Training

Learn about Minder, an automatic faulty machine detection system designed for large-scale distributed model training environments in this 14-minute conference presentation from NSDI '25. Discover how researchers from Tsinghua University, ByteDance, Northeastern University, and Harvard University developed a solution to address the critical challenge of detecting machine failures in distributed training tasks that can involve thousands of machines simultaneously. Explore the key technical approach of automatically detecting distinctive monitoring metric patterns that occur before training tasks halt, replacing time-consuming manual scrutiny processes. Understand the real-world performance metrics of this production-deployed system, which has been monitoring daily distributed training tasks for over a year and can detect faults within 3.6 seconds on average with a precision of 0.904 and F1-score of 0.893. Gain insights into how this system addresses the significant operational challenge where training tasks typically encounter two faults per day on average, potentially causing hours-long halts in large-scale machine learning operations.