Primus - Unified Training System for Large-Scale Deep Learning Recommendation Models

Learn about Primus, a unified training system designed to handle large-scale deep learning recommendation models (DLRM) in production environments through this 17-minute conference presentation from USENIX ATC '25. Discover how ByteDance developed this comprehensive system to address the growing computational and storage demands of modern recommendation models by unifying training resources, data management, and training paradigms. Explore the system's three key innovations: a unified resource abstraction that enables dynamic horizontal and vertical scaling across multiple scheduling systems, a three-tier data definition framework with data task graph generation for orchestrating multi-source training samples from both batch and stream data, and a novel hybrid training paradigm that maintains high model timeliness through controlled parameter updates and fine-grained prioritization of mixed data types. Examine the real-world performance results from five years of deployment at ByteDance, including 17.1% reduction in training costs at the cluster level, increased CPU utilization from 50% to 80% per job, 23× acceleration in task generation, improved training throughput, and 0.4%-2.4% increases in advertising revenue across four different DLRM implementations. Gain insights into enterprise-grade solutions for scaling deep learning recommendation systems and understand the practical challenges and solutions in managing large-scale machine learning infrastructure.