Google, IBM & Meta Certificates — All 10,000+ Courses at 40% Off
One annual plan covers every course and certificate on Coursera. 40% off for a limited time.
Get Full Access
Explore an innovative approach to automating model-parallel training for large deep learning models in this 18-minute conference talk from OSDI '22. Discover how Alpa generates execution plans that unify data, operator, and pipeline parallelism, addressing the limitations of existing model-parallel training systems. Learn about Alpa's hierarchical view of parallelisms, its new space for massive model-parallel execution plans, and the compilation passes designed to derive efficient parallel execution plans. Understand how Alpa's runtime orchestrates two-level parallel execution on distributed compute devices, and examine its performance compared to hand-tuned systems. Gain insights into Alpa's versatility in handling models with heterogeneous architectures and those without manually-designed plans. Access the source code and explore the potential of this groundbreaking approach to scaling out complex deep learning models on distributed systems.
