Distributed Training Courses and Certifications

MIT HAN Lab

Distributed Training Methods and Parallelization Techniques - Lecture 19

Explore advanced distributed training concepts, from data parallelism and communication primitives to pipeline, tensor, and sequence parallelism methods for efficient machine learning systems.

• YouTube
• 1 hour 10 minutes
• Self-Paced
• Free Video

IEEE Signal Processing Society

Distributed Training of Modern AI Systems

Discover distributed training techniques for modern AI systems and learn scalable approaches to handle large-scale machine learning workloads efficiently.

• YouTube
• 1 hour 15 minutes
• Self-Paced
• Free Video

OpenInfra Foundation

Heterogeneous Hybrid Distributed Training for Large-Scale Language Models

Explore the technical challenges and solutions in distributed training for heterogeneous GPU clusters, focusing on accelerating large language model development across different computing architectures.

• YouTube
• 11 minutes
• Self-Paced
• Free Video

USENIX

LeapGNN: Accelerating Distributed GNN Training Leveraging Feature-Centric Model Migration

Discover LeapGNN, a feature-centric framework that accelerates distributed GNN training by bringing models to vertex features, reducing communication bottlenecks through micrograph-based training and feature pre-gathering techniques.

• YouTube
• 14 minutes
• Self-Paced
• Free Video

USENIX

Minder - Faulty Machine Detection for Large-scale Distributed Model Training

Discover Minder, an automatic faulty machine detector for large-scale distributed model training that identifies faults within 3.6 seconds, achieving 0.904 precision in production environments.

• YouTube
• 14 minutes
• Self-Paced
• Free Video

AWS Events

NVIDIA Run:ai and Amazon SageMaker HyperPod Integration for Distributed Training

Explore how NVIDIA Run:ai integrates with Amazon SageMaker HyperPod to streamline large-scale AI training with centralized GPU management and hybrid-cloud bursting capabilities.

• YouTube
• 51 minutes
• Self-Paced
• Free Video

Anyscale

How to Get Started with Distributed Training at Scale

Master distributed training strategies for scaling deep learning models using data, model, and pipeline parallelism with PyTorch and Ray to overcome bottlenecks and system failures.

• YouTube
• 31 minutes
• Self-Paced
• Free Video

Open Compute Project

Networking for Multi-Region Distributed Training

Explore networking challenges and innovative solutions for multi-region distributed training of large language models across datacenters.

• YouTube
• 13 minutes
• Self-Paced
• Free Video

USENIX

ZEN - Empowering Distributed Training with Sparsity-driven Data Synchronization

Discover ZEN, a gradient synchronization system that leverages tensor sparsity to accelerate distributed deep learning training by up to 5.09x in communication time.

• YouTube
• 17 minutes
• Self-Paced
• Free Video

Anyscale

Getting Started with Distributed Training at Scale

Master distributed training techniques using PyTorch and Ray to scale ML models from single-GPU to massive clusters with DDP, ZeRO, and FSDP methods.

• YouTube
• 1 hour 12 minutes
• Self-Paced
• Free Video

Custom and Distributed Training with TensorFlow

438 ratings at Coursera

Explore advanced TensorFlow techniques for custom training loops, graph mode optimization, and distributed training on GPUs and TPUs. Gain flexibility and efficiency in model development.

• Coursera
• 1 day 15 minutes
• Self-Paced
• Paid Course

Let's Build Pipeline Parallelism from Scratch - Tutorial

Master pipeline parallelism by building distributed AI training systems from scratch, implementing GPipe micro-batching and 1F1B algorithms across multiple GPUs step-by-step.

• freeCodeCamp
• 3 hours 22 minutes
• Self-Paced
• Free Video

USENIX

Accelerating Neural Recommendation Training with Embedding Scheduling

Optimizing deep learning recommendation models through embedding scheduling, reducing communication overhead and improving training performance without compromising accuracy.

• YouTube
• 15 minutes
• Self-Paced
• Free Video

Anyscale

Redesigning Scheduling in Ray to Improve Cost-Efficiency at Scale

Explore Ray scheduling features to optimize AI applications, enhancing performance and cost-efficiency. Learn placement groups, graceful node draining, and label-based scheduling for faster, cheaper operations.

• YouTube
• 30 minutes
• Self-Paced
• Free Video

USENIX

SimAI - Unifying Architecture Design and Performance Tuning for Large-Scale Large Language Model Training with Scalability and Precision

Discover SimAI, a unified simulator for large-scale LLM training that achieves 98.1% alignment with real-world results through high-fidelity integration and multi-thread acceleration.

• YouTube
• 11 minutes
• Self-Paced
• Free Video