Distributed Training Courses and Certifications

Stanford CS149 - Parallel Computing 2023

18 ratings

Master parallel computing fundamentals from multi-core CPUs to GPUs, covering hardware design, programming techniques, and performance optimization for modern systems.

• YouTube
• 1 day 19 minutes
• Self-Paced
• Free Video

Custom and Distributed Training with TensorFlow

438 ratings at Coursera

Explore advanced TensorFlow techniques for custom training loops, graph mode optimization, and distributed training on GPUs and TPUs. Gain flexibility and efficiency in model development.

• Coursera
• 1 day 15 minutes
• Self-Paced
• Paid Course

Let's Build Pipeline Parallelism from Scratch - Tutorial

Master pipeline parallelism by building distributed AI training systems from scratch, implementing GPipe micro-batching and 1F1B algorithms across multiple GPUs step-by-step.

• freeCodeCamp
• 3 hours 22 minutes
• Self-Paced
• Free Video

Algorithms for Big Data

11 ratings

Explore advanced algorithmic techniques for handling massive datasets, enhancing your ability to process and analyze big data efficiently.

• YouTube
• 1 day 10 hours
• Self-Paced
• Free Video

Systems Thinking In Public Health

5 ratings

Explore systems thinking in public health, learning to analyze complex problems, create models, and develop strategies that account for real-world complexities and unintended consequences.

• Coursera
• 14 hours 54 minutes
• Self-Paced
• Paid Course

Computer Architecture

6 ratings

In this course, you will learn to design the computer architecture of complex modern microprocessors. All the features of this course are available for free. It does not offer a certificate upon completion.

• Coursera
• 2 days 1 hour 38 minutes
• Self-Paced
• Paid Course

Parallel programming

6 ratings

Apprenez les fondamentaux de la programmation parallèle, de la parallélisation des tâches à celle des données. Maîtrisez les collections parallèles et appliquez ces concepts à travers des exemples pratiques analysant des données réelles.

• Coursera
• 1 day 8 hours 59 minutes
• Self-Paced
• Paid Course

HOTI - Hot Interconnects Symposium

ASTRA-sim and Chakra - Co-design Exploration for Distributed Machine Learning Platforms

Explore co-design methodologies for distributed ML platforms using ASTRA-sim and Chakra frameworks, focusing on system optimization and performance analysis.

• YouTube
• 1 hour 58 minutes
• Self-Paced
• Free Video

USENIX

Conspirator - SmartNIC-Aided Control Plane for Distributed ML Workloads

Innovative control plane design using SmartNICs to optimize distributed ML workloads, reducing bottlenecks and improving resource utilization for enhanced performance and efficiency.

• YouTube
• 18 minutes
• Self-Paced
• Free Video

USENIX

Accelerating Neural Recommendation Training with Embedding Scheduling

Optimizing deep learning recommendation models through embedding scheduling, reducing communication overhead and improving training performance without compromising accuracy.

• YouTube
• 15 minutes
• Self-Paced
• Free Video

Anyscale

Overcoming Distributed ML Challenges with Ray Train

Unlock efficient distributed ML pipelines with Ray Train. Explore solutions for setup, integration, debugging, fault tolerance, and data processing challenges in scalable machine learning workflows.

• YouTube
• 33 minutes
• Self-Paced
• Free Video

Anyscale

Redesigning Scheduling in Ray to Improve Cost-Efficiency at Scale

Explore Ray scheduling features to optimize AI applications, enhancing performance and cost-efficiency. Learn placement groups, graceful node draining, and label-based scheduling for faster, cheaper operations.

• YouTube
• 30 minutes
• Self-Paced
• Free Video

Introduction to Distributed ML Workloads with Ray on Kubernetes

Learn to scale ML workloads using Ray and Kubernetes. Explore basic Ray concepts, set up a Ray cluster in Kubernetes, and run distributed ML training jobs efficiently.

• YouTube
• 26 minutes
• Self-Paced
• Free Video

USENIX

Primus - Unified Training System for Large-Scale Deep Learning Recommendation Models

Discover Primus, ByteDance's unified training system for large-scale deep learning recommendation models, featuring dynamic scaling, data orchestration, and hybrid training paradigms.

• YouTube
• 17 minutes
• Self-Paced
• Free Video

USENIX

Holmes - Localizing Irregularities in LLM Training with Mega-scale GPU Clusters

Discover Holmes, a system that localizes training irregularities in large-scale GPU clusters for LLMs, achieving 97.21% accuracy and 6.52× speedup over traditional methods.

• YouTube
• 16 minutes
• Self-Paced
• Free Video