Distributed Training Courses and Certifications

Custom and Distributed Training with TensorFlow

438 ratings at Coursera

Explore advanced TensorFlow techniques for custom training loops, graph mode optimization, and distributed training on GPUs and TPUs. Gain flexibility and efficiency in model development.

• Coursera
• 1 day 15 minutes
• Self-Paced
• Paid Course

Let's Build Pipeline Parallelism from Scratch - Tutorial

Master pipeline parallelism by building distributed AI training systems from scratch, implementing GPipe micro-batching and 1F1B algorithms across multiple GPUs step-by-step.

• freeCodeCamp
• 3 hours 22 minutes
• Self-Paced
• Free Video

Algorithms for Big Data

10 ratings

Explore advanced algorithmic techniques for handling massive datasets, enhancing your ability to process and analyze big data efficiently.

• YouTube
• 1 day 10 hours
• Self-Paced
• Free Video

Stanford CS149 - Parallel Computing 2023

10 ratings

Master parallel computing fundamentals from multi-core CPUs to GPUs, covering hardware design, programming techniques, and performance optimization for modern systems.

• YouTube
• 1 day 19 minutes
• Self-Paced
• Free Video

Systems Thinking In Public Health

5 ratings

Explore systems thinking in public health, learning to analyze complex problems, create models, and develop strategies that account for real-world complexities and unintended consequences.

• Coursera
• 14 hours 54 minutes
• Self-Paced
• Paid Course

Computer Architecture

6 ratings

In this course, you will learn to design the computer architecture of complex modern microprocessors. All the features of this course are available for free. It does not offer a certificate upon completion.

• Coursera
• 2 days 1 hour 38 minutes
• Self-Paced
• Paid Course

Parallel programming

6 ratings

Apprenez les fondamentaux de la programmation parallèle, de la parallélisation des tâches à celle des données. Maîtrisez les collections parallèles et appliquez ces concepts à travers des exemples pratiques analysant des données réelles.

• Coursera
• 1 day 8 hours 59 minutes
• Self-Paced
• Paid Course

USENIX

SimAI - Unifying Architecture Design and Performance Tuning for Large-Scale Large Language Model Training with Scalability and Precision

Discover SimAI, a unified simulator for large-scale LLM training that achieves 98.1% alignment with real-world results through high-fidelity integration and multi-thread acceleration.

• YouTube
• 11 minutes
• Self-Paced
• Free Video

USENIX

Holmes - Localizing Irregularities in LLM Training with Mega-scale GPU Clusters

Discover Holmes, a system that localizes training irregularities in large-scale GPU clusters for LLMs, achieving 97.21% accuracy and 6.52× speedup over traditional methods.

• YouTube
• 16 minutes
• Self-Paced
• Free Video

USENIX

Primus - Unified Training System for Large-Scale Deep Learning Recommendation Models

Discover Primus, ByteDance's unified training system for large-scale deep learning recommendation models, featuring dynamic scaling, data orchestration, and hybrid training paradigms.

• YouTube
• 17 minutes
• Self-Paced
• Free Video

USENIX

Accelerating Neural Recommendation Training with Embedding Scheduling

Optimizing deep learning recommendation models through embedding scheduling, reducing communication overhead and improving training performance without compromising accuracy.

• YouTube
• 15 minutes
• Self-Paced
• Free Video

Anyscale

Redesigning Scheduling in Ray to Improve Cost-Efficiency at Scale

Explore Ray scheduling features to optimize AI applications, enhancing performance and cost-efficiency. Learn placement groups, graceful node draining, and label-based scheduling for faster, cheaper operations.

• YouTube
• 30 minutes
• Self-Paced
• Free Video

Career Certificate

Functional Programming in Scala

10659 ratings at Coursera

Explore functional programming in Scala, from basic concepts to advanced applications. Learn to write elegant code, analyze data sets, and leverage parallel and distributed programming using Apache Spark.

• Coursera
• 20 weeks, 10 hours/week
• Self-Paced
• Paid Course
• 5 courses

NPTEL;  IIT Patna

Big Data Computing

13 ratings

Comprehensive exploration of big data technologies, covering Hadoop, Spark, key-value stores, streaming analytics, machine learning, and graph processing for large-scale data analysis and insights.

• YouTube
• 19 hours
• Self-Paced
• Free Video

Cloud Computing Concepts: Part 2

6 ratings

Explore distributed computing concepts for cloud systems, including MapReduce, NoSQL, scalability, and trending areas. Implement core techniques in C++ through hands-on exercises.

• Coursera
• 20 hours
• Self-Paced
• Paid Course