Coursera Plus Annual Is 40% Off. Ends Soon!

Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

IBM

Compilation for Dataflow

IBM via Coursera

Go to class Write review

Overview

Google, IBM & Meta Certificates – 40% Off a Year
One Coursera Plus subscription covers most Professional Certificates on Coursera. 40% off a full year through Dec 2.
Unlock All Certificates
A PyTorch model passes through several compiler stages before it runs on an accelerator, and each stage makes decisions that change what the hardware executes. In this course you will follow a model through that stack and connect each compiler decision to the kernels that run on the device. You will study the PyTorch 2 compilation path through TorchDynamo, AOTAutograd, and TorchInductor, including graph breaks, guards, and recompilation. You will then examine the intermediate representations and compiler passes that matter most for accelerators: decomposition, fusion, tiling, layout, scheduling, and code generation. The course compares three compiler stacks on the same classes of problems: TorchInductor with Triton on GPUs, XLA with HLO on TPUs, and the Spyre compilation stack with SuperDSC and the emerging KTIR representation. Worked examples show how a feed-forward layer is mapped across Spyre cores and how one attention graph is handled by XLA and torch-spyre. In the lab, you will benchmark eager and compiled execution, inspect a captured graph, repair a graph break, and connect compiler artifacts to the code your run produced. A 18-question graded quiz at the end covers every module of the course.

Syllabus

  • From PyTorch to the accelerator
    • This module covers the part of compilation that every PyTorch backend shares. It starts from the FX graph introduced in week two and explains why torch.compile exists. You will then follow a model through TorchDynamo, AOTAutograd and Core ATen decomposition, and see how graph breaks and guards affect what the backend receives.
  • Intermediate representations
    • A compiler uses several intermediate representations between source code and machine code. This module covers five of them: FX, Core ATen, HLO, MLIR and KTIR. For each one, you will see what it makes explicit and what it no longer records, and why different transformations need different levels.
  • Key compiler passes for dataflow
    • Five compiler passes matter most for dataflow hardware: decomposition, operator fusion, layout, tiling for on-chip memory, and scheduling. This module explains what each pass decides and how a decision in one pass limits the others. It also introduces tile-level kernel languages such as Triton.
  • XLA on TPU
    • XLA is a compiler built for TPU, and its design follows from the hardware. A large systolic matrix unit is efficient only when it receives regular, correctly shaped data on every cycle. This module covers why TPU needs a whole-graph compiler, what the HLO representation makes possible, the two ways a PyTorch model reaches XLA, and the cost of requiring static shapes.
  • The Spyre compiler
    • On Spyre, many decisions that a GPU leaves to caches or runtime scheduling are made by the compiler before execution. This module studies torch-spyre, the open-source frontend for Spyre, and the plans it produces. It distinguishes features that are published, features that are implemented today and features that are planned.
  • XLA and torch-spyre compared
    • One attention block is compiled with XLA for TPU and with torch-spyre for Spyre. You will see where the two pipelines make the same choices and where they differ. Some transformations follow from the graph alone. Others depend on local memory size, matrix-array shape, the interconnect or the scheduling model.
  • Lab 3: Compiling Models for Dataflow Execution
    • This lab compiles a transformer block with torch.compile in Google Colab and measures the speedup over eager execution, weighing one-time compilation latency against the per-call steady-state saving. Students ablate the eager, aot_eager, and inductor backends to locate where the speedup originates, then detect and repair graph breaks that prevent whole-graph compilation. Reading the generated Triton and cuBLAS kernels connects operator fusion to dataflow execution. The lab adds the Week 3 row to the Accelerator Landscape Map.
  • Compilation for Dataflow Graded Quiz
    • This quiz covers all six modules of the course: the PyTorch compilation frontend, intermediate representations from FX to KTIR, the main compiler passes, XLA on TPU, torch-spyre on Spyre, and the two compilers side by side. It has 18 questions, including calculations of fusion traffic, padding waste and scratchpad slice size. Allow about 45 minutes and keep a calculator nearby. You need 80% to pass, and you can take the quiz up to 3 times. Your highest score counts.

Taught by

Kaoutar El Maghraoui

Tags

Reviews

Start your review of Compilation for Dataflow

Go to class

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Sign up for free
Someone learning on their laptop while sitting on the floor.