Coursera Plus Annual Is 40% Off. Ends Soon!

Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

IBM

Dataflow Execution Models

IBM via Coursera

Go to class Write review

Overview

Google, IBM & Meta Certificates – 40% Off a Year
One Coursera Plus subscription covers most Professional Certificates on Coursera. 40% off a full year through Dec 2.
Unlock All Certificates
By the end of this course, you will be able to explain how dataflow execution works, trace dependencies through a computational graph, and compare how different accelerators schedule and execute the same AI workload. You will begin with the classical dataflow model: operations as nodes, dependencies as edges, and execution triggered when inputs are ready. From there, you will compare static and dynamic scheduling and examine how compile-time planning introduces constraints such as shape specialization. You will use torch.fx to inspect model graphs and learn how guards capture assumptions about shapes and program state. The course then compares four implementations of dataflow: TPU systolic arrays, Groq's deterministic tensor streaming, the Cerebras wafer-scale fabric, and IBM Spyre's 32-core programmable dataflow design. Triton is used as a contrasting example of tile-level programming on GPUs. A final case study follows one transformer attention block through TPU and Spyre compilation flows. In the lab, you will trace a transformer block, visualize the graph, annotate FLOPs and memory footprint, and predict the critical path on two hardware profiles. You should be comfortable with Python, PyTorch, and transformer architectures.

Syllabus

  • Dataflow Execution Principles
    • This module defines the classical dataflow abstraction: a program expressed as a graph of nodes and edges, in which a node becomes eligible to execute when all of its inputs are available. An attention block is traced step by step to show how concurrency follows from graph topology. The module then separates two properties that are often confused: scheduling commitment, meaning how much of the schedule is fixed at compile time, and runtime dynamism. It introduces a six-layer execution model that applies to any accelerator, explains the energy savings of dataflow execution, and examines the main cost of compile-time commitment, which is the requirement for static shapes.
  • Graph Representation with torch.fx
    • This module covers how a PyTorch program becomes a graph that a compiler can process. It describes the structure of a torch.fx graph, including node types, targets, arguments, and the output node. It then identifies what the traced graph does not contain, such as concrete tensor shapes and memory layout, and explains the guards that record the assumptions under which a traced graph is valid. The module concludes with the sequence of compiler stages that take a captured graph to hardware execution.
  • Comparing Dataflow Architectures: GPU, TPU, and Groq
    • This module places accelerator architectures in a two-dimensional design space defined by memory model and compile-time scheduling commitment. It describes the GPU warp as the unit of SIMT execution and then presents the TPU systolic array step by step, from weight loading to the completed wavefront. It covers the TPU's compiler-managed dataflow and the organization of TPU pods connected by the inter-chip interconnect. It then describes the Groq Tensor Streaming Processor, including its functional slices, superlanes, and the separate paths for instructions and tensor data. The module ends with a comparison of five architectures across six design dimensions, which forms the Week 2 row of the Accelerator Landscape Map.
  • Cerebras deep dive
    • Moving data between chips is slow and costs energy. Cerebras keeps most of that traffic on a single wafer-sized chip. This module looks at how that choice affects memory, routing, compilation and the handling of manufacturing defects.
  • Spyre deep dive
    • Each core on IBM Spyre has a small scratchpad that software manages directly. As a result, the compiler’s choices about tile size and data lifetime decide much of the performance. This module is based on public architecture and software documentation, and it separates the software path in use today from the KTIR work that is still being developed.
  • Case study
    • The attention computation stays the same and the hardware changes. You will follow it through the Spyre and TPU compiler stacks and explain each difference in their output from the hardware, not from the names of compiler passes.
  • Lab 2: Visualizing Transformer Dataflow
    • This lab traces a transformer block into a dataflow graph with torch.fx and visualizes it as a DAG in Google Colab. Students annotate every node with FLOPs and bytes, compute per-node arithmetic intensity, and classify each node as compute-bound or memory-bound against their GPU's ridge point. Re-classifying the same graph against a dataflow accelerator's scratchpad ridge shows the partition shift while the graph stays unchanged.
  • Dataflow Execution Models Graded Quiz
    • This quiz covers all six modules of the course: dataflow firing rules and the six-layer execution model, PyTorch FX graphs, the six-question comparison of accelerators, Cerebras, Spyre, and the attention case study on Spyre and TPU. It has 19 questions, including shape-bucket padding and SRAM capacity calculations. Allow about 50 minutes and keep a calculator nearby. You need 80% to pass, and you can take the quiz up to 3 times. Your highest score counts.

Taught by

Kaoutar El Maghraoui

Tags

Reviews

Start your review of Dataflow Execution Models

Go to class

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Sign up for free
Someone learning on their laptop while sitting on the floor.