Coursera Plus Annual Is 40% Off. Ends Soon!

Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

IBM

Memory Systems in Dataflow Architectures

IBM via Coursera

Go to class Write review

Overview

Google, IBM & Meta Certificates – 40% Off a Year
One Coursera Plus subscription covers most Professional Certificates on Coursera. 40% off a full year through Dec 2.
Unlock All Certificates
In LLM inference, moving data often takes longer than computing on it. This course covers how memory capacity, bandwidth, placement and data movement affect inference on different accelerator architectures. You will use arithmetic intensity and the ridge point to explain why low-batch decode is often bandwidth-bound and how the KV cache changes that balance as context grows. You will compare prefill and decode across multiple accelerator designs and study how data placement differs between hardware-managed caches and compiler-managed local memories. The course covers tiling, layout, alignment, software-managed scratchpads, and FlashAttention from a memory-systems perspective. The second half focuses on the KV cache: how to size it, how MHA and GQA change its footprint, and how serving systems manage it with continuous batching, PagedAttention, prefix caching, chunked prefill, speculative decoding, and tiered storage. You will also examine how these techniques change on a compiled-schedule architecture such as Spyre. In the lab, you will measure decode time and memory use on a Colab GPU across batch sizes and context lengths, compare the results with analytical predictions, and explain the gap between the model and the measurement. A 22-question graded quiz at the end covers every module of the course.

Syllabus

  • Introduction
  • The memory problem
    • In many inference workloads, moving data takes longer than computing on it. This module gives you five questions for analyzing the memory system of any accelerator and two numbers for deciding which resource limits a workload: the ridge point of the hardware and the arithmetic intensity of the workload. You will compute both for decode on an H100, first for the weights alone and then with the KV cache included.
  • Prefill and decode
    • Inference has two phases. Prefill processes the whole prompt at once and is usually limited by compute. Decode produces one token at a time and is usually limited by memory bandwidth. You will place both on a roofline, compare the ridge points of five accelerators, and work out what one decode step reads on Spyre.
  • Who decides where data lives
    • On some accelerators the hardware decides what to keep close to the compute units. On others the compiler decides. Most machines, including GPUs, split the work between the two. This module describes that range, the three kinds of local memory, and the jobs a compiler must take on when there is no hardware cache.
  • Five architectures compared
    • The five questions from module 1 are applied to GPU, TPU, Cerebras, Groq and Spyre. You will compare their memory hierarchies, capacity and bandwidth, and identify the resource that is scarce on each machine.
  • Tiling and layout
    • Tiling decides how much data fits on chip and how often it is reused. In this module you will size matmul tiles against the scratchpad of one Spyre core and measure how tile size changes arithmetic intensity. You will also look at how data layout, memory banks, access order and alignment affect bandwidth. The module ends with FlashAttention, which applies the same tiling ideas to attention.
  • Sizing the KV cache
    • The KV cache stores the keys and values of every earlier token, and it grows as a request gets longer. Its size is the product of six numbers: layers, batch size, context length, KV heads, head dimension and bytes per element. You will compute it, find the context length at which it becomes larger than the model weights, and see how four techniques reduce it.
  • Managing KV at runtime
    • Earlier modules looked at memory decisions made before execution. When a model serves many users, the size and lifetime of each request's KV cache are known only at runtime. Part 1 covers serving engines and six techniques for managing that memory: continuous batching, PagedAttention, prefix caching, chunked prefill and disaggregation, KV tiering, and speculative decoding. Each is shown on a GPU and then on a dataflow accelerator with a compiled schedule. Part 2 looks at how Spyre handles the same problems, then compares KV management across the five accelerators.
  • Lab 4: Profiling Memory Access Patterns in LLMs
    • This lab profiles memory access during distilgpt2 generation in Google Colab using torch.profiler with memory tracking enabled. Students identify the memory-intensive operators and the peak allocation during decode, then measure effective bandwidth against an analytically known byte count to see why allocator activity is not bandwidth. They derive the KV-cache size formula, confirm it against measured peak-memory growth, compare its cost under MHA, GQA, and MQA, and size it for a scratchpad.
  • Memory Systems in Dataflow Architectures Graded Quiz
    • This quiz covers all seven modules of the course: ridge points and arithmetic intensity, prefill and decode on the roofline, who manages on-chip memory, memory across five accelerators, tiling and layout, KV cache sizing, and KV management at runtime. It has 22 questions, and many need a short calculation. Allow about 55 minutes and keep a calculator nearby. You need 80% to pass, and you can take the quiz up to 3 times. Your highest score counts.

Taught by

Kaoutar El Maghraoui

Tags

Reviews

Start your review of Memory Systems in Dataflow Architectures

Go to class

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Sign up for free
Someone learning on their laptop while sitting on the floor.