Performance Modeling and Optimization

This course shows how to predict how fast an LLM inference step should run on given hardware, check that prediction against measurements, and use the difference to find practical optimizations. You will start by bounding the time of an operation from its FLOPs and bytes, then use the roofline model and ridge point to distinguish compute-limited from memory-limited execution. You will learn to count traffic at different memory levels and reason about how compiler placement decisions change off-chip movement on scratchpad-based architectures. The course then moves from analytical models to measurement. You will use PyTorch Profiler to read execution traces and separate time dominated by compute, memory movement, synchronization, and software overhead. A worked example takes a slow decode loop from diagnosis to fix. You will also study benchmarking methodology for compiled code, compiler cost models, shape bucketing, warmup precompilation, and batched decode. In the lab, you will measure achieved compute and bandwidth on a GPU, place prefill and decode on a roofline, run a batch sweep, report time to first token and time per output token, and predict how a scratchpad target would change off-chip traffic. A 20-question graded quiz at the end covers every module of the course.