In many inference workloads, moving data takes longer than computing on it. This module gives you five questions for analyzing the memory system of any accelerator and two numbers for deciding which resource limits a workload: the ridge point of the hardware and the arithmetic intensity of the workload. You will compute both for decode on an H100, first for the weights alone and then with the KV cache included.

Inference has two phases. Prefill processes the whole prompt at once and is usually limited by compute. Decode produces one token at a time and is usually limited by memory bandwidth. You will place both on a roofline, compare the ridge points of five accelerators, and work out what one decode step reads on Spyre.

On some accelerators the hardware decides what to keep close to the compute units. On others the compiler decides. Most machines, including GPUs, split the work between the two. This module describes that range, the three kinds of local memory, and the jobs a compiler must take on when there is no hardware cache.

The five questions from module 1 are applied to GPU, TPU, Cerebras, Groq and Spyre. You will compare their memory hierarchies, capacity and bandwidth, and identify the resource that is scarce on each machine.

Tiling decides how much data fits on chip and how often it is reused. In this module you will size matmul tiles against the scratchpad of one Spyre core and measure how tile size changes arithmetic intensity. You will also look at how data layout, memory banks, access order and alignment affect bandwidth. The module ends with FlashAttention, which applies the same tiling ideas to attention.

The KV cache stores the keys and values of every earlier token, and it grows as a request gets longer. Its size is the product of six numbers: layers, batch size, context length, KV heads, head dimension and bytes per element. You will compute it, find the context length at which it becomes larger than the model weights, and see how four techniques reduce it.

Earlier modules looked at memory decisions made before execution. When a model serves many users, the size and lifetime of each request's KV cache are known only at runtime. Part 1 covers serving engines and six techniques for managing that memory: continuous batching, PagedAttention, prefix caching, chunked prefill and disaggregation, KV tiering, and speculative decoding. Each is shown on a GPU and then on a dataflow accelerator with a compiled schedule. Part 2 looks at how Spyre handles the same problems, then compares KV management across the five accelerators.

This lab profiles memory access during distilgpt2 generation in Google Colab using torch.profiler with memory tracking enabled. Students identify the memory-intensive operators and the peak allocation during decode, then measure effective bandwidth against an analytically known byte count to see why allocator activity is not bandwidth. They derive the KV-cache size formula, confirm it against measured peak-memory growth, compare its cost under MHA, GQA, and MQA, and size it for a scratchpad.

This quiz covers all seven modules of the course: ridge points and arithmetic intensity, prefill and decode on the roofline, who manages on-chip memory, memory across five accelerators, tiling and layout, KV cache sizing, and KV management at runtime. It has 22 questions, and many need a short calculation. Allow about 55 minutes and keep a calculator nearby. You need 80% to pass, and you can take the quiz up to 3 times. Your highest score counts.