Prefill and decode want different machines

When comparing inference hardware, I want to know how long it takes to read the prompt and how quickly it writes the answer. Those are separate phases, and a machine can be good at one without being good at the other.

Prefill takes your whole prompt and pushes it through the model in one shot. A 2,000-token prompt means the first matmul has a 2,000-row activation matrix against each weight matrix. Each weight is loaded from memory once and reused 2,000 times. That is a dense GEMM, it saturates tensor cores, and it sets time-to-first-token.

Decode produces one token, then the next, then the next. Each step multiplies a single row of activations against every weight in the model. Each weight is loaded from memory once and used once. The GPU spends its time waiting on memory and its FLOPs sit idle. Decode sets inter-token latency, and therefore the tokens/sec number you actually watch scroll.

That is why I look at prompt-processing and generation benchmarks separately.

The decode arithmetic you can do on a napkin

At batch size 1, decode has to read the whole model out of memory to emit one token. So:

tokens/sec ≈ memory bandwidth ÷ bytes read per token
           ≈ memory bandwidth ÷ model file size

This is a bandwidth ceiling for the simplified batch-one case. It is a starting estimate, not a prediction of measured speed.

Take Llama-2-7B at Q4_0, which is 3.83 GB on disk. Apple states 400GB/s for M1 Max and M2 Max, and 800GB/s for M1 Ultra and M2 Ultra. The measured tg128 figures come from the long-running llama.cpp Apple Silicon benchmark thread.

The ceiling column is mine: 400e9 / 3.83e9 = 104.4. It leaves out runtime overhead and other memory traffic.

Two things fall out. Real decode lands at roughly 40–60% of the bandwidth ceiling, so the napkin number is an upper bound you should discount, not a prediction. And the Ultra parts, which are two Max dies fused together at twice the paper bandwidth, return only about 40% more decode throughput than the Max they're built from (1.37x on M1, 1.43x on M2). If you were buying an Ultra for single-stream generation on the strength of the 800GB/s number, that is the number you should be looking at instead.

The same arithmetic explains why quantization buys speed and not just VRAM headroom: cutting a model from FP16 to 4-bit cuts bytes-read-per-token by roughly 4×, which moves the decode ceiling by roughly 4×. Decode is bound by the exact quantity quantization shrinks.

Why big-FLOPs, small-bandwidth boxes disappoint

NVIDIA's DGX Spark is the cleanest illustration currently shipping. NVIDIA claims 1 petaFLOP of sparse FP4 on the GB10 superchip, with 128 GB of unified LPDDR5X at 273 GB/s. The compute figure alone does not tell you how quickly it will generate tokens.

Measured on gpt-oss-120b MXFP4 in the llama.cpp DGX Spark thread: 1,956 tok/s prompt processing, 60.57 tok/s generation. A 32× gap between the two phases on one box, from one set of weights. The two measurements illustrate why the advertised compute figure cannot stand in for a decode benchmark.

Apple Silicon has the mirror-image problem. High bandwidth, modest matmul throughput, so a Mac Studio punches above its FLOPs on single-stream decode and falls behind badly on long prompts. Tom's Hardware measured exactly this shape, titling their Mac Studio piece "M4 Max beats GB10 and Strix Halo in decode throughput, but memory bandwidth isn't everything". My read: if you paste 40k-token files into a local coding agent, prefill is the wall you'll hit, and it's the wall Apple hardware is worst at. More on the machine-by-machine tradeoffs in local inference hardware.

How batching changes decode

Decode at batch 1 reads 2P bytes (FP16) to produce one token. Decode at batch 64 reads the same 2P bytes to produce 64 tokens, because all 64 sequences multiply against the same weights in the same pass. Weight traffic per token falls by 64×. Sharing those weight reads is one reason hosted inference can serve tokens cheaply.

You can find the crossover point with a roofline. NVIDIA's H100 SXM has 3.35 TB/s of HBM3 and 1,979 FP16 tensor TFLOPS with sparsity, so 989 dense. The ridge point:

989e12 FLOP/s ÷ 3.35e12 B/s ≈ 295 FLOP per byte

Decode's arithmetic intensity in the weight matmuls is about B FLOP/byte at FP16 (2·P·B FLOPs against 2·P bytes read). So you need a batch of roughly 300 concurrent sequences before an H100 stops being memory-bound during decode. Prefill with a 2,000-token prompt sits at intensity ~2,000 and is compute-bound at batch 1, seven times over.

Which is why batching does much less for prefill. A single 2,000-token prompt already fills the machine; stacking a second one just queues behind the first. Batching converts decode from memory-bound to compute-bound. For prefill it mostly just adds work.

A single local conversation has no other sequences to share those weight reads with. A busy provider can spread that work across many requests. That is one part of the cost difference I would consider when comparing local and hosted inference.

The KV cache is what grows

Batching decode is limited by memory capacity, not just bandwidth, because every active sequence carries a KV cache that grows one entry per layer per token.

Llama 3.3 70B's config is 80 layers, 64 attention heads, 8 KV heads, hidden size 8192 (so head dim 128). At FP16:

per token = 2 (K and V) × 8 kv_heads × 128 head_dim × 80 layers × 2 bytes
          = 327,680 bytes  ≈ 320 KiB / token

At its full 131,072-token context that's 43 GB for one sequence. Batch 32 at only 8k context each is 86 GB, which already exceeds a single 80 GB H100. Long context turns a bandwidth problem into a capacity problem, and the capacity problem caps your batch size, which drags you back onto the bad side of the decode roofline.

Two fixes are now standard:

• Grouped-query attention. Those 8 KV heads serve 64 query heads. Full multi-head attention would need 64 KV heads and 2.56 MiB per token — the same 128k context would cost 343 GB. GQA is an 8× cut in KV traffic and footprint, and it's why 128k contexts are servable at all.
• Paged attention. vLLM allocates KV in fixed-size pages instead of one contiguous reservation per sequence, so you don't pre-reserve for the worst-case length. The paper reports 2–4× throughput at equal latency versus FasterTransformer and Orca, with "near-zero waste" in KV memory. The throughput comes from fitting more sequences, which is the same batching lever again.

Prompt caching is the third lever, and it attacks prefill instead — see prompt caching across harnesses.

Scheduling the two phases

Running prefill and decode on the same GPU means they fight. A long prefill occupying the GPU stalls every in-flight decode, and users see the stream freeze.

Chunked prefill splits a prompt into fixed-size chunks and slots decode steps in alongside them. Sarathi-Serve introduced this as "stall-free scheduling". It is now on by default in vLLM V1, whose scheduler batches all pending decodes first, then fills the remaining token budget with prefill chunks. The docs are explicit about why: it gets "better GPU utilization by locating compute-bound (prefill) and memory-bound (decode) requests to the same batch." max_num_batched_tokens is the dial — around 2048 favours inter-token latency, above 8192 favours TTFT and raw throughput.

Disaggregation goes further and puts the two phases on different machines. DistServe made the case that prefill/decode interference costs enough goodput to justify separate GPU pools. Every major stack now ships it: vLLM with KV connectors under vllm/distributed/kv_transfer, SGLang with separate prefill and decode pools, and TensorRT-LLM under NVIDIA Dynamo. The prefill pool runs high tensor parallelism to chew through matmuls; the decode pool runs lower TP with more replicas for concurrency. KV cache moves between them over RDMA.

For hardware selection, I take this as a reason to measure both phases against the intended workload. A configuration chosen for a busy serving pool may be a poor fit for one local conversation.

What to buy

I would use bandwidth ÷ quantized file size as a first check, then compare it with measurements for the model and runtime I intend to use. For long prompts and short answers, I would pay particular attention to prefill time. Fast generation does not help much if most of the wait happens before the first token.