Two phases, opposite bottlenecks

Every request to an LLM runs in two phases, and they stress completely different parts of a machine.

Prefill takes your whole prompt and pushes it through the model in one shot. A 2,000-token prompt means the first matmul has a 2,000-row activation matrix against each weight matrix. Each weight is loaded from memory once and reused 2,000 times. That is a dense GEMM, it saturates tensor cores, and it sets time-to-first-token.

Decode produces one token, then the next, then the next. Each step multiplies a single row of activations against every weight in the model. Each weight is loaded from memory once and used once. The GPU spends its time waiting on memory and its FLOPs sit idle. Decode sets inter-token latency, and therefore the tokens/sec number you actually watch scroll.

Same weights, same kernels, and the hardware you'd buy to make each one fast is nearly the opposite.

The decode arithmetic you can do on a napkin

At batch size 1, decode has to read the whole model out of memory to emit one token. So:

tokens/sec ≈ memory bandwidth ÷ bytes read per token
           ≈ memory bandwidth ÷ model file size

That's it. That's the ceiling. Nothing about FLOPs appears in it.

Take Llama-2-7B at Q4_0, which is 3.83 GB on disk. Apple states 400GB/s for M1 Max and M2 Max, and 800GB/s for M1 Ultra and M2 Ultra. The measured tg128 figures come from the long-running llama.cpp Apple Silicon benchmark thread.

The ceiling column is mine: 400e9 / 3.83e9 = 104.4. Check it yourself.

Two things fall out. Real decode lands at roughly 40–60% of the bandwidth ceiling, so the napkin number is an upper bound you should discount, not a prediction. And the Ultra parts, which are two Max dies fused together at twice the paper bandwidth, return only about 40% more decode throughput than the Max they're built from (1.37x on M1, 1.43x on M2). If you were buying an Ultra for single-stream generation on the strength of the 800GB/s number, that is the number you should be looking at instead.

The same arithmetic explains why quantization buys speed and not just VRAM headroom: cutting a model from FP16 to 4-bit cuts bytes-read-per-token by roughly 4×, which moves the decode ceiling by roughly 4×. Decode is bound by the exact quantity quantization shrinks.

Why big-FLOPs, small-bandwidth boxes disappoint

NVIDIA's DGX Spark is the cleanest illustration currently shipping. NVIDIA claims 1 petaFLOP of sparse FP4 on the GB10 superchip, with 128 GB of unified LPDDR5X at 273 GB/s. That is datacenter-class compute bolted to laptop-class memory.

Measured on gpt-oss-120b MXFP4 in the llama.cpp DGX Spark thread: 1,956 tok/s prompt processing, 60.57 tok/s generation. A 32× gap between the two phases on one box, from one set of weights. The petaFLOP shows up in the first number and is entirely absent from the second.

Apple Silicon has the mirror-image problem. High bandwidth, modest matmul throughput, so a Mac Studio punches above its FLOPs on single-stream decode and falls behind badly on long prompts. Tom's Hardware measured exactly this shape, titling their Mac Studio piece "M4 Max beats GB10 and Strix Halo in decode throughput, but memory bandwidth isn't everything". My read: if you paste 40k-token files into a local coding agent, prefill is the wall you'll hit, and it's the wall Apple hardware is worst at. More on the machine-by-machine tradeoffs in local inference hardware.

Batching is why the API is cheap and your Mac is not

Decode at batch 1 reads 2P bytes (FP16) to produce one token. Decode at batch 64 reads the same 2P bytes to produce 64 tokens, because all 64 sequences multiply against the same weights in the same pass. Weight traffic per token falls by 64×. This is the entire economic basis of hosted inference.

You can find the crossover point with a roofline. NVIDIA's H100 SXM has 3.35 TB/s of HBM3 and 1,979 FP16 tensor TFLOPS with sparsity, so 989 dense. The ridge point:

989e12 FLOP/s ÷ 3.35e12 B/s ≈ 295 FLOP per byte

Decode's arithmetic intensity in the weight matmuls is about B FLOP/byte at FP16 (2·P·B FLOPs against 2·P bytes read). So you need a batch of roughly 300 concurrent sequences before an H100 stops being memory-bound during decode. Prefill with a 2,000-token prompt sits at intensity ~2,000 and is compute-bound at batch 1, seven times over.

Which is why batching does much less for prefill. A single 2,000-token prompt already fills the machine; stacking a second one just queues behind the first. Batching converts decode from memory-bound to compute-bound. For prefill it mostly just adds work.

Your local single stream never gets any of this. You pay full weight-read cost for every single token. A provider amortises that read across hundreds of users, which is how per-token prices land where they do while your Mac Studio does one conversation at a time.

The KV cache is what grows

Batching decode is limited by memory capacity, not just bandwidth, because every active sequence carries a KV cache that grows one entry per layer per token.

Llama 3.3 70B's config is 80 layers, 64 attention heads, 8 KV heads, hidden size 8192 (so head dim 128). At FP16:

per token = 2 (K and V) × 8 kv_heads × 128 head_dim × 80 layers × 2 bytes
          = 327,680 bytes  ≈ 320 KiB / token

At its full 131,072-token context that's 43 GB for one sequence. Batch 32 at only 8k context each is 86 GB, which already exceeds a single 80 GB H100. Long context turns a bandwidth problem into a capacity problem, and the capacity problem caps your batch size, which drags you back onto the bad side of the decode roofline.

Two fixes are now standard:

• Grouped-query attention. Those 8 KV heads serve 64 query heads. Full multi-head attention would need 64 KV heads and 2.56 MiB per token — the same 128k context would cost 343 GB. GQA is an 8× cut in KV traffic and footprint, and it's why 128k contexts are servable at all.
• Paged attention. vLLM allocates KV in fixed-size pages instead of one contiguous reservation per sequence, so you don't pre-reserve for the worst-case length. The paper reports 2–4× throughput at equal latency versus FasterTransformer and Orca, with "near-zero waste" in KV memory. The throughput comes from fitting more sequences, which is the same batching lever again.

Prompt caching is the third lever, and it attacks prefill instead — see prompt caching across harnesses.

What production stacks actually do about it

Running prefill and decode on the same GPU means they fight. A long prefill occupying the GPU stalls every in-flight decode, and users see the stream freeze.

Chunked prefill splits a prompt into fixed-size chunks and slots decode steps in alongside them. Sarathi-Serve introduced this as "stall-free scheduling". It is now on by default in vLLM V1, whose scheduler batches all pending decodes first, then fills the remaining token budget with prefill chunks. The docs are explicit about why: it gets "better GPU utilization by locating compute-bound (prefill) and memory-bound (decode) requests to the same batch." max_num_batched_tokens is the dial — around 2048 favours inter-token latency, above 8192 favours TTFT and raw throughput.

Disaggregation goes further and puts the two phases on different machines. DistServe made the case that prefill/decode interference costs enough goodput to justify separate GPU pools. Every major stack now ships it: vLLM with KV connectors under vllm/distributed/kv_transfer, SGLang with separate prefill and decode pools, and TensorRT-LLM under NVIDIA Dynamo. The prefill pool runs high tensor parallelism to chew through matmuls; the decode pool runs lower TP with more replicas for concurrency. KV cache moves between them over RDMA.

The important part for a buyer: the people who serve models at scale concluded the two phases want physically different hardware allocations. That is the strongest available evidence that the split is real and not a modelling curiosity.

What to buy

Two rules I'd give anyone shopping. First, compute bandwidth ÷ quantized file size, discount to 50%, and treat that as your realistic ceiling before you read a single review. Second, if your workload is long prompts and short answers (coding agents, document QA, RAG), you are buying a prefill machine, and bandwidth is the wrong headline number to be optimising.