Quantization is the field with the widest gap between what people repeat and what anyone has measured. "Q4 is basically lossless" gets said constantly, usually by someone quoting a perplexity number from a 2023 forum post about a model that no longer exists. The real answer depends on which format, which layers, and which task, and there is now enough published measurement to give it properly.

The formats

GPTQ (Frantar et al., 2022) quantizes one layer at a time and corrects the remaining weights for the error already introduced, which is why it can hit 3–4 bits on a 175B model in about four GPU hours. AWQ (MLSys 2024 best paper) took the opposite route: look at activation magnitudes, find the ~1% of channels that matter, and scale them up so rounding hurts them less. No backpropagation, no reconstruction, which is why it generalises off its calibration set better than GPTQ does.

NF4 came out of QLoRA. Its 16 levels are placed so each bin holds equal probability mass under a standard normal rather than being evenly spaced. Weights are roughly normal, so this is a better use of 16 codes. NF4 exists mainly to make fine-tuning fit in memory; as a serving format it is slower than a Marlin-kernel GPTQ or AWQ model on the same hardware.

MXFP4 is the odd one, because for gpt-oss it is not a compression step applied afterwards. OpenAI post-trained with the MoE weights already in MXFP4, and those weights are 90%+ of the parameter count, so the 120B fits on a single 80GB GPU and the 20B in 16GB. There is no fp16 reference version of those tensors to be worse than.

Dynamic quants are the actual recent development

Everything above except the _M/_S split applies one width to the whole model. Unsloth's dynamic quants pick a width per layer from measured sensitivity, so a nominal "2-bit" build is 2-bit in the layers that tolerate it and 4- or 6-bit in the ones that don't. This is why models that "cannot fit" now fit.

The measurement that convinced me is Unsloth's Qwen3.5-35B-A3B GGUF table, which reports mean KL-divergence against the reference for the same model quantized by several people:

The two bolded rows are the point. The dynamic build is 0.6 GB smaller than the uniform Q4_K_M and 25% closer to the reference distribution. At 2 bits the gap widens to nearly 2×. Layer-width selection buys more at low bit counts, which is exactly where you need it.

PTQ vs QAT

Post-training quantization is everything above: take finished weights, calibrate on a few hundred thousand tokens, write out a smaller file. Minutes to hours, no training loop.

Quantization-aware training simulates the rounding during training so the weights learn to survive it. Google shipped this for Gemma 3 with ~5,000 QAT steps against the non-quantized checkpoint's own probabilities (April 2025), and reported that it cut the Q4_0 perplexity drop by 54% versus plain PTQ, taking the 27B from 54 GB to 14.1 GB.

QAT is worth it if you publish the weights and can amortise the training cost over every download. It is almost never worth it if you are quantizing someone else's model for your own use — a good dynamic PTQ build closes most of that gap for free. Unsloth reports their dynamic quants reaching lower KL-divergence than Gemma 3's QAT builds at comparable size, and while that is a vendor claim about their own product, the third-party KLD table above makes it plausible.

What it costs, measured

Three numbers get used, and they do not measure the same thing.

Perplexity is the one everyone quotes, and it understates the damage. Its failure mode is that per-token differences from the reference cancel out in the average. Accuracy is Not All You Need (Microsoft Research, 2024) documents this: quantized models produce large numbers of flips, answers changing from right to wrong and wrong to right in roughly equal proportion, so the aggregate score barely moves while the model's actual behaviour has shifted. Flips correlate with KL-divergence at Spearman 0.981 on MMLU, and with perplexity much less well.

KL-divergence against the fp16 reference is the metric I'd use if I could only have one. It measures distributional distance per token, so cancellation is not possible, and it is what the credible quant publishers now report.

Task benchmarks are the ground truth, and they show the damage is not evenly distributed. From a unified llama.cpp evaluation on Llama-3.1-8B-Instruct (Kurt, January 2026):

Read the bottom row across. Perplexity rises 22%, HellaSwag falls 0.6 points, MMLU falls 4.2, GSM8K falls 9.3. Multi-step arithmetic degrades roughly fifteen times harder than sentence completion at the same bit width, and no single perplexity number tells you that. The paper also finds schemes with identical perplexity diverging on instruction-following, which is the flips phenomenon showing up in a different dataset.

One honesty note on that table: Q5_K_M scores 78.54 on GSM8K against the F16 baseline's 77.63. Quantization did not make the model better at arithmetic. That is benchmark noise, and it is a useful reminder that sub-point differences in these tables mean nothing.

On the GPU-serving side, Red Hat/Neural Magic's half-million-evaluation study found all schemes recovering over 99% of baseline average on OpenLLM v1, with HumanEval recovery at 99.9% for 8-bit and 98.9% for 4-bit. Their 4-bit W4A16 results drop more on AIME and GPQA-Diamond than elsewhere, which is the same reasoning-first pattern.

The ladder

8-bit is close to free. Q8_0 moved perplexity by 0.01 and every task score by less than 0.2 points in the table above. Mean KLD around 0.003. If you have the memory, stop thinking about it.

4-bit is the sweet spot, and it is not free. Expect roughly 1–3 points on reasoning-heavy tasks and near-zero on everything else. This is the default for a reason.

3-bit is where reasoning starts to break while the model still sounds completely fine. This is the dangerous tier, because the failure is invisible in chat and shows up in arithmetic, tool arguments, and long code edits.

Sub-3-bit is a different proposition. Unsloth's Aider Polyglot runs on DeepSeek V3.1 put real numbers on it:

Going 4-bit → 2-bit costs 3.9 points and saves 132 GB. Going 2-bit → 1-bit costs 10.1 points and saves 49 GB. The curve has a knee and it sits just below 2-bit. Unsloth's own Dynamic 3.0 documentation says the same thing more bluntly: below their UD-Q2_K_XL tier, models degrade badly on tool-calling and agentic use, loop, and return empty responses.

My read: sub-3-bit only makes sense on models large enough that the alternative is not running the model at all. A 1-bit 400B beats a 4-bit 30B on most work. A 1-bit 30B is worse than a 4-bit 8B and you should not build anything on it.

What quantization buys beyond capacity

Decoding one token requires reading every weight the token touches out of memory. That makes decode memory-bandwidth-bound rather than compute-bound, which is the mechanism /prefill-vs-decode covers. Halving the bytes per weight halves the bytes read per token, so quantization buys throughput on the same hardware, not only the ability to load the model.

The Llama-3.1-8B CPU measurements in Kurt's paper show generation going from 2.83 tok/s at F16 to 5.12 at Q4_K_M and 9.91 at Q3_K_S on a dual Xeon 8488C. I trust the direction and the rough magnitude; I do not trust the fine ordering, since Q5_0 beats Q4_K_S in that same table, which cannot be a bandwidth effect and is more likely kernel quality or thread contention. On GPUs the story is cleaner, with Neural Magic reporting ~2.4× single-stream speedup for W4A16 and ~1.8× for W8A8.

What it does not shrink: the KV cache

Weight quantization does nothing to the KV cache. Cache size scales with context length, batch size, and attention-layer count, and at long context it can rival or exceed the weights. A model that fits at 4-bit with a 4K context may not fit at 128K.

The lever is separate. llama.cpp and vLLM both quantize K and V independently. On the quality side, a measurement on Qwen 2.5 Coder 7B moved perplexity from 8.3891 to 8.3934 going from f16 to q8_0 KV, a change of 0.0043. That is nothing, and it halves the cache. The same write-up puts an 8B model's 32K cache at ~6 GB f16, ~3 GB q8_0, ~2 GB q4_0.

q4_0 KV is a real trade rather than a free one, and its cost is architecture-dependent — reported deltas span roughly -0.7% to +3% perplexity depending on the model. Run Q8_0 KV by default and treat q4_0 as a long-context-only measure you verify on your own workload. Note also that quantized KV wants a Flash Attention path with dequant in-kernel; without it you may lose more speed than the memory is worth.

Picking a format for your runtime

The serving stack decides this more than quality does.

vLLM's hardware compatibility matrix is worth reading before you download 200 GB of the wrong thing: AWQ needs Turing or newer, llm-compressor FP8 needs Ada or Hopper, and bitsandbytes works nearly everywhere while being the wrong choice for serving nearly everywhere.

MLX quantizes with group sizes of 64 for 4/6-bit and 32 for 2/3-bit, and the community convention is to keep embeddings and the final projection at higher width than the body — the same sensitivity principle as dynamic GGUF, applied by hand. Apple Silicon's practical constraint is that MLX and GGUF are separate ecosystems, so the model you want may only exist in one of them on any given day.

What I'd actually do

Take the largest model that fits at 4-bit dynamic with room for a full-context KV cache at q8_0, and prefer a dynamic build over a uniform one at equal file size, because the KLD data says you get it for free. Reach below 3-bit only when the model is large enough that the alternative is not running it. Test on your own reasoning and tool-calling traffic rather than on perplexity, because perplexity is the number that will tell you everything is fine right up until the agent starts mangling JSON arguments.

Related: /local-inference-hardware for what fits on what, /prefill-vs-decode for why fewer bytes per token means faster decode, and /context-engineering-for-coding-agents for keeping the context small enough that the KV cache question stops being the binding one.