What changes when you quantize a model

I would start with a 4-bit build when trying a model locally, but I would still test the tasks I need it for. A smaller model file is easy to measure. Whether it changes the answers I care about takes more work.

The useful comparisons below keep the model fixed and vary the quantization. They show why a result on sentence completion cannot tell you much about a coding agent.

The formats

GPTQ (Frantar et al., 2022) quantizes one layer at a time and corrects the remaining weights for the error already introduced, which is why it can hit 3–4 bits on a 175B model in about four GPU hours. AWQ (MLSys 2024 best paper) took the opposite route: look at activation magnitudes, find the ~1% of channels that matter, and scale them up so rounding hurts them less. No backpropagation, no reconstruction, which is why it generalises off its calibration set better than GPTQ does.

NF4 came out of QLoRA. Its 16 levels are placed so each bin holds equal probability mass under a standard normal rather than being evenly spaced. Weights are roughly normal, so this is a better use of 16 codes. NF4 exists mainly to make fine-tuning fit in memory; as a serving format it is slower than a Marlin-kernel GPTQ or AWQ model on the same hardware.

MXFP4 is the odd one, because for gpt-oss it is not a compression step applied afterwards. OpenAI post-trained with the MoE weights already in MXFP4, and those weights are 90%+ of the parameter count, so the 120B fits on a single 80GB GPU and the 20B in 16GB. There is no fp16 reference version of those tensors to be worse than.

Choosing precision per layer

Everything above except the _M/_S split applies one width to the whole model. Unsloth's dynamic quants pick a width per layer from measured sensitivity, so a nominal "2-bit" build is 2-bit in the layers that tolerate it and 4- or 6-bit in the ones that don't. This gives the publisher another way to balance file size and error.

The measurement that convinced me is Unsloth's Qwen3.5-35B-A3B GGUF table, which reports mean KL-divergence against the reference for the same model quantized by several people:

The two bolded rows are the comparison I would look at first. The dynamic build is 0.6 GB smaller than the uniform Q4_K_M and 25% closer to the reference distribution. At 2 bits the gap widens to nearly 2×. Layer-width selection buys more at low bit counts, which is exactly where you need it.

PTQ vs QAT

Post-training quantization is everything above: take finished weights, calibrate on a few hundred thousand tokens, write out a smaller file. Minutes to hours, no training loop.

Quantization-aware training simulates the rounding during training so the weights learn to survive it. Google shipped this for Gemma 3 with ~5,000 QAT steps against the non-quantized checkpoint's own probabilities (April 2025), and reported that it cut the Q4_0 perplexity drop by 54% versus plain PTQ, taking the 27B from 54 GB to 14.1 GB.

QAT is worth it if you publish the weights and can amortise the training cost over every download. It is almost never worth it if you are quantizing someone else's model for your own use — a good dynamic PTQ build closes most of that gap for free. Unsloth reports their dynamic quants reaching lower KL-divergence than Gemma 3's QAT builds at comparable size, and while that is a vendor claim about their own product, the KLD table above is also published by Unsloth and should be read with that source in mind.

What it costs, measured

Three numbers get used, and they do not measure the same thing.

Perplexity is the one everyone quotes, and it understates the damage. Its failure mode is that per-token differences from the reference cancel out in the average. Accuracy is Not All You Need (Microsoft Research, 2024) documents this: quantized models produce large numbers of flips, answers changing from right to wrong and wrong to right in roughly equal proportion, so the aggregate score barely moves while the model's actual behaviour has shifted. Flips correlate with KL-divergence at Spearman 0.981 on MMLU, and with perplexity much less well.

KL-divergence against the fp16 reference is the metric I'd use if I could only have one. It measures distributional distance per token, so cancellation is not possible, and it is what the credible quant publishers now report.

Task benchmarks are the ground truth, and they show the damage is not evenly distributed. From a unified llama.cpp evaluation on Llama-3.1-8B-Instruct (Kurt, January 2026):

Read the bottom row across. Perplexity rises 22%, HellaSwag falls 0.6 points, MMLU falls 4.2, GSM8K falls 9.3. Multi-step arithmetic degrades roughly fifteen times harder than sentence completion at the same bit width, and no single perplexity number tells you that. The paper also finds schemes with identical perplexity diverging on instruction-following, which is the flips phenomenon showing up in a different dataset.

One caution when reading that table: Q5_K_M scores 78.54 on GSM8K against the F16 baseline's 77.63. That single result is not enough to conclude quantization improved arithmetic. I would want repeated measurements before drawing a conclusion from a small difference.

On the GPU-serving side, Red Hat/Neural Magic's half-million-evaluation study found all schemes recovering over 99% of baseline average on OpenLLM v1, with HumanEval recovery at 99.9% for 8-bit and 98.9% for 4-bit. Their 4-bit W4A16 results drop more on AIME and GPQA-Diamond than elsewhere, which is the same reasoning-first pattern.

How I would read the bit-depth tradeoff

8-bit is close to free. Q8_0 moved perplexity by 0.01 and every task score by less than 0.2 points in the table above. Mean KLD around 0.003. If memory allows, that makes it a reasonable starting point.

4-bit is the sweet spot, and it is not free. Expect roughly 1–3 points on reasoning-heavy tasks and near-zero on everything else. This is the default for a reason.

3-bit is where reasoning starts to break while the model still sounds completely fine. A fluent chat response may not reveal errors in arithmetic, tool arguments or long code edits.

Sub-3-bit is a different proposition. Unsloth's Aider Polyglot runs on DeepSeek V3.1 put real numbers on it:

Going 4-bit → 2-bit costs 3.9 points and saves 132 GB. Going 2-bit → 1-bit costs 10.1 points and saves 49 GB. The curve has a knee and it sits just below 2-bit. Unsloth's own Dynamic 3.0 documentation says the same thing more bluntly: below their UD-Q2_K_XL tier, models degrade badly on tool-calling and agentic use, loop, and return empty responses.

I would consider sub-3-bit when it is the only way to fit a larger model I have a reason to use. I would still compare it with a smaller model at higher precision on the actual workload; parameter count alone cannot settle that choice.

What quantization buys beyond capacity

Decoding one token requires reading every weight the token touches out of memory. That makes decode memory-bandwidth-bound rather than compute-bound, which is the mechanism /prefill-vs-decode covers. Halving the bytes per weight halves the bytes read per token, so quantization buys throughput on the same hardware, not only the ability to load the model.

The Llama-3.1-8B CPU measurements in Kurt's paper show generation going from 2.83 tok/s at F16 to 5.12 at Q4_K_M and 9.91 at Q3_K_S on a dual Xeon 8488C. I trust the direction and the rough magnitude; I do not trust the fine ordering, since Q5_0 beats Q4_K_S in that same table, which cannot be a bandwidth effect and is more likely kernel quality or thread contention. On GPUs the story is cleaner, with Neural Magic reporting ~2.4× single-stream speedup for W4A16 and ~1.8× for W8A8.

What it does not shrink: the KV cache

Weight quantization does nothing to the KV cache. Cache size scales with context length, batch size, and attention-layer count, and at long context it can rival or exceed the weights. A model that fits at 4-bit with a 4K context may not fit at 128K.

The lever is separate. llama.cpp and vLLM both quantize K and V independently. On the quality side, a measurement on Qwen 2.5 Coder 7B moved perplexity from 8.3891 to 8.3934 going from f16 to q8_0 KV, a change of 0.0043. That is a small perplexity change in this measurement, alongside a halving of cache memory. The same write-up puts an 8B model's 32K cache at ~6 GB f16, ~3 GB q8_0, ~2 GB q4_0.

q4_0 KV is a real trade rather than a free one, and its cost is architecture-dependent — reported deltas span roughly -0.7% to +3% perplexity depending on the model. Run Q8_0 KV by default and treat q4_0 as a long-context-only measure you verify on your own workload. Note also that quantized KV wants a Flash Attention path with dequant in-kernel; without it you may lose more speed than the memory is worth.

Picking a format for your runtime

The serving stack decides this more than quality does.

vLLM's hardware compatibility matrix is worth reading before you download 200 GB of the wrong thing: AWQ needs Turing or newer, llm-compressor FP8 needs Ada or Hopper, and bitsandbytes works nearly everywhere while being the wrong choice for serving nearly everywhere.

MLX quantizes with group sizes of 64 for 4/6-bit and 32 for 2/3-bit, and the community convention is to keep embeddings and the final projection at higher width than the body — the same sensitivity principle as dynamic GGUF, applied by hand. Apple Silicon's practical constraint is that MLX and GGUF are separate ecosystems, so the model you want may only exist in one of them on any given day.

Choosing a build

Take the largest model that fits at 4-bit dynamic with room for a full-context KV cache at q8_0, and prefer a dynamic build over a uniform one at equal file size, because the KLD data says you get it for free. Reach below 3-bit only when the model is large enough that the alternative is not running it. Before using it in an agent, I would check reasoning tasks and tool arguments directly. Those are the failures I need to catch, even when the perplexity score barely moves.

Related: /local-inference-hardware for what fits on what, /prefill-vs-decode for why fewer bytes per token means faster decode, and /context-engineering-for-coding-agents for keeping the context small enough that the KV cache question stops being the binding one.