# Ong Jun Xiong — Notes > Working notes on AI systems: prompt caching across agent harnesses, prefill vs decode economics, quantization trade-offs, local inference hardware, and how agent integrations get built inside large organisations. Source: https://notes.junxiong.dev --- # How agent integrations get built inside large organisations URL: https://notes.junxiong.dev/agent-integrations-in-large-orgs Updated: 2026-08-28 ## The shape of the problem An agent is worth exactly the set of systems it can reach. Everything else is a demo. Inside a company of any size, reaching a system means clearing four separate gates for every service: an auth scheme somebody else chose, a rate limit somebody else tuned, a data-access policy somebody else owns, and a team whose roadmap does not include you. Four gates times fifty services is the actual project. The model is not the hard part and has not been for a while. This is the generic version of a problem I keep seeing solved the same way in different buildings. The convergent shape is worth writing down, because if you know the shape you can skip about six months of rediscovering it. ## Why everyone builds a registry The first version of an internal agent always hand-rolls HTTP calls. Someone writes a Python function that hits the ticketing API, pastes a bearer token into a config, and it works. The second and third teams do the same thing against the same API with different tokens, different retry logic and no shared record that any of it exists. Then the org builds a registry. Not because registries are elegant, but because four questions have no answer without one: | Question | What the registry answers | |---|---| | What can an agent do here? | Discovery — one catalogue instead of tribal knowledge | | Who authorised this call? | A single auth path per connector, not per caller | | How do we turn it off? | One revocation point when a credential leaks | | What did it touch last Tuesday? | Audit that spans agents, not per-team logs | ```d2 What changes is the number of paths. Above, each agent carries its own credential and reaches the service directly, so there is no place to look and nothing single to switch off. Below, the broker is the only route, which is what makes one audit log and one revocation point possible. direction: down a1: Agent A\nown token { style: { fill: "#fbe8de"; stroke: "#eb6834"; stroke-width: 2; font-size: 22 } } a2: Agent B\nown token { style: { fill: "#fbe8de"; stroke: "#eb6834"; stroke-width: 2; font-size: 22 } } a3: Agent C\nown token { style: { fill: "#fbe8de"; stroke: "#eb6834"; stroke-width: 2; font-size: 22 } } svc1: Internal service { style: { fill: transparent; stroke: "#6b6459"; stroke-width: 1; font-size: 22 } } pain: Three credentials, three logs.\nA leak means hunting every copy. { style: { fill: "#fffdf9"; stroke: "#eb6834"; stroke-width: 2; font-size: 22 } } a1 -> svc1 { style: { stroke: "#eb6834"; stroke-width: 2 } } a2 -> svc1: pasted credential { style: { stroke: "#eb6834"; stroke-width: 2; font-size: 20 } } a3 -> svc1 { style: { stroke: "#eb6834"; stroke-width: 2 } } svc1 -> pain { style: { stroke: "#eb6834"; stroke-width: 2 } } reg: TOOL REGISTRY\none catalogue, live { style: { fill: "#e4edf9"; stroke: "#2a78d6"; stroke-width: 2; font-size: 22 } } pain -> reg: the change { style: { stroke: "#6b6459"; stroke-width: 2; stroke-dash: 4; font-size: 20 } } b1: Agent A\nno token { style: { fill: "#e4edf9"; stroke: "#2a78d6"; stroke-width: 2; font-size: 22 } } b2: Agent B\nno token { style: { fill: "#e4edf9"; stroke: "#2a78d6"; stroke-width: 2; font-size: 22 } } b3: Agent C\nno token { style: { fill: "#e4edf9"; stroke: "#2a78d6"; stroke-width: 2; font-size: 22 } } reg -> b1 { style: { stroke: "#2a78d6"; stroke-width: 2 } } reg -> b2: schemas { style: { stroke: "#2a78d6"; stroke-width: 2; font-size: 20 } } reg -> b3 { style: { stroke: "#2a78d6"; stroke-width: 2 } } broker: AUTH BROKER\nthe only path { style: { fill: "#e0f4ec"; stroke: "#1baf7a"; stroke-width: 2; font-size: 22 } } b1 -> broker { style: { stroke: "#1baf7a"; stroke-width: 2 } } b2 -> broker: call tool { style: { stroke: "#1baf7a"; stroke-width: 2; font-size: 20 } } b3 -> broker { style: { stroke: "#1baf7a"; stroke-width: 2 } } svc2: Internal service { style: { fill: transparent; stroke: "#6b6459"; stroke-width: 1; font-size: 22 } } audit: One audit log.\nOne switch to revoke. { style: { fill: "#e0f4ec"; stroke: "#1baf7a"; stroke-width: 2; font-size: 22 } } broker -> svc2: scoped token { style: { stroke: "#1baf7a"; stroke-width: 2; font-size: 20 } } broker -> audit: every call { style: { stroke: "#1baf7a"; stroke-width: 2; font-size: 20 } } ``` The prior art is not from the agent world. Backstage's software catalogue does this for services: teams commit a metadata YAML alongside the code, the catalogue harvests it, and the stated goal is that "no more orphan software" hides in dark corners of the org ([Backstage docs](https://backstage.io/docs/features/software-catalog/)). Ownership lives with the team, discovery is central. An agent tool registry that works has the same split. Ownership decentralised, index centralised. MCP encodes the same split at the protocol level. A server declares a `tools` capability and answers `tools/list`; a client discovers what exists at connect time rather than at build time. Servers that declare `listChanged` push a `notifications/tools/list_changed` when the set changes, so the catalogue is live rather than a checked-in manifest ([MCP tools spec](https://modelcontextprotocol.io/specification/2025-06-18/server/tools)). My read: the registry is the cheap part and it is still the part orgs get wrong, because a registry with no owner rots into a list of dead endpoints within two quarters. More on that below. ## Auth brokering is the whole project Everything else on this page is a week of work. This part is a year. There are two modes, and confusing them is the most common design error I see. **Service-to-service.** The agent platform holds its own identity and calls downstream with its own credential. Easy to build, and wrong for anything touching user data, because every downstream audit log now reads `agent-platform` and every access check has to be re-implemented inside the agent. **On-behalf-of-user.** The agent carries a credential scoped to the human who asked. Downstream permission checks work unchanged. Audit logs name a person. This is the correct default and it is genuinely hard. The public app platforms model the distinction cleanly. Slack's OAuth v2 splits the request into `scope` (bot token, the app's own identity) and `user_scope` (user token, acting on behalf of a user), and you request both when you need both ([Slack OAuth docs](https://docs.slack.dev/authentication/installing-with-oauth)). GitHub Apps have three credentials for three situations: a JWT to authenticate as the app, an installation access token to act as the installation, and a user access token so the app "only takes actions that could be performed by a specific user" ([GitHub docs](https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/about-authentication-with-a-github-app)). GitHub's installation token also demonstrates the two properties an internal broker needs. It **expires after 1 hour**, and when you mint one you can pass `repositories` and `permissions` to narrow it below what the app was granted ([token generation docs](https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/generating-an-installation-access-token-for-a-github-app)). Short-lived, and downscoped at mint time rather than at grant time. Copy that. The mechanism for doing this generically is [RFC 8693 token exchange](https://www.rfc-editor.org/rfc/rfc8693.html). Grant type `urn:ietf:params:oauth:grant-type:token-exchange`, a `subject_token` for the user whose rights are being used, an optional `actor_token` for the thing doing the acting. The RFC's distinction between **impersonation** (the agent becomes indistinguishable from the user) and **delegation** (the agent keeps its own identity, expressed in an `act` claim, while carrying the user's rights) is the design decision, and delegation is the one you want. When the audit trail says "agent X acting for user Y," an incident review takes an afternoon rather than a week. `audience` and `scope` on the exchange request are how you narrow per call. ```d2 Delegation via RFC 8693. The agent presents two things and receives one: a short-lived token whose audience is a single service, carrying an act claim that keeps the agent's own identity alongside the user's rights. The service accepts it only because the audience names the service. shape: sequence_diagram direction: down agent: Agent { style: { font-size: 22 } } broker: Auth broker { style: { font-size: 22 } } svc: Service { style: { font-size: 22 } } agent -> agent: a user asks\nfor something { style: { font-size: 20 } } agent -> broker: own cred +\nuser subject token { style: { font-size: 20 } } broker -> broker: downscope\naud = 1 service\nTTL: minutes { style: { font-size: 20 } } broker -> agent: token, act claim\nagent for user { style: { font-size: 20 } } agent -> svc: that token only { style: { font-size: 20 } } svc -> svc: aud is me?\nscope? expiry? { style: { font-size: 20 } } svc -> agent: result\nchecks unchanged { style: { font-size: 20 } } ``` ### The confused deputy, made worse An agent reads text and then acts on it. That text can contain instructions. This turns a classic confused-deputy problem into a routine one. MCP's security best practices document spells out the specific version: a proxy server with a static client ID to a third-party auth server, plus dynamic client registration, plus a consent cookie set on first authorisation, equals an attacker who can register a client with their own `redirect_uri` and have the auth server skip the consent screen entirely because the cookie is already there ([MCP security best practices](https://modelcontextprotocol.io/specification/2025-06-18/basic/security_best_practices)). Required mitigations: per-client consent stored server-side and checked *before* forwarding, exact-string `redirect_uri` matching with no wildcards, and single-use `state` values stored only after consent is approved. The related rule is short enough to memorise. **Token passthrough is forbidden.** An MCP server "MUST NOT accept any tokens that were not explicitly issued for the MCP server," and if it calls an upstream API it must obtain a separate token rather than forwarding the one it received. The enforcement mechanism is [RFC 8707](https://www.rfc-editor.org/rfc/rfc8707.html) resource indicators: clients MUST send `resource` on authorisation and token requests, servers MUST validate they are the intended audience ([MCP authorization spec](https://modelcontextprotocol.io/specification/2025-06-18/basic/authorization)). If your internal broker forwards the caller's token to the downstream service unchanged, you have built the anti-pattern. It will pass review the first time, because it works. ## Tool definitions are an interface contract Hand-written tool schemas drift. The API adds a required field, the schema does not, and the agent starts failing in a way that looks like a model problem for the two days it takes anyone to check the spec. Generate them. FastMCP's `from_openapi()` turns every endpoint in a spec into a tool by default, deriving names from `operationId`, with `RouteMap` rules that can mark internal or admin routes as `EXCLUDE` ([FastMCP OpenAPI docs](https://gofastmcp.com/integrations/openapi)). Protobuf service definitions work the same way. The generator is a day of work and it removes an entire class of drift bug permanently. The one field you cannot generate is `description`, and it is the field that decides whether the agent works. That field is prompt text. The model reads it and nothing else when choosing between two similar tools. Anthropic's guidance on this is blunt: "even small refinements to tool descriptions can yield dramatic improvements," and "too many tools or overlapping tools can also distract agents from pursuing efficient strategies" ([Writing tools for agents](https://www.anthropic.com/engineering/writing-tools-for-agents)). Namespacing under common prefixes is the recommended way to draw boundaries when there are many. A generic example of a description doing real work: ```json { "name": "ticketing_search_issues", "description": "Search issues in the ticketing system by text query. Returns at most 25 results, newest first. Use this to FIND an issue when you only know words from its title or body. Do NOT use this to fetch a known issue by ID — use ticketing_get_issue, which is cheaper and returns full body text. This tool cannot see issues in restricted projects the calling user lacks read access to; those are silently omitted.", "inputSchema": { "type": "object", "properties": { "query": { "type": "string", "description": "Free-text search terms. Not a query language; boolean operators are ignored." }, "project": { "type": "string", "description": "Optional project key to restrict the search, e.g. PLAT." } }, "required": ["query"] } } ``` Three things in there are absent from any OpenAPI spec: when *not* to use it, what the cheaper alternative is, and that results are silently filtered by the caller's permissions. All three are the difference between a working agent and a confusing one. Write them by hand, keep them next to the generator config, and diff them in review like code. Two mechanical consequences worth knowing. Tool definitions sit **first** in the cacheable prefix (the order is `tools`, `system`, then `messages`) and modifying tool definitions "invalidates the entire cache" at every level ([prompt caching docs](https://platform.claude.com/docs/en/build-with-claude/prompt-caching)). A registry that returns tools in nondeterministic order destroys the cache on every request. Sort the list. See [prompt caching across harnesses](/prompt-caching-across-harnesses) for what that costs, and [prefill vs decode](/prefill-vs-decode) for why a 40k-token tool prefix is a prefill bill you pay on every turn. ## Shipping it The deployment path is boring and should stay boring: the connector is a service, so it ships like a service. Internal PaaS or a Kubernetes target, staged environments, a review gate before production credentials are issued. Kubernetes operators are the closest structural analogue for the runtime — a control loop reconciling declared connector state against what is actually registered. Observability is where agent work differs from normal service work, and the three things you need are specific: 1. **A trace that spans the agent loop**, not just individual HTTP calls. One trace ID from user turn through every tool call and back, or you cannot answer "why did it do that." 2. **Per-tool latency and error rate**, broken out by tool name. Tool sprawl is invisible until you can see that eleven tools have never been called and three account for 90% of errors. 3. **Replayable transcripts.** The full message list including tool schemas as they were at the time. Without the schema snapshot you cannot reproduce a failure after someone edits a description. ## What actually kills these projects Not the model, and not the protocol. Roughly in order of how often they are the cause: **Data-owner sign-off has no SLA.** Every connector touching user data needs approval from whoever owns that data. That review is unbudgeted, unqueued and unowned, so it takes as long as it takes. Fifteen connectors at three weeks each, serialised through one privacy reviewer, is your actual delivery date. **Nobody owns the registry.** It gets built by whichever team needed it first, then that team reorgs. Entries rot. New teams route around it because the thing they need is not in it, which makes it less complete, which makes more teams route around it. **Tool sprawl.** This one has numbers now. GitHub cut Copilot's default toolset to 13 core tools and grouped the rest into four virtual categories, reporting a 2–5 percentage point improvement in success rate on SWE-Lancer and SWE-bench Verified across GPT-5 and Sonnet 4.5, plus roughly 400ms lower response latency in online A/B testing ([GitHub blog](https://github.blog/ai-and-ml/github-copilot/how-were-making-github-copilot-smarter-with-fewer-tools/)). A registry with 200 entries and no selection layer is worse than one with 15. Treat it as a [context engineering](/context-engineering-for-coding-agents) problem, because that is what it is: filter to a task-relevant subset before the request, or accept the degradation. **The long tail has no API.** A meaningful fraction of internal systems expose a web UI, a shared spreadsheet, or a nightly CSV drop. There is no clean connector for these. Someone will propose browser automation. Budget for it being flaky and scope it to read-only. MCP's own scope-minimisation guidance names the pattern that makes all of this worse — publishing every possible scope in `scopes_supported` and using omnibus scopes like `*` or `full-access`, which drives consent abandonment and makes revocation disruptive. Start with a minimal read-only scope set and elevate on demand via `WWW-Authenticate` challenges. ## What MCP actually changed It did not solve auth. The hard parts of auth are still hard, and the spec mostly points at OAuth 2.1, RFC 8707 and RFC 9728 rather than replacing them. What it changed is the coupling. Before a standard protocol, an internal tool server was written against one agent framework, and moving to a different framework meant rewriting the integration layer. Now the server is written once and any client speaking the protocol consumes it — an IDE assistant, a chat surface, a CI job, a different vendor's model entirely. `tools/list` and `tools/call` are the whole contract for tools. My read on why that matters organisationally rather than technically: it changes who has to say yes. The team that owns a service can stand up a server for their own data and put it in the registry, without the agent platform team writing a line of code and without a framework migration hanging over the decision. That removes the single worst bottleneck in this whole design, which was never a technical one. --- # Where your tokens actually go in a coding agent URL: https://notes.junxiong.dev/context-engineering-for-coding-agents Updated: 2026-08-28 ## Your prompt is about half a percent of the request Anthropic publishes an interactive simulation of a Claude Code session filling its context window, with token counts on every block. Before you type anything, this is what has already loaded: | Block | Tokens | |---|---:| | System prompt | 4,200 | | Project `CLAUDE.md` | 1,800 | | Auto memory | 680 | | Skill descriptions | 450 | | User-level `CLAUDE.md` | 320 | | Environment info (cwd, shell, OS, git) | 280 | | MCP tool names, schemas deferred | 120 | | **Total before you type** | **7,850** | | Your prompt | 45 | Source: [Explore the context window](https://code.claude.com/docs/en/context-window), Claude Code docs. Forty-five tokens out of 7,895. That ratio only gets worse from there, because the prompt is the one part that never grows. The 120-token line for MCP tools is the interesting one, and it is small only because tool schemas are deferred by default now. Load them eagerly and the number changes shape entirely: Anthropic measured 58 tools across five MCP servers at [roughly 55,000 tokens before the conversation starts](https://www.anthropic.com/engineering/advanced-tool-use), and a heavier catalog at [150,000 tokens, cut to 2,000 by loading on demand](https://www.anthropic.com/engineering/code-execution-with-mcp). If your harness or your org's platform hands every engineer the same twelve integrations by default, that is where your window went. I wrote about how that sprawl happens in [agent integrations in large orgs](/agent-integrations-in-large-orgs). ## Then the reads arrive The startup cost is fixed and cacheable. The part that actually kills you is what the agent pulls in while working. From the same published trace, a single ordinary auth-token task: | Event | Tokens | |---|---:| | Read `src/api/auth.ts` | 2,400 | | Read `middleware.ts` | 1,800 | | Read `auth.test.ts` | 1,600 | | Read `src/lib/tokens.ts` | 1,100 | | `npm test` output | 1,200 | | `grep "refreshToken"` | 600 | | Two edits plus formatter hooks | 1,220 | Just under ten thousand tokens of file contents and command output against 45 tokens of instruction. And that is a well-behaved session. A `cat` of a 4,000-line generated client, a `pytest` run that prints every passing test name, an `npm install` log, a stack trace with 200 frames of framework internals — any one of those lands 20k to 50k tokens in the window, permanently, and you pay for it on every subsequent turn. That last clause is the part people miss. The API is stateless. Every turn re-sends the entire conversation from the top, so a bad read at turn 6 is still in the request at turn 60. Total billed input over a session grows with roughly the square of the turn count, not linearly. My arithmetic, not a published figure: a 20k baseline plus 3k of new content per turn bills about 365k tokens over 10 turns and about 17.2M over 100. Doubling the session length costs about 3.6x, and caching changes the price of those tokens without changing the count. The fix is to give the agent a way to *search* rather than a pile to read. A grep that returns 40 matching lines costs a few hundred tokens; the file it came from costs thousands. The same applies to test output, and it is worth wiring up once: ```bash # Instead of: npm test # Feed the agent only what it needs to act on. npm test 2>&1 | grep -E -A5 '(FAIL|✕|Error:)' | head -100 ``` Claude Code's docs suggest [doing this in a `PreToolUse` hook](https://code.claude.com/docs/en/costs#offload-processing-to-hooks-and-skills) so the agent never sees the raw output at all. Any harness with command interception can do the same thing. On a self-hosted model the cost of a long context is memory rather than money, since the KV cache competes with the weights for VRAM — see [local inference hardware](/local-inference-hardware) and [what quantization costs](/quantization-what-it-costs). Either way, the re-sent history is prefill work, which is the cheap-per-token but latency-dominant half of the equation ([prefill vs decode](/prefill-vs-decode)). ## A full window gives worse answers This is the part I have to argue with people about, because "200k context" reads like a capacity you can fill. It is a limit, not a working range. The clearest evidence is **NoLiMa** ([arXiv:2502.05167](https://arxiv.org/abs/2502.05167)), which hides a fact in a long document and strips the literal word overlap between the question and the fact, so lexical matching cannot rescue the model. It defines a model's *effective length* as the longest tested context at which it still exceeds 85% of its base score, where the base is its best average across 250-, 500- and 1K-token inputs. Table 3 of the paper: | Model | Advertised window | Base score | Effective length | Score at 32K | |---|---|---:|---|---:| | GPT-4o | 128K | 99.3 | 8K | 69.7 | | Llama 3.3 70B | 128K | 97.3 | 2K | 42.7 | | Gemini 1.5 Pro | 2M | 92.6 | 2K | 48.2 | | Command R+ | 128K | 90.9 | under 1K | 7.4 | | Claude 3.5 Sonnet | 200K | 87.6 | 4K | 29.8 | The paper's summary line: "Out of the 13 models, 11 exhibit performance at 32K lengths that is half or less of their base scores." Reasoning does not buy you out of it either. On the hard subset, o1 scores 99.9 at base and 31.1 at 32K. ```vega-lite Every one of these models starts between 87.6 and 99.3 on the same metric. At 32K, four of the five have lost more than half of it. {"title":{"text":"NoLiMa score at 32K context","subtitle":"NoLiMa Table 3 (arXiv:2502.05167). Same metric as each model's base score, which runs 87.6-99.3 across these five."}, "height":{"step":38}, "data":{"values":[ {"model":"GPT-4o (base 99.3)","v":69.7}, {"model":"Gemini 1.5 Pro (base 92.6)","v":48.2}, {"model":"Llama 3.3 70B (base 97.3)","v":42.7}, {"model":"Claude 3.5 Sonnet (base 87.6)","v":29.8}, {"model":"Command R+ (base 90.9)","v":7.4}]}, "encoding":{ "y":{"field":"model","type":"nominal","sort":"-x","title":null,"axis":{"labelFontSize":13}}, "x":{"field":"v","type":"quantitative","title":"score at 32K context","scale":{"domain":[0,100]},"axis":{"grid":true}}}, "layer":[ {"mark":{"type":"bar","height":24},"encoding":{"color":{"value":"#2a78d6"}}}, {"mark":{"type":"text","align":"left","dx":8,"fontWeight":600,"fontSize":13}, "encoding":{"text":{"field":"v","type":"quantitative","format":",.3~f"}}}]} ``` **Lost in the Middle** ([arXiv:2307.03172](https://arxiv.org/abs/2307.03172), Liu et al., TACL) is older and more specific: accuracy is highest when the relevant passage sits at the start or the end of the input and sags in between. Their GPT-3.5-Turbo row across 20 documents runs 75.8 at the first position, 53.8 in the middle, 63.2 at the last. The number I keep coming back to is the control: 56.1 with no documents at all. Handing the model twenty documents with the answer buried in the middle scored worse than handing it nothing. ```vega-lite Ranked, the control lands above the middle position: twenty documents with the answer buried in them scored worse than supplying no documents at all. {"title":{"text":"GPT-3.5-Turbo accuracy by answer position, 20 documents","subtitle":"Lost in the Middle (arXiv:2307.03172, Liu et al., TACL). The orange bar is the closed-book control, with no documents supplied."}, "height":{"step":38}, "data":{"values":[ {"case":"Answer at first position","v":75.8,"kind":"20 documents supplied"}, {"case":"Answer at last position","v":63.2,"kind":"20 documents supplied"}, {"case":"No documents at all","v":56.1,"kind":"Closed-book control"}, {"case":"Answer in the middle","v":53.8,"kind":"20 documents supplied"}]}, "encoding":{ "y":{"field":"case","type":"nominal","sort":"-x","title":null,"axis":{"labelFontSize":13}}, "x":{"field":"v","type":"quantitative","title":"accuracy (%)","scale":{"domain":[0,80]},"axis":{"grid":true}}}, "layer":[ {"mark":{"type":"bar","height":24}, "encoding":{"color":{"field":"kind","type":"nominal","title":null, "scale":{"domain":["20 documents supplied","Closed-book control"],"range":["#2a78d6","#eb6834"]}}}}, {"mark":{"type":"text","align":"left","dx":8,"fontWeight":600,"fontSize":13}, "encoding":{"text":{"field":"v","type":"quantitative","format":",.3~f"}}}]} ``` Chroma's [context rot report](https://www.trychroma.com/research/context-rot) (18 models) adds the finding that matters most operationally: "models do not use their context uniformly; instead, their performance grows increasingly unreliable as input length grows." You do not get a gentle slope you can budget against. The same report found that shuffling the haystack to destroy its logical ordering *improved* scores, which should unsettle anyone who thinks of a long context as a well-organised briefing document. It also splits the failure by family — Claude models "tend to abstain when uncertain," while GPT models show "the highest rates of hallucination, often generating confident but incorrect responses." One of those failures is easy to notice. The other is not. Anthropic's own guidance calls context an ["attention budget"](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) that gets drawn down, which is a vendor telling you not to use the capacity it sells you. I find that more persuasive than the papers, honestly. The practical consequence is blunt. A fresh session with a tight brief beats a 200k session that has been running since lunch. Clearing between unrelated tasks is the cheapest quality lever available to you, and it is the one people treat as optional housekeeping. ## What compaction actually costs When a harness runs out of room it summarises the history to reclaim space. Two things happen. First, detail is discarded and you do not choose which. The summary keeps what the summariser thought mattered. Anthropic's own warning is that ["overly aggressive compaction can result in the loss of subtle but critical context whose importance only becomes apparent later"](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents). In practice that means the constraint you gave at turn 3 about never touching the migration files. Their advice follows from it: put durable rules in the instruction file, because conversation history is not storage. ```d2 The blue block is written once and read from cache on every turn. The orange loop grows underneath it until compaction fires, which throws away detail you did not choose and takes the cached prefix with it. direction: down prefix: FIXED PREFIX\ntool schemas · system prompt\ninstruction file\n\nWritten once. Read from cache. { style: { fill: "#e4edf9"; stroke: "#2a78d6"; stroke-width: 2; font-size: 22 } } turn: Your prompt { style: { stroke: "#6b6459"; fill: transparent; stroke-width: 1; font-size: 22 } } grow: THE LOOP THAT GROWS\nfile reads · test output\nedits · hook output\n\nRe-sent in full every turn. { style: { fill: "#fbe8de"; stroke: "#eb6834"; stroke-width: 2; font-size: 22 } } full: Window full { style: { fill: transparent; stroke: "#eb6834"; stroke-width: 2; font-size: 22 } } compact: COMPACTION\nhistory summarised { style: { fill: "#fbe8de"; stroke: "#eb6834"; stroke-width: 2; font-size: 22 } } lost: Detail discarded —\nchosen for you, not by you { style: { fill: "#fffdf9"; stroke: "#eb6834"; stroke-width: 2; font-size: 22 } } prefix -> turn: every turn { style: { stroke: "#2a78d6"; stroke-width: 2; font-size: 20 } } turn -> grow { style: { stroke: "#6b6459"; font-size: 20 } } grow -> grow: each tool call { style: { stroke: "#eb6834"; stroke-width: 2; font-size: 20 } } grow -> full { style: { stroke: "#eb6834"; stroke-width: 2; font-size: 20 } } full -> compact { style: { stroke: "#eb6834"; stroke-width: 2; font-size: 20 } } compact -> lost { style: { stroke: "#eb6834"; stroke-width: 2; font-size: 20 } } compact -> prefix: cache prefix dies { style: { stroke: "#eb6834"; stroke-width: 2; stroke-dash: 4; font-size: 20 } } ``` Second, the prompt cache. Caching is a strict prefix match, so compaction [invalidates the conversation layer by design](https://code.claude.com/docs/en/prompt-caching#compacting-the-conversation) — the new, shorter history shares no prefix with the old one. One correction to the folk wisdom here: while the cache is still warm, the summarisation call itself reads the old prefix from cache and is cheaper than the context size suggests. It is when you resume a cold session that compaction reprocesses the whole history at full price. The docs are explicit that `/clear` costs nothing by comparison. I go through the prefix-stability mechanics, and how they differ across harnesses, in [prompt caching across harnesses](/prompt-caching-across-harnesses). Rule I follow: if I want continuity, compact at a task boundary. If I want a fresh start, clear. Compacting to avoid re-explaining is usually slower than re-explaining. ## Scoping is the whole skill The public numbers on agent-authored code are real, and they are all narrow. **Published, primary source.** Stripe reports that ["over 1,300 Stripe pull requests merged each week are completely minion-produced, human-reviewed, but containing no human-written code"](https://stripe.dev/blog/minions-stripes-one-shot-end-to-end-coding-agents-part-2), up from a thousand ten days earlier. Part 1 notes the code involved ["moves well over $1 trillion per year of payment volume live in production"](https://stripe.dev/blog/minions-stripes-one-shot-end-to-end-coding-agents). The work they describe is bounded: fixing flaky tests, clearing small on-call issues, LLM-assisted migrations across the codebase, running linters. Their pipeline literally has a node called "Fix CI failures". **Published, primary source.** Monzo reports agents ["authoring ~10% of all merged PRs"](https://monzo.com/blog/building-agent-chip) and "routinely running more than 1800 tasks every day". **Company claim, secondhand publisher.** Shopify's Head of Engineering told Bessemer that ["engineer productivity has increased by roughly 20%"](https://www.bvp.com/atlas/inside-shopifys-ai-first-engineering-playbook), and in the same interview rejects lines of code and PR count as measures, preferring weekly demos. That 20% is his estimate, not an instrumented number, which is a direct consequence of refusing to instrument it. I'd treat it as a direction. **Company claim, official talk.** Booking.com's developer-experience team describes an enablement program across [3,000+ developers](https://www.youtube.com/watch?v=v2GirPD0gf4), taking GenAI adoption from under 10% to over 75%, with 65% of those users on it daily. The "65% higher adoption" figure that circulates from this talk is a garble of that daily-use share, and I have left it out. Notice what none of these are. Nobody published "the agent built the feature." Every headline number is made of work with an edge you can point at: migrations, lint, flaky tests, small issues with a machine-checkable definition of done. That is the difference between the two engineers. One hands the agent a task with a verifiable boundary and gets a merged PR. The other hands it "make the checkout flow better," gets 900 lines of plausible code, and spends the afternoon reviewing it. Same tool, same model, same repo. My read: scoping ability is doing almost all of the work that people attribute to prompt wording, and it is a skill engineers already have from writing tickets for other humans. They just don't apply it, because the agent doesn't push back the way a junior would. The Stack Overflow 2025 survey found [66% of developers naming "AI solutions that are almost right, but not quite" as their top frustration](https://survey.stackoverflow.co/2025/ai), with 45.2% saying debugging AI-generated code takes longer. Almost-right is what unscoped work produces. It looks finished, so it reaches review, and the defect is found late by a person instead of early by a test. ## What actually works **Search, don't dump.** Name the exact file when you know it. When you don't, let the agent grep. Never paste a file you haven't read yourself. **Keep the instruction file small and stable.** Anthropic's guidance is to [aim for under 200 lines](https://code.claude.com/docs/en/costs#move-instructions-from-claude-md-to-skills). Bigger files don't get followed harder, they get followed less. Move workflow-specific instructions into on-demand skills so they cost nothing when you're doing something else. One mechanical trap: in Claude Code, `CLAUDE.md` is [read once at session start](https://code.claude.com/docs/en/prompt-caching#editing-claude-md-mid-session), so a mid-session edit neither breaks the cache nor takes effect. People lose real time to this. **Use sub-agents for exploration.** A sub-agent can read 40 files in its own window and return a paragraph. Given quadratic growth, every token kept out of the main thread is kept out of every later turn too. Don't overdo it: Anthropic's own [multi-agent post](https://www.anthropic.com/engineering/multi-agent-research-system) reports a 90.2% win on a *research* eval while conceding "most coding tasks involve fewer truly parallelizable tasks than research," and multi-agent setups use about 15x the tokens of a chat. Cognition argues the [other side from experience](https://cognition.com/blog/dont-build-multi-agents): split agents make conflicting implicit decisions and the merge is a mess. Both sides agree on the same line — parallelise reading, never parallelise editing shared files. **Commit often.** Checkpoints are how you throw away a bad direction without throwing away the session. Rewinding to a commit is also cheaper than compacting, since it truncates back to a prefix that is still cached. **Write the test first.** This has nothing to do with TDD purity. A failing test is a machine-readable definition of done, which is the only thing that stops an agent at "looks finished." Everything else on this list matters less than giving the loop a target it can evaluate without you. ## Measuring it at all Lines of code and PR count are worse than useless here, because agents inflate both by construction and both are trivially gameable by a tool that never gets tired. Shopify's engineering lead rejecting them is the correct instinct. The counter-evidence worth carrying: METR's randomised trial found 16 experienced open-source developers were [19% *slower* with AI tools](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) on 246 issues in repos they knew well, while believing afterwards that they had been 20% faster. Self-report is not a measurement. That is an early-2025 snapshot on mature codebases with high review standards, and it does not generalise to greenfield work, but it should end any argument that settles on "it feels faster." What I'd actually track: | Instead of | Track | |---|---| | Lines written | Time from task start to merged | | PRs opened | Review rounds per PR | | Agent adoption % | Change failure rate | | Tokens spent | Share of PRs merged without a human rewrite | Cost is worth a glance but not a target. Claude Code's docs put enterprise usage at [around $13 per developer per active day](https://code.claude.com/docs/en/costs), under $30 for 90% of users. Against a loaded engineer salary that is noise. Optimise context because it makes the answers better; the bill falling is a side effect. --- # What to actually buy to run open models locally URL: https://notes.junxiong.dev/local-inference-hardware Updated: 2026-08-28 ## Two axes, and most people optimise the wrong one Local inference hardware is decided by two numbers: how much memory you have, and how fast you can read it. Capacity decides *which* models load at all. Bandwidth decides how fast tokens come out once they do. There is no consumer product that maximises both, and the gap between the extremes is roughly 30×. | Hardware | Memory | Bandwidth | Source | |---|---|---|---| | Ryzen AI Max+ 395 (Strix Halo) | up to 128GB | 256 GB/s (~215 GB/s measured) | [llm-tracker](https://llm-tracker.info/AMD-Strix-Halo-(Ryzen-AI-Max+-395)-GPU-Performance) | | NVIDIA DGX Spark (GB10) | 128GB LPDDR5x | ~273 GB/s | [IntuitionLabs](https://intuitionlabs.ai/articles/nvidia-dgx-spark-review) | | Apple M5 Max | 36–128GB | 460 GB/s, 614 GB/s on the 40-core GPU | [Apple specs](https://www.apple.com/mac-studio/specs/) | | Apple M3 Ultra | up to 512GB | 819 GB/s | [Trusted Reviews](https://www.trustedreviews.com/versus/apple-m4-max-vs-m3-ultra-4594325) | | RTX 4090 | 24GB GDDR6X | 1,008 GB/s | [Spheron](https://www.spheron.network/blog/nvidia-rtx-5090-specs/) | | Apple M5 Ultra | 96 / 256 / 512GB | 1.2 TB/s | [Apple specs](https://www.apple.com/mac-studio/specs/) | | RTX 5090 | 32GB GDDR7 | 1,792 GB/s | [Spheron](https://www.spheron.network/blog/nvidia-rtx-5090-specs/) | | H100 SXM | 80GB HBM3 | 3.35 TB/s | [RunPod](https://www.runpod.io/articles/guides/nvidia-h100) | | B200 | 180GB HBM3e | ~8 TB/s | [RunPod](https://www.runpod.io/articles/guides/nvidia-b200) | The shape of that table is the whole argument. A 512GB unified-memory desktop holds a model no consumer GPU can touch, at a third of a 5090's bandwidth. A 5090 will out-generate it on anything that fits in 32GB and simply cannot run anything that doesn't — once llama.cpp starts pushing layers into system RAM, throughput [collapses rather than degrades gracefully](https://bmdpat.com/blog/llama-cpp-n-gpu-layers-explained-2026), because every offloaded layer now reads over a PCIe link instead of GDDR7. ```vega-lite Bandwidth spans 7x across the consumer range, and the fastest box is the one with the least capacity. Sources: llm-tracker (Strix Halo), IntuitionLabs (DGX Spark), Apple specs (M5 Max 40-core GPU, M5 Ultra), Trusted Reviews (M3 Ultra), Spheron (RTX 5090). {"title":{"text":"Memory bandwidth, the number that sets decode speed","subtitle":"Unified memory also buys capacity; the 5090's 1,792 GB/s only reaches 32GB."}, "height":{"step":38}, "data":{"values":[ {"hw":"RTX 5090","v":1792,"kind":"Discrete VRAM"}, {"hw":"Apple M5 Ultra","v":1200,"kind":"Unified memory"}, {"hw":"Apple M3 Ultra","v":819,"kind":"Unified memory"}, {"hw":"Apple M5 Max (40-core)","v":614,"kind":"Unified memory"}, {"hw":"NVIDIA DGX Spark","v":273,"kind":"Unified memory"}, {"hw":"Ryzen AI Max+ 395","v":256,"kind":"Unified memory"}]}, "encoding":{ "y":{"field":"hw","type":"nominal","sort":"-x","title":null,"axis":{"labelFontSize":13}}, "x":{"field":"v","type":"quantitative","title":"GB/s","axis":{"grid":true}}}, "layer":[ {"mark":{"type":"bar","height":24},"encoding":{"color":{"field":"kind","type":"nominal","legend":{"title":null,"orient":"bottom"}}}}, {"mark":{"type":"text","align":"left","dx":8,"fontWeight":600,"fontSize":13}, "encoding":{"text":{"field":"v","type":"quantitative","format":",.0f"}}}]} ``` Which axis binds you depends entirely on decode versus prefill. Generation speed tracks bandwidth; prompt processing tracks compute. See [/prefill-vs-decode](/prefill-vs-decode) for why. That split is visible in the DGX Spark numbers: on gpt-oss-120b it does ~1,723 tok/s of prompt processing but only ~38.6 tok/s of generation, against ~124 tok/s of generation from 3× RTX 3090 ([IntuitionLabs](https://intuitionlabs.ai/articles/nvidia-dgx-spark-review)). Same box, world-class at one phase, mediocre at the other. ## The "will it fit" calculation almost everyone gets wrong Three things claim memory. Nearly every online sizing table counts one of them. **1. Weights.** The number you look up. Q4 of a 120B MoE is about 63GB ([Unsloth's gpt-oss-120b GGUFs](https://huggingface.co/unsloth/gpt-oss-120b-GGUF) run 62.6–63GB across the 2-bit through 4-bit range, because the MXFP4 MoE weights are already quantized natively). What quantizing costs you in quality is a separate question — [/quantization-what-it-costs](/quantization-what-it-costs). **2. KV cache, which scales linearly with context** and is wildly architecture-dependent. The per-token cost is `4 × num_kv_heads × head_dim` bytes per attention layer at bf16, summed over layers that actually cache. Sebastian Raschka [publishes the worked numbers per model](https://sebastianraschka.com/llm-architecture-gallery/kv-cache-calculations/): ``` Qwen3 8B 144 KiB/token (36 layers × 8 KV heads × 128 dim × 4) Gemma 4 31B 840 KiB/token (hybrid sliding-window + global) DeepSeek V3 68.6 KiB/token (MLA compression) Qwen3-Next 80B-A3B 24 KiB/token (only 12 full-attention layers cache) ``` Multiply those out — my arithmetic, from Raschka's per-token figures: ``` Qwen3 8B @ 128K ctx: 131,072 × 147,456 B = 18.0 GiB KV ...against roughly 4.5 GB of Q4 weights. KV is 4× the model. Gemma 4 31B @ 128K ctx: 131,072 × 860,160 B = 105 GiB KV ...against roughly 17 GB of Q4 weights. KV is 6× the model. Qwen3-Next 80B @ 1M ctx: 1,048,576 × 24,576 B = 24 GiB KV ...against roughly 45 GB of Q4 weights. KV is half the model. ``` An 8B model can need more memory for its context than a 31B model needs for its weights. This is the single most common sizing error I see, and it is why "a 24GB card runs 8B models" is true for chat and false for a coding agent with 100K of repo in context. **3. Prompt-cache retention**, which trades memory for latency and is the difference between an always-on box feeling instant and feeling broken. LM Studio's mlx-engine reports a 40K-token context taking ~200 seconds to process cold versus ~5 seconds with cache reuse ([LM Studio](https://lmstudio.ai/blog/mlx-engine-agentic-workloads)). You pay for that in resident memory. More on the harness-level differences in [/prompt-caching-across-harnesses](/prompt-caching-across-harnesses). **The rule that falls out:** budget Q4 weights plus the KV cache at the context length you will actually use, plus ~20% for the OS and cache retention. For most 2026 architectures that lands between 1.5× and 2× the weight size. Do not trust the multiplier — look up your model's KV-per-token and multiply, because hybrid-attention models like Qwen3-Next are 30× cheaper per token than Gemma 4. A config that only just fits the weights cannot use the context window the model card advertises. ```d2 Three claims on memory, summed. Nearly every sizing table budgets the first box and stops there, which is how an "it fits" config loses the context window the model card advertises. direction: down w: 1 · Q4 WEIGHTS\n63 GB for a 120B MoE\n\nThe only line most\nsizing tables count. { style: { fill: "#e4edf9"; stroke: "#2a78d6"; stroke-width: 2; font-size: 22 } } kv: 2 · KV CACHE\nat your real context\n\nGemma 4 31B @ 128K = 105 GiB\nQwen3-Next 80B @ 1M = 24 GiB { style: { fill: "#fbe8de"; stroke: "#eb6834"; stroke-width: 2; font-size: 22 } } os: 3 · ~20% OS +\nprompt-cache retention\n\n40K ctx back in 5s, not 200s. { style: { fill: "#fbe8de"; stroke: "#eb6834"; stroke-width: 2; font-size: 22 } } sum: TOTAL RESIDENT\n= 1.5–2× the weight size { style: { fill: "#e0f4ec"; stroke: "#1baf7a"; stroke-width: 2; font-size: 22 } } ok: Advertised context\nis usable { style: { stroke: "#6b6459"; fill: transparent; stroke-width: 1; font-size: 22 } } no: Cut context, or\nquantise harder { style: { stroke: "#eb6834"; fill: "#fffdf9"; stroke-width: 2; font-size: 22 } } w -> kv: plus { style: { stroke: "#6b6459"; stroke-width: 2; font-size: 20 } } kv -> os: plus { style: { stroke: "#6b6459"; stroke-width: 2; font-size: 20 } } os -> sum: equals { style: { stroke: "#6b6459"; stroke-width: 2; font-size: 20 } } sum -> ok: fits RAM { style: { stroke: "#1baf7a"; stroke-width: 2; font-size: 20 } } sum -> no: over budget { style: { stroke: "#eb6834"; stroke-width: 2; font-size: 20 } } ``` ## Runtime choice is worth as much as hardware choice On Apple Silicon the spread between inference runtimes is larger than the spread between adjacent hardware tiers. Measured on a Mac mini M4 Pro 64GB running Qwen3-Coder-30B-A3B: **MLX ~130 tok/s, Ollama ~43 tok/s** ([yage.ai](https://yage.ai/share/mlx-apple-silicon-en-20260331.html)). On an M4 Max 128GB with Qwen3.5-35B-A3B the same writeup measures MLX 130, raw llama.cpp on the Metal backend 89.4, Ollama 43.5. ```vega-lite One machine, one model, 3x apart — the entire difference is which runtime you installed. Source: yage.ai, 2026-03-31. {"title":{"text":"Same machine, same model: runtime is worth a hardware tier","subtitle":"One M4 Max 128GB, Qwen3.5-35B-A3B, decode throughput. Ollama 0.19 later swapped Metal for MLX \u2014 re-benchmark."}, "height":{"step":38}, "data":{"values":[ {"runtime":"MLX","v":130}, {"runtime":"llama.cpp (Metal)","v":89.4}, {"runtime":"Ollama","v":43.5}]}, "encoding":{ "y":{"field":"runtime","type":"nominal","sort":"-x","title":null,"axis":{"labelFontSize":13}}, "x":{"field":"v","type":"quantitative","title":"tokens / sec (decode)","axis":{"grid":true}}}, "layer":[ {"mark":{"type":"bar","height":24},"encoding":{"color":{"field":"runtime","type":"nominal","legend":null}}}, {"mark":{"type":"text","align":"left","dx":8,"fontWeight":600,"fontSize":13}, "encoding":{"text":{"field":"v","type":"quantitative","format":",.4~f"}}}]} ``` Picking the convenient runtime cost you 3× throughput. That is more than the entire generational jump from M4 Max to M5 Ultra on many models. Ollama shipped 0.19 on 2026-03-30 replacing its llama.cpp Metal backend with MLX ([Ollama](https://ollama.com/blog/mlx)), which should close most of that gap — verify on your own model before assuming it has. Two caveats worth carrying: - MLX's advantage is in decode, not prefill. On an M1 Max with a ~650-token prompt the same source measured MLX at 13 tok/s combined against GGUF's 20, with MLX spending 94% of its time in prefill. The gap narrows above ~27B where bandwidth becomes the binding constraint for both. - Prefix cache reuse [is broken for hybrid-architecture models in mlx-lm](https://github.com/ml-explore/mlx-lm/issues/980) (sliding-window, SSM/Mamba). Since hybrid attention is exactly what makes long context affordable, check this before building a workflow on it. **My read:** if you buy Apple Silicon for inference and run Ollama out of habit, you have wasted roughly a tier of hardware. Install MLX first, benchmark second, buy third. ## MoE changed the arithmetic This is the fact that invalidates most pre-2025 sizing advice. In a sparse mixture-of-experts model, **throughput tracks active parameters while capacity tracks total parameters.** gpt-oss-120b is 117B total with 5.1B active ([model card](https://huggingface.co/openai/gpt-oss-120b)). It occupies memory like a 120B model and decodes like a 5B one. The roofline follows directly. My arithmetic, not a measurement: ``` bytes read per token ≈ active_params × bytes_per_param gpt-oss-120b at MXFP4 (4.25 bits ≈ 0.53 B/param): 5.1e9 × 0.53 ≈ 2.7 GB per token ceiling on 1.2 TB/s = 1200 / 2.7 ≈ 440 tok/s ceiling on 273 GB/s = 273 / 2.7 ≈ 100 tok/s ``` Real systems land far below the roofline — the measured DGX Spark number is 38.6 tok/s against that ~100 ceiling, so figure 35–50% realised at best. Use the ratio, not the absolute: it tells you a 400B sparse model with 17B active will generate roughly 3× slower than a 120B/5B one, not 3× *faster than a 400B dense model would be*, which is the intuition people carry over and get wrong. The practical consequence is that capacity, not bandwidth, has become the binding constraint on high-end local inference. GLM-5.2 is 744B total / 40B active; Unsloth's 2-bit dynamic quant is 239GB and they explicitly note it "can directly fit on a 256GB unified memory Mac" ([Unsloth](https://unsloth.ai/docs/models/glm-5.2)). Its 4-bit is 372–475GB. Nothing with 32GB of VRAM participates in that conversation at any price. ## The ladder, opinionated **24–32GB.** A used 3090 or a 4090 if you find one sane. This tier runs 8–30B models at high speed and nothing else, and the KV arithmetic above means "30B" really means "30B at modest context." Buy here only if you already game on the card. An M4/M5 Pro laptop covers the same models portably. **64–128GB unified.** The sweet spot, and where I would point most people. An M5 Max Mac Studio starts at $2,499 ([Apple](https://www.apple.com/newsroom/2026/08/apple-introduces-new-mac-studio-with-m5-max-and-m5-ultra/)) and 614 GB/s at 128GB runs a 120B-class sparse model with real context headroom. The 128GB LPDDR5x boxes (DGX Spark at $4,699, Strix Halo systems) hold the same models at 256–273 GB/s, roughly 2.4× slower on decode. Strix Halo's ~340 tok/s prompt processing on gpt-oss-120b makes long-context agent work unpleasant. Buy the Spark for CUDA compatibility, not for speed. **256GB.** $5,499 for the base 96GB M5 Ultra plus $4,000 for the 256GB step ([Apple](https://www.apple.com/newsroom/2026/08/apple-introduces-new-mac-studio-with-m5-max-and-m5-ultra/), [9to5Mac](https://9to5mac.com/2026/08/25/apple-unveils-next-generation-mac-studio-with-m5-max-and-m5-ultra/)). This tier exists to run 400–750B sparse models at aggressive quants. Justified only if you have a specific model in that class you need running privately and continuously. **512GB.** A bet that open weights keep growing faster than memory gets cheap. It buys quality-quant headroom on models that fit 256GB only at 2-bit. Everything else is speculation about models that don't exist yet. **Rent instead** for anything bursty, and for anything you'd need multiple H100s to serve. Median on-demand H100 pricing is $3.39/GPU-hour across 38 providers ([getdeploying](https://getdeploying.com/gpus/nvidia-h100)); a full day of experimentation costs less than a GPU fan. ## Where buying is plainly the wrong call If your workload is intermittent inference on models that are already available as an API, the economics are not close. gpt-oss-120b runs at roughly $0.03/M input and $0.17/M output on OpenRouter ([OpenRouter](https://openrouter.ai/openai/gpt-oss-120b)). A $9,499 Mac Studio at 256GB buys, at that rate, on the order of 55 billion output tokens of the same model before the hardware has paid for itself — my arithmetic, ignoring electricity, which makes the comparison worse. At 50 tok/s of local generation you would need decades of continuous output to consume that. Buy hardware when the driver is one of: data that legally cannot leave your machine, a workload running continuously enough that the box is never idle, or wanting a model no provider hosts. Those are real reasons. "It'll pay for itself" is not one, and neither is a GPU market where the RTX 5090's $1,999 MSRP has become a [median street price near $4,700](https://tech-insider.org/gpu-prices-2026/) on GDDR7 shortage. One more thing to price in. Apple claims the M5 Ultra delivers "up to 4.3x the peak AI compute performance when compared to M3 Ultra" in the [Mac Studio announcement](https://www.apple.com/newsroom/2026/08/apple-introduces-new-mac-studio-with-m5-max-and-m5-ultra/), and "up to 4.5x the peak GPU compute for AI compared to M3 Ultra" in the [chip announcement](https://www.apple.com/newsroom/2026/08/apple-introduces-m6-and-m5-ultra-for-a-big-leap-in-performance-and-ai-compute/) published the same month. Two numbers, same comparison, same vendor. That gap is a useful reminder of what these multipliers are: peak compute is not what decodes your tokens — bandwidth went up 50%, from 819 GB/s to 1.2 TB/s, and that is the number that moves generation speed. Treat AI-compute multipliers as marketing until someone measures tokens per second, which for prefill-heavy agent workloads may well vindicate them. See [/context-engineering-for-coding-agents](/context-engineering-for-coding-agents) for why prefill dominates that particular workload. --- # Prefill and decode want different machines URL: https://notes.junxiong.dev/prefill-vs-decode Updated: 2026-08-28 ## Two phases, opposite bottlenecks Every request to an LLM runs in two phases, and they stress completely different parts of a machine. **Prefill** takes your whole prompt and pushes it through the model in one shot. A 2,000-token prompt means the first matmul has a 2,000-row activation matrix against each weight matrix. Each weight is loaded from memory once and reused 2,000 times. That is a dense GEMM, it saturates tensor cores, and it sets **time-to-first-token**. **Decode** produces one token, then the next, then the next. Each step multiplies a *single* row of activations against every weight in the model. Each weight is loaded from memory once and used once. The GPU spends its time waiting on memory and its FLOPs sit idle. Decode sets **inter-token latency**, and therefore the tokens/sec number you actually watch scroll. Same weights, same kernels, and the hardware you'd buy to make each one fast is nearly the opposite. ```d2 Prefill reads the whole prompt in one compute-bound pass. Decode then loops once per token, and every pass re-reads the entire model out of memory — which is why the two halves want different hardware. direction: down prompt: Prompt\nN tokens { style: { stroke: "#6b6459"; fill: transparent; stroke-width: 1; font-size: 22 } } prefill: PREFILL\ncompute-bound\n\nOne N-row GEMM over every token\nat once. Saturates FLOPs. { style: { fill: "#e4edf9"; stroke: "#2a78d6"; stroke-width: 2; font-size: 22 } } step: DECODE\nmemory-bandwidth-bound\n\nOne 1-row matmul. One token out. { style: { fill: "#fbe8de"; stroke: "#eb6834"; stroke-width: 2; font-size: 22 } } mem: Read EVERY weight\n+ the whole KV cache { style: { fill: "#fffdf9"; stroke: "#eb6834"; stroke-width: 2; font-size: 22 } } prompt -> prefill: all N tokens { style: { stroke: "#6b6459"; font-size: 20 } } prefill -> step: first token · TTFT { style: { stroke: "#6b6459"; stroke-width: 2; font-size: 20 } } step -> mem: every pass { style: { stroke: "#eb6834"; stroke-width: 2; font-size: 20 } } mem -> step: KV cache +1 { style: { stroke: "#eb6834"; stroke-width: 2; stroke-dash: 4; font-size: 20 } } ``` ## The decode arithmetic you can do on a napkin At batch size 1, decode has to read the whole model out of memory to emit one token. So: ``` tokens/sec ≈ memory bandwidth ÷ bytes read per token ≈ memory bandwidth ÷ model file size ``` That's it. That's the ceiling. Nothing about FLOPs appears in it. Take Llama-2-7B at `Q4_0`, which is [3.83 GB on disk](https://huggingface.co/TheBloke/Llama-2-7B-GGUF). Apple states 400GB/s for [M1 Max](https://www.apple.com/newsroom/2021/10/introducing-m1-pro-and-m1-max-the-most-powerful-chips-apple-has-ever-built/) and M2 Max, and 800GB/s for [M1 Ultra](https://www.apple.com/newsroom/2022/03/apple-unveils-m1-ultra-the-worlds-most-powerful-chip-for-a-personal-computer/) and [M2 Ultra](https://www.apple.com/newsroom/2023/06/apple-introduces-m2-ultra/). The measured `tg128` figures come from the long-running [llama.cpp Apple Silicon benchmark thread](https://github.com/ggml-org/llama.cpp/discussions/4167). | Chip | Bandwidth (Apple) | Ceiling, my arithmetic | Measured (llama.cpp) | Fraction of ceiling | |---|---|---|---|---| | M1 Max | 400 GB/s | 104 tok/s | 61.19 | 59% | | M2 Max | 400 GB/s | 104 tok/s | 65.95 | 63% | | M1 Ultra | 800 GB/s | 209 tok/s | 83.73 | 40% | | M2 Ultra | 800 GB/s | 209 tok/s | 94.27 | 45% | The ceiling column is mine: `400e9 / 3.83e9 = 104.4`. Check it yourself. Two things fall out. Real decode lands at roughly 40–60% of the bandwidth ceiling, so the napkin number is an upper bound you should discount, not a prediction. And the Ultra parts, which are two Max dies fused together at twice the paper bandwidth, return only about **40% more decode throughput** than the Max they're built from (1.37x on M1, 1.43x on M2). If you were buying an Ultra for single-stream generation on the strength of the 800GB/s number, that is the number you should be looking at instead. ```vega-lite Measured decode lands at 40–60% of the arithmetic ceiling, and doubling paper bandwidth (Max to Ultra) buys about 40% more real throughput. {"title":{"text":"Napkin ceiling vs measured decode, Llama-2-7B Q4_0","subtitle":"Ceilings are my arithmetic (bandwidth ÷ 3.83 GB file size); measured tg128 from the llama.cpp Apple Silicon benchmark thread."}, "height":{"step":38}, "data":{"values":[ {"label":"Ceiling, 400 GB/s","v":104,"kind":"Ceiling (arithmetic)"}, {"label":"M1 Max measured","v":61.19,"kind":"Measured (llama.cpp)"}, {"label":"M2 Max measured","v":65.95,"kind":"Measured (llama.cpp)"}, {"label":"Ceiling, 800 GB/s","v":209,"kind":"Ceiling (arithmetic)"}, {"label":"M1 Ultra measured","v":83.73,"kind":"Measured (llama.cpp)"}, {"label":"M2 Ultra measured","v":94.27,"kind":"Measured (llama.cpp)"}]}, "encoding":{ "y":{"field":"label","type":"nominal","title":null,"sort":["Ceiling, 400 GB/s","M1 Max measured","M2 Max measured","Ceiling, 800 GB/s","M1 Ultra measured","M2 Ultra measured"],"axis":{"labelFontSize":13}}, "x":{"field":"v","type":"quantitative","title":"tokens / sec","axis":{"grid":true}}}, "layer":[ {"mark":{"type":"bar","height":24},"encoding":{"color":{"field":"kind","type":"nominal","title":null}}}, {"mark":{"type":"text","align":"left","dx":8,"fontWeight":600,"fontSize":13}, "encoding":{"text":{"field":"v","type":"quantitative","format":",.4~f"}}}]} ``` The same arithmetic explains why [quantization](/quantization-what-it-costs) buys speed and not just VRAM headroom: cutting a model from FP16 to 4-bit cuts bytes-read-per-token by roughly 4×, which moves the decode ceiling by roughly 4×. Decode is bound by the exact quantity quantization shrinks. ## Why big-FLOPs, small-bandwidth boxes disappoint NVIDIA's DGX Spark is the cleanest illustration currently shipping. NVIDIA [claims](https://www.nvidia.com/en-us/products/workstations/dgx-spark/) 1 petaFLOP of sparse FP4 on the GB10 superchip, with 128 GB of unified LPDDR5X at **273 GB/s**. That is datacenter-class compute bolted to laptop-class memory. Measured on `gpt-oss-120b` MXFP4 in the [llama.cpp DGX Spark thread](https://github.com/ggml-org/llama.cpp/discussions/16578): **1,956 tok/s prompt processing, 60.57 tok/s generation**. A 32× gap between the two phases on one box, from one set of weights. The petaFLOP shows up in the first number and is entirely absent from the second. ```vega-lite Prefill and decode differ by ~32x on the same box, from one set of weights. Source: llama.cpp DGX Spark thread. {"title":{"text":"Same box, same weights: prefill vs decode","subtitle":"DGX Spark, gpt-oss-120b MXFP4, llama.cpp DGX Spark thread. pp2048 against tg32."}, "height":{"step":38}, "data":{"values":[{"phase":"Prefill (pp2048)","v":1956},{"phase":"Decode (tg32)","v":60.57}]}, "encoding":{ "y":{"field":"phase","type":"nominal","sort":"-x","title":null,"axis":{"labelFontSize":13}}, "x":{"field":"v","type":"quantitative","title":"tokens / sec","axis":{"grid":true}}}, "layer":[ {"mark":{"type":"bar","height":24},"encoding":{"color":{"field":"phase","type":"nominal","legend":null}}}, {"mark":{"type":"text","align":"left","dx":8,"fontWeight":600,"fontSize":13}, "encoding":{"text":{"field":"v","type":"quantitative","format":",.4~f"}}}]} ``` Apple Silicon has the mirror-image problem. High bandwidth, modest matmul throughput, so a Mac Studio punches above its FLOPs on single-stream decode and falls behind badly on long prompts. Tom's Hardware measured exactly this shape, titling their Mac Studio piece ["M4 Max beats GB10 and Strix Halo in decode throughput, but memory bandwidth isn't everything"](https://www.tomshardware.com/desktops/exploring-apple-silicons-local-ai-performance-with-the-mac-studio-and-m4-max-m4-max-beats-gb10-and-strix-halo-in-decode-throughput-but-memory-bandwidth-isnt-everything). My read: if you paste 40k-token files into a local coding agent, prefill is the wall you'll hit, and it's the wall Apple hardware is worst at. More on the machine-by-machine tradeoffs in [local inference hardware](/local-inference-hardware). ## Batching is why the API is cheap and your Mac is not Decode at batch 1 reads `2P` bytes (FP16) to produce one token. Decode at batch 64 reads the same `2P` bytes to produce **64** tokens, because all 64 sequences multiply against the same weights in the same pass. Weight traffic per token falls by 64×. This is the entire economic basis of hosted inference. You can find the crossover point with a roofline. NVIDIA's [H100 SXM](https://www.nvidia.com/en-us/data-center/h100/) has 3.35 TB/s of HBM3 and 1,979 FP16 tensor TFLOPS with sparsity, so 989 dense. The ridge point: ``` 989e12 FLOP/s ÷ 3.35e12 B/s ≈ 295 FLOP per byte ``` Decode's arithmetic intensity in the weight matmuls is about `B` FLOP/byte at FP16 (2·P·B FLOPs against 2·P bytes read). So you need a batch of roughly **300 concurrent sequences** before an H100 stops being memory-bound during decode. Prefill with a 2,000-token prompt sits at intensity ~2,000 and is compute-bound at batch 1, seven times over. Which is why batching does much less for prefill. A single 2,000-token prompt already fills the machine; stacking a second one just queues behind the first. Batching converts decode from memory-bound to compute-bound. For prefill it mostly just adds work. Your local single stream never gets any of this. You pay full weight-read cost for every single token. A provider amortises that read across hundreds of users, which is how per-token prices land where they do while your Mac Studio does one conversation at a time. ## The KV cache is what grows Batching decode is limited by memory *capacity*, not just bandwidth, because every active sequence carries a KV cache that grows one entry per layer per token. Llama 3.3 70B's [config](https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/raw/main/config.json) is 80 layers, 64 attention heads, 8 KV heads, hidden size 8192 (so head dim 128). At FP16: ``` per token = 2 (K and V) × 8 kv_heads × 128 head_dim × 80 layers × 2 bytes = 327,680 bytes ≈ 320 KiB / token ``` At its full 131,072-token context that's **43 GB for one sequence**. Batch 32 at only 8k context each is 86 GB, which already exceeds a single 80 GB H100. Long context turns a bandwidth problem into a capacity problem, and the capacity problem caps your batch size, which drags you back onto the bad side of the decode roofline. Two fixes are now standard: - **Grouped-query attention.** Those 8 KV heads serve 64 query heads. Full multi-head attention would need 64 KV heads and 2.56 MiB per token — the same 128k context would cost 343 GB. [GQA](https://arxiv.org/abs/2305.13245) is an 8× cut in KV traffic and footprint, and it's why 128k contexts are servable at all. - **Paged attention.** vLLM allocates KV in fixed-size pages instead of one contiguous reservation per sequence, so you don't pre-reserve for the worst-case length. The [paper](https://arxiv.org/abs/2309.06180) reports 2–4× throughput at equal latency versus FasterTransformer and Orca, with "near-zero waste" in KV memory. The throughput comes from fitting more sequences, which is the same batching lever again. Prompt caching is the third lever, and it attacks prefill instead — see [prompt caching across harnesses](/prompt-caching-across-harnesses). ## What production stacks actually do about it Running prefill and decode on the same GPU means they fight. A long prefill occupying the GPU stalls every in-flight decode, and users see the stream freeze. **Chunked prefill** splits a prompt into fixed-size chunks and slots decode steps in alongside them. [Sarathi-Serve](https://arxiv.org/abs/2403.02310) introduced this as "stall-free scheduling". It is now on by default in vLLM V1, whose [scheduler](https://docs.vllm.ai/en/latest/configuration/optimization.html) batches all pending decodes first, then fills the remaining token budget with prefill chunks. The docs are explicit about why: it gets "better GPU utilization by locating compute-bound (prefill) and memory-bound (decode) requests to the same batch." `max_num_batched_tokens` is the dial — around 2048 favours inter-token latency, above 8192 favours TTFT and raw throughput. **Disaggregation** goes further and puts the two phases on different machines. [DistServe](https://arxiv.org/abs/2401.09670) made the case that prefill/decode interference costs enough goodput to justify separate GPU pools. Every major stack now ships it: [vLLM](https://docs.vllm.ai/en/latest/features/disagg_prefill/) with KV connectors under `vllm/distributed/kv_transfer`, [SGLang](https://docs.sglang.ai/advanced_features/pd_disaggregation.html) with separate prefill and decode pools, and [TensorRT-LLM](https://nvidia.github.io/TensorRT-LLM/features/disagg-serving.html) under NVIDIA Dynamo. The prefill pool runs high tensor parallelism to chew through matmuls; the decode pool runs lower TP with more replicas for concurrency. KV cache moves between them over RDMA. The important part for a buyer: the people who serve models at scale concluded the two phases want **physically different hardware allocations**. That is the strongest available evidence that the split is real and not a modelling curiosity. ## What to buy | You care about | Optimise for | Watch out for | |---|---|---| | Time-to-first-token on long prompts | FLOPs, tensor cores, high TP | Unified-memory machines; prefill is their weak phase | | Tokens/sec, single stream | Memory bandwidth ÷ quantized model size | Paper FLOPs; Ultra-tier bandwidth that doesn't convert | | Many concurrent users | VRAM capacity for KV, then bandwidth | Long contexts eating your batch size | | Cost per token | Someone else's batch | Running one local stream and calling it cheap | Two rules I'd give anyone shopping. First, compute `bandwidth ÷ quantized file size`, discount to 50%, and treat that as your realistic ceiling before you read a single review. Second, if your workload is long prompts and short answers (coding agents, document QA, RAG), you are buying a prefill machine, and bandwidth is the wrong headline number to be optimising. --- # Prompt caching, and why the same prompt costs 10x more on some days URL: https://notes.junxiong.dev/prompt-caching-across-harnesses Updated: 2026-08-28 Add a `Generated at {timestamp}` line to the top of a system prompt and your input spend can multiply overnight. The prompt is otherwise identical. The code is identical. Nothing errors, nothing logs a warning, and the only visible symptom is the invoice. That failure is the whole topic. Prompt caching is one of the few inference-cost levers that is free when it works and invisible when it doesn't. ## One rule, and everything else follows A transformer's prefill computes key/value tensors for every input token. Prompt caching stores those tensors and reuses them when a new request starts with byte-identical text. OpenAI's cookbook puts the mechanism plainly: the model "reuses the cached tensors and only computes attention for the new tokens" ([Prompt Caching 201](https://developers.openai.com/cookbook/examples/prompt_caching_201)). The consequence is that a cache hit needs an **exact prefix match**, not a similar one, and not a fuzzy one. Change byte 400 of a 200,000-token prompt and tokens 400 through 200,000 are all recomputed at full price. There is no per-file, per-section, or per-document caching underneath. Anthropic's Claude Code docs say it in one sentence: "The match is exact, so a change anywhere in the prefix recomputes everything after it. There is no per-file or per-segment caching" ([code.claude.com](https://code.claude.com/docs/en/prompt-caching)). ```d2 The API matches from token 0. An exact prefix hit is read at 0.1x; an edit anywhere forces everything below it to be recomputed at full price. direction: down hit: CACHE HIT\nprefix byte-identical { style: { fill: "#e4edf9"; stroke: "#2a78d6"; stroke-width: 2; font-size: 22 } } h1: 1 · System + tools { style: { fill: "#e4edf9"; stroke: "#2a78d6"; stroke-width: 2; font-size: 22 } } h2: 2 · Project context { style: { fill: "#e4edf9"; stroke: "#2a78d6"; stroke-width: 2; font-size: 22 } } h3: 3 · Conversation { style: { fill: "#e4edf9"; stroke: "#2a78d6"; stroke-width: 2; font-size: 22 } } h4: 4 · New turn { style: { stroke: "#6b6459"; fill: transparent; stroke-width: 1; font-size: 22 } } miss: CACHE MISS\none byte edited in layer 2 { style: { fill: "#fbe8de"; stroke: "#eb6834"; stroke-width: 2; font-size: 22 } } m1: 1 · System + tools { style: { fill: "#e4edf9"; stroke: "#2a78d6"; stroke-width: 2; font-size: 22 } } m2: 2 · Project context { style: { fill: "#fbe8de"; stroke: "#eb6834"; stroke-width: 2; font-size: 22 } } m3: 3 · Conversation { style: { fill: "#fbe8de"; stroke: "#eb6834"; stroke-width: 2; font-size: 22 } } m4: 4 · New turn { style: { fill: "#fbe8de"; stroke: "#eb6834"; stroke-width: 2; font-size: 22 } } hit -> h1: match from 0 { style: { stroke: "#2a78d6"; stroke-width: 2; font-size: 20 } } h1 -> h2: 0.1x { style: { stroke: "#2a78d6"; stroke-width: 2; font-size: 20 } } h2 -> h3: 0.1x { style: { stroke: "#2a78d6"; stroke-width: 2; font-size: 20 } } h3 -> h4: write { style: { stroke: "#6b6459"; font-size: 20 } } miss -> m1: match from 0 { style: { stroke: "#eb6834"; stroke-width: 2; font-size: 20 } } m1 -> m2: 0.1x { style: { stroke: "#2a78d6"; stroke-width: 2; font-size: 20 } } m2 -> m3: full price { style: { stroke: "#eb6834"; stroke-width: 2; font-size: 20 } } m3 -> m4: full price { style: { stroke: "#eb6834"; stroke-width: 2; font-size: 20 } } ``` That is why the ordering advice is not a style preference. Content must be laid out in descending order of stability: 1. System prompt and tool definitions (change on deploy) 2. Retrieved documents, repo context, few-shot examples (change per session) 3. Conversation history (changes per turn) 4. The user's actual question (changes per request) Anything volatile placed above something stable destroys the cache for the stable thing. A timestamp at position 0 makes the entire prompt uncacheable, forever, silently. ## Three vendors, three interfaces to the same idea All three major APIs do prefix caching. They disagree about who places the breakpoint and who pays for the write. Checked 2026-08-28. | | Anthropic | OpenAI | Google (Gemini) | |---|---|---|---| | How it turns on | Explicit `cache_control` breakpoints, or one top-level auto breakpoint | Implicit by default; explicit `prompt_cache_breakpoint` on GPT-5.6+ | Implicit by default on 2.5+; explicit cache objects available | | Breakpoints | Max 4 per request | Auto at end of latest eligible message (5.6+) | n/a for implicit | | Minimum prefix | 512 tok (Opus 5, Fable 5), 1,024 (Opus 4.8, Sonnet 5), up to 4,096 (Opus 4.6, Haiku 4.5) | 1,024 visible input tokens (GPT-5.6+), 2,048 earlier | 2,048 (Gemini 2.5), 4,096 (3.x) | | Read price | 0.1x base input | 0.1x base input (GPT-5.6+) | 0.1x base input | | Write price | 1.25x (5m TTL), 2x (1h TTL) | 1.25x (GPT-5.6+); no write charge on earlier models | No write charge; hourly **storage** fee | | TTL | 5 min default, 1 hour opt-in | 30 min default on 5.6+; `in_memory` (~5-10 min) or `24h` on earlier | Implicit is opportunistic; explicit caches have a TTL | Sources: [Anthropic pricing](https://platform.claude.com/docs/en/about-claude/pricing), [Anthropic prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching), [OpenAI prompt caching](https://developers.openai.com/api/docs/guides/prompt-caching), [Gemini caching](https://ai.google.dev/gemini-api/docs/caching), [Gemini pricing](https://ai.google.dev/gemini-api/docs/pricing). ```vega-lite Reads are a tenth of base input, writes a small premium over it. The whole game is moving token volume from the top two bars to the bottom one. {"title":{"text":"What an input token costs, as a multiple of the base rate","subtitle":"Published multipliers; the 1h and 5m write tiers are Anthropic's. Sources: Anthropic pricing; OpenAI prompt caching, GPT-5.6+. Google charges no write premium and bills hourly storage instead. Checked 2026-08-28."}, "height":{"step":38}, "data":{"values":[ {"kind":"Cache write, 1h TTL","v":2,"d":"2.0x","g":"write"}, {"kind":"Cache write, 5m TTL","v":1.25,"d":"1.25x","g":"write"}, {"kind":"Uncached input","v":1,"d":"1.0x","g":"base"}, {"kind":"Cache read","v":0.1,"d":"0.1x","g":"read"}]}, "encoding":{ "y":{"field":"kind","type":"nominal","sort":"-x","title":null,"axis":{"labelFontSize":13,"labelLimit":260}}, "x":{"field":"v","type":"quantitative","title":"multiple of base input rate","axis":{"grid":true}}}, "layer":[ {"mark":{"type":"bar","height":24},"encoding":{"color":{"field":"g","type":"nominal","legend":null,"scale":{"domain":["read","base","write"],"range":["#1baf7a","#2a78d6","#eb6834"]}}}}, {"mark":{"type":"text","align":"left","dx":8,"fontWeight":600,"fontSize":13}, "encoding":{"text":{"field":"d","type":"nominal"}}}]} ``` Two differences actually change how you write code. **Google bills storage, not writes.** Gemini's context caching charges the cached rate at 10% of input plus a per-hour storage fee, e.g. $1.00 per million tokens per hour on 3.5 Flash. That inverts the calculus: on Anthropic and OpenAI you worry about paying the write premium too often, on Google you worry about paying rent on a cache nobody reads. **OpenAI has a routing problem the others don't expose.** Cache hits require your request to land on a machine that holds the prefix, so OpenAI gives you `prompt_cache_key` to pin related requests together. Their cookbook reports one customer going from 60% to 87% hit rate by setting it, and warns that a single prefix+key combination saturates at roughly 15 requests per minute before traffic spills to other machines and misses. If you run high-QPS traffic on OpenAI and haven't thought about that key, that's probably where your money is. ## The arithmetic Take a plausible agent loop on Claude Opus 5 at the published rates ($5/MTok input, $0.50/MTok cache read, $6.25/MTok 5-minute cache write, $25/MTok output — [pricing page](https://platform.claude.com/docs/en/about-claude/pricing), checked 2026-08-28): - 30,000-token stable prefix: system prompt, tool definitions, project context - 20 turns, each appending ~2,000 tokens of tool results and user input - 500 output tokens per turn Every turn resends everything before it, so total input across the session is 980,000 tokens either way. What differs is the rate each token is billed at. | | Tokens | Rate | Cost | |---|---:|---:|---:| | **No caching** | 980,000 uncached | $5.00 | **$4.90** | | **Cached** — reads | 912,000 | $0.50 | $0.456 | | **Cached** — writes | 68,000 | $6.25 | $0.425 | | **Cached total** | | | **$0.88** | That's 5.6x on input for a session that ran identical text through the model. Output cost ($0.25) is unchanged by caching, so the session total moves from $5.15 to $1.13. The 5.6x isn't the ceiling. Because reads bill at exactly one tenth of base input, the asymptote is 10x, approached as the stable prefix grows relative to per-turn additions. A 100-turn session against a 100K-token repo context gets much closer to it than this example does. My read: for coding agents, cache read tokens routinely make up 90%+ of input volume and under 50% of input cost, and that ratio is the single best health metric you have. **Now break it once.** Same session, but you switch models on turn 11. The turn-11 request has 50,000 tokens of history and none of it hits, so it writes 50,000 tokens at $6.25/MTok = **$0.31 for one turn**, against $0.037 for a healthy turn 11. That single keystroke costs 8.5x a normal turn, and the session's input bill rises from $0.88 to $1.16, up 31%. ```vega-lite Caching cuts input cost 5.6x on this workload. One model switch mid-session claws back a third of the saving. {"title":{"text":"Input cost for the same 980,000-token modelled session","subtitle":"The model switch happens at turn 11. Arithmetic over published Claude Opus 5 rates ($5 input, $0.50 cache read, $6.25 5-minute cache write per MTok), not a measurement. Token counts are a modelled workload. Output cost excluded."}, "height":{"step":38}, "data":{"values":[ {"case":"No caching","v":4.90,"d":"$4.90"}, {"case":"Cached, one model switch","v":1.16,"d":"$1.16"}, {"case":"Cached, prefix intact","v":0.88,"d":"$0.88"}]}, "encoding":{ "y":{"field":"case","type":"nominal","sort":"-x","title":null,"axis":{"labelFontSize":13,"labelLimit":260}}, "x":{"field":"v","type":"quantitative","title":"input cost, USD","axis":{"grid":true}}}, "layer":[ {"mark":{"type":"bar","height":24},"encoding":{"color":{"field":"case","type":"nominal","legend":null}}}, {"mark":{"type":"text","align":"left","dx":8,"fontWeight":600,"fontSize":13}, "encoding":{"text":{"field":"d","type":"nominal"}}}]} ``` (These are arithmetic on published prices, not measurements from my own logs. The token counts are a modelled workload.) ## What each harness does with your history Every harness resends the full conversation on every turn. They differ in how carefully they preserve the prefix while doing it. **Claude Code** is the most explicit about this, and its docs are worth reading even if you use something else. It orders each request system prompt → project context → conversation, and publishes the full list of what invalidates each layer. The non-obvious entries: - `/model` and `/effort` both change the cache key, not the prompt text. Switching either re-reads the entire history with zero hits. - Editing CLAUDE.md mid-session does *not* invalidate the cache, because it also doesn't apply. It's loaded once at session start and takes effect on the next `/clear`, `/compact`, or restart. - Skills, slash commands, `/recap` and plan mode all append as messages, so they're cache-safe by construction. - `/rewind` truncates back to a prefix that is already cached; `/compact` builds a new one. Given the choice, rewind. - Cache scope is effectively per-machine and per-directory, because the system prompt embeds the working directory, platform and shell. Two git worktrees of the same repo do not share a cache. TTL is configurable there via `promptCacheTtl` / `CLAUDE_CODE_PROMPT_CACHE_TTL`, and on an API key the main conversation defaults to 5 minutes rather than the 1 hour a subscription gets. **Codex CLI** takes the append-only route and pins `prompt_cache_key` to the thread, so turns in one session route to the same cached prefix. A [third-party writeup](https://codex.danielvaughan.com/2026/04/21/codex-cli-prompt-caching-maximise-cache-hits-cost-reduction/) measured 80-90% hit rates and 40-55% input cost reduction across sessions, and notes that `/compact` resets the cache because the prefix changes. Treat those as one person's numbers, not vendor-published ones. The actionable part is verifiable from OpenAI's own docs: dynamic content in AGENTS.md (generated line counts, timestamps) sits in the prefix and breaks it, and MCP servers connecting mid-session change the tool list. **Cursor** passes provider caching through and bills you for both halves. Its [pricing docs](https://cursor.com/docs/account/pricing) show a per-model Input / Cache write / Cache read table with cache writes at 1.25x and reads at 10% for Anthropic models. You have far less control here: Cursor owns the system prompt, decides what rules and MCP descriptions go into the prefix, and decides when to trim context. The lever you do have is not attaching and detaching things mid-conversation. **Raw API** is the only place you control all of it, and the only place you can get it badly wrong. If you're building your own loop, the two things that matter more than breakpoint placement are keeping the system prompt frozen and serializing tool definitions deterministically (`sort_keys=True`, no set iteration). ## Failure modes, ranked by how often I've hit them **Prefix drift from injected state.** A date, a user ID, a feature flag, a "you have N credits remaining" line. Anything interpolated into the system prompt is at position 0 and costs you the entire prompt. On Claude Opus 5 and Opus 4.8 there's a clean fix: append `{"role": "system", "content": "..."}` to `messages[]` instead of editing top-level `system`. It sits after the cached history and invalidates nothing. **Non-deterministic serialization.** `json.dumps()` without `sort_keys`, iterating a `set`, a library that reorders keys between versions. The prompt is logically identical and byte-different, which is the only thing the cache cares about. **Compaction and context editing.** Both rewrite history, which by definition destroys the conversation-layer prefix. The turn *after* compaction is cheap, since the new history is short. The expensive one is compaction itself when it runs cold: Anthropic notes that a mid-session `/compact` reads your prefix from cache and costs a fraction of what the context size suggests, but after a break longer than the TTL there's nothing to read and the summarization request reprocesses the full history uncached. **Expiry mid-session.** The TTL clock runs from the *start* of the request. A four-minute generation leaves about one minute of a five-minute entry. Agent loops with long tool calls or slow human review drift past the window without anyone noticing, and every re-warm is a full write. **Parallel fan-out.** N identical requests fired simultaneously all miss, because an entry only becomes readable once the first response starts streaming. Send one, wait for first token, then fire the rest. **Resuming an old session after an upgrade.** New harness version means a new system prompt, which means the entire resumed history sits behind a different prefix. The first turn back into a long conversation can be the most expensive request you send all week. ## The check that would have caught mine Log `cache_read_input_tokens` and `cache_creation_input_tokens` on every response and alert when the ratio inverts. In a healthy loop, reads grow turn over turn and writes stay roughly the size of the last exchange. If creation is near the full conversation size on every request, something upstream is rewriting your prefix. Make it an assertion, not a dashboard. Send the same request twice in CI and fail the build if the second one reports zero cache reads. Caching regressions don't announce themselves — the requests keep succeeding, and you find out on the invoice six weeks later. If you're structuring the prefix itself, [context engineering for coding agents](/context-engineering-for-coding-agents) is the other half of this problem: what goes in the window at all, before you worry about what order it's in. --- # What quantization actually costs you URL: https://notes.junxiong.dev/quantization-what-it-costs Updated: 2026-08-28 Quantization is the field with the widest gap between what people repeat and what anyone has measured. "Q4 is basically lossless" gets said constantly, usually by someone quoting a perplexity number from a 2023 forum post about a model that no longer exists. The real answer depends on which format, which layers, and which task, and there is now enough published measurement to give it properly. ## The formats | Format | Bits | Where it runs | How it decides precision | |---|---|---|---| | GGUF k-quants (`Q4_K_M`, `Q5_K_M`, `Q6_K`) | ~3.4–6.6 effective | llama.cpp, Ollama, LM Studio | Super-blocks with quantized scales; `_M`/`_S` variants keep attention and `feed_forward.w2` at higher width | | GGUF i-quants (`IQ2_XXS`, `IQ3_S`, `IQ4_XS`) | ~2.1–4.3 | llama.cpp | Codebook lookup plus an importance matrix from calibration data | | GPTQ | 3–4 | vLLM, TGI, ExLlama | Layer-wise error minimisation using approximate second-order (Hessian) information | | AWQ | 4 | vLLM, TGI | Scales up the ~1% of weight channels with the largest activations before quantizing | | NF4 (bitsandbytes) | 4 | Transformers, PEFT | Quantile bins under a normal prior, plus double-quantized scales | | MXFP4 | 4.25 | vLLM, llama.cpp, MLX | 4-bit float (E2M1) in blocks of 32 with a shared 8-bit exponent | | Unsloth dynamic (`UD-*`) | ~1.6–8 mixed | llama.cpp and anything reading GGUF | Per-layer width chosen by measured sensitivity, not a uniform setting | [GPTQ](https://arxiv.org/abs/2210.17323) (Frantar et al., 2022) quantizes one layer at a time and corrects the remaining weights for the error already introduced, which is why it can hit 3–4 bits on a 175B model in about four GPU hours. [AWQ](https://arxiv.org/abs/2306.00978) (MLSys 2024 best paper) took the opposite route: look at activation magnitudes, find the ~1% of channels that matter, and scale them up so rounding hurts them less. No backpropagation, no reconstruction, which is why it generalises off its calibration set better than GPTQ does. [NF4](https://arxiv.org/abs/2305.14314) came out of QLoRA. Its 16 levels are placed so each bin holds equal probability mass under a standard normal rather than being evenly spaced. Weights are roughly normal, so this is a better use of 16 codes. NF4 exists mainly to make fine-tuning fit in memory; as a serving format it is slower than a Marlin-kernel GPTQ or AWQ model on the same hardware. MXFP4 is the odd one, because for `gpt-oss` it is not a compression step applied afterwards. OpenAI post-trained with the MoE weights already in MXFP4, and those weights are [90%+ of the parameter count](https://arxiv.org/pdf/2508.10925), so the 120B fits on a single 80GB GPU and the 20B in 16GB. There is no fp16 reference version of those tensors to be worse than. ### Dynamic quants are the actual recent development Everything above except the `_M`/`_S` split applies one width to the whole model. Unsloth's dynamic quants pick a width per layer from measured sensitivity, so a nominal "2-bit" build is 2-bit in the layers that tolerate it and 4- or 6-bit in the ones that don't. This is why models that "cannot fit" now fit. The measurement that convinced me is Unsloth's [Qwen3.5-35B-A3B GGUF table](https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks), which reports mean KL-divergence against the reference for the same model quantized by several people: | Build | Disk | Perplexity | Mean KLD | |---|---|---|---| | Unsloth `Q8_K_XL` | 36.04 GB | 6.5352 | 0.0026 | | Unsloth `Q6_K_XL` | 28.22 GB | 6.5392 | 0.0041 | | Unsloth `Q5_K_XL` | 23.22 GB | 6.5489 | 0.0069 | | Unsloth `UD-Q4_K_XL` | 19.17 GB | 6.5918 | **0.0137** | | bartowski `Q4_K_M` | 19.77 GB | 6.6097 | **0.0182** | | Unsloth `Q3_K_XL` | 16.06 GB | 6.7245 | 0.0308 | | Unsloth `Q2_K_XL` | 12.04 GB | 7.0438 | 0.0970 | | Unsloth `IQ2_XXS` | 9.09 GB | 7.7160 | 0.1846 | | bartowski `IQ2_XXS` | 8.15 GB | 9.3427 | 0.3457 | The two bolded rows are the point. The dynamic build is 0.6 GB *smaller* than the uniform `Q4_K_M` and 25% closer to the reference distribution. At 2 bits the gap widens to nearly 2×. Layer-width selection buys more at low bit counts, which is exactly where you need it. ## PTQ vs QAT Post-training quantization is everything above: take finished weights, calibrate on a few hundred thousand tokens, write out a smaller file. Minutes to hours, no training loop. Quantization-aware training simulates the rounding during training so the weights learn to survive it. Google shipped this for Gemma 3 with [~5,000 QAT steps against the non-quantized checkpoint's own probabilities](https://developers.googleblog.com/en/gemma-3-quantized-aware-trained-state-of-the-art-ai-to-consumer-gpus/) (April 2025), and reported that it **cut the Q4_0 perplexity drop by 54%** versus plain PTQ, taking the 27B from 54 GB to 14.1 GB. QAT is worth it if you publish the weights and can amortise the training cost over every download. It is almost never worth it if you are quantizing someone else's model for your own use — a good dynamic PTQ build closes most of that gap for free. Unsloth [reports their dynamic quants reaching lower KL-divergence than Gemma 3's QAT builds](https://unsloth.ai/blog/dynamic-v2) at comparable size, and while that is a vendor claim about their own product, the third-party KLD table above makes it plausible. ## What it costs, measured Three numbers get used, and they do not measure the same thing. **Perplexity** is the one everyone quotes, and it understates the damage. Its failure mode is that per-token differences from the reference cancel out in the average. [Accuracy is Not All You Need](https://arxiv.org/abs/2407.09141) (Microsoft Research, 2024) documents this: quantized models produce large numbers of *flips*, answers changing from right to wrong and wrong to right in roughly equal proportion, so the aggregate score barely moves while the model's actual behaviour has shifted. Flips correlate with KL-divergence at Spearman 0.981 on MMLU, and with perplexity much less well. **KL-divergence against the fp16 reference** is the metric I'd use if I could only have one. It measures distributional distance per token, so cancellation is not possible, and it is what the credible quant publishers now report. **Task benchmarks** are the ground truth, and they show the damage is not evenly distributed. From [a unified llama.cpp evaluation on Llama-3.1-8B-Instruct](https://arxiv.org/abs/2601.14277) (Kurt, January 2026): | Quant | Size cut | PPL (F16 = 7.32) | GSM8K | MMLU | HellaSwag | |---|---|---|---|---|---| | F16 | — | 7.32 | 77.63 | 63.50 | 72.51 | | Q8_0 | 46.9% | 7.33 | 77.48 | 63.43 | 72.52 | | Q5_K_M | 64.4% | 7.40 | 78.54 | 62.80 | 72.33 | | Q4_K_M | 69.4% | 7.56 | 77.41 | 62.43 | 72.35 | | Q3_K_M | 75.0% | 7.96 | 73.16 | 62.01 | 73.41 | | Q3_K_S | 77.2% | 8.96 | **68.31** | 59.31 | 71.87 | Read the bottom row across. Perplexity rises 22%, HellaSwag falls 0.6 points, MMLU falls 4.2, GSM8K falls **9.3**. Multi-step arithmetic degrades roughly fifteen times harder than sentence completion at the same bit width, and no single perplexity number tells you that. The paper also finds schemes with *identical* perplexity diverging on instruction-following, which is the flips phenomenon showing up in a different dataset. ```vega-lite The same bit width, the same model, three very different amounts of damage. Multi-step arithmetic degrades roughly fifteen times harder than sentence completion. {"title":{"text":"Same model, same bit width, very different damage","subtitle":"Accuracy points lost going F16 \u2192 Q3_K_S, Llama-3.1-8B-Instruct. Lower is better. Kurt, January 2026 (arxiv.org/abs/2601.14277)."}, "height":{"step":38}, "data":{"values":[{"task":"GSM8K (arithmetic)","v":9.3},{"task":"MMLU (knowledge)","v":4.2},{"task":"HellaSwag (completion)","v":0.6}]}, "encoding":{ "y":{"field":"task","type":"nominal","sort":"-x","title":null,"axis":{"labelFontSize":13}}, "x":{"field":"v","type":"quantitative","title":"accuracy points lost","axis":{"grid":true}}}, "layer":[ {"mark":{"type":"bar","height":24},"encoding":{"color":{"field":"task","type":"nominal","legend":null}}}, {"mark":{"type":"text","align":"left","dx":8,"fontWeight":600,"fontSize":13}, "encoding":{"text":{"field":"v","type":"quantitative","format":".1f"}}}]} ``` One honesty note on that table: Q5_K_M scores 78.54 on GSM8K against the F16 baseline's 77.63. Quantization did not make the model better at arithmetic. That is benchmark noise, and it is a useful reminder that sub-point differences in these tables mean nothing. On the GPU-serving side, Red Hat/Neural Magic's [half-million-evaluation study](https://developers.redhat.com/articles/2024/10/17/we-ran-over-half-million-evaluations-quantized-llms) found all schemes recovering over 99% of baseline average on OpenLLM v1, with HumanEval recovery at **99.9% for 8-bit and 98.9% for 4-bit**. Their 4-bit W4A16 results drop more on AIME and GPQA-Diamond than elsewhere, which is the same reasoning-first pattern. ## The ladder **8-bit is close to free.** Q8_0 moved perplexity by 0.01 and every task score by less than 0.2 points in the table above. Mean KLD around 0.003. If you have the memory, stop thinking about it. **4-bit is the sweet spot, and it is not free.** Expect roughly 1–3 points on reasoning-heavy tasks and near-zero on everything else. This is the default for a reason. **3-bit is where reasoning starts to break** while the model still sounds completely fine. This is the dangerous tier, because the failure is invisible in chat and shows up in arithmetic, tool arguments, and long code edits. **Sub-3-bit is a different proposition.** Unsloth's [Aider Polyglot runs on DeepSeek V3.1](https://unsloth.ai/docs/basics/dynamic-3.0-ggufs/unsloth-dynamic-ggufs-on-aider-polyglot) put real numbers on it: | Build | Disk | Aider pass-2 (non-reasoning) | |---|---|---| | Full precision | 671 GB | 71.6% | | Dynamic 4-bit | 387 GB | 69.7% | | Dynamic 3-bit | 300 GB | 68.4% | | Dynamic 2-bit | 255 GB | 65.8% | | Dynamic 1-bit | 206 GB | 55.7% | ```vega-lite Read down the bars: the fall is gentle to 2-bit, then it drops. The knee sits just below 2-bit, where 49 GB of savings costs 10 points. {"title":{"text":"DeepSeek V3.1: Aider Polyglot pass-2 by bit depth","subtitle":"Unsloth dynamic GGUFs, non-reasoning mode. Bars in bit-depth order, not rank order. Source: unsloth.ai/docs/basics/dynamic-3.0-ggufs/unsloth-dynamic-ggufs-on-aider-polyglot"}, "height":{"step":38}, "data":{"values":[ {"build":"Full precision (671 GB)","v":71.6}, {"build":"Dynamic 4-bit (387 GB)","v":69.7}, {"build":"Dynamic 3-bit (300 GB)","v":68.4}, {"build":"Dynamic 2-bit (255 GB)","v":65.8}, {"build":"Dynamic 1-bit (206 GB)","v":55.7}]}, "encoding":{ "y":{"field":"build","type":"nominal","title":null, "sort":["Full precision (671 GB)","Dynamic 4-bit (387 GB)","Dynamic 3-bit (300 GB)","Dynamic 2-bit (255 GB)","Dynamic 1-bit (206 GB)"], "axis":{"labelFontSize":13}}, "x":{"field":"v","type":"quantitative","title":"Aider Polyglot pass-2 (%)","axis":{"grid":true}}}, "layer":[ {"mark":{"type":"bar","height":24},"encoding":{"color":{"field":"build","type":"nominal","legend":null}}}, {"mark":{"type":"text","align":"left","dx":8,"fontWeight":600,"fontSize":13}, "encoding":{"text":{"field":"v","type":"quantitative","format":".1f"}}}]} ``` Going 4-bit → 2-bit costs 3.9 points and saves 132 GB. Going 2-bit → 1-bit costs 10.1 points and saves 49 GB. The curve has a knee and it sits just below 2-bit. Unsloth's own [Dynamic 3.0 documentation](https://unsloth.ai/docs/basics/dynamic-3.0-ggufs) says the same thing more bluntly: below their `UD-Q2_K_XL` tier, models degrade badly on tool-calling and agentic use, loop, and return empty responses. My read: sub-3-bit only makes sense on models large enough that the alternative is not running the model at all. A 1-bit 400B beats a 4-bit 30B on most work. A 1-bit 30B is worse than a 4-bit 8B and you should not build anything on it. ## What quantization buys beyond capacity Decoding one token requires reading every weight the token touches out of memory. That makes decode memory-bandwidth-bound rather than compute-bound, which is the mechanism [/prefill-vs-decode](/prefill-vs-decode) covers. Halving the bytes per weight halves the bytes read per token, so quantization buys throughput on the same hardware, not only the ability to load the model. The Llama-3.1-8B CPU measurements in [Kurt's paper](https://arxiv.org/abs/2601.14277) show generation going from **2.83 tok/s at F16 to 5.12 at Q4_K_M and 9.91 at Q3_K_S** on a dual Xeon 8488C. I trust the direction and the rough magnitude; I do not trust the fine ordering, since Q5_0 beats Q4_K_S in that same table, which cannot be a bandwidth effect and is more likely kernel quality or thread contention. On GPUs the story is cleaner, with [Neural Magic reporting](https://developers.redhat.com/articles/2024/10/17/we-ran-over-half-million-evaluations-quantized-llms) ~2.4× single-stream speedup for W4A16 and ~1.8× for W8A8. ## What it does not shrink: the KV cache Weight quantization does nothing to the KV cache. Cache size scales with context length, batch size, and attention-layer count, and at long context it can rival or exceed the weights. A model that fits at 4-bit with a 4K context may not fit at 128K. The lever is separate. llama.cpp and vLLM both quantize K and V independently. On the quality side, a [measurement on Qwen 2.5 Coder 7B](https://smcleod.net/2024/12/bringing-k/v-context-quantisation-to-ollama/) moved perplexity from 8.3891 to 8.3934 going from f16 to q8_0 KV, a change of 0.0043. That is nothing, and it halves the cache. The same write-up puts an 8B model's 32K cache at ~6 GB f16, ~3 GB q8_0, ~2 GB q4_0. q4_0 KV is a real trade rather than a free one, and its cost is architecture-dependent — reported deltas span roughly -0.7% to +3% perplexity depending on the model. Run `Q8_0` KV by default and treat `q4_0` as a long-context-only measure you verify on your own workload. Note also that quantized KV wants a Flash Attention path with dequant in-kernel; without it you may lose more speed than the memory is worth. ## Picking a format for your runtime The serving stack decides this more than quality does. | Runtime | Load this | Don't bother with | |---|---|---| | llama.cpp / Ollama / LM Studio | GGUF `UD-Q4_K_XL` or `Q4_K_M`; `Q5_K_M` if memory allows | AWQ, GPTQ, NF4 (not loadable) | | vLLM on Ampere/Ada/Hopper | AWQ or compressed-tensors W4A16 for latency; FP8 W8A8 on Ada/Hopper for throughput | bitsandbytes for serving — it loads, it is slow | | vLLM on Turing | GPTQ (AWQ needs Turing+, Marlin has gaps) | FP8, which needs Ada or newer | | MLX on Apple Silicon | MLX 4-bit or 6-bit community builds; mixed-precision variants where published | GGUF, unless you specifically want llama.cpp | vLLM's [hardware compatibility matrix](https://docs.vllm.ai/en/latest/features/quantization/) is worth reading before you download 200 GB of the wrong thing: AWQ needs Turing or newer, llm-compressor FP8 needs Ada or Hopper, and bitsandbytes works nearly everywhere while being the wrong choice for serving nearly everywhere. ```d2 Two questions decide the download, and neither is about quality: which runtime you serve on, then which GPU generation it sits on. Source: vLLM quantization compatibility matrix. direction: down q1: WHICH RUNTIME? { style: { stroke: "#6b6459"; fill: transparent; stroke-width: 1; font-size: 22 } } gguf: GGUF\n\nUD-Q4_K_XL\nor Q4_K_M { style: { fill: "#e0f4ec"; stroke: "#1baf7a"; stroke-width: 2; font-size: 22 } } mlx: MLX\n\n4-bit or 6-bit { style: { fill: "#e0f4ec"; stroke: "#1baf7a"; stroke-width: 2; font-size: 22 } } q2: WHICH GPU\nGENERATION? { style: { fill: "#fbe8de"; stroke: "#eb6834"; stroke-width: 2; font-size: 22 } } q1 -> gguf: llama.cpp\nOllama { style: { stroke: "#6b6459"; font-size: 20 } } q1 -> q2: vLLM / TGI { style: { stroke: "#eb6834"; stroke-width: 2; font-size: 20 } } q1 -> mlx: Apple Silicon { style: { stroke: "#6b6459"; font-size: 20 } } turing: Turing\n\nGPTQ only { style: { fill: "#e4edf9"; stroke: "#2a78d6"; stroke-width: 2; font-size: 22 } } ampere: Ampere\n\nAWQ / W4A16 { style: { fill: "#e4edf9"; stroke: "#2a78d6"; stroke-width: 2; font-size: 22 } } hopper: Ada+\n\nFP8 W8A8 { style: { fill: "#e4edf9"; stroke: "#2a78d6"; stroke-width: 2; font-size: 22 } } q2 -> turing { style: { stroke: "#eb6834"; stroke-width: 2 } } q2 -> ampere { style: { stroke: "#eb6834"; stroke-width: 2 } } q2 -> hopper { style: { stroke: "#eb6834"; stroke-width: 2 } } ``` MLX quantizes with group sizes of 64 for 4/6-bit and 32 for 2/3-bit, and the community convention is to keep embeddings and the final projection at higher width than the body — the same sensitivity principle as dynamic GGUF, applied by hand. Apple Silicon's practical constraint is that MLX and GGUF are separate ecosystems, so the model you want may only exist in one of them on any given day. ## What I'd actually do Take the largest model that fits at 4-bit dynamic with room for a full-context KV cache at q8_0, and prefer a dynamic build over a uniform one at equal file size, because the KLD data says you get it for free. Reach below 3-bit only when the model is large enough that the alternative is not running it. Test on your own reasoning and tool-calling traffic rather than on perplexity, because perplexity is the number that will tell you everything is fine right up until the agent starts mangling JSON arguments. Related: [/local-inference-hardware](/local-inference-hardware) for what fits on what, [/prefill-vs-decode](/prefill-vs-decode) for why fewer bytes per token means faster decode, and [/context-engineering-for-coding-agents](/context-engineering-for-coding-agents) for keeping the context small enough that the KV cache question stops being the binding one.