The AI stack, one constraint per layer

Every layer of the AI stack is organised around something it is short of, and the shortage usually starts lower down. Chips ran out of memory bandwidth, so inference engines batch and cache. The web ran out of good text, so labs buy expert tasks and build RL environments. Models ran out of context, so harnesses decide what goes in.

I use that as my map of the field. When a new product, layer or job title appears, I ask what it is short of and which layer caused the shortage. Tools turn over every few months. The shortages are physical or economic, so they last, and most tools are built around one of them.

This post walks the ten layers from the power plant to the product, with evals, gateways and safety running beside them. Each layer gets the number that shows its constraint, the terms worth learning, what changed in 2025 and 2026, where to go deeper, and one question you should be able to answer before moving on. The companies building each layer, and who has been buying them, are in the sister post, who builds the AI stack.

Infrastructure

The bottom four layers are shared: the same power, chips, racks and money serve both training and serving.

Power and sites

A frontier campus is bought in gigawatts, and the scarce input is steady power delivered to one site on a schedule. A Blackwell NVL72-class rack draws roughly 120–140 kW, and Microsoft designed its Fairwater sites for about 140 kW per rack (Microsoft). At that density, one gigawatt of IT load is about 7,000 racks. That is my arithmetic: 1 GW ÷ 140 kW.

Grid connections for loads that size take five years or more in PJM and Virginia (Cleanview), so builders stopped waiting. xAI's Colossus 2 runs on about 1.3 GW of its own gas turbines with no utility power at all (same source), for roughly 946 MW of IT load (Epoch).

Chips and memory

Compute has outrun memory. Microsoft's Maia 200 offers over 10 PFLOPS of FP4 against 7 TB/s of HBM, the high-bandwidth memory stacked beside the chip, roughly 1,400 operations for every byte it can fetch (Microsoft). Generating a token does one or two operations per weight byte, so during decode almost all of that arithmetic sits idle, waiting on memory. I worked through the same roofline in prefill vs decode: bandwidth sets tokens per second, and capacity sets which models fit.

That one ratio explains most of the layer. HBM sits on a CoWoS interposer beside the logic die, which makes HBM and CoWoS the supply chokepoints, and the IEA expects the HBM shortage to last until at least the end of 2027 (IEA). Four-bit formats exist to halve the bytes per weight. Inference chips lean on SRAM because that is where bandwidth is cheap: Nvidia licensed Groq's LPU design in December 2025 (Groq) and put the Groq 3 LPX rack into full production in August 2026 (Nvidia).

Racks and networks

Bandwidth drops at every boundary a byte crosses. A Blackwell GPU reads its own HBM at about 8 TB/s. It reaches the other 71 GPUs in an NVL72 rack at 1.8 TB/s over NVLink, and the next rack at about 0.1 TB/s through an 800 Gb/s network card. The HBM and NVLink figures are Nvidia's; the network figure is unit conversion. A model whose weights and KV cache (the stored attention state of each conversation) fit inside one fast domain runs very differently from one that crosses the slow link on every step.

That is why vendors stopped selling chips. Nvidia sells Vera Rubin as a 72-GPU rack and a five-rack POD (Nvidia), and AWS sells Trainium3 as a 144-chip UltraServer (Amazon). What used to be facility detail, such as copper reach, optics, power delivery and liquid cooling, now decides how large a model or a batch runs efficiently.

Cloud and capital

A GPU is a fast-depreciating asset bought with borrowed money. Epoch's site estimates imply roughly $35–40B of capex per gigawatt of IT load: $35.8B for Colossus 2's 946 MW (Epoch) and $15.9B for Stargate Abilene's 421 MW (Epoch). Over a four-to-five-year hardware life that is $7–10B a year of depreciation per gigawatt, by my arithmetic, whether or not the chips are busy.

So this layer behaves like project finance. Owners want multi-year take-or-pay contracts, and they care about goodput, meaning the work that survives failures. Failures are constant at this scale: Meta's Llama 3 run saw 419 unexpected interruptions in 54 days on 16,384 H100s, about 78% of them hardware (Llama 3 paper).

Models

These three layers run once per model, and each is short of something different: good text, affordable compute and a training signal.

Data

Compute keeps growing, and good human text does not. Epoch puts the effective stock of quality public text at about 300T tokens, with a wide 100T–1,000T interval, and expects it to be fully used between 2026 and 2032 (Epoch). Quality filters already throw away about 90% of a web crawl (Nemotron-CC), and 2026 runs train on 20–45T tokens: 20T for Nemotron 3 Ultra (arXiv) and 33T for DeepSeek-V4-Pro (arXiv).

So the next useful token has to be bought, manufactured or made executable. Bought means licences and expert-written data. Manufactured means rewriting real documents into new forms. Executable means an RL environment: a task, a grader and a sandbox to act in. Epoch reports RL tasks selling for $200–2,000 each and a high-fidelity Slack clone for about $300k (Epoch). My read is that the price only makes sense because one good task trains across thousands of attempts.

Pretraining

Training compute is about 6 × active parameters × tokens, and loss falls only as a power law in compute (Chinchilla is the canonical fit). Each equal step of improvement costs multiplicatively more chip-months, so the layer is a fight over the cost of a useful operation.

Mixture-of-experts is the biggest lever, because a token only pays for the experts it visits. DeepSeek-V4-Pro has 1.6T parameters and activates 49B of them, about 3% (arXiv). Memory still has to hold all 1.6T, so MoE moves cost from training compute into serving memory, which becomes the inference layer's problem. Precision is the next lever: Nemotron 3 Ultra was pretrained in NVFP4 at 550B parameters on 20T tokens, and its loss stayed within 0.4% of the BF16 segments (arXiv). The Muon optimizer, about twice as compute-efficient as AdamW (Moonlight), has spread from Kimi to DeepSeek-V4.

Post-training

A pretrained model predicts text. Post-training turns it into something that follows instructions, uses tools and reasons. Since DeepSeek-R1 in January 2025 the main tool has been reinforcement learning with verifiable rewards (arXiv), and its constraint is that RL is bound by inference and its signal is sparse. Each training sample is a whole rollout, often run in a sandbox that has to boot and run tests, and it comes back with a few bits of reward. GRPO, the default algorithm family, learns only from groups of attempts whose rewards differ, so a task the model always passes or always fails teaches it nothing.

Dense signals are cheaper. On-policy distillation grades every token of the student's own attempt against a teacher, and Thinking Machines measured it at 9–30× cheaper than continuing off-policy fine-tuning (Thinking Machines). The 2026 recipe follows: use RL to build specialists, then distil them into one model. DeepSeek-V4 trains maths, code and agent specialists and merges them this way (arXiv).

Serving and product

These three run on every request, so whatever they cost repeats for as long as the product runs.

Inference

Every new token re-reads the active weights plus that conversation's KV cache. Batching shares the weight read across every request in the batch, which is why hosted tokens are cheap, but each conversation's cache is its own. Llama 3 70B's cache is about 320 KiB per token at FP16, so a 32k-token conversation holds about 10 GB, and a few long agent sessions can fill a GPU by themselves. I did that arithmetic in prefill vs decode. Nearly every technique in this layer moves fewer bytes per useful token.

The list is long, and it all serves that goal: paged KV memory, prefix caching, quantisation, speculative decoding, sparse attention, and running prefill and decode on separate machines. The DistServe authors wrote in late 2025 that "almost every production-grade" serving framework now splits the two phases (retrospective).

Harness

A model acts only on what is in its context window, and its recall over that window degrades as the window fills. Chroma measured the degradation across 18 models, even on simple tasks (Chroma). The harness is the loop around the model, and its real job is choosing which tokens go in. It loads instructions and tool definitions only when needed, retrieves just in time, compacts or resets, and hands subtasks to subagents with clean windows. Anthropic's code-execution approach to MCP cut one workflow from 150,000 tokens to 2,000 (Anthropic).

The second constraint is blast radius. An agent can damage whatever it can reach, and Anthropic reports that users approved about 93% of permission prompts (Anthropic). Asking before every step stops working as a control, so sandboxes have to do that job instead.

Apps

Output is worth something only once someone has checked it. An AI product's value is capped by how cheap that check is and how fast the answer feels. Coding broke out first because tests and a running app make checking cheap, and on OpenRouter, programming passed half of all tokens during 2025 (OpenRouter). The rest of the product layer is moving the same way: interfaces the user can inspect instead of walls of text, and models trained to start answering sooner. OpenAI says GPT-6 Instant starts answering 44% sooner on web-search questions (OpenAI).

Beside the stack

Three concerns run beside the layers rather than inside them. Gateways and evals each serve a few layers near the top, and safety spans all ten.

Gateways and observability

Model prices differ by orders of magnitude, and a cached prefix costs a fraction of normal input: up to 90% off on GPT-6 (OpenAI), and 0.05× the input price for Claude Opus and Sonnet 5.5 cache reads (Anthropic). Cost, latency and quality are therefore decided call by call: which model, which cached prefix, how many turns. You can only steer that from a point that sees every call, a gateway or router, and you can only debug it from a record of what happened, a trace. This runs beside the layers rather than inside them: it sits on every call, but no layer is built on top of it.

Evals

A benchmark score is a task set, a grader and a budget. It measures the model only while the tasks are unseen, the cheap grader agrees with an expert, and the budget and harness stay fixed, and optimisation erodes all three. OpenAI stopped reporting SWE-bench Verified in February 2026. It had found that 59.4% of 138 audited hard tasks had tests that reject correct fixes, and that the frontier models it tested could reproduce the reference patches (OpenAI). METR found maintainers would merge only about half of agent PRs that pass the tests (METR), and Anthropic measured a 6-point swing on Terminal-Bench 2.0 from container settings alone (Anthropic).

Evals now also sit next to post-training, because an eval environment and an RL environment are the same object: Terminal-Bench's Harbor runs both eval runs and RL rollouts (Terminal-Bench). A grader good enough to score a model is good enough to train on, and once a lab trains on it, it stops being a fair score.

Safety

Safety is a band across the stack, not a layer. Deployment risk is the likelihood of a failure times its blast radius; training pushes the first down, while capability and access push the second up (Anthropic). The measurements behind release decisions get harder to trust as models notice they are being tested. Anthropic's Opus 5.5 system card reports unverbalised eval awareness in 36% of audit transcripts, against 0.4% of real Claude Code sessions (system card). In April 2026 Anthropic withheld Claude Mythos Preview from general release on cyber-capability grounds and gave it to defenders instead (Anthropic).

The same stack, six kinds of model

The infrastructure layers do not care what a model produces. Data through inference do, because each kind of model is short of something different.

Image and video are a step count. A diffusion model runs its whole network many times per output, so cutting steps by distillation is the main speed lever. Video multiplies the work. Using Wan's video encoder settings (arXiv), a 5-second 720p clip is about 75k tokens against 4k for a 1024² image. That is about 18× the tokens and roughly 340× the attention per step. Providers price per second of output: Gemini Omni 1.1 Flash costs about $0.10 a second at 720p (Google).

Voice is a latency budget. People leave about 200 ms between turns. Moshi targets 160–200 ms (arXiv), and ElevenLabs claims about 100 ms to audible speech for Eleven v4 Turbo (ElevenLabs). Transcribed speech is plentiful, but recordings of two people talking over each other are not. That is why NVIDIA's PersonaPlex trained on 1,217 hours of real conversation plus synthetic data (NVIDIA).

World models are video with a steering wheel. Genie 3 generates 720p at 24 fps from user actions and stays consistent for "a few minutes" (DeepMind). World Labs' RTFM targets one H100 per stream (World Labs). The scarce data is video with the actions attached, which games supply for free.

Robotics pays for every hour. Each hour of data needs a robot or a person. DROID took 50 collectors 12 months to gather 350 hours (arXiv). Generalist's GEN-1 trained on over 500k hours from wearable devices on humans, with no robot data in pretraining (Generalist).

The interfaces are converging. Google's video flagship is now Gemini Omni rather than a new Veo (model card), and Black Forest Labs trained FLUX 3 jointly on image, video and audio (BFL). The pixels underneath still mostly come from diffusion, and Google's own card describes Omni as combining Gemini with its generative media models. "One model" is often a product boundary, not one network.

Where I would start

My rule for using this map is to learn the layer your constraint comes from, which is usually somewhere below the one you work in.

If you build agents, learn inference: cache hits and batch size decide your bill and your latency. If you run inference, learn chips and memory, because bandwidth sets your ceiling, and then post-training, because RL is itself an inference workload, run in hundreds of thousands of sandboxes per cluster at the labs that publish it (Hugging Face), and it looks nothing like chat traffic. If you train models, learn the data layer, since the next gain is more likely a task you can buy or build than a crawl you already have. If you work in infrastructure, the layer below is power, and it is the one layer nobody can speed up with software.