Context windows

The shared budget every agent lives in — sizes, cost, and compute.

Every agent lives inside a context window

Before we talk about tools, skills, memory, or RAG, you need a clear mental model of the thing they all share: the context window. It is the single piece of working memory the model gets on every call, and everything else in this module is a strategy for spending it well.

The first thing to internalise: the model has no memory of its own. Every time the user types a new message, the client sends the entire conversation so far — system prompt, every prior user message, every prior assistant reply, plus the new user message — back to the server as one big payload. There is no server-side "session" holding the chat; the server is stateless and the history is the prompt. Each turn the prompt gets longer, even if the user only adds a sentence.

That re-sending is what makes prompt caching and conversation compaction so important: by turn 20 of a working session you may be sending hundreds of thousands of tokens on every keystroke's reply, and you're paying for them every single time.

The context window is a fixed-size token budget. On each call the model can see:

• The system prompt you wrote.
• The tool definitions the agent has been given.
• Any skills loaded into the conversation.
• The conversation history so far.
• Any retrieved documents injected by RAG.
• Any scratchpad / reasoning the model produced earlier.

When all of that together exceeds the window, the call fails or older content is dropped. Hence the discipline: what gets space, and what earns its keep.

A real example — Claude Code's /context

Claude Code ships a /context slash command that shows you exactly what has been spent. Here's a real snapshot from a session building this course:

Observations:

Note that with Claude Opus, the context window is very large.

• The conversation dominates. 88% of everything spent is message history. System plumbing (prompt + tools + skills + agents) is a rounding error by comparison.
• Free space is not really free. The auto-compact buffer (~3% here) is reserved for the moment the window fills and the tool has to summarise and compress older messages to keep going.
• Tool definitions are cheap. 14k for all tools is a tiny price to pay, which is why it's usually wrong to prune tools for context reasons. Prune them for confusion reasons.

Context sizes vary by ~16× across the frontier

Not all models have the same window. Pick a model whose window fits the shape of the problem.

The spread matters:

• 128K (Llama 4, DeepSeek V3.2, GPT-OSS) — enough for a long document or a short agent session; not enough for a multi-hour coding run.
• 200K (Claude Haiku 4.5, OpenAI o3) — comfortable for most single-task agent work.
• 1M (Claude Opus/Sonnet 4.x, Gemini 3, GPT-5.4, Qwen3-Coder Plus) — the current frontier ceiling for most closed models. Enough to hold a whole codebase or a week of chat.
• 2M (Grok 4) — headroom for exceptionally long agent runs or corpus-scale ingestion.

The pattern in 2026 is clear: closed-model vendors have converged on ~1M as the standard; open models are still mostly 128K. If your workflow genuinely needs 1M of live context, that narrows your provider choice significantly.

Bigger is not free, and isn't always better

A larger window sounds like pure upside. It is not. Three real costs come with spending more:

1. Compute and latency

Attention is the technology at the heart of a transformer, and classic attention scales quadratically with sequence length. Modern models use tricks (sparse attention, sliding windows, KV-cache optimisations) to make long context feasible, but the economics still rhyme: more tokens in, more compute per call, more waiting before the first output token, more wall-clock time to the last.

In practice: a 500k-token input is not just expensive, it is noticeably slower to respond than a 5k-token one, even on the same model.

2. Cost

Every token in the input is billed on every turn. If the conversation has grown to 400k tokens and you take 20 more turns, you are paying for 400k×20 = 8M input tokens even if the user says very little new.

Prompt caching — where the provider stores the KV-cache for repeated parts of the prompt and bills you at a fraction of the usual price. It costs more upfront, but much less over time With caching on, the unchanged part of the prompt (system, tools, earlier messages) can be billed at around 10% of its usual price. Without it, long sessions get expensive fast.

3. Quality decays — the "dumb zone"

A model that can hold 1M tokens rarely uses all of them equally well. Research has shown that models are generally best at paying attention to the beginning and end of their context windows, leading to the model appearing to forget things from the middle.

The qualitative U-shape from the Lost in the Middle paper holds up; the Context Rot paper shows the curve collapses further when you add realistic distractor load.

The practical implications for agent builders:

• Put the most important content at the end. System prompt and tools up top (cacheable, ignorable); retrieved facts, user query, and required output format near the bottom.
• Fewer, cleaner retrievals beat more. If you can pull five very relevant chunks, do that rather than 20 loosely-relevant ones — distractors cost more than missing context.
• Don't treat 1M as a license to stuff. A model that scores well at 5k tokens may score materially worse at 500k on the same task. Measure on realistic input sizes.
• Summarise aggressively for older turns. The middle of a long conversation is the dumb zone. Rolling summaries are not just a cost mitigation — they are a quality mitigation.

Sources

• Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023) — arxiv.org/abs/2307.03172.
• Chroma Research, Context Rot: How Input Length Degrades LLM Performance (2025) — trychroma.com/research/context-rot.

Strategies for living inside the window

Four moves that every serious agent uses:

1. Rolling summaries. When conversation history starts crowding other content, compact older turns to a summary. Claude Code's auto-compact is an automatic version of this.
2. RAG instead of stuffing. If a whole document library is relevant, don't paste it — index it and retrieve the relevant chunks per turn. Covered in RAG and context engineering.
3. Per-turn selection. At each step, decide what memory and which skills are actually relevant, and load only those. This is the unglamorous part of agent engineering that most moves the quality needle past prototype.
4. Prompt caching — covered in detail below.

Prompt caching

The single biggest cost lever for any multi-turn agent is prompt caching. Because every turn re-sends the same system prompt, the same tool definitions, and the same earlier messages, most of the payload on turn N was already sent on turn N−1. Prompt caching lets the provider recognise that unchanged prefix and charge you a fraction of the usual price to reuse it.

The picture is the same one as the top of the lesson — only now the system prompt carries a cache marker. Turn 1 writes the prefix to the cache; turns 2+ read it back.

The catch: writing to the cache costs 2× the normal input price. That one-time premium means a single-turn interaction with caching on is more expensive than one without. The question is how quickly the 0.1× read price pays that premium back.

Read the shape of the chart, not the absolute numbers. With a realistic cacheable prefix, the two curves cross at around turn 3 — and from then on the gap widens every turn. The implications:

• One-shot classifiers and single-turn calls: caching is usually not worth it. The write premium never amortises.
• Any multi-turn agent (>3 turns): caching is almost always a win, and the savings compound. A 20-turn working session typically ends up costing a third or less of the uncached equivalent.
• Put stable things at the top. System prompt, tool definitions, long skills, reference docs. Anything that changes every turn goes at the bottom — once a cached prefix's tail changes, the cache is invalidated from that point on.
• Measure your real prefix. The 5k prefix in the chart is conservative; most production agents have 20–50k tokens of system prompt + tools + skills, which makes the break-even even earlier and the long-tail savings bigger.

Play with the cost

Drag the sliders to see how context fill, output size, and session length move cost — and how different models compare at each setting.

Three things worth noticing as you play with it:

• With prompt caching on, doubling input tokens roughly doubles cost — but without caching, it roughly doubles cost per turn, every turn. Caching is the single biggest lever.
• Small-context models disappear off the chart as you push past 128k — they simply can't run. That is the cliff they hit in production.
• The cost-per-turn gap between Sonnet and GPT-5.4 is modest; the gap between either of them and DeepSeek V3.2 is an order of magnitude.