Tokens & Context Windows

The intuition

Think of the model as a postal system that only ships numbered parcels, never loose letters. Before anything reaches the transformer, your string is cut into parcels called tokens — common words often one parcel, rare words several. The model never opens a parcel to count letters inside. It only sees parcel IDs.

The context window is the size of the truck: system instructions, past turns, tool schemas, retrieved documents, images, and the answer still being written must fit in one load. A bigger truck helps, but longer loads cost more, leave later (prefill), and the driver remembers the start and end of the manifest better than the middle — lost-in-the-middle.

Once packing and truck size click, pricing, rate limits, agent memory, and "why still RAG with a 1M window?" stop being separate mysteries — all consequences of tokens as the unit of work and one finite window per request.

Why it exists

Models need an open vocabulary without a fixed character alphabet. Whole-word lists explode and break on typos, code, and new names; character sequences are far too long. Subword tokens (BPE and cousins) are the compromise: fixed vocab, any Unicode representable, no hard OOV crash.

Serving needs a hard cap: attention and the KV cache grow with sequence length, so providers publish a context window. Billing needs a meter — a 200-token call and a 200k-token call are not the same job. Quality depends on what is in the window, not only whether it fits: stuffing everything still fails on lost-in-the-middle, prefill latency, and cost. Curation (RAG, summarization, pruning) exists because relevance density beats raw capacity. Ignoring tokenization produces silent cost bugs — non-English and code often tokenize 2–4× worse than English prose.

On the governed enterprise platform, token accounting is how the gateway enforces per-team budgets and how retrieval packs chunks without blowing the window — every page competes for the same budget as the system prompt and tool results.

The core idea

Models do not see characters or words. They see tokens: integer IDs from a fixed vocabulary produced by a tokenizer trained separately from the model. Modern tokenizers use byte-pair encoding (BPE) — start from raw bytes, repeatedly merge the most frequent adjacent pair, stop at a target vocab size (~50k–200k). Common words become one token; rare words split into subwords; any byte sequence is representable.

Rough English-prose guide: ~4 characters or ~0.75 words per token. Code, non-English text, and odd formatting are worse — some scripts cost 2–4× more tokens per unit of meaning. Counts are tokenizer-specific: the same text differs across providers and even across generations from one provider (Anthropic's newer tokenizer yields roughly ~30% more tokens than its predecessor). Never reuse counts across models.

The context window is the maximum tokens the model can attend over in one request — system prompt, history, tools, tool results, documents, and the generated output, all in one budget. Frontier models offer large windows (hundreds of thousands to ~1M tokens) with a separate, smaller output cap. Capacity is not free: cost scales with input, latency grows with prefill, and quality degrades as the window fills.

Lost-in-the-middle (Liu et al., 2023): on multi-document QA, accuracy is U-shaped in the position of the relevant document — highest at the start and end, lowest in the middle. Broader degradation as input grows is context rot (Chroma, 2025): even when needle-in-a-haystack scores look fine, multi-fact and semantic recall get less reliable. That is why context engineering — the smallest high-signal token set for the next step — still matters when everything technically fits.

How it actually works

From raw text to a billable request:

Encoding details. GPT-class BPE works on bytes (any Unicode, including emoji) with regex pre-splitting so merges do not cross word boundaries. encode() / decode() are lossless inverses. "strawberry" may be one or two tokens — the model never sees its letters, so counting "r"s is memorized pattern, not inspection. Numbers split inconsistently ("2023" one token, "2039" maybe two), which breaks digit-place reasoning.

Counting. Always use the target model's tools:

Do not estimate non-OpenAI models with tiktoken — cross-tokenizer error often runs 15–30%.

Pricing and window occupancy. Input and output are priced separately; output usually costs several times input (each output token is a sequential forward pass). Cached input and batch APIs dominate real savings (LLM APIs). In agent loops, re-sent history is the main cost driver. Tool schemas, images (often thousands of tokens), thinking/reasoning tokens where kept, and tool results all share the window. Overlong input → 400-class reject. Hitting the limit mid-generation → truncation stop reason (recent Anthropic: often model_context_window_exceeded). Rates rot monthly — treat dollar figures as illustrative.

The flows

A worked example

On the governed enterprise platform, an employee uploads a 40-page policy PDF and asks: "What is the pre-approval threshold for international travel?"

Illustrative packing (not a measured production trace):

Order: system at the start, strongest evidence toward the edges of the document block, question at the end. A threshold buried mid-6k tokens is where the U-shaped curve hurts.

Cost sketch at illustrative rates (~$3 / M input, ~$15 / M output), single turn, no cache: input ≈ $0.036, output 200 tokens ≈ $0.003. An agent that re-sends growing history for 20 turns pays roughly for turns 1 + 2 + … + 20 of repeated prefixes — superlinear cumulative tokens.

Production concerns

Long context degrades before it overflows. Needle-in-a-haystack near 100% is a weak lower bound; multi-fact reasoning and mid-context evidence fail earlier. Put critical instructions first, the current question last, and rank retrieved chunks so the best evidence sits at the edges of the document block. Measure your real task with position-ablated evals.

TTFT is prefill-bound — stuffing hundreds of thousands of tokens can mean multi-second waits before the first token. Curated retrieval usually beats full-context stuffing on latency, cost, and accuracy together (latency.md). For agents, escalate: sliding window → summarization/compaction (often server-side) → prune stale tool results → externalize state and re-retrieve. Keep system prompt and tool list byte-stable for cache hits (agent-foundations.md). Providers enforce tokens-per-minute (TPM) separately from RPM; one long request can burn a full minute's quota. Alert on input_tokens in agent loops — re-appending history yields superlinear cumulative growth (cost.md, observability.md).

Common drill-downs

Why do token counts differ from word counts? BPE splits by corpus frequency, not linguistic words. Frequent words are one token; rare words and odd casing split further (" hello" ≠ "hello"). English averages ~0.75 words/token; code and non-Latin scripts are worse.

Lost-in-the-middle vs context rot? Lost-in-the-middle is the U-shaped position bias. Context rot is broader: reliability falls as length grows, even on simple tasks once distractors and semantic matching enter. Edge-ordering helps the first; shorter, denser context helps both.

Same prompt, two models — different counts? Different tokenizers (vocab + merge rules). Recount with the target model every time.

Test yourself

Go deeper

• Let's build the GPT Tokenizer — Andrej Karpathy — BPE from scratch; tokenizer-rooted LLM quirks.
• Lost in the Middle (Liu et al., 2023) — U-shaped position-accuracy result.
• Context Rot (Chroma, 2025) — performance degrades with input length.
• tiktoken (OpenAI) — reference BPE library.
• Anthropic: Context windows — what counts, overflow, compaction.
• Anthropic: Token counting — free count endpoint; tokenizer-generation caveats.
• Effective context engineering for AI agents — Anthropic — context as a finite attention budget.

Where this connects

• How LLMs Work — why prefill tracks input tokens and decode tracks output tokens.
• LLM APIs — usage, prompt caching, and batch pricing that turn token counts into money.
• The RAG Pipeline — putting the right tokens in the window instead of all of them.
• Cost — token accounting, caching, and history growth as primary bill drivers.