Tokens & Context Windows

The intuition

Think of the model as a postal system that only ships numbered parcels, never loose letters. Before anything reaches the transformer, your string is cut into parcels called tokens — common words often one parcel, rare words several, punctuation and spaces part of the packing rules. The model never opens a parcel to count letters inside. It only sees parcel IDs.

The context window is the size of the truck: every system instruction, past turn, tool schema, retrieved document, image, and the answer still being written must fit in one load. A bigger truck helps, but it is not free warehouse space. Longer loads cost more, leave later (prefill), and the driver remembers the start and the end of the manifest better than the middle — the famous lost-in-the-middle effect.

Once packing and truck size click, pricing, rate limits, agent memory, and "why still RAG with a 1M window?" stop being separate mysteries. They are all consequences of tokens as the unit of work and one finite window per request.

Why it exists

Without an explicit tokenization and context model, applied systems hit the same walls:

1. Open vocabulary without a fixed input alphabet. Characters are too long; whole words explode the vocabulary and fail on typos, code, and new names. Subword tokens (BPE and cousins) are the compromise: fixed vocab, any Unicode representable, no hard OOV crash.
2. Finite compute per request. Attention and KV cache grow with sequence length. Providers publish a hard max — the context window — because serving cost and memory are real.
3. Billing needs a meter. "Per request" ignores that a 200-token call and a 200k-token call are not the same job. Tokens are the meter for input, output, and cache discounts.
4. Product quality depends on what is in the window. Stuffing everything "because it fits" still fails on lost-in-the-middle, latency, and cost. Curation (RAG, summarization, pruning) exists because relevance density beats raw capacity.

Alternatives lose. Character-level models make sequences enormous. Word-level vocabularies break on morphologically rich languages and identifiers. Unlimited context is not free at serving time — memory and prefill dominate. Ignoring tokenization produces silent cost bugs (non-English and code tokenizing 2–4× worse) and naive "letter counting" product promises the model cannot keep.

On the governed enterprise platform, token accounting is how the gateway enforces per-team budgets, how retrieval packs chunks without blowing the window, and why temporary document uploads need TTL cleanup — every page competes for the same budget as the system prompt and tool results.

The core idea

Models don't see characters or words — they see tokens, integer IDs from a fixed vocabulary produced by a tokenizer trained separately from the model. Modern tokenizers use byte-pair encoding (BPE): start from raw bytes, repeatedly merge the most frequent adjacent pair into a new vocabulary entry, stop at a target vocab size (~50k for GPT-2's BPE, ~100k–200k for current models). Common words become one token; rare words split into subwords; any byte sequence is representable, so there's no out-of-vocabulary failure.

Practical rules of thumb: ~4 characters or ~0.75 words per token for English prose. Code, non-English languages, and unusual formatting tokenize less efficiently — some scripts cost 2–4x more tokens per unit of meaning, which directly multiplies cost and eats context. Token counts are tokenizer-specific: the same text yields different counts on different providers and even across model generations from one provider (Anthropic's newer tokenizer produces ~30% more tokens than its predecessor for identical text — never reuse counts across models).

The context window is the maximum tokens the model can attend over in one request — system prompt + conversation history + tool definitions + tool results + documents + the generated output, all in one budget. Frontier models in 2026 offer 200k–1M token windows with separate, smaller output caps (typically 64k–128k). But a big window isn't free real estate: cost scales linearly with input tokens, latency grows with prefill length, and retrieval quality degrades as context fills — the lost-in-the-middle effect (Liu et al., 2023) showed models recall facts placed at the beginning or end of a long context far better than facts in the middle. Long-context degradation in general is now called "context rot," and it's why context curation (RAG, summarization, pruning) still matters even when everything technically fits.

How it actually works

From raw text to a billable request:

BPE training and encoding. Training: count adjacent symbol pairs in a corpus, merge the most frequent pair into a new token, repeat for N merges. Encoding applies the learned merge rules in order. GPT-4-class tokenizers operate on bytes (so emoji and any Unicode work) with regex pre-splitting so merges don't cross word/category boundaries. encode() and decode() are exact inverses — lossless.

Why LLMs fail at spelling and arithmetic. "strawberry" may be 1–2 tokens; the model never observes its letters, so counting "r"s is inference from training data, not inspection. Numbers split inconsistently ("2023" one token, "2039" maybe two), which breaks digit-place reasoning. These are tokenizer artifacts, not intelligence gaps — models write code to count characters precisely because the tokenizer hides them.

Counting tokens.

Do not use tiktoken to estimate for non-OpenAI models — cross-tokenizer error runs 15–30%.

Pricing mechanics. Billed per token, input and output priced separately; output is typically ~5x input (it costs a sequential forward pass per token). Representative mid-2026 magnitudes: frontier-tier ~$3–15 per million input tokens / ~$15–75 per million output; small-tier ~$0.5–1 / $2–5. Two standing discounts change the math more than model choice: cached input at ~10% of base input price, and batch processing at 50% off everything (see llm-apis.md). Cost per request = (uncached_input x in_rate) + (cached_input x 0.1 x in_rate) + (output x out_rate) — in agent loops the re-sent conversation history dominates, which is why caching and history pruning are the top cost levers.

Everything counts toward the window. Tool schemas (hundreds to thousands of tokens each), images (a high-res image can be 1.5k–5k tokens), thinking/reasoning tokens, and tool results all consume budget. Exceeding the window on input returns a 400; hitting it mid-generation stops with a truncation stop reason.

The flows

These flows turn "context window" from a model-card number into an operational control plane.

A worked example

On the governed enterprise platform, an employee uploads a 40-page policy PDF to a temporary session store and asks:

"What is the pre-approval threshold for international travel?"

Illustrative packing (not a measured production trace):

Positioning: system instructions at the start, strongest policy chunks toward the edges of the document block (or ranked so best evidence is not buried mid-pack), question at the end. A buried threshold sentence in the middle of 6k tokens of policy prose is exactly where Liu et al.'s U-shaped accuracy curve hurts.

Cost sketch at frontier-ish rates (~$3 / M input, ~$15 / M output), single turn, no cache: input ≈ $0.036, output 200 tokens ≈ $0.003. With a warm cache on the 2.5k system prefix at ~0.1×, savings are small this turn but dominate when thousands of users share the same gateway system prompt all day. An agent that re-sends growing history for 20 turns without pruning pays roughly for turns 1 + 2 + … + 20 of repeated prefixes — superlinear cumulative tokens.

Common drill-downs

Why do token counts differ from word counts? BPE splits text into subword units by corpus frequency, not linguistic words. "tokenization" might be 2–3 tokens; frequent words are 1; whitespace and casing variants tokenize differently (" hello" ≠ "hello"). English prose averages ~0.75 words/token; code and non-Latin scripts are worse.

Why can't GPT count the letters in 'strawberry'? It never sees letters — the word arrives as one or two token IDs. Character-level facts must be memorized from training data rather than read off the input. Same root cause as weak digit arithmetic.

What counts against the context window? Everything in the request and response: system prompt, all messages, tool definitions, tool results, images/documents, reasoning tokens, and the output being generated. Output also has its own separate max cap.

You have a 1M-token window. Why still use RAG? Cost (1M input tokens per request, every request), latency (prefill scales with length), and accuracy (lost-in-the-middle / context rot — recall over huge contexts is unreliable for multi-fact tasks). Retrieval sends the relevant 5k tokens instead. Long windows and RAG are complements: bigger windows let you retrieve more generously.

What is lost-in-the-middle? Liu et al. 2023: on multi-document QA, accuracy is U-shaped in the position of the relevant document — highest at the start and end of context, lowest in the middle. Practical rule: order retrieved chunks so the strongest evidence sits at the edges.

How do you manage context in a long-running agent? Prune or clear old tool results first, compact/summarize old turns when approaching a threshold, keep the system prompt stable (for cache hits), and move durable state into external memory (files/DB) retrieved on demand. Measure with per-request usage.

How would you estimate the monthly cost of an LLM feature? requests/day x (avg input tokens x input rate + avg output tokens x output rate), then apply cache hit rate to the repeated prefix (~90% discount on hits) and batch discount for offline portions. Instrument usage from day one; estimates from character counts are off by 25%+.

Same prompt, two models — different token counts. Why? Different tokenizers: different vocabularies and merge rules. Even one provider changes tokenizers between generations. Always count with the target model's tokenizer/endpoint.

Production concerns

• Long context degrades before it overflows. Needle-in-a-haystack scores near 100% are marketing; multi-fact reasoning over long contexts degrades much earlier, worst for mid-context information. Place critical instructions at the start (system prompt) and the current question at the end; put retrieved documents in between and rank the best ones toward the edges.
• Latency scales with prompt length. TTFT is prefill-bound. Stuffing 500k tokens "because it fits" can mean tens of seconds before the first output token. RAG over a curated subset usually beats full-context stuffing on latency, cost, and accuracy simultaneously. See latency.md.
• Context management strategies for agents/chat, in escalation order: (1) sliding window — drop oldest turns; (2) summarization/compaction — replace old turns with a model-written summary (providers now offer this server-side); (3) pruning stale tool results — usually the bulkiest, least-referenced content; (4) externalize state — write notes to files/memory and retrieve on demand instead of carrying everything in-window. Agent-oriented framing: agent-foundations.md.
• Token-based rate limits. Providers enforce tokens-per-minute quotas separately from requests-per-minute; one long-context request can consume a whole minute's TPM. Count before sending in high-throughput pipelines. See reliability.md.
• Budget monitoring. Alert on input_tokens per request in agent loops — a loop that re-appends growing history shows superlinear cumulative token growth (turn N re-sends turns 1..N-1) and is the classic silent cost bug. See cost.md and observability.md.

Test yourself

Go deeper

• Let's build the GPT Tokenizer — Andrej Karpathy — builds BPE from scratch; also a tour of every weird LLM behavior traceable to tokenization.
• Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023) — the U-shaped position-accuracy result.
• tiktoken (OpenAI) — reference BPE implementation; the README's educational notes are worth reading.
• Anthropic docs: Context windows — what counts toward the window, overflow behavior, context rot, compaction.
• Anthropic docs: Token counting — the free count endpoint and cross-tokenizer caveats.
• Effective context engineering for AI agents — Anthropic Engineering — context as a finite resource; curation strategies for agents.

Where this connects

• How LLMs Work — why prefill cost tracks input tokens and decode cost tracks output tokens.
• LLM APIs — usage fields, prompt caching discounts, and batch pricing that make token accounting real money.
• The RAG Pipeline — the main production strategy for putting the right tokens in the window instead of all tokens.
• Cost — token accounting, caching, and history growth as the primary bill drivers.