How LLMs Work
Transformer and attention at survival depth; next-token prediction — enough to understand what the model is and isn't when you build on top of it.
Prerequisites
- None — this is a starting point. If you already know tokens, still read Tokens & Context Windows next: the model never sees characters, and that fact explains failure modes that look like "intelligence gaps."
The intuition
Imagine a skilled autocomplete that has read a large fraction of the public internet. You type a few words; it proposes the most likely next word; you accept it; it proposes the next. That loop — predict, sample, append, repeat — is all generation is. Chat, tool calling, and multi-step reasoning are not separate engines. They are sequences shaped so that "what comes next" is the answer, the tool call, or the step you wanted.
Under the hood, every token becomes a vector. Those vectors pass through dozens to over a hundred identical layers. In each layer, every token can look at earlier tokens and selectively pull information from them — that is attention — then each token is transformed alone by a small network that holds much of the stored knowledge. After the last layer, the model scores every vocabulary entry and picks one. No database. No search. No "I know this fact" flag. Just a probability distribution over the next token, over and over.
That picture unlocks the rest of the book: hallucination, sampling, structured output, and caching all fall out of the same loop.
Key insight
An LLM has no separate "reasoner" and no separate "memory." It is one next-token loop. Everything that looks like planning, tool use, or knowledge is either patterns compressed into the weights during training, or text you put in the context window for this request. Context engineering and retrieval matter as much as model choice.
Why it exists
Before transformers, sequence models struggled at modern scale. RNNs and LSTMs could not fully parallelize training and squeezed long-range dependencies through a fixed hidden state. Hand-built NLP pipelines were brittle. Retrieval-only systems cannot compose, paraphrase, or follow open-ended instructions. One fine-tuned model per workflow cannot cover every internal task a firm has.
The transformer won because attention gives every position direct access to every earlier position and trains in parallel. Pretraining compresses language and world statistics into weights; post-training turns a document-completer into something that follows roles, tools, and safety norms. You need this picture so API knobs and product failures stop looking mysterious. In the governed enterprise platform, it is why one managed model service can power chat, retrieval-augmented answers, and tool-using agents: one generation loop, many shapes of context.
The core idea
An LLM is a next-token predictor. It takes a sequence of tokens and outputs a probability distribution over its vocabulary (~50k–200k tokens) for what comes next. One token is sampled, appended, and the process repeats. That loop is generation. Chat, tools, and chain-of-thought are sequences the model has learned to continue — not separate engines.
The architecture is the transformer. Tokens map to embeddings, then flow through a stack of identical layers (dozens to 100+). Each layer has self-attention (every token selectively pulls from previous tokens) and an MLP (each token transformed alone — roughly where stored knowledge lives). Unlike RNNs, every token can look at every earlier token directly, in parallel, so training scales across GPUs.
Training has two phases. Pretraining predicts the next token across internet-scale text. Post-training applies supervised fine-tuning on instruction/response pairs, then preference optimization (RLHF, DPO-family) and often RL on verifiable tasks. The base model continues text; post-training is what makes "Answer:" come next, and what teaches roles, tools, and refusals.
At inference there are two phases with very different costs. Prefill processes the whole prompt in one parallel pass. Decode generates tokens one at a time; each needs a full forward pass. Decode is sequential because token N+1 depends on the sampled token N — the autoregressive constraint. That is why output tokens dominate latency and price.
How it actually works
Generation is a loop. The diagram is the mechanism, not a product architecture:
Attention (QKV). For each token, the layer computes three vectors via learned projections: a query ("what am I looking for?"), a key ("what do I contain?"), and a value ("what do I pass along if attended to?"). The score between token i and earlier token j is the scaled dot product q_i · k_j / √d_k, then softmaxed over all j ≤ i into weights that sum to 1. The output for i is the weighted sum of values. Causal masking zeros future positions so the model cannot cheat during training. Multi-head attention runs this many times in parallel; heads specialize (syntax, coreference, copying earlier patterns).
The rest of the layer. Attention output feeds an MLP that expands each token's vector (often ~4×) and projects back. Residual connections add each block's output to its input — the residual stream — and layer norms keep activations stable; both make deep stacks trainable. Position comes from RoPE (rotary embeddings), which rotate Q/K by position and extrapolate better to long contexts. The final layer is multiplied by an unembedding matrix to produce one logit per vocabulary entry; softmax turns logits into probabilities; a decoding strategy (greedy, temperature, top-p — see Sampling & Determinism) picks the token.
Key insight: prefill vs decode is the latency story
Prefill is one big parallel pass over the prompt — compute-bound. Decode is one full forward pass per output token — memory-bandwidth-bound and sequential. A long prompt with a short answer is TTFT-dominated; a short prompt with a long essay is decode-dominated. Different problems, different fixes (latency.md).
KV cache. Naively, generating token 1000 would recompute attention over all 999 prior tokens. Instead, each token's K and V are computed once and cached per layer; each decode step computes Q only for the new token. Decode cost per token stays roughly constant, but KV memory grows linearly with context — often gigabytes per long request. Provider prompt caching persists that cache for a shared prefix across requests (see LLM APIs). Grouped-query attention (GQA) and multi-query attention (MQA) shrink the cache further: several query heads share the same key/value heads, so fewer K/V tensors are stored and streamed at decode time.
Why decode cannot just run in parallel. Token N+1 is conditioned on the sampled token N. Speculative decoding works around this without changing the output distribution: a small draft model proposes several tokens; the large model verifies them in one parallel pass and keeps the accepted prefix.
Common misconception
"The model looks up facts in a knowledge base inside the weights." There is no database and no lookup. Parametric knowledge is statistical regularities compressed into matrices — frozen at the training cutoff. That is why RAG and tools exist, and why hallucination is structural rather than a glitch.
The flows
| Flow | Sequence | When it applies | What breaks it |
|---|---|---|---|
| Prefill-then-decode | Tokenize → embed → prefill → write KV → sample → decode until stop | Every chat completion and agent step | Context overflow; max_tokens mid-answer; empty prompts |
| Cached-prefix generation | Stable prefix hits stored KV → skip most prefill → decode the new suffix | Multi-turn chat, agent loops, shared system prompts | Any byte change in the prefix (timestamp, reordered tools) → full prefill again |
| Speculative decode | Draft proposes k tokens → large model verifies in one pass → accept matching prefix | Serving where decode is the bottleneck | Poor draft quality; only on stacks that implement it |
| Base vs instruct (training) | Pretrain → SFT → preference/RL → serve chat model | Explains roles, tools, refusals | Expecting a base model to follow chat format; expecting post-training to refresh knowledge past cutoff |
For product work, the first two flows dominate. The last is why a "chat model" is not a raw pretrained checkpoint.
A worked example
Trace one request through the governed enterprise platform. An employee asks: "Summarize our Q3 travel policy in three bullets." Assume a ~2 000-token system prompt, a short user message (~20 tokens), and no retrieval. Illustrative scale only.
| Stage | What happens | Illustrative numbers |
|---|---|---|
| 1. Tokenize | Text → integer IDs (BPE) | ~2 020 input tokens |
| 2. Embed | Each ID → dense vector; RoPE encodes position | Hidden size not exposed by the API |
| 3. Prefill | One parallel pass; build K/V for every prompt token | Dominates time-to-first-token |
| 4. First logits | Unembedding → ~100k–200k scores → softmax | Mass may land on "Here", "1.", policy jargon |
| 5. Sample | Temperature/top-p (or reasoning-model defaults) → one token | Token 1 appends; KV already holds the prompt |
| 6. Decode loop | Q for new position, attend to cached K/V, sample | ~80–150 output tokens → that many sequential passes |
| 7. Stop | End-of-turn / stop token or max_tokens | Client streams deltas (LLM APIs) |
TTFT is prefill-dominated; the rest is decode at the host's tokens-per-second. Output tokens cost more per unit (often ~3–5× input — check current pricing) and sequential compute, so "answer in three bullets" is both UX and cost control.
What each misunderstanding looks like in production
- No prefill vs decode split → a "faster model" does not fix a 200k-token prefill; first token still crawls.
- No caching design → every agent turn re-prefills a 10k-token system prompt; cost and TTFT climb with traffic that could be mostly cache hits.
- Model as database → invents a policy clause never in training data or the prompt (hallucination.md).
- Ignoring autoregression → product assumes the whole answer can finish in one parallel pass. Speculative decoding helps; free-running future tokens does not.
- Base model in chat → completions continue the user's sentence instead of answering.
Production concerns
Measure latency as two numbers. Time-to-first-token (TTFT) is mostly prefill and scales with prompt length. Tokens-per-second (TPS) is decode throughput under a given load. A 100k-token prompt with a 200-token answer is TTFT-dominated — mitigate with shorter contexts and prompt caching, not a model that only improves decode (latency.md). Output tokens are the expensive axis: higher price and one sequential forward pass each — constrain length deliberately (cost.md).
On the server, continuous batching lets new requests join and finished ones leave mid-flight so GPUs stay full (vLLM-class stacks). Decode is memory-bandwidth-bound, so batching amortizes weight loads — and is one reason temperature 0 is not bit-identical across runs (Sampling & Determinism). PagedAttention-style KV management stores cache in fixed blocks; without it, long-context concurrency collapses under fragmentation.
If you self-host, memory is weights plus KV. Weights: params × bytes (illustrative: 70B at FP16 ≈ 140 GB; ~35 GB at 4-bit). KV grows with context × layers × KV heads × concurrent sequences — a handful of 100k-context sessions can dwarf the weights. Quantization and GQA are operational decisions (open-source-self-hosting.md). Downstream of the objective: no built-in "I don't know" (hallucination.md), no character-level view (tokens-and-context.md), and timeouts that work for short answers fail for long generations without streaming (reliability.md).
Common drill-downs
What problem does attention solve that RNNs could not? Direct, content-based access between any two positions, plus full parallelization across the sequence during training. RNNs forced history through a fixed hidden state and sequential compute.
Why does the KV cache exist, and what is its hard limit? It avoids recomputing K/V on every decode step. Memory then grows linearly with context × layers × KV heads × concurrency — the long-context serving tax. GQA/MQA cut the KV-head dimension; prompt caching reuses a shared prefix across requests.
Prefill is compute-bound; decode is memory-bandwidth-bound. So what? Long-prompt / short-answer work needs shorter prompts and prefix caching. Short-prompt / long-answer work needs higher TPS (batching, speculative decoding, smaller or quantized models). Optimizing the wrong phase wastes the budget.
Where does "knowledge" live? Distributed across weights, with a large share in MLP layers; attention mostly routes and copies. Still no lookup table — cutoff and hallucination follow.
Test yourself
A feature has a 50k-token system prompt and usually returns 30 tokens. Users see a long empty spin, then the answer appears almost at once. Prefill or decode — and what would you change first?
Self-hosting a 70B model at FP16: roughly how much memory for weights alone, and what else grows with concurrent long-context users?
A PM asks: 'Can we parallelize generation so the answer finishes in one forward pass like prefill?' What do you say?
Where do you look first if 'the model isn't following instructions' after switching from a chat model to a base model endpoint?
Go deeper
- Intro to Large Language Models — Andrej Karpathy — canonical ~1-hour survival-depth overview: training, finetuning, LLM-as-OS, security.
- But what is a GPT? Visual intro to transformers — 3Blue1Brown — best visual intuition for embeddings and the transformer stack.
- Attention in transformers, step-by-step — 3Blue1Brown — clearest QKV walkthrough available.
- The Illustrated Transformer — Jay Alammar — classic diagram-driven written explanation for QKV matrices.
- Attention Is All You Need (Vaswani et al., 2017) — original transformer paper; skim architecture diagram and scaled dot-product attention.
- Anatomy of a High-Throughput LLM Inference System — vLLM — prefill/decode, PagedAttention, continuous batching, and speculative decoding in production servers.
Where this connects
- Tokens & Context Windows — the model only ever sees token IDs; context is a hard budget that includes tools, images, and output.
- Sampling & Determinism — how one token is chosen from logits, and why temperature 0 is still not a correctness guarantee.
- Hallucination — structural consequence of next-token prediction without a truth check; mitigation is layered engineering.
- LLM APIs — the stateless request/response envelope around prefill, decode, streaming, and KV-cache-backed prompt caching.