How LLMs Work

The intuition

Imagine a very skilled autocomplete that has read a large fraction of the public internet. You type a few words; it proposes the single most likely next word; you accept it; it proposes the next; and so on. That loop — predict, sample, append, repeat — is all generation is. Chat, tool calling, and multi-step reasoning are not separate engines. They are sequences shaped so that "what comes next" is the answer, the tool call, or the intermediate step you wanted.

The surprising part is what the autocomplete is made of. Every token becomes a vector. Those vectors pass through dozens to over a hundred identical layers. In each layer, every token can look at every earlier token and selectively pull information from them — that is attention — then each token is transformed alone by a small network that holds much of the stored knowledge. After the last layer, the model scores every word in its vocabulary and picks one. No database lookup. No search. No internal "I know this fact" flag. Just a probability distribution over the next token, over and over.

Once you hold that picture, later pages stop feeling like magic. Hallucination is plausible next tokens when truth is thin. Sampling is how you pick from the distribution. Structured output is how you constrain which tokens are even legal. Caching is how you avoid redoing the expensive first pass over a repeated prefix.

Why it exists

Before transformers, the practical options for sequence models were worse for the scale modern systems need:

1. RNNs and LSTMs processed tokens one by one. Training could not fully parallelize across the sequence, so wall-clock training on web-scale data was brutal. Long-range dependencies had to squeeze through a fixed-size hidden state — easy to lose.
2. Hand-built NLP pipelines (parsers, entity extractors, rules) were interpretable but brittle. Every new domain meant new features and new failure modes. They could not absorb "write a coherent paragraph in the style of a support agent" as a single learned skill.
3. Retrieval-only systems answer from documents but do not compose, paraphrase, plan tool use, or follow open-ended instructions. You still need a generative front end for product UX.
4. Fine-tuned task models per use case work for narrow classification, but a firm cannot train a separate model for every internal workflow. One general next-token model, steered by prompts and tools, is operationally cheaper.

The transformer won because attention gives every position direct access to every earlier position and trains in parallel across the sequence. Pretraining on next-token prediction over trillions of tokens compresses language and world statistics into weights. Post-training turns a document-completer into something that follows roles, tools, and safety norms. The applied engineer does not re-derive this from scratch — but without this picture, API knobs (max tokens, caching, temperature) and product failures (hallucination, cutoff, slow decode) stay mysterious.

In the governed enterprise platform, this architecture is why a single managed model service can power chat, retrieval-augmented answers, and tool-using agents: one generation loop, many shapes of context.

The core idea

An LLM is a next-token predictor. It takes a sequence of tokens and outputs a probability distribution over its vocabulary (~50k–200k tokens) for what comes next. One token is sampled, appended, and the process repeats — that loop is all "generation" is. Everything else (chat, tool calling, reasoning) is built by shaping what sequences the model has learned to continue.

The architecture is the transformer. Input tokens are mapped to embedding vectors, then flow through a stack of identical layers (dozens to over a hundred in frontier models). Each layer has two parts: a self-attention block, where every token's representation is updated by selectively pulling in information from previous tokens, and an MLP (feed-forward) block, where each token is transformed independently — roughly where stored "knowledge" lives. Attention is the piece that made transformers win: unlike RNNs, every token can look at every other token directly, in parallel, so training scales across GPUs.

Training has two phases. Pretraining: predict the next token across trillions of tokens of internet-scale text — this builds a compressed model of language and world knowledge. Post-training: supervised fine-tuning on instruction/response examples, then preference optimization (RLHF/DPO-family) and increasingly RL on verifiable tasks — this turns a document-completer into an assistant. The base model after pretraining doesn't answer questions; it continues text. Post-training is what makes "Answer:" come next.

At inference there are two phases with very different costs: prefill (the whole prompt is processed in one parallel pass) and decode (tokens generated one at a time, each requiring a full forward pass). Decode is sequential because token N+1 depends on token N — that's the autoregressive constraint, and it's why output tokens dominate latency and price.

How it actually works

Generation is a loop. The diagram below is the mechanism, not a product architecture:

Attention mechanics (QKV). For each token, the layer computes three vectors via learned projections: a query ("what am I looking for?"), a key ("what do I contain?"), and a value ("what do I pass along if attended to?"). The attention score between token i and earlier token j is the dot product q_i · k_j, scaled by 1/√d_k, then softmaxed over all j ≤ i into weights that sum to 1. The output for token i is the weighted sum of the values. Causal masking zeroes out future positions so the model can't cheat during training. Multi-head attention runs this 16–128 times in parallel with smaller projections; different heads specialize (syntax, coreference, induction — copying patterns seen earlier in context).

The rest of the layer. Attention output feeds an MLP that expands each token's vector ~4x and projects back. Residual connections add each block's output to its input (the "residual stream"), and layer norms keep activations stable — both are what make 100+ layer stacks trainable. Position information comes from positional encodings; modern models use RoPE (rotary embeddings), which rotate Q/K vectors by position and extrapolate better to long contexts.

From vectors to tokens. The final layer's output is multiplied by an unembedding matrix producing one logit per vocabulary entry; softmax converts logits to probabilities; a decoding strategy (greedy, temperature, top-p — see sampling-and-determinism.md) picks the token.

KV cache. Naively, generating token 1000 would recompute attention for all 999 prior tokens. Instead, each token's K and V vectors are computed once and cached per layer per head. Each decode step then computes Q only for the new token and attends against cached K/V. Consequences:

• Decode cost per token is roughly constant, but KV cache memory grows linearly with context length — for large models it can run to gigabytes per long-context request, which is why context length is expensive to serve.
• Prefill is compute-bound (big parallel matmuls); decode is memory-bandwidth-bound (streaming weights + KV cache for one token of work). This asymmetry drives all serving economics.
• Provider "prompt caching" is persisting the KV cache for a shared prefix across requests — that's why cached input is ~10x cheaper (see llm-apis.md).

Why sequential decoding can't just be parallelized. Token N+1's distribution is conditioned on the sampled token N — you cannot compute step N+1 before N resolves. Speculative decoding works around this: a small draft model proposes several tokens, the large model verifies them in one parallel pass, keeping the accepted prefix. That accelerates decode without changing output distribution.

Cross-links on first contact with neighboring topics. Tokenization and context budgets live in tokens-and-context.md. How one token is chosen from logits lives in sampling-and-determinism.md. The HTTPS request shape that wraps this loop is llm-apis.md.

The flows

For product work, the first two flows dominate day-to-day design. The last flow is the reason a "chat model" is not the same object as a raw pretrained checkpoint.

A worked example

Trace one request through the governed enterprise platform. An employee types into the internal assistant:

"Summarize our Q3 travel policy in three bullets."

Assume a ~2 000-token system prompt (role, safety, tool policy), a short user message (~20 tokens), and no retrieval for this turn — pure parametric + prompt behavior. Illustrative scale only.

Wall-clock intuition: TTFT might be a few hundred milliseconds to a couple of seconds depending on load and prompt size; the rest of the latency is decode at whatever tokens-per-second the host provides. Output tokens are billed ~5× input and cost sequential compute — so "answer in three bullets" is both a UX choice and a cost control.

Now break stages on purpose — this is what production incidents look like:

Common drill-downs

These are the questions a careful engineer asks one level down — the same depth you need when a design review challenges the basics.

Explain how an LLM generates text. Tokenize the prompt, run one forward pass through the transformer to get a probability distribution over the vocabulary, sample one token, append, repeat until a stop token or max length. Prefill processes the prompt in parallel; decode is one token per pass.

What problem does attention solve? Direct access between any two positions in the sequence. RNNs squeezed history through a fixed-size hidden state, losing long-range information and forcing sequential training. Attention gives content-based lookup over the whole context and is fully parallelizable in training.

What are Q, K, and V? Learned per-token projections. Query = what this token is looking for; key = what each token advertises; value = what gets passed along. Score = scaled dot product of query with each key, softmaxed into weights; output = weighted sum of values. Multi-head runs many of these in parallel.

What is the KV cache and why does it matter? Cached key/value vectors for every processed token, per layer. It makes decode O(context) per token instead of O(context²) total recomputation, but its memory grows linearly with context — it's the binding constraint on long-context serving and the mechanism behind prompt caching discounts.

Why is generation slow relative to reading the prompt? Prefill is one parallel pass over all prompt tokens (compute-bound). Decode requires a full sequential forward pass per output token (memory-bandwidth-bound). 1000 output tokens = 1000 dependent passes.

Base model vs instruct/chat model? Base model = pretrained next-token predictor; completes documents, won't reliably follow instructions. Instruct model = same weights further trained on instruction data + preference optimization; follows the chat format. The chat "roles" are just special tokens the post-trained model learned to respect.

Where does the model's knowledge live? Distributed across the weights, mostly in MLP layers — pretraining compresses statistical regularities of the corpus. There's no database and no lookup; that's why parametric knowledge is frozen at the training cutoff and why RAG exists.

Do application engineers need to know backprop? Application engineers consume models; you should know training happens by gradient descent on next-token cross-entropy loss, and the pretraining/post-training split, because it explains behavior (hallucination, cutoff, instruction-following). Implementing training is the model provider's job.

Production concerns

• Latency splits into TTFT and TPS. Time-to-first-token ≈ prefill time, scales with prompt length; tokens-per-second ≈ decode throughput, roughly constant per model. A 100k-token prompt with a 200-token answer is TTFT-dominated; mitigate with prompt caching and shorter contexts, not a faster model. Full treatment: latency.md.
• Output tokens are the expensive axis. Priced ~5x input across providers, and each one costs a sequential forward pass. Constraining output length (schemas, "answer in one line") is a real latency/cost lever. Full treatment: cost.md.
• Batching. Servers batch many requests per forward pass to amortize weight-loading; continuous batching (requests join/leave mid-flight) is standard in vLLM-class servers. Batch composition varies per request — one root cause of nondeterminism at temperature 0 (see sampling-and-determinism.md).
• Memory math for self-hosting. Weights: params x bytes (a 70B model at FP16 = ~140 GB; ~35 GB at 4-bit). Plus KV cache per concurrent request. Quantization (INT8/FP8/4-bit) trades small quality loss for large memory/throughput gains. Full treatment: open-source-self-hosting.md.
• Failure modes downstream of the objective. The model always emits a plausible next token — it has no built-in "I don't know" state (see hallucination.md), and it can't count characters or do reliable arithmetic on digits because it sees tokens, not characters (see tokens-and-context.md).
• Reliability under load. Retries, timeouts, and fallbacks sit at the API layer, but their correct settings depend on prefill/decode economics — a timeout that works for short answers fails for long generations without streaming (reliability.md).

Test yourself

Go deeper

• Intro to Large Language Models — Andrej Karpathy — the canonical 1-hour survival-depth overview: training, finetuning, LLM-as-OS, security.
• But what is a GPT? Visual intro to transformers — 3Blue1Brown — best visual intuition for embeddings and the transformer stack.
• Attention in transformers, step-by-step — 3Blue1Brown — the clearest QKV walkthrough available.
• Let's build GPT: from scratch, in code, spelled out — Andrej Karpathy — 2 hours implementing a working transformer (nanoGPT); one level deeper than interviews need, permanently clarifying.
• The Illustrated Transformer — Jay Alammar — the classic diagram-driven written explanation; cited in Stanford/MIT courses.
• Attention Is All You Need (Vaswani et al., 2017) — the original transformer paper; skim for the architecture diagram and scaled dot-product attention.

Where this connects

• Tokens & Context Windows — the model only ever sees token IDs; context is a hard budget that includes tools, images, and output.
• Sampling & Determinism — how one token is chosen from logits, and why temperature 0 is still not a correctness guarantee.
• Hallucination — structural consequence of next-token prediction without a truth check; mitigation is layered engineering, not a single prompt line.
• LLM APIs — the stateless request/response envelope around prefill, decode, streaming, and KV-cache-backed prompt caching.