Sampling & Determinism

The intuition

After the transformer runs, it does not emit a sentence. It emits a scorecard over the whole vocabulary — one raw score (logit) per possible next token. Softmax turns that scorecard into a probability pie. Sampling is how you cut the pie: always take the largest slice (greedy), or spin a weighted wheel (random sampling), optionally after throwing away the tiny slices (top-k / top-p) or reshaping the pie (temperature).

Temperature is a dimmer switch on confidence, not a creativity gland. Turn it down and the biggest slices dominate; turn it up and the long tail gets more mass. Top-p is a smart bouncer: when the model is sure, only a few tokens get in; when it is torn, the room expands.

The trap is believing the dimmer switch is a lock. Temperature 0 makes the wheel deterministic — always pick the current argmax — but the scorecard itself can still twitch between runs on real GPU servers. One flipped token, then autoregression does the rest: a different novel.

Why it exists

If every step always took the single most likely token, systems would still work for many extraction tasks — and they would fail at everything that needs variety or exploration. The knobs exist because product needs differ:

1. Classification and extraction want conservatism. Spurious diversity breaks parsers and invents enum values. Low temperature (or constrained decoding) is the default.
2. Assistants and brainstorming want coverage. Multiple plausible phrasings, ideas, or candidates need probability mass on non-argmax tokens.
3. Open-ended sampling without truncation is messy. Pure full-vocab sampling at high temperature drifts into incoherence; top-p / top-k / min-p cut the pathological tail.
4. Hosts must batch for economics. Serving many users on one GPU forces numerical paths that are not identical run-to-run — so "deterministic decoding" as a product promise collides with throughput.

Alternatives lose in different ways. Always greedy loops and dulls creative tasks. Always high-T unrestricted raises garbage and schema failures. Pretending T=0 is a unit test oracle produces flaky CI and false confidence. Seeds alone fix the RNG, not the logits under variable batching.

On the governed enterprise platform, extraction routes stay cold and schema-constrained; brainstorming and synthetic data routes run warmer with nucleus sampling; compliance-sensitive paths cache answers or avoid depending on bit-identical regeneration.

The core idea

Every generation step ends with the model producing one logit (raw score) per vocabulary token. Softmax turns logits into a probability distribution; the sampling strategy decides which token to take from it. Greedy decoding always takes the argmax; pure sampling draws from the full distribution; everything else is a truncation or reshaping of that distribution.

Temperature divides logits before softmax: p_i = softmax(logits / T). T < 1 sharpens the distribution toward the top tokens; T > 1 flattens it toward uniform; T → 0 approaches greedy. It rescales relative confidence — it does not add knowledge or "creativity," it just widens or narrows how far from the modal continuation you're willing to sample. Top-k keeps only the k highest-probability tokens and renormalizes. Top-p (nucleus) keeps the smallest set of tokens whose cumulative probability ≥ p — adaptive where top-k is fixed: on a confident step the nucleus might be 2 tokens, on an open-ended step 200. Providers apply cutoffs first, then sample with temperature; the standard advice is to tune temperature or top-p, not both.

Practical settings: extraction/classification/code — T ≈ 0–0.3; general assistant — T ≈ 0.7–1.0; brainstorming — T ≈ 1.0+ with top-p ≈ 0.95 to clip the incoherent tail. And a 2026 caveat worth stating clearly: reasoning-first models increasingly remove sampling knobs entirely (Anthropic's current Opus-tier models reject temperature/top_p/top_k) — steering is expected to happen via prompting, and determinism was never guaranteed anyway.

That last point is the trap: temperature 0 still isn't fully deterministic. Greedy decoding is deterministic given identical logits — but the logits themselves vary run to run on real serving stacks.

How it actually works

From logits to a token, and why two identical HTTP requests can still diverge:

Why identical requests produce different logits:

1. Floating-point non-associativity. (a + b) + c ≠ a + (b + c) in floating point. GPU kernels sum in whatever order maximizes parallel throughput, and that order can differ between runs, kernel versions, and hardware. Tiny differences (~1e-7) in logits are harmless until two tokens are near-tied — then the argmax flips, and because generation is autoregressive, one flipped token cascades into a completely different continuation.

2. Batching (the dominant cause on APIs). The 2025 Thinking Machines analysis pinned the mechanism precisely: most inference kernels are not batch-invariant — your request's logits depend on the batch it was grouped with. Server load determines batch size and composition, so the same request at 2pm and 2:05pm runs through numerically different reductions. From your seat, the endpoint is load-dependent, hence nondeterministic. Fixing it requires batch-invariant kernels (which vLLM/SGLang have since shipped as opt-in modes, at some throughput cost) — not just setting T=0.

3. Mixture-of-Experts routing. In MoE models, a router picks which experts process each token. Expert-parallel implementations can route based on batch-level load balancing, so which experts your tokens hit can depend on co-batched traffic — a larger perturbation than float noise.

4. Mundane sources. Silent model/backend updates behind a stable model alias, different GPU types across a fleet, and speculative-decoding acceptance edge cases.

Two more mechanisms worth naming: seeds (some APIs accept one; it fixes the RNG for sampling but cannot fix logit-level variance, so OpenAI only ever promised "mostly deterministic" plus a system_fingerprint to detect backend changes) and min-p (a newer truncation rule — keep tokens above a fraction of the max probability — popular in open-source serving for high-temperature stability).

The flows

A worked example

Suppose the gateway on the governed enterprise platform classifies an inbound email intent into a fixed set: refund, password_reset, address_change, other.

Illustrative single-step distribution over the label token (simplified — real models emit multi-token labels):

At T=0.2 the classifier almost always emits refund when logits look like this. At T=1.5, refund vs other is a coin-flip neighborhood — bad for routing. Top-p=0.9 at T=1.0 would keep mostly {refund, other} and discard the long tail of random words.

Now run the same request twice at T=0 on a busy multi-tenant endpoint. Batch A groups this request with short prompts; batch B with long multimodal jobs. Non-invariant matmuls nudge logits so refund and other swap order by 1e-5. Greedy decoding flips; the ticket goes to the wrong queue. Temperature did not cause it — serving numerics + a near tie did.

Mitigation that actually works: constrain the label with structured output (enum schema), validate, and if the product needs identical re-answers for identical mail, cache the classification keyed on content hash. Optionally read logprobs: if top-two are within a tight margin, route to a human or a larger model.

Common drill-downs

Walk me through what happens between logits and the output token. Optionally truncate the vocabulary (top-k / top-p / min-p), divide logits by temperature, softmax to probabilities, sample (or argmax at T=0), append the token, repeat.

What exactly does temperature do? Scales logits before softmax. T<1 exaggerates gaps between token scores (conservative), T>1 compresses them (diverse). At T→0 it converges to greedy argmax. It doesn't change what the model knows — only how much probability mass reaches non-modal tokens.

Top-p vs top-k? Top-k: fixed count cutoff — same k whether the model is certain or torn. Top-p: probability-mass cutoff — the candidate set shrinks when the model is confident and expands when it's uncertain. Top-p is generally the better default for that adaptivity.

Why is temperature 0 not deterministic? T=0 makes the sampling step deterministic, not the logits. Floating-point addition is non-associative and GPU reduction order varies; serving batches your request with others through kernels that aren't batch-invariant, so logits depend on server load; MoE routing can vary with batch composition. Near-tied logits then flip the argmax and the divergence compounds autoregressively.

How would you get reproducible outputs anyway? In order of practicality: cache responses by request hash; pin model snapshot versions and use seeds + fingerprint checks where offered; self-host with deterministic/batch-invariant inference mode. Accept that hosted-API determinism is best-effort.

When would you raise temperature above 1? Diversity-first tasks: brainstorming, synthetic data generation, multiple-candidate sampling (best-of-N followed by reranking). Pair with top-p/min-p to cut the degenerate tail.

What temperature for a JSON extraction pipeline? Low (0–0.2) to minimize format drift and fabrication — but rely on constrained decoding/schema validation for correctness, and note some current models no longer expose temperature at all.

Model outputs repeat or loop — what do you reach for? Frequency/presence penalties (or repetition penalty in open-source stacks) which down-weight already-emitted tokens; slightly higher temperature; check for degenerate greedy decoding at T=0, where loops are a classic failure mode.

Production concerns

• Don't build correctness on determinism. If a pipeline breaks when the model phrases something differently, the fix is structured outputs + validation (see structured-output.md), not T=0. Treat sampling params as a quality dial, never a correctness guarantee.
• Evals must account for run-to-run variance. A/B-ing prompts on one run each measures noise. Run N samples per case, compare pass rates; for flaky judgment tasks report variance. This is the most common junior eval mistake. See evals-and-testing.md.
• Caching for true determinism. Where identical output for identical input is a hard requirement (billing, compliance, snapshot tests), cache responses keyed on the request hash — the only real guarantee on hosted APIs. Self-hosting with batch-invariant kernels and pinned versions is the heavyweight alternative (and what reproducible RL training uses).
• Temperature interacts with retries and JSON. Retrying a failed parse at T=0 often re-fails identically; a retry with slight temperature or a modified prompt escapes the loop. Conversely, high temperature raises schema-violation and hallucination rates — keep extraction paths cold.
• Logprobs as a signal. Several APIs return token logprobs; low top-token probability on a classification answer is a usable, if imperfect, confidence proxy for routing to a human or a bigger model.
• Observability. Log sampling parameters with each trace so regressions after a "minor" config change are diagnosable (observability.md).

Test yourself

Go deeper

• Defeating Nondeterminism in LLM Inference — Thinking Machines Lab — the definitive writeup: batch-invariance as the real culprit, with kernel-level fixes; basis for deterministic modes in vLLM/SGLang.
• Deep Dive into LLMs like ChatGPT — Andrej Karpathy — sampling, inference, and model psychology inside a full-stack walkthrough.
• But what is a GPT? Visual intro to transformers — 3Blue1Brown — the final section visualizes softmax-with-temperature over logits.
• OpenAI docs: Structured Outputs — the correctness-over-determinism pattern in practice: constrain the output instead of chasing reproducibility.

Where this connects

• Structured Output — constrain tokens with grammars so correctness does not depend on sampling luck.
• Evals & Testing — how to measure quality under non-determinism without fooling yourself.
• LLM APIs — where sampling parameters sit on the request, and how seeds/fingerprints are exposed.
• How LLMs Work — logits come from the final unembedding; sampling is the last mile of each decode step.