Sampling & Determinism

The intuition

After the transformer runs, it does not emit a sentence. It emits a scorecard — one raw score (logit) per vocabulary token. Softmax turns that scorecard into a probability pie. Sampling is how you cut the pie: always take the largest slice (greedy decoding), or spin a weighted wheel, optionally after throwing away tiny slices or reshaping the pie.

Temperature is a dimmer switch on confidence, not a creativity gland. Turn it down and the biggest slices dominate; turn it up and the long tail gets more mass. The trap is treating the dimmer as a lock. Temperature 0 makes the wheel deterministic — always pick the current argmax — but the scorecard itself can still twitch between runs on real GPU servers. One flipped token, then autoregression does the rest: a different novel.

Why it exists

If every step always took the single most likely token, extraction would often work — and anything that needs variety would fail. Classification wants conservatism; assistants and brainstorming need mass on non-argmax tokens; unrestricted high-temperature sampling drifts into incoherence, so top-p / top-k / min-p cut the pathological tail. Hosts also batch for economics, which forces numerical paths that are not identical run-to-run.

Alternatives lose differently. Always greedy loops and dulls creative work. Always high temperature unrestricted raises garbage and schema failures. Pretending T=0 is a unit-test oracle produces flaky CI. Seeds alone fix the RNG after logits exist, not the logits under variable batching.

On the governed enterprise platform, extraction routes stay cold and schema-constrained; brainstorming runs warmer with nucleus sampling (top-p); compliance paths cache answers rather than depending on bit-identical regeneration.

The core idea

Every generation step ends with one logit per vocabulary token. Softmax turns logits into probabilities. The sampling strategy decides which token to take.

Temperature divides logits before softmax: p_i = softmax(logits / T). T < 1 sharpens toward the top tokens; T > 1 flattens; T → 0 approaches greedy. It rescales relative confidence — it does not add knowledge. Top-k keeps only the k highest-scoring tokens and renormalizes — a fixed-size shortlist. Top-p (nucleus sampling) keeps the smallest set whose cumulative probability is at least p — adaptive where top-k is fixed: two tokens when the model is sure, hundreds when it is torn. Min-p (common in open-source stacks) keeps tokens above a fraction of the max probability, so the floor rises with confidence.

Typical stacks scale logits by temperature, apply top-k / top-p / min-p, then sample (or argmax near T=0). Warper order varies by library. Tune temperature or top-p hard, not both — they both move mass into the tail. Illustrative ranges: extraction / code when knobs exist — T ≈ 0–0.3; assistant — T ≈ 0.7–1.0; brainstorming — T ≈ 1.0+ with top-p ≈ 0.9–0.95. Some reasoning-tier models (notably Anthropic's current Opus-class endpoints) reject those parameters and expect prompting and effort controls instead.

The trap remains: temperature 0 is still not fully deterministic on multi-tenant APIs. Greedy decoding is deterministic given identical logits. The logits are what still move.

How it actually works

From logits to a token, and why two identical HTTP requests can diverge:

Floating-point non-associativity is the underlying math: (a + b) + c ≠ a + (b + c) in finite precision. That alone does not explain API nondeterminism. Thinking Machines Lab's 2025 analysis showed the usual "GPU concurrency makes every matmul random" story is incomplete: forward-pass kernels are typically run-to-run deterministic for a fixed batch. What they lack is batch invariance — the same request alone vs co-batched can take different reduction strategies (split-K matmul, different attention splits), so logits change. Server load sets batch size; from your seat the endpoint is load-dependent. Near ties make this product-visible: a 10^{-5} nudge swaps argmax, and autoregression turns that into a different string.

Mixture-of-Experts (MoE) routing can amplify this when expert choice depends on batch composition. Mundane sources remain: silent alias updates, mixed GPU types, speculative-decoding edge cases. Seeds fix the RNG after logits exist — not the logits. OpenAI documents seed as best-effort "mostly" deterministic and exposes system_fingerprint for backend changes. Frequency / presence penalties (open-source: repetition penalty) down-weight already-emitted tokens when low-T decoding loops.

The flows

A worked example

The gateway on the governed enterprise platform classifies an inbound email into refund, password_reset, address_change, or other.

Illustrative single-step distribution over the label token (real models often emit multi-token labels):

At T=0.2 the classifier almost always emits refund. At T=1.5, refund vs other is a coin-flip neighborhood — bad for routing. Top-p = 0.9 at T=1.0 would keep mostly {refund, other} and drop the long tail.

Now run the same request twice at T=0 on a busy multi-tenant endpoint. Batch A groups it with short prompts; batch B with long multimodal jobs. Non-invariant matmuls nudge logits so refund and other swap by 10^{-5}. Greedy flips; the ticket goes to the wrong queue. Temperature did not cause it — serving numerics plus a near tie did.

Mitigations that work: constrain the label with structured output (enum schema), validate, and if identical re-answers matter, cache on a content hash. Optionally read logprobs: if the top two are within a tight margin, escalate.

Production concerns

Do not build correctness on determinism. If a pipeline breaks when the model phrases something differently, fix it with structured outputs and validation — not T=0. High T on extraction invents enum values; pure T=0 retries of an unchanged prompt often re-fail identically (feed the validation error back, or change the prompt); greedy loops without penalties burn max_tokens on repeated politeness.

Evals must account for variance. A/B-ing prompts on one run each measures noise — T=0 golden files flake with load too. Run multiple samples per case and compare pass rates; see Evals & Testing.

Where identical output is a hard requirement (billing, compliance, snapshots), cache by request hash — the practical guarantee on hosted APIs. Self-hosting with batch-invariant kernels (vLLM's VLLM_BATCH_INVARIANT=1, similar modes in SGLang) plus pinned weights is the heavyweight path, at a throughput cost. Token logprobs, when available, are a rough confidence signal. Log sampling parameters on every trace (observability).

Common drill-downs

Top-p vs top-k? Top-k: fixed count. Top-p: probability-mass budget that shrinks when the model is confident and expands when uncertain. Top-p is usually better for assistants; top-k is a hard engineering cap.

How do you get reproducible outputs? Cache by request hash first. Then pin snapshots, use seeds + fingerprints where offered, or self-host with batch-invariant inference. Hosted determinism is best-effort.

When would you raise temperature above 1? Diversity-first work: brainstorming, synthetic data, best-of-N. Pair with top-p or min-p so the flattened tail does not dominate.

Model outputs loop — what do you reach for? Frequency/presence (or repetition) penalties; slight temperature increase; check for degenerate greedy decoding at T=0.

Test yourself

Go deeper

• Defeating Nondeterminism in LLM Inference — Thinking Machines Lab — batch-invariance as the real culprit; basis for deterministic modes in vLLM/SGLang.
• Batch invariance — vLLM docs — enabling VLLM_BATCH_INVARIANT for load-independent self-hosted outputs.
• The Curious Case of Neural Text Degeneration (Holtzman et al., 2019) — the nucleus sampling paper: why fixed top-k and pure sampling fail.
• Reproducible outputs with the seed parameter — OpenAI Cookbook — seed + system_fingerprint as best-effort, not a guarantee.
• OpenAI docs: Structured Outputs — constrain tokens instead of chasing bit-identical free text.

Where this connects

• Structured Output — constrain tokens with grammars so correctness does not depend on sampling luck.
• Evals & Testing — measure quality under non-determinism without fooling yourself.
• LLM APIs — where sampling parameters sit on the request, and how seeds/fingerprints are exposed.
• How LLMs Work — logits come from the final unembedding; sampling is the last mile of each decode step.