Latency

The intuition

Waiting for an LLM feels like waiting for a courier who must read your entire letter before writing a single word of the reply. That reading phase is prefill: the model processes every input token before it can emit anything. The writing phase is decode: one token at a time until the answer finishes.

If the courier starts speaking as soon as the first sentence is ready, the conversation feels underway even while the full reply continues. That is streaming. If they wait until the whole letter is done, you stare at a blank screen and leave. Two numbers therefore matter, not one: time to first token (TTFT) — when speech starts — and total generation time — when it ends.

Why it exists

LLM products break the user's patience budget in ways classical APIs do not. Decode is serial: one token per forward pass, so a 500-token answer at 100 tok/s is five seconds of wall clock no matter how fast the rest of the stack is. Prefill grows with everything you stuff in the prompt. A RAG or agent request is a pipeline, not a single call, and sequential awaits add up before the model even starts. Users abandon blank UIs; progressive tokens feel alive even when total time is long.

The alternatives lose. Ignoring latency until launch produces a product people try once. Optimizing only total time with no streaming still feels slow. Throwing a frontier model at everything costs many times more than routing easy work to a small model. Serializing every retrieval and tool call is the multi-second tax teams invent for themselves. Latency engineering exists so you measure the ledger, fix the dominant stage, and make waiting feel shorter.

The core idea

LLM serving has three operational metrics. TTFT is time from request until the first output token — network, queueing, and prefill. Time per output token (TPOT) (also called inter-token latency, ITL) is the gap between successive tokens during decode; inverse TPOT is the tokens-per-second the user feels after speech starts. Total latency is roughly TTFT + TPOT × output_tokens. Conflating TTFT with total time is the junior mistake: a 500-token answer at ~100 tok/s still costs about five seconds of decode even if prefill was instant. For end-to-end wall clock, output length usually dominates; input length mainly moves TTFT.

Streaming does not change actual total latency — only perceived latency. The SLO for a streaming product should be on TTFT (and time-to-first-useful-token). For machine-consumed JSON into a pipeline, only total time matters.

For a RAG or agent request the LLM call is only one line in a latency ledger. The big four levers: parallelize independent I/O, cache (prompt caching for repeated prefixes, semantic caching for repeated questions), route easy queries to a smaller model, and generate fewer tokens. Cutting half the output tokens often cuts roughly half the generation latency.

How it actually works

Prefill and decode have different physics. Prefill processes input tokens in parallel and is compute-bound — TTFT grows with prompt length. Decode is autoregressive, one forward pass per token, and is memory-bandwidth-bound: each step reloads model weights and the KV cache from GPU memory. Hosted APIs typically deliver tens to a couple of hundred tokens per second — measure your own. Serving stacks like vLLM (PagedAttention, continuous batching) target decode throughput; providers price output higher than input for the same reason.

A typical RAG ledger (illustrative ranges — build yours from traces):

Decode is often 60–90% of wall clock. See the RAG pipeline for stage composition and cost for spend coupling.

Parallel retrieval. Fan out concurrent work: dense + keyword, multiple collections, query-expansion variants — then merge. Agents should do the same for independent tool calls; sequential await in a loop is the common self-inflicted tax. Mechanics: hybrid search, tool calling.

Prompt caching (provider-side) reuses the computed KV cache for a repeated prompt prefix. Prefill shrinks, so TTFT drops and cached input bills cheaper (Anthropic cache reads about 0.1× base input; OpenAI caches stable prefixes automatically above a minimum length). Static content first, variable content last — a timestamp at the top of the system prompt kills every hit. See cost and LLM APIs.

Semantic caching (application-side) stores final responses, keyed by query embedding. On a hit above a similarity threshold you return the stored answer with zero LLM calls. The threshold is a precision/recall knob: too loose and "cancel my order" matches "track my order." Start conservative (often ~0.92–0.95 cosine for FAQ traffic), sample-audit near-threshold hits, bypass personalized or time-sensitive queries, scope keys by tenant, and tie TTLs to source freshness.

Model routing. A small model answers easy queries at lower cost and usually lower TTFT; escalate hard work via a classifier or a cascade. See choosing models. Speculative decoding (consumer-level): a draft model proposes tokens; the large model verifies them in one parallel pass — providers ship this as "fast modes."

The flows

A worked example

Trace one request through the governed enterprise platform. An employee asks: "What is the approval threshold for international travel under the current expense policy?"

Naive serial path (illustrative): embed 40 ms → dense 35 ms → BM25 30 ms (sequential) → rerank top-50 200 ms → prefill ~8k tokens 900 ms TTFT → decode ~400 tokens at ~100 tok/s 4 000 ms. Total ~5.2 s, first token ~1.2 s, blank UI until complete.

With levers: concurrent dense+BM25 (35 ms) → rerank top-20 keep 5 (90 ms) → prompt-cache hit (TTFT ≈ 350 ms) → concise instruction + max_tokens 200, ~180 tokens (~1 500 ms) → stream. Total ~2.0 s, perceived start ≈ 350–500 ms. A semantic-cache hit at 0.96 (threshold 0.93) collapses to embed + return under 100 ms.

Production concerns

Tail latency, not averages. Report p95/p99 for TTFT and total time — LLM APIs have fat tails from queueing, long generations, and provider load. Cap the tail with per-stage timeouts and a hard max_tokens ceiling. Streaming needs TTFT and inter-token idle timeouts. Pair with reliability.

Streaming infrastructure is a path property. SSE must flow through every hop. A buffering proxy, CDN, or serverless gateway that waits for the full body silently destroys the TTFT win. Structured JSON cannot paint token-by-token like prose — stream field-by-field with partial parsing, or show progress states. See structured output and LLM APIs.

Semantic cache failure mode is correctness. A false-positive hit returns the wrong answer with full confidence. Log hits with similarity scores, sample-audit near the threshold, and start conservative. Similarity matching fails on close-but-distinct intents ("credit card" vs "debit card"). Join hits to traces in observability. Track TTFT and tok/s per model and fall back when a provider degrades (reliability). Most latency levers also cut spend; hedged requests buy tail latency with money (cost).

Common drill-downs

Your RAG endpoint takes 8 seconds. Walk me through diagnosing it. Instrument per-stage spans and build the ledger. If decode dominates: cap or condense output, stream, consider a faster model. If TTFT dominates: prompt-cache the static prefix, trim retrieved context. If retrieval dominates: sequential calls that should be parallel, or an oversized rerank set.

How does semantic caching differ from prompt caching? Prompt caching is provider-side KV reuse of an identical prompt prefix — the model still generates; you save prefill time and input cost. Semantic caching is app-side reuse of a full response for a similar query — it skips the LLM entirely; wrong-answer hits need threshold, TTL, and bypass rules.

How would you cut latency without changing the frontier model for hard queries? Stream interactive paths; parallelize independent I/O; prompt-cache static prefix with variables last; trim context after rerank; concise output and hard max_tokens; route or cascade easy traffic; semantic-cache the FAQ head conservatively; move judges and logging off the request path.

Test yourself

Go deeper

• Latency optimization — OpenAI docs — seven principles from fewer tokens through parallelize and perceived wait.
• Prompt caching — Anthropic docs — mechanics, minimum lengths, TTLs, pre-warming with max_tokens: 0.
• Redis LangCache docs — managed semantic caching: thresholds, TTLs, hit-rate monitoring.
• LLM Inference Performance Engineering — Databricks — TTFT/TPOT, continuous batching, memory-bandwidth-bound decode.
• Fast LLM Serving with vLLM and PagedAttention — Anyscale — PagedAttention and continuous batching by vLLM's authors.

Where this connects

• Cost — the same levers (caching, routing, shorter outputs) usually cut spend.
• Reliability — timeouts, retries, and fallbacks cap the latency tail when a provider stalls.
• Observability — per-stage spans and TTFT metrics make the ledger visible in production.
• Choosing models — routing and cascade need a fast enough model for each tier.