Latency

The intuition

Waiting for an LLM feels like waiting for a courier who first has to read your entire letter before they write a single word of the reply. That reading phase is prefill: the model must process every input token before it can emit anything. The writing phase is decode: one word (token) at a time, at a steady pace, until the answer is finished.

If the courier starts speaking as soon as the first sentence is ready, you feel the conversation is underway even though the full reply is still minutes away. That is streaming. If they wait until the whole letter is done, you stare at a blank screen for six seconds and abandon the product.

Two numbers therefore matter, not one: time to first token (TTFT) — when the courier starts talking — and total generation time — when they finish. Optimizing the wrong number is the classic self-inflicted latency bug.

Why it exists

LLM products fail the user's patience budget in ways classical APIs do not. The hard constraints:

1. Generation is serial by nature. Decode emits one token per forward pass. A 500-token answer at 100 tok/s is five seconds of wall clock no matter how fast the rest of your stack is.
2. Prefill grows with everything you stuff in the prompt. System instructions, tool schemas, retrieved chunks, and chat history all lengthen TTFT. A 100k-token context is not free real estate.
3. A RAG or agent request is a pipeline, not a single call. Embed → retrieve → optional rerank → generate. Sequential awaits on independent stages add up before the model even starts.
4. Users abandon blank UIs. Hundreds of milliseconds of silence after a click feels broken; progressive tokens feel alive even when total time is long.

The alternatives lose in practice. Ignoring latency until launch produces a product people try once. Optimizing only total time with no streaming still feels slow. Throwing a faster frontier model at everything costs 5–25× more than routing easy work to a small model. Serializing every retrieval and tool call is the most common self-inflicted multi-second tax.

Latency engineering exists so you measure the ledger, optimize the dominant stage, and make waiting feel shorter — not so you micro-tune the vector index while decode still owns 80% of the request.

The core idea

LLM latency has two distinct numbers, and conflating them is the classic junior mistake. Time to first token (TTFT) is how long until the model starts responding — it covers network, queueing, and prefill (processing the entire input prompt). Total generation time adds decode: producing output tokens one at a time, typically 50–200 tokens/second on hosted APIs. A 500-token answer at 100 tok/s takes 5 seconds of decode no matter how fast prefill was — so output length, not input length, usually dominates total time.

Streaming changes perceived latency without changing actual latency: if TTFT is 800ms and users read as tokens arrive, the app feels sub-second even though full completion takes 6 seconds. That's why chat UIs stream by default, and why the latency SLO for a streaming product should be on TTFT (and time-to-first-useful-token), not on total time.

For a RAG or agent request, the LLM call is only one line in a latency ledger: embed the query, retrieve, optionally rerank, then prefill and decode. Each stage is a lever. The big four optimizations: parallelize everything without a data dependency (multi-index retrieval, independent tool calls), cache (prompt caching for repeated prefixes, semantic caching for repeated questions), route easy queries to a smaller, faster model, and generate fewer tokens (concise-output instructions, lower max_tokens, structured output instead of prose).

How it actually works

Prefill and decode have different physics. Prefill processes all input tokens in parallel and is compute-bound — TTFT grows roughly linearly with prompt length (another reason 100k-token contexts hurt). Decode generates autoregressively, one forward pass per token, and is memory-bandwidth-bound: the bottleneck is loading model weights and KV cache from GPU memory each step. This is why serving stacks like vLLM (PagedAttention, continuous batching) target decode throughput, and why output tokens cost more than input tokens on every provider.

The latency ledger of a typical RAG request:

Decode is usually 60–90% of the total — measure your own ledger before optimizing; teams routinely tune the vector DB while decode dominates. See the RAG pipeline for how those stages compose end to end, and cost for why the same levers often cut spend.

Parallel retrieval. Fan out concurrently: dense + keyword search in hybrid retrieval, multiple collections, query expansion variants — then merge. Same for agents: independent tool calls should execute concurrently (most SDKs support parallel tool use). Sequential awaits in a pipeline are the most common self-inflicted latency bug. Mechanics of hybrid retrieval: hybrid search; tool concurrency: tool calling.

Prompt caching (provider-side) reuses the computed KV cache for a repeated prompt prefix — system prompt, tool definitions, large documents. It cuts both TTFT and input cost: Anthropic cache reads bill at 0.1x input price; OpenAI caches automatically for prompts ≥1024 tokens. Requires prefix stability — put static content first, variable content last. Full economics: cost.

Semantic caching (application-side) is different: it caches final responses, keyed by query embedding. On a new query, embed it, search cached entries, and if similarity clears a threshold, return the stored answer with zero LLM calls — milliseconds instead of seconds. The threshold is the whole game: ~0.92–0.95 works for FAQ-style traffic; too permissive and "cancel my order" matches "track my order" — a wrong answer served confidently. Invalidation risks: underlying data changes (stale answers need TTLs tied to source freshness), and personalized or time-sensitive queries must bypass the cache entirely (scope cache keys by user/tenant where answers differ).

Model routing. A small model (Haiku-class) answers easy queries at ~5x lower cost and noticeably lower TTFT and higher tok/s; a cheap classifier (or the small model itself with an escalation instruction) routes hard queries to the large model. Cascade variant: try small first, escalate on low confidence or failed validation — trades a second call on hard queries for savings on easy ones. Landscape context: choosing models.

Speculative decoding (mention-level): a small draft model proposes several tokens; the large model verifies them in one parallel pass, accepting matches. Output is identical to the large model alone, but decode gets 2–3x faster. It's how providers ship "fast modes" — as an API consumer you select it, not build it.

The flows

A worked example

Trace one request through the governed enterprise platform. An employee asks:

"What is the approval threshold for international travel under the current expense policy?"

Without optimizations (naive serial path):

With the latency levers applied:

If the same question was asked earlier in the session and semantic cache similarity is 0.96 against a stored answer (threshold 0.93), the path becomes: embed (~40 ms) → cache hit → return in under 100 ms with zero generation.

Common drill-downs

Your RAG endpoint takes 8 seconds. Walk me through diagnosing it. Instrument per-stage spans, get the ledger. Typical finding: decode dominates (long answers) → cap/condense output, stream, smaller model. If TTFT dominates: huge prompt → prompt caching, trim retrieved context, fewer chunks. If retrieval dominates: sequential calls that should be parallel, or an oversized rerank set.

What's the difference between TTFT and total latency, and which do you optimize for a chatbot? TTFT = prefill + queueing until first token; total adds decode at ~50–200 tok/s. For a streaming chat UI, optimize TTFT — users read along, so perceived latency ≈ TTFT. For machine-consumed output (JSON into a pipeline), only total time matters.

How does semantic caching differ from prompt caching? Prompt caching is provider-side KV-cache reuse of an identical prompt prefix — the model still generates fresh output; saves prefill time and input cost. Semantic caching is app-side reuse of a full response for a semantically similar query — skips the LLM entirely; risks wrong-answer hits and staleness, so it needs threshold tuning, TTLs, and bypass rules.

When does a semantic cache hurt you? Personalized answers (same question, different user, different truth), time-sensitive data, low-traffic long-tail queries (hit rate too low to justify complexity), and thresholds tuned too loose — a wrong cached answer is worse than a slow correct one.

How would you cut latency without changing models? Parallelize independent I/O; prompt-cache static prefix; trim retrieved context to what's needed; instruct concise output and cap max_tokens; stream; semantic-cache the head of the query distribution; move non-blocking work (logging, judge scoring, summarization) off the request path.

What is speculative decoding, one level deep? Draft model emits k cheap tokens; target model validates all k in one parallel forward pass, accepting the longest matching prefix. Acceptance rate determines speedup (2–3x typical); output distribution is provably unchanged. Consumers get it via provider fast modes rather than implementing it.

Why are output tokens slower (and pricier) than input tokens? Input is processed in one parallel, compute-bound prefill pass. Output is sequential — one memory-bandwidth-bound forward pass per token. Serial generation consumes far more accelerator time per token, which pricing reflects (output commonly ~5x input).

Production concerns

• Tail latency. Report p95/p99, not averages — LLM APIs have fat tails (queueing, long generations, provider load). Cap the tail with per-stage timeouts and a max_tokens ceiling; an unbounded generation is an unbounded SLO. Pair with reliability timeouts and circuit breakers.
• Streaming infrastructure. SSE/streamed responses must flow through every hop — a buffering proxy or serverless gateway that waits for the full body silently destroys the TTFT win. Structured output (JSON) can't be shown token-by-token; stream field-by-field with partial parsing, or show progress states. See structured output and LLM APIs.
• Semantic cache failure mode is correctness, not staleness only: a false-positive cache hit returns the wrong answer with full confidence. Log cache hits with similarity scores, sample-audit them, and start with a conservative threshold. Join hits to traces in observability.
• Provider variance. TTFT and tok/s vary by provider load and region and shift with model updates — track them as first-class metrics per model, and route or fall back when a provider degrades (reliability).
• Cost coupling. Most latency levers (routing, caching, shorter outputs) also cut cost; the exception is redundancy tricks like hedged/parallel duplicate requests, which buy tail latency with money (cost).
• Perceived latency beyond streaming: optimistic UI states ("Searching your documents…"), retrieval results shown before generation, skeleton states. Users tolerate seconds when something visibly progresses.

Test yourself

Go deeper

• Latency optimization — OpenAI docs — the seven principles: fewer tokens, fewer requests, parallelize, make waiting feel shorter.
• Prompt caching — Anthropic docs — mechanics, minimum cacheable lengths, TTLs, cache-warming with max_tokens: 0.
• Redis LangCache docs — managed semantic caching: thresholds, TTLs, eviction, hit-rate monitoring.
• LLM Inference Performance Engineering: Best Practices — Databricks — TTFT/TPOT, batching, why decode is memory-bandwidth-bound.
• Fast LLM Serving with vLLM and PagedAttention — Anyscale — the canonical talk on PagedAttention and continuous batching, by vLLM's authors.

Where this connects

• Cost — the same levers (caching, routing, shorter outputs) usually cut spend; learn the unit economics next.
• Reliability — timeouts, retries, and fallbacks are how you cap the latency tail when a provider stalls.
• Observability — per-stage spans and TTFT metrics are how the ledger becomes visible in production.
• Choosing models — routing and cascade only work if you know which models are fast enough for each tier.