Reliability

The intuition

Calling a hosted LLM is like ordering from a kitchen you do not own, during dinner rush, with a menu that sometimes changes without notice. The dish usually arrives. Sometimes the ticket is rejected (your fault — bad order). Sometimes the kitchen is overloaded (their fault — try again later). Sometimes the waiter never returns with the first course (timeout mid-stream). And sometimes a second kitchen across town can cook a similar dish — but only if you already translated the recipe and tested it cold.

Reliability engineering for LLMs is the set of rules that decide: retry or not, how long to wait, when to switch kitchens, and what to serve if every kitchen fails. Classical distributed-systems patterns apply; the twists are expensive tokens, streaming timeouts, semantic failures that return HTTP 200, and agent actions that must not run twice.

Why it exists

An LLM API call is a slow, expensive, rate-limited call to a dependency you do not control. That creates hard constraints:

1. Transient failures are normal. 429 rate limits, 5xx, provider overload (e.g. Anthropic 529), and network resets happen at scale. Without disciplined retries, availability collapses on noise.
2. Blind retries are dangerous. Deterministic 4xx failures waste full cost on identical failures. Side-effecting tool calls double-execute. Layered retry loops (SDK × wrapper × caller) become self-inflicted DDoS.
3. A single global timeout is the wrong shape. Embedding needs seconds; generation needs tens of seconds; streaming needs TTFT and inter-token idle timeouts, not only a total cap.
4. Quality degrades before availability does. The model can return 200 with bad JSON, a refusal, or a wrong answer. Transport success is not task success.
5. Fallback without rehearsal is a second outage. Cross-provider prompts do not port cleanly; an unevaluated fallback path swaps an availability incident for a silent quality incident.

The alternatives lose. No retries fails open on blips. Infinite retries burns money and worsens overload. One provider, no breaker means every request waits out full timeout × attempts during an incident. Hard 500s with no degradation throw away the fact that a partial answer or human handoff is often better than nothing.

Reliability exists so user-facing systems stay available and safe when the model tier, the network, or the agent tool plane misbehaves — by design, not by hope.

The core idea

An LLM API call is a slow, expensive, rate-limited call to a dependency you don't control — so every classical distributed-systems pattern applies, with LLM-specific twists. Retries: only for transient errors — 429 (rate limit), 5xx, and Anthropic's 529 (overloaded), plus network timeouts — with exponential backoff and jitter, honoring the retry-after header. Never retry 4xx like 400/401/413: the request is deterministically bad and will fail identically at full cost. Timeouts per pipeline stage, not one global one — an embedding call gets seconds, a generation gets tens of seconds, and with streaming you time-out on time-to-first-token and inter-token gaps rather than total duration. Fallback chains: same provider smaller model, then a second provider — with the caveat that prompts don't port cleanly across models, so a fallback path you've never evaluated is a silent quality incident waiting for an outage to trigger it. Circuit breakers stop hammering a provider mid-incident and make fallback automatic. Idempotency matters the moment agents take actions: retrying a generation is safe; retrying an executed tool call ("send email", "issue refund") is not, so side-effecting tools need idempotency keys. And because LLM quality degrades gracefully by nature, design graceful degradation deliberately: RAG falls back to a no-context answer with a disclaimer, an agent falls back to a human handoff — a degraded answer beats a 500.

How it actually works

Retry classification:

That last row is LLM-specific: a schema-invalid output or an unwarranted refusal is a 200 at the HTTP layer. Handle it above transport retries — feed the validation error back ("your output failed schema X, correct it"), and count these separately in metrics. Pairs with structured output.

Backoff: exponential with full jitter (sleep = random(0, min(cap, base × 2^attempt))). Jitter is not optional garnish — synchronized clients retrying on the same schedule create waves that re-overload a recovering service (Marc Brooker's AWS analysis shows jitter dramatically cuts both total work and time-to-recovery). Cap attempts (2–3 for interactive traffic; more for batch), and set a retry budget — if 20% of traffic is retrying, retries are now the problem.

Timeouts per stage: embedding 2–5s, vector search 1–2s, rerank ~5s, generation 30–120s depending on expected output. For streaming: TTFT timeout (~10s) catches a hung request early, and an inter-token idle timeout catches mid-stream stalls — far better than waiting out a 120s total timeout. Every timeout maps to a defined action: retry, fall back, or degrade — never a bare exception to the user. See latency.

Provider fallback and the portability caveat. Chain: primary model → cheaper same-provider model → second provider. Same-provider fallback is easy (shared API shape, shared prompt conventions). Cross-provider fallback needs: prompt translation (system-prompt conventions, tool-calling formats differ), separate keys/quota you actually pay to keep warm, and — critically — its own eval run. A prompt tuned for model A can drop 20 quality points on model B; run your eval suite against the fallback path before the outage, and monitor fallback traffic quality separately (evals and testing, choosing models).

Circuit breaker: per provider+model, classic three states. Closed: traffic flows, failures counted in a sliding window. Open (failure rate over threshold, e.g. >50% in 30s): fail fast to fallback without paying the timeout each time. Half-open: probe requests test recovery. Without a breaker, a provider incident means every request burns its full timeout + retries before failing — latency amplification across your whole service. The breaker turns a provider outage into an instant, automatic reroute.

Idempotency for agent actions. The LLM call itself is stateless — safe to retry. Tool executions with side effects are not. Mechanics: generate an idempotency key per intended action (derived from request ID + step), pass it to downstream APIs that support it (payments APIs do), and dedupe in the tool layer for those that don't. The dangerous window: timeout after the action executed but before the response arrived — a naive retry double-executes. Rule: consequential actions (spend money, send messages, mutate records) are idempotent, confirmed by a human, or at minimum logged for reconciliation. See tool calling and agent reliability.

Rate-limit management (client side). Providers enforce RPM plus token-per-minute limits (Anthropic: RPM/ITPM/OTPM via token bucket, with anthropic-ratelimit-* response headers; OpenAI: RPM/TPM with x-ratelimit-* headers). Don't discover limits via 429s: throttle client-side with your own token bucket set below the provider limit, queue excess with backpressure, and prioritize — interactive traffic preempts background jobs; batch workloads go to the Batch API, which has separate limits entirely. On Anthropic, cache reads don't count toward ITPM, so prompt caching directly raises effective throughput (cost). Watch the remaining-capacity headers and shed load before the 429.

The flows

A worked example

Trace a grounded answer request on the governed enterprise platform when the primary model provider starts failing.

Happy path (steady state):

Incident path — primary model 503 rate hits 60% over 30 s:

1. Per-request retries (max 2, full jitter, honor retry-after) absorb the first blips.
2. Failure rate in the sliding window crosses 50% → circuit opens for providerA/model-large.
3. New traffic fails fast (milliseconds) to same-provider smaller model with a pre-translated prompt and its own eval baseline.
4. If the small model also trips, chain continues to provider B equivalent — only because keys are warm and the fallback suite already ran offline.
5. If all model paths fail, RAG degradation ladder: semantic-cache answer if present → model-only answer with "I couldn't access the primary model; treating this as general guidance…" → static help + human ticket link.
6. Half-open probes every N seconds send a tiny fraction back to primary; success closes the breaker.

Agent side-effect path: the model emits send_email with idempotency key req_9f3a:step_4. The mail API accepts the key. A timeout occurs after send but before the tool result returns. The orchestrator retries the LLM observation path with the recorded success, not a second bare send_email. Without the key, the customer receives two emails — the classic reliability bug that looks like "the model went wrong" but is really retry semantics.

Common drill-downs

Which HTTP errors do you retry on an LLM API, and how? 429/5xx/529 and timeouts: exponential backoff, full jitter, respect retry-after, cap 2–3 attempts with a global retry budget. 400/401/403/413: never — deterministic failures. Invalid-but-200 outputs (bad JSON): semantic retry with the validation error fed back, capped separately.

Why jitter? Without it, clients that failed together retry together — synchronized waves that keep re-overloading a recovering service. Full jitter (uniform random up to the exponential cap) desynchronizes retries, reducing both aggregate load and time-to-recovery.

Design fallback for a provider outage. Circuit breaker per provider+model detects the incident and fails fast. Chain: primary → same-provider cheaper model → cross-provider equivalent. Cross-provider requires translated prompts/tool schemas, warm credentials, and a pre-run eval suite — otherwise you've swapped an availability incident for a quality incident. Tag and monitor fallback traffic separately.

Your agent sent a customer email twice. What went wrong and how do you prevent it? A retry re-executed a side-effecting tool call — likely a timeout after execution but before acknowledgment. Fix: idempotency keys on every consequential action, deduplication in the tool layer, and retries only around the LLM call, never blindly around executed actions. For high-stakes actions, add human confirmation.

How do you handle provider rate limits at scale? Client-side throttling below the published limit (token bucket on both requests and tokens), priority queueing with backpressure, prompt caching to shrink counted tokens (Anthropic excludes cache reads from ITPM), Batch API for async work (separate limits), and monitoring the rate-limit response headers to shed load pre-429. 429s should be the backstop, not the mechanism.

Timeouts for a streaming LLM response? Three: connection/TTFT timeout (~10s — catches hung requests), inter-token idle timeout (catches mid-stream stalls), and a generous total cap. This fails fast on genuine hangs without killing legitimately long generations.

When is a circuit breaker better than retries? Retries handle transient, isolated failures; breakers handle sustained dependency failure. During a real outage retries only add load and latency (every request waits out timeout × attempts). The breaker converts that to instant failover, then probes for recovery. Use both: retries inside, breaker around.

Production concerns

• Retry storms are self-inflicted DDoS. Layered retries (SDK × your wrapper × upstream caller) multiply: 3×3×3 = 27 attempts per user action. Own retries at exactly one layer; disable the SDK's or yours.
• Cost of reliability. Every retry and every hedged request is billed. A 120s generation that times out and retries costs double. Prefer streaming + idle timeouts (fail fast mid-stream) over long total timeouts (cost, latency).
• Degradation ladder, designed in advance. Example for RAG: full pipeline → skip reranker → cached/semantic-cache answer → model-only answer with "I couldn't access the knowledge base; based on general knowledge…" disclaimer → static help response. For agents: bounded iterations → summarize progress and hand off to human. Each rung is a product decision — make it explicitly, not at 3am. See production RAG and agent reliability.
• Queueing beats dropping for spiky traffic, but queues need depth limits and staleness cutoffs — a chat answer delivered 4 minutes late is a failure that also cost money. Shed low-priority load first (Google SRE: criticality-based load shedding).
• Fallback drills. An unexercised fallback path is broken by default: expired keys, drifted prompt versions, missing evals. Chaos-test it — force the breaker open in staging (or a small prod slice) monthly.
• Multi-tenant fairness: one heavy tenant can exhaust org-level provider quota for everyone. Per-tenant client-side quotas mirror the provider's global ones (cost attribution helps detect this).

Test yourself

Go deeper

• Exponential Backoff and Jitter — Marc Brooker, AWS Architecture Blog — the canonical simulation-backed analysis; full jitter wins.
• Handling Overload — Google SRE Book — client-side adaptive throttling, criticality, load shedding.
• Rate limits — Anthropic docs — RPM/ITPM/OTPM token-bucket limits, retry-after, cache-aware ITPM, response headers.
• Rate limits — OpenAI docs — RPM/TPM, headers, backoff implementations (tenacity/backoff), Batch API as pressure relief.

Where this connects

• Latency — stage timeouts and streaming idle timeouts are how you enforce latency SLOs under failure.
• Cost — retries, hedges, and fallbacks spend money; budgets and caps are reliability controls too.
• Observability — error, timeout, and fallback rates are the signals that trip breakers and pages.
• Evals & testing — fallback paths and model swaps need offline suites before an outage exercises them.