Reliability

The intuition

Calling a hosted LLM is like ordering from a kitchen you do not own during dinner rush. The dish usually arrives. Sometimes the ticket is rejected (your fault — fix the order). Sometimes the kitchen is overloaded (their fault — wait and try again). Sometimes the waiter never returns with the first course (timeout mid-stream). Sometimes a second kitchen can cook something close — but only if you already translated the recipe and tested it cold.

Reliability engineering is that rulebook: retry or not, how long to wait, when to switch providers, and what to serve if every kitchen fails. Classical patterns still apply; the twists are expensive tokens, streaming timeouts, semantic failures that return HTTP 200, and agent actions that must not run twice.

Why it exists

An LLM API call is a slow, expensive, rate-limited dependency you do not control. Transient failures are normal at scale — 429s, 5xx, Anthropic's 529, network resets — and without retries availability collapses on noise. Blind retries are worse: deterministic 4xx burn full cost on identical failures, side-effecting tools double-execute, and layered loops (SDK × wrapper × caller) become self-inflicted DDoS.

A single global timeout is the wrong shape. Embedding needs seconds; generation needs tens of seconds; streaming needs a time-to-first-token (TTFT) budget and an inter-token idle timeout, not only a total cap. Quality degrades before availability does — HTTP 200 with bad JSON or a wrong answer is still a product failure. Fallback without rehearsal is a second outage: an unevaluated cross-provider path swaps availability pain for a silent quality incident.

The alternatives lose. No retries fails open on blips. Infinite retries burns money and worsens overload. No breaker means every request waits out timeout × attempts during an incident. Hard 500s throw away partial answers and human handoff. Reliability exists so systems stay available and safe when the model, network, or tool plane misbehaves — by design, not hope.

The core idea

LLM reliability is a layered policy stack. Each layer answers a different question; stacking them keeps a blip from becoming an outage and an outage from becoming a cascade.

Classify first. Transient errors (429, 5xx, 529, network timeouts) may retry. Deterministic client errors (400, 401, 403, 413) must not. Valid HTTP with invalid content is a third class: a semantic retry that feeds the validation error back into the next prompt, counted and capped separately from transport retries.

Then bound the damage. Retries use exponential backoff with full jitter, honor retry-after, and live under a retry budget. Timeouts are per stage, with separate TTFT and inter-token idle limits for streams. When failure rate stays high, a circuit breaker per provider+model fails fast into an evaluated fallback chain (same-provider smaller model, then cross-provider). If every model path fails, a degradation ladder still returns something useful. When agents take actions, idempotency separates "safe to re-call the model" from "safe to re-execute the tool."

How it actually works

Retry classification is the first gate:

The last row is LLM-specific — still HTTP 200. Handle it above transport retries; prefer constrained decoding so you need this loop less often (structured output).

Backoff and budget. Full jitter: sleep = random(0, min(cap, base × 2^attempt)). Without jitter, clients that failed together retry together and re-crush a recovering service (Marc Brooker / AWS). Cap 2–3 attempts for interactive traffic. Own a retry budget: if ~10–20% of traffic is already retrying, stop and fail fast or fall back. Own retries at exactly one layer — OpenAI's official SDKs already retry rate limits and honor Retry-After; wrapping them multiplies attempts.

Timeouts per stage. Illustrative: embedding 2–5 s, vector search 1–2 s, rerank ~5 s, generation 30–120 s. Streaming needs a TTFT timeout (~10 s) and an inter-token idle timeout — better than a long total cap alone. Every timeout maps to retry, fallback, or degrade (latency).

Circuit breaker (per provider+model). Closed: traffic flows, failures counted in a sliding window. Open (e.g. >50% failures in 30 s): fail fast to fallback. Half-open: probe a tiny fraction for recovery. Without a breaker, every request burns full timeout + retries during an incident.

Fallback and portability. Chain: primary → same-provider cheaper model → second provider. Cross-provider needs prompt and tool-schema translation, warm keys/quota, and its own eval suite run before the outage. Monitor fallback traffic quality separately (evals and testing, choosing models).

Idempotency for agent actions. The LLM call is stateless — safe to retry. Tool side effects are not. Generate an idempotency key per intended action (request ID + step), pass it to APIs that support it, and dedupe otherwise. The dangerous window is timeout after commit but before ack. Consequential actions are idempotent, human-confirmed, or logged for reconciliation (tool calling, agent reliability).

Client-side rate limits. Anthropic: RPM + ITPM / OTPM via token bucket (anthropic-ratelimit-*, retry-after on 429). For most Claude models, cache-read tokens do not count toward ITPM, so prompt caching raises effective throughput. OpenAI: RPM/TPM (often RPD/TPD) with x-ratelimit-* and Retry-After. Throttle client-side below the published limit, prioritize interactive over batch, move offline work to the Batch API, and shed from remaining-capacity headers before you hit 429 (cost).

The flows

A worked example

Trace a grounded answer on the governed enterprise platform when the primary model starts failing.

Happy path (illustrative stage budgets): embed 3 s (retry once), retrieve 2 s (retry once), rerank 5 s (skip on timeout — use raw top-5), stream TTFT 10 s (retry once then fallback), inter-token idle 8 s (cancel; retry or fallback), total generation cap 90 s (cancel; degrade).

Incident — primary 503 rate hits 60% over 30 s. Per-request retries (max 2, full jitter) absorb the first blips. When the sliding window crosses 50%, the circuit opens for providerA/model-large. New traffic fails fast to a same-provider smaller model with a pre-evaluated prompt. If that trips too, continue to provider B — only because keys are warm and the fallback suite already ran offline. If every model path fails: semantic-cache answer → model-only with an explicit "couldn't access the primary model" disclaimer → static help + human ticket. Half-open probes close the breaker on recovery.

Agent side effect. The model emits send_email with key req_9f3a:step_4. A timeout hits after send but before the tool result. The orchestrator retries the LLM observation with the recorded success, not a second bare send. Without the key, the customer gets two emails.

Production concerns

Retry storms are self-inflicted DDoS. SDK × wrapper × caller = 3×3×3 = 27 attempts. Own retries at one layer and enforce a global retry budget so a partial outage cannot convert most traffic into retries.

Reliability has a bill. Every retry and every hedged request (delayed backup; take first success) is billed. Hedging can cut p99 when stragglers dominate, but multiplies spend — a deliberate cost trade. Prefer streaming + idle timeouts over long total caps that burn most of a generation before retrying (cost, latency).

Degradation is a product decision. Design the ladder in advance: full pipeline → skip reranker → cache → model-only with disclaimer → static help; for agents, bound iterations then hand off to a human (production RAG, agent reliability).

Queues need limits. Queueing beats dropping for spikes, but depth limits and staleness cutoffs matter — a chat answer four minutes late is a failure that also cost money. Shed low-criticality load first (Google SRE practice).

Drill fallbacks. An unexercised path is broken by default: expired keys, drifted prompts, missing evals. Force the breaker open in staging (or a small prod slice) on a schedule.

Multi-tenant fairness. One heavy tenant can exhaust org-level provider quota. Per-tenant client-side quotas; cost attribution detects the abuser before the 429 storm.

Common drill-downs

Why full jitter? Clients that failed together otherwise retry together and re-overload recovery. Full jitter desynchronizes them.

When is a breaker better than retries? Retries handle blips; breakers handle sustained failure. During an outage, retries only add load and latency. Use both: retries inside, breaker around.

Cross-provider fallback without a quality incident? Translated prompts and tool schemas, warm credentials, a pre-run eval suite, and separate monitoring. Untested fallback is a latent incident.

Double email from an agent? Timeout after tool commit, before ack, then a blind retry. Idempotency keys, tool-layer dedupe, retries around the LLM state machine — not around executed actions.

Test yourself

Go deeper

• Exponential Backoff and Jitter — Marc Brooker, AWS Architecture Blog — the canonical simulation-backed analysis; full jitter wins.
• Handling Overload — Google SRE Book — client-side adaptive throttling, criticality, load shedding.
• Rate limits — Anthropic docs — RPM/ITPM/OTPM token-bucket limits, cache-aware ITPM, retry-after, response headers.
• Rate limits — OpenAI docs — RPM/TPM, headers, SDK retries that honor Retry-After, Batch API as pressure relief.

Where this connects

• Latency — stage and streaming idle timeouts enforce latency SLOs under failure.
• Cost — retries, hedges, and fallbacks spend money; budgets are reliability controls too.
• Observability — error, timeout, and fallback rates trip breakers and pages.
• Evals & testing — fallback paths need offline suites before an outage exercises them.