Design: LLM Gateway

The problem

Design the internal platform through which every team at the company calls LLMs — or, equivalently: dozens of teams are calling OpenAI/Anthropic/Gemini directly with shared API keys; fix it.

This is platform engineering applied to LLM traffic: multi-tenancy, cost attribution, governance vs developer velocity, and the specific ways LLM traffic differs from normal API traffic (streaming, token-based cost, non-deterministic latency, provider volatility). The trap is designing a generic API gateway and ignoring everything token-shaped.

In the governed enterprise platform, this is the model gateway: fronting managed cloud model services so data stays inside the tenancy, access is via IAM, and quotas, audit, and routing wrap every call from RAG, agents, and batch jobs.

Why it is hard

1. Cost is token-shaped, not request-shaped. Ten huge-context calls can cost more than a thousand short ones. RPM-only limits fail; TPM and dollar budgets are first-class.
2. Streaming breaks naive gateway assumptions. Buffering a full response destroys TTFT UX; output tokens are not known pre-flight, so TPM enforcement is partly retrospective.
3. Providers are volatile. 429s, regional incidents, and model deprecations are normal. The gateway must be more available than any single provider and offer failover without app rewrites.
4. Governance fights velocity. Central control without an OpenAI-compatible facade and model aliases becomes a ticket queue that teams bypass with shadow keys.
5. Logging fights privacy. Full payload retention helps debugging and hurts legal; metadata + hashes + opt-in payloads is a deliberate policy, not an accident.
6. Buy vs build is real. LiteLLM and edge gateways exist; the org still owns identity, data-class policy, and showback — but reinventing the proxy core is rarely the hard part that matters.

Alternatives that lose: shared API keys in every repo (no attribution, no kill switch), per-team direct provider access (no org failover or audit), and a heavy enterprise ESB that adds 200 ms and gets shadow-IT'd.

Requirements and scale

Clarifying questions (each tied to a design consequence)

1. How many teams/apps, and what's the traffic profile? 200 devs and 5 apps is a LiteLLM deployment; 5k devs and 200 apps at 500 QPS is a real distributed system with HA requirements.
2. Which providers/models, and any self-hosted? Multi-provider is the point of a gateway; self-hosted models add a serving stack behind the same facade.
3. What must be governed — cost, data, or both? Cost-only means metering and budgets; data governance adds PII redaction, payload-logging policy, region routing, and audit — a much bigger system.
4. Latency sensitivity of the workloads? Interactive products can't afford a gateway adding 50 ms; batch pipelines don't care. Decides how thin the hot path must be.
5. Is prompt/response content loggable at all? Legal/privacy stance decides the observability design — some orgs forbid payload retention entirely, which changes debugging strategy.
6. Buy vs. build appetite? Acknowledge LiteLLM/Cloudflare AI Gateway exist and specify what you would build around or instead of them, with reasons.

Requirements (assumed)

Capacity estimates

Explicit arithmetic:

• Daily requests at peak flat: 500 × 86,400 = 43.2M/day theoretical; design uses ~15M req/day mixed interactive + batch (peak ≠ sustained).
• Tokens/request: ~4,000 in + 400 out = 4,400 tokens. 15M × 4,400 = 66B if every request were that size — design states ~2B tokens/day, so average request is smaller or many cache/ embed/short calls: 2e9 / 15e6 ≈ 133 tokens average across the full mix, with a heavy tail of large chat completions. Use ~2B tokens/day as the spend driver.
• Spend band: mid-tier blend $3–5/M in and $15–25/M out on the large calls drives ~$8–10k/day blended org spend ($240–300k/month order). Gateway infra ~$2–3k/month is noise against ~$250k/month provider spend.
• Pods: 500 / 80 ≈ 6.25 → 6–8 pods with headroom.
• Metadata logs: 15M × 1 KB ≈ 15e9 bytes ≈ 15 GB/day; full payloads ~100× → ~1.5 TB/day if retained — hence default metadata-only.

Cost framing: the gateway's own infra (~$2–3k/month) is noise against ~$250k/month provider spend — its job is making that spend visible, attributable, and reducible (caching + routing can realistically cut 20–30%).

The design

API surface

One OpenAI-compatible endpoint — it's the de facto wire format every SDK speaks, so teams migrate by changing base URL + key. Requests name model aliases (chat-large, chat-fast, embed-default), never raw provider models: aliases let the platform swap chat-large from one provider's flagship to another's — or run a canary — without any team changing code. Version pinning available per app for teams whose evals are sensitive.

AuthN/Z

No provider keys ever reach teams — the gateway holds them in a secrets manager and is the only egress (enforced at the network layer: direct provider domains blocked from prod). Apps authenticate with short-lived OIDC service tokens or issued virtual keys, each mapped to {team, app, environment}. Authorization = per-team model allowlist plus data-class rules (e.g., apps tagged pii may only use providers with zero-retention agreements, or the self-hosted model).

Quotas and rate limits

Two distinct mechanisms, both per {team, app}:

• Rate limits (RPM and TPM — token-per-minute is the one that matters; LLM cost and provider limits are token-shaped, and an app sending 10 huge-context requests can starve everyone at a request-based limit) enforced in Redis with sliding windows on the hot path (<2 ms).
• Budgets (monthly $ per team) enforced softly — alert at 80%, throttle to a cheap model at 100%, hard-block only batch traffic, because silently breaking a customer-facing product over an internal budget is the wrong failure.

Priority tiers: interactive traffic preempts batch when a provider rate-limits us.

Routing and resilience

Health-checked provider pool per alias; on 429/5xx/timeout: retry once with jitter → failover to the alias's secondary provider → shed batch traffic first. Track live TTFT and error rate per provider; route on p95 TTFT when both are healthy. Streaming passthrough with no buffering (overhead only on headers). Circuit breaker per provider trips at 25% error rate over 30s. Per-model kill switch is the governance win: one flag removes a deprecated or incident-struck model org-wide.

Self-hosted path: open source and self-hosting behind the same alias facade (e.g. vLLM).

Logging vs. privacy — the deliberate tradeoff

Always logged (1-year retention): request metadata — who, which alias→model, token counts, latency, cost, cache/failover flags, and a prompt hash so incidents can be correlated without content.

Payloads (prompt/response bodies): off by default, opt-in per app for debugging with 30-day retention and PII redaction on write; apps tagged pii cannot opt in. Audit events (key issuance, allowlist changes, kill-switch flips) go to an append-only store. This answers "can we see everything?" with "we can account for everything, and see content only where a team consented" — a policy choice, not an accident.

Cost attribution

Meter usage-reported tokens (not client estimates) per request, priced from a versioned model-price table, aggregated to {team, app, model, day}. Monthly showback per team; anomaly alert on any app whose $/day jumps >3× baseline (usually a retry loop or a runaway agent — this alert pays for the gateway by itself).

Caching

Exact-match response cache (hash of model + full request, tenant-scoped) — high hit rates only for idempotent internal workloads, but nearly free. Semantic caching offered per-app opt-in only: cross-user semantic hits risk leaking one user's answer to another, and false-positive hits are a correctness bug — default off, and the reason matters. Provider-side prompt caching (cache-control on stable prefixes) is passed through and encouraged; it cuts input cost up to ~90% on repeated prefixes and is where most real savings live. See cost.

Serving path — gateway overhead budget

p50 sum: 2+1+2+1+2 = 8 ms (≤15 ms target). p95 sum: 5+3+5+3+5 = 21 ms ≈ ~20 ms.

Metering/audit is written async (local buffer → queue) — never on the hot path; a logging outage degrades attribution freshness, not traffic.

Evaluation and operations

The gateway is judged on SLOs, not model quality: added-latency p95, availability, failover correctness (chaos-test by black-holing a provider in staging monthly), attribution accuracy (reconcile metered spend vs. provider invoices — target <2% drift). Dashboards per team: spend, tokens, error rate, cache hit rate, model mix. Rollouts: alias re-pointing is done as a canary (5% → 50% → 100%) with the owning teams' eval suites as the gate — the gateway gives platform teams the mechanism for model migration; the decision stays with app teams' evals.

Observability patterns: observability.

Key decisions and tradeoffs

The flows

Flow A — Interactive completion (cache miss)

1. App SDK → OpenAI-compatible endpoint with virtual key / OIDC token.
2. AuthN → AuthZ allowlist → RPM/TPM + budget check in Redis.
3. Exact cache miss → route alias to healthy primary → stream passthrough.
4. Async meter tokens, cost, trace; debit TPM as usage frames complete.

Breaks when: gateway buffers the stream, Redis becomes required for data plane, or alias maps to a single provider with no failover.

Flow B — Cache hit / prompt-cache pass-through

1. Same auth path.
2. Exact response cache hit returns stored body (tenant-scoped key).
3. Or provider prompt-cache hits on stable prefix — gateway forwards cache-control headers and records cache-hit tokens for showback.

Breaks when: cache key omits tenant/app, or prompt order puts volatile content first (provider prefix never stable).

Flow C — Provider failure and failover

1. Primary returns 429/5xx or breaker open (25% errors / 30s).
2. Retry once with jitter → secondary provider for alias → shed batch first.
3. Interactive continues; kill switch available for bad models.

Breaks when: apps hard-code provider model IDs (bypass aliases) or circuit breaker flaps without cooldown.

Flow D — Budget exhaustion

1. Team hits 80% monthly $ → alert.
2. At 100% → route to cheap model for eligible traffic; hard-block batch; keep interactive degraded not dark.
3. Showback dashboard explains which app burned the budget (often a runaway agent).

Breaks when: hard-kill all traffic at budget cap (self-inflicted product outage).

Failure modes and degradation

Single point of failure defense: Stateless horizontal pods behind a load balancer across 3 AZs; Redis is the only stateful hot-path dependency — on Redis loss, fail open on rate limits rather than dropping traffic. Config cached in-pod so the control plane can die without touching the data plane.

Common drill-downs

Your gateway is now a single point of failure. Defend it.

Stateless horizontal pods behind a load balancer across 3 AZs; Redis is the only stateful hot-path dependency — on Redis loss, fail open on rate limits (log and reconcile later) rather than dropping traffic: losing quota enforcement for minutes is cheaper than an org-wide LLM outage. Config (allowlists, aliases) is cached in-pod with background refresh, so the control plane can die without touching the data plane.

How do you rate-limit a streaming response? You don't know output tokens upfront.

Admit the mechanism: TPM limits on input are enforced pre-flight; output tokens are debited as usage frames arrive at stream end — so enforcement is retrospective by up to one request. Bound the damage with max_tokens caps per app tier and concurrent-stream limits. Anyone who claims exact pre-flight output limiting has not built one.

A team says the gateway added 200 ms to their p99. Investigate.

Per-stage spans exist because every hop is traced: usually it's (a) missing streaming — the app buffered the full response through an SDK default, (b) cross-region hop to the gateway, or (c) retry-on-429 hidden inside the gateway counted as gateway time. Show the debugging story — and note the gateway's own overhead SLO chart proves innocence or guilt in one look.

Why not let teams call providers directly with their own keys?

Enumerate what's lost: cost attribution (finance), kill switch and audit (security/compliance), failover (reliability), alias-based migration leverage (platform), zero-retention enforcement for PII apps (legal). Then concede the real cost — the gateway team becomes on-call for everyone's LLM traffic — and staff it.

Buy or build?

Start from open-source (LiteLLM proxy is the de facto standard) or a managed edge gateway (Cloudflare AI Gateway) for the proxy/metering core; you'll still own the org-specific 30% — identity integration, data-classification policy, showback pipelines, and the migration playbook. Building the proxy itself from scratch is justified mainly at very high QPS or unusual compliance surface.

Test yourself

Go deeper

• LiteLLM AI Gateway docs — the reference open-source implementation: virtual keys, budgets, routing, spend tracking — read it to steal vocabulary and defaults.
• Cloudflare AI Gateway docs — the managed-edge version of the same ideas: caching, rate limits, fallbacks, analytics.
• Prompt caching — Claude Docs — the provider-side caching mechanics (TTLs, prefix rules) a gateway should exploit rather than reinvent.
• Building A Generative AI Platform — Chip Huyen — situates the gateway among guardrails, caches, and observability in a full platform.

Where this connects

• Enterprise RAG and Document Q&A at scale — every production retrieval product should call models through this gateway for budgets and failover.
• Customer support agent — agents need TPM budgets and kill switches when loops run away.
• Cost — token accounting and caching levers the gateway meters and exposes.
• Choosing models — alias routing decisions need a clear capability/cost/latency framework.