Design: LLM Gateway
Governed multi-model access — quotas, audit, routing — the enterprise middleware question.
Prerequisites
- How to reason about AI system design — requirements, capacity, ops phases.
- LLM APIs — chat completion anatomy, streaming, caching, batch.
- Cost — token accounting, caching, tiering.
- Reliability — retries, timeouts, fallback, circuit breakers.
- Security — data handling and abuse surfaces for LLM traffic.
The problem
Design the internal platform through which every team at the company calls LLMs. Equivalently: dozens of teams call OpenAI, Anthropic, and Gemini with shared API keys — fix that.
Think of it as a switchboard, not a thin reverse proxy. Every call is authenticated, authorized against a model allowlist, metered in tokens and dollars, and routed to a healthy provider — without each product team knowing which backend ran today. The trap is building a generic API gateway and ignoring what makes LLM traffic different: streaming, token-shaped cost, non-deterministic latency, and provider volatility.
In the governed enterprise platform, this is the model gateway: managed cloud models behind IAM, with quotas, audit, and routing wrapping every call from RAG, agents, and batch jobs.
Why it is hard
- Cost is token-shaped, not request-shaped. Ten huge-context calls can cost more than a thousand short ones. RPM-only limits fail; TPM (tokens per minute) and dollar budgets are first-class.
- Streaming breaks naive gateways. Buffering destroys TTFT (time to first token). Output tokens are unknown pre-flight, so TPM enforcement is partly retrospective.
- Providers are volatile. 429s, regional incidents, and deprecations are normal. The gateway must be more available than any single provider, with failover that needs no app rewrites.
- Governance fights velocity. Without an OpenAI-compatible facade and model aliases, central control becomes a ticket queue that teams bypass with shadow keys.
- Logging fights privacy. Full payload retention helps debugging and hurts legal; metadata + hashes + opt-in payloads is a deliberate policy.
- Buy vs build is real. LiteLLM, Cloudflare AI Gateway, and peers exist. The org still owns identity, data-class policy, and showback — reinventing the proxy core is rarely what matters.
Alternatives that lose: shared keys in every repo, per-team direct provider access, and a heavy enterprise ESB that adds hundreds of milliseconds and gets shadow-IT'd.
Requirements and scale
Clarifying questions (each tied to a design consequence)
- How many teams/apps, and traffic profile? Low QPS → LiteLLM deploy; high QPS across many apps → real HA system.
- Which providers/models, any self-hosted? Multi-provider is the point; self-hosted sits behind the same facade.
- Cost, data, or both? Cost-only is metering and budgets. Data governance adds PII redaction, payload policy, region routing, and audit.
- Latency sensitivity? Interactive cannot afford a fat hot path; batch does not care.
- Are prompts/responses loggable? Legal stance decides observability design.
- Buy vs build? Name what you adopt (LiteLLM, Cloudflare AI Gateway, or similar) and what you still build: identity, policy, showback.
Requirements (assumed)
| Kind | Requirement |
|---|---|
| Scale | 5k developers, ~150 apps; peak 500 QPS, ~2B tokens/day (illustrative) |
| Backends | 3 external providers + 1 self-hosted open-weight model |
| Overhead | Gateway adds ≤15 ms p50 |
| Availability | 99.95% (more available than any provider) |
| Attribution / audit | Per-team cost to the token; who called which model when |
| Control | Central kill switch per model/provider |
Capacity estimates
| Quantity | Estimate | Derivation |
|---|---|---|
| Request load | 500 QPS peak, ~15M req/day | Mixed interactive + batch; peak ≠ sustained |
| Token volume | ~2B tokens/day | Short embed/cache hits + long chat tail |
| Daily provider spend | ~$8–10k/day | Blended mid-tier on the heavy tail (order-of-magnitude, mid-2020s) |
| Gateway compute | ~6–8 pods | ~80 QPS/pod; CPU-bound on TLS + JSON |
| Metadata logs | ~15 GB/day | ~1 KB/request; full payloads ~100× if retained |
Gateway infra (~$2–3k/month) is noise against ~$250k/month provider spend. The job is visibility and reduction — caching and routing can cut 20–30% of that bill.
The design
API surface and identity
One OpenAI-compatible endpoint — the de facto wire format every SDK speaks. Teams migrate by
changing base_url and key. Requests name model aliases (chat-large, chat-fast,
embed-default), never raw provider IDs, so the platform can re-point or canary without app code
changes. Version pinning is available for eval-sensitive teams.
No provider keys reach teams. The gateway holds them in a secrets manager and is the only egress
(network-deny direct provider domains from prod). Apps authenticate with short-lived OIDC service
tokens or issued virtual keys, each mapped to {team, app, environment}. Authorization is a
per-team model allowlist plus data-class rules: apps tagged pii may only use zero-retention
providers or the self-hosted model.
Quotas and rate limits
Two mechanisms, both per {team, app}. Rate limits — RPM and TPM. Providers enforce
multi-dimensional limits (RPM, TPM, often per-day; some split input vs output TPM). Ten huge-context
requests can starve the org under a request-only cap. Enforce in Redis with sliding windows (<2 ms).
Budgets — monthly $ per team, soft: alert at 80%, throttle eligible traffic to a cheap model at
100%, hard-block only batch. Killing a customer-facing product over an internal budget is the wrong
failure. Interactive preempts batch when a provider rate-limits you.
Common misconception
Request-per-minute limits feel like normal API-gateway work. For LLMs they are insufficient. Pair RPM with TPM and dollar budgets, or one agent loop will dominate the org bill.
Routing, caching, and metering
Health-checked provider pool per alias. On 429 / 5xx / timeout: retry once with jitter → failover to secondary → shed batch first. Prefer lower p95 TTFT when both are healthy. Stream passthrough — no buffering. Circuit breaker per provider (e.g. 25% errors over 30s) with cooldown. A per-model kill switch removes a bad model org-wide. Self-hosted models sit behind the same facade (e.g. vLLM) — see open source and self-hosting.
Always log metadata: who, alias→model, tokens, latency, cost, cache/failover flags, prompt hash.
Payload bodies off by default, opt-in per app with short retention and PII redaction; pii apps
cannot opt in. Meter usage-reported tokens from a versioned price table, aggregate to
{team, app, model, day}, reconcile vs invoices (target <2% drift). Alert on $/day >3× baseline —
usually a retry loop or runaway agent.
Exact-match cache (tenant-scoped): nearly free; hits only for idempotent workloads. Semantic caching: per-app opt-in — cross-user hits risk leaks; false positives are correctness bugs. Provider-side prompt caching (stable prefixes): pass through and encourage; most real input-cost savings live here. See cost.
Serving path and ops
Hot-path budget (illustrative): TLS+parse ~2 ms, AuthN hit ~1 ms, AuthZ+rate limit ~2 ms, cache ~1 ms, route+forward ~2 ms → ~8 ms p50 (≤15 ms target), ~20 ms p95. Provider TTFT is hundreds of ms to seconds — the gateway must not become the story. Metering writes async: logging outages degrade attribution freshness, not traffic.
Judge on SLOs: added-latency p95, availability, failover correctness (chaos-test monthly), attribution accuracy. Alias re-points ship as canaries (5% → 50% → 100%) gated by owning teams' evals — the gateway provides the mechanism; app teams own the decision. See observability.
Key decisions and tradeoffs
| Decision | Choice | Tradeoff |
|---|---|---|
| Wire API | OpenAI-compatible + aliases | Occasional feature lag vs universal SDK support |
| Keys | Gateway-held; network deny direct egress | Platform on-call for all LLM traffic; huge security win |
| Budgets | Soft throttle; hard-block batch only | Some interactive overage; protects customer UX |
| Semantic cache | Off by default; per-app opt-in | Lower hit rate; avoids cross-user leaks |
| Payload logs | Metadata always; bodies opt-in | Harder debugging for pii apps; correct default |
| Redis failure | Fail open on rate limits | Temporary quota bypass beats org-wide outage |
| Buy vs build | OSS/managed proxy + custom policy/showback | Less greenfield control; faster path to value |
Key insight
The gateway succeeds when teams migrate by changing base_url and never touch provider keys
again — and when finance can see $/team without reading Slack. Mechanism over mandate.
The flows
A — Interactive (cache miss). AuthN → AuthZ → RPM/TPM + budget → cache miss → route alias to healthy primary → stream passthrough. Async meter as usage frames complete. Breaks when: stream buffering, Redis required for the data plane, or no failover on the alias.
B — Cache hit / prompt-cache pass-through. Exact response cache hit (tenant-scoped key), or provider prompt-cache on a stable prefix — gateway forwards cache-control headers and records cache-hit tokens. Breaks when: cache key omits tenant/app, or volatile content sits first so the provider prefix never stabilizes.
C — Provider failure. Primary 429/5xx or breaker open → retry once with jitter → secondary → shed batch first. Kill switch ready. Breaks when: apps hard-code provider model IDs or the breaker flaps without cooldown.
D — Budget exhaustion. 80% → alert. 100% → cheap model for eligible traffic; hard-block batch; keep interactive degraded, not dark. Showback names the app (often a runaway agent). Breaks when: hard-kill all traffic at the budget cap.
Failure modes and degradation
| Failure | Response |
|---|---|
| Gateway pod loss | Stateless scale; LB across 3 AZs |
| Redis loss | Fail open on rate limits; reconcile later |
| Control plane down | In-pod cached config; background refresh |
| Provider incident | Failover + batch shed + kill switch |
| Spend anomaly >3× baseline | Alert; optional auto-throttle that app |
| Logging pipeline down | Traffic continues; attribution lag, not 5xx |
| Client buffers full body | Fix client/SDK; gateway SLO chart shows low overhead |
SPOF defense: multi-AZ stateless pods; fail open on Redis rather than take the org offline; config cached in-pod so the control plane can die without touching the data plane.
Common drill-downs
How do you rate-limit a streaming response when output tokens are unknown? TPM on input is
pre-flight. Output tokens debit as usage frames arrive at stream end — retrospective by up to one
request. Bound damage with max_tokens caps per app tier and concurrent-stream limits. Claims of
exact pre-flight output limiting are not from people who have built one.
A team says the gateway added 200 ms to their p99. Every hop is traced. Usual causes: SDK buffering (no stream), cross-region hop, or gateway retries counted as gateway time. Compare span sum (~8–20 ms) to client delta — the gap is almost always client or network.
Why not let teams call providers with their own keys? You lose cost attribution, kill switch, audit, failover, alias migration, and zero-retention enforcement for PII apps. The real cost: the gateway team is on-call for everyone's LLM traffic — staff it.
Buy or build? Start from LiteLLM proxy or Cloudflare AI Gateway for proxy and metering. You still own the org-specific ~30%: identity, data-class policy, showback, migration playbook. Building the proxy from scratch is justified mainly at very high QPS or unusual compliance surface.
Test yourself
You only enforce 100 RPM per app. One app sends 10 requests/min with 200k-token prompts. What happens?
Gateway p50 overhead is 8 ms but a chat app sees +200 ms TTFT after migration. Where do you look?
Finance says metered spend is 6% below provider invoices. Is that acceptable?
Should semantic cache be on by default at the gateway?
Alias chat-large moves from Provider A to B at 5% canary. Who decides full cutover?
Go deeper
- LiteLLM AI Gateway docs — reference OSS proxy: virtual keys, budgets, routing, spend tracking.
- Cloudflare AI Gateway docs — managed-edge version: caching, rate limits, fallbacks, analytics.
- Prompt caching — Claude Docs — provider-side caching mechanics a gateway should exploit rather than reinvent.
- Building A Generative AI Platform — Chip Huyen — gateway among guardrails, caches, and observability in a full platform.
Where this connects
- Enterprise RAG and Document Q&A at scale — production retrieval should call models through this gateway for budgets and failover.
- Customer support agent — agents need TPM budgets and kill switches when loops run away.
- Cost — token accounting and caching levers the gateway meters.
- Choosing models — alias routing needs a clear capability/cost/latency framework.