LLM APIs

The intuition

An LLM API is a stateless function over the network. You send a model id, role-tagged messages, and optional tools. You get assistant text (or tool calls) plus a receipt (usage). Unless you opt into a special stateful product, the server forgets you between calls — so you re-send the full conversation every turn. That fact alone drives growing bills, prompt caching, and history pruning as a product feature. Streaming delivers tokens as they generate; prompt caching reuses work on a repeated prefix; batch runs bulk jobs overnight at half price.

Why it exists

Hosted APIs productize the generation loop so teams do not own GPU fleets for every experiment: elasticity, a portable contract (messages, tools, stream, usage), economic levers (caching, batch, model tiers), and documented HTTP for rate limits and multimodal uploads. Enterprises put a gateway in front for IAM, audit, and routing — see design-llm-gateway.md.

Alternatives lose for many products. Self-host wins on data control and unit economics at huge steady load, but loses on ops and model freshness for most teams. SDK calls scattered through the app block failover. No streaming dies on proxy timeouts. No caching donates margin on every repeated system prompt.

On the governed enterprise platform, apps never call the public model API with user keys. They call an internal gateway that attaches IAM, enforces quotas, injects the stable system policy (cached), picks a model tier, and logs usage for chargeback.

The core idea

Every major LLM API is the same shape: a stateless HTTPS endpoint that takes a model ID, role-tagged messages, generation parameters, and optional tools, and returns assistant content plus metadata.

Request concepts: model; system instructions (top-level system at Anthropic, a system-role message at OpenAI); messages with typed content blocks (text, image, document, tool_use, tool_result); output cap (max_tokens); sampling params; tools + tool_choice; stream. Response: content blocks; a stop reason (end_turn, max_tokens, tool_use, stop_sequence, refusal-type reasons); a usage object with input, output, and cached token counts — the field you bill and alert on.

Three features to know cold: streaming (SSE tokens as they generate), prompt caching (exact-prefix KV reuse for cheaper/faster prefill), and batch APIs (async bulk at 50% off, 24-hour window). Plus table stakes: keys server-side only; exponential backoff with jitter on 429/5xx/529; honor retry-after; rate limits in both RPM and TPM.

How it actually works

Streaming (SSE). With stream: true the response is text/event-stream. Anthropic's flow is typical: message_start → per content block content_block_start / N× content_block_delta / content_block_stop → message_delta (stop reason + cumulative usage) → message_stop, plus ping keep-alives. Deltas include text_delta, input_json_delta for tool args (partial JSON strings — accumulate until block stop), and thinking_delta when extended thinking is on. A 529-style overload can arrive as an in-stream error event after HTTP 200 already opened. SDKs offer "stream internally, return the final message" so large max_tokens does not hit client timeouts.

Multimodal inputs. Images and documents are content-block types: base64, URL, or file_id from a Files API. Images tokenize by size (illustrative: ~1.5k–5k tokens for a detailed image) and count fully toward context and cost. Image/audio generation is usually a separate endpoint family.

Prompt caching. The cache key is the exact rendered prefix (typically tools → system → messages). A hit restores stored KV instead of re-running prefill — cheaper input and lower TTFT. Any byte change invalidates from that point: timestamps in the system prompt, shuffled JSON keys, reordered tools, model switches. Structure prompts stable-first, volatile-last.

Anthropic: automatic top-level cache_control and/or up to four explicit breakpoints; 5-minute TTL default (1-hour at 2× write); reads ~0.1×, 5-minute writes ~1.25×; minimum length model-dependent (~1k–4k tokens often). OpenAI: automatic on prefixes ≥~1024 tokens; cached input commonly ~50% of base (check pricing); newer families add explicit breakpoints and optional write fees. Verify with cache_read_input_tokens / cached_tokens. A one-shot write is a net loss.

Batch. Submit many requests each with a custom_id; poll or webhook; results arrive in any order — key by id, never array index. Caps are large (Anthropic ~100k / 256 MB; OpenAI ~50k / 200 MB — confirm in docs). Use for evals, backfills, bulk jobs. Caching stacks with batch.

The flows

A worked example

Employee chat on the governed enterprise platform, turn 4 (token counts illustrative):

Production concerns

Timeouts and streaming. Long generations can run minutes; default HTTP timeouts kill them. Stream for large expected output; set timeouts deliberately; retries multiply wall-clock. See latency.md and reliability.md.

Errors and rate limits. Do not retry 400 (malformed / context overflow), 401/403, or 404. Do backoff on 429 (honor retry-after) and 500/529 (jittered retry; consider fallback). Many production failures are capacity limits, not bad prompts. Limits are multi-dimensional — RPM and input/output TPM — so one large request can burn a minute of budget. High-throughput systems need client-side token accounting and load-shedding. Know what your SDK auto-retries before stacking another layer, or you create retry storms during outages.

Abstraction and cost. Field names, streaming shapes, tool formats, and cache semantics differ. Wrap providers behind your own interface (or a gateway) so fallback is a config change. Log usage with route tags; alert on tokens-per-request drift and cache hit-rate drops. Cached-vs-uncached and batch-vs-live move the bill more than swapping mid-tier models. See cost.md and observability.md. Pin model snapshot IDs; re-run evals before adopting a new alias. See choosing-models.md.

Common drill-downs

Why resend the whole conversation every turn? The default API is stateless. The model conditions only on tokens in this request's window, so per-turn cost grows with history unless you cache prefixes and compact history.

How does streaming matter beyond chat UX? Perceived latency (first tokens in under a second), reliability (long generations exceed HTTP timeouts unless streamed), and pipelining (TTS or progressive UI can start early). Mid-stream errors still need explicit handling.

How do you design cost controls for a chat product? Cache the system prompt and stable history; compact history past a threshold; cap max_tokens per route; route cheap intents to small models; batch offline jobs; per-user token budgets; usage logging with anomaly alerts.

Test yourself

Go deeper

• Anthropic: Streaming Messages — SSE event taxonomy, tool-use and in-stream errors.
• Anthropic: Prompt caching — automatic/explicit breakpoints, TTLs, multipliers, invalidation.
• OpenAI: Prompt caching — automatic caching, minimums, retention, explicit breakpoints.
• OpenAI: Batch API — 50% discount, 24h window, custom_id mapping.
• Anthropic: Batch processing — Message Batches limits and result ordering.
• Intro to LLMs — Andrej Karpathy — mental model for API design choices.

Where this connects

• Cost — token accounting, caching, batch, and routing as bill design.
• Latency — TTFT, streaming UX, and prefill vs decode.
• Reliability — retries, timeouts, fallbacks, circuit breakers around these endpoints.
• Design an LLM Gateway — governed multi-model access, quotas, and audit on top of the same API shape.