LLM APIs

The intuition

An LLM API is a stateless vending machine for next-token generation. You insert a model id, a stack of role-tagged messages, and optional tools; it returns assistant text (or tool calls) plus a receipt (usage). Because the machine forgets you between requests, you re-bring the entire conversation every turn. That single fact explains growing bills, the need for prompt caching, and why history pruning is a product feature rather than a nice-to-have.

Streaming is watching the cups fill as coffee pours instead of waiting for a sealed carafe — better UX and fewer HTTP timeouts on long pours. Prompt caching is the shop pre-grinding the house blend (the shared system prompt) so the hundredth customer of the day does not pay full prep time. Batch is the overnight wholesale order at half price when nobody needs espresso in 200 ms.

Why it exists

Teams do not want to own GPU fleets to try a feature. Hosted APIs productize the generation loop:

1. Elasticity without cluster ops. Burst traffic without buying H100s for the average load.
2. A stable application contract. Messages, tools, stream, usage — portable concepts even when field names differ.
3. Economic controls. Caching, batch, and model tiers are how serious cost curves get bent (cost.md).
4. Operational realities. Rate limits, retries, and multimodal uploads need a documented HTTP protocol, not ad-hoc SSH to a box.
5. Governance hook. Enterprises put a gateway in front of the same APIs for IAM, audit, and routing (design-llm-gateway.md).

Alternatives lose for many products. Pure self-host wins on data control and unit economics at huge steady load but loses on ops and model freshness for most teams. One SDK scattered everywhere blocks failover. Ignoring streaming dies on timeouts. Ignoring caching donates margin to repeated prefixes.

On the governed enterprise platform, applications never call the public internet model API with user keys. They call an internal gateway that attaches IAM identity, enforces quotas, injects the stable system policy (cached), chooses a model tier, and logs usage for chargeback.

The core idea

Every major LLM API is the same shape: a stateless HTTPS endpoint that takes a model ID, a list of role-tagged messages, generation parameters, and optional tool definitions, and returns assistant content plus metadata. Statelessness is the fact that drives everything else: the server keeps no conversation memory, so you re-send the full history every turn — which is why context grows, why cost grows per turn, and why prompt caching exists.

Request anatomy (Anthropic Messages API / OpenAI equivalents differ in field names, not concepts): model; system prompt (top-level field at Anthropic, a system-role message at OpenAI); messages alternating user/assistant, where content is a list of typed blocks (text, image, document, tool_use, tool_result); max_tokens (output cap); sampling params; tools + tool_choice; stream. Response anatomy: content blocks, a stop_reason (end_turn, max_tokens, tool_use, stop_sequence, refusal-type reasons), and a usage object with input/output/cached token counts — the field you bill, monitor, and alert on.

Three API features every applied-AI engineer must know cold:

• Streaming via Server-Sent Events — tokens delivered as they're generated, for UX (perceived latency) and to keep long generations from hitting HTTP timeouts.
• Prompt caching — reuse of server-side KV-cache for a repeated prompt prefix: cached input bills at ~10% of base rate (Anthropic; writes cost 1.25x) or ~50–90% discount (OpenAI, automatic ≥1024-token prefixes). The single biggest cost lever in chat/agent workloads.
• Batch APIs — asynchronous bulk processing at a flat 50% discount with a 24-hour completion window, for anything that doesn't need a synchronous answer.

Plus operational table stakes: API keys server-side only, retry-with-exponential-backoff on 429/5xx/529, respect retry-after, idempotency via your own request keys, and per-model rate limits counted in both requests and tokens per minute.

How it actually works

Streaming (SSE). With stream: true the response is text/event-stream — a long-lived HTTP response of event:/data: lines. Anthropic's event flow: message_start → per content block: content_block_start, N x content_block_delta (with text_delta, input_json_delta for tool args, or thinking_delta), content_block_stop → message_delta (carries final stop_reason + usage) → message_stop, with ping keep-alives and in-stream error events (a 529 overload can arrive mid-stream as an event, not an HTTP status — handle it). Tool-call arguments stream as partial JSON strings you accumulate and parse at block stop. SDKs wrap all this and offer "stream internally, return final message" helpers — the standard way to run large max_tokens without timeouts.

Multimodal inputs. Images and documents are just more content-block types in a user message: base64-encoded inline, by URL, or by file_id from a Files API upload (upload once, reference many times). Images are tokenized by size — roughly 1.5k–5k tokens for a detailed image — and count fully toward context and cost; requests carry limits on image count and pixel dimensions. PDFs can be processed with page rendering + text so models read both layout and content. Output multimodality (image/audio generation) is a separate endpoint family; chat APIs are text-out.

Prompt caching mechanics. The cache key is the exact byte prefix of the rendered request (tools → system → messages, in order). A hit means the server restores the stored KV-cache instead of re-running prefill — that's why it's both cheaper and faster (TTFT drops dramatically on long prefixes). Consequences:

• Any byte change invalidates everything after it: a timestamp in the system prompt, unsorted JSON keys, a reordered tool list = 0% hit rate.
• Structure prompts stable-first, volatile-last; put per-request content after the last cached segment.
• Anthropic: explicit cache_control breakpoints (up to 4), 5-min TTL (1-hour at 2x write cost), reads 0.1x, writes 1.25x, ~1024–4096-token minimum prefix. OpenAI: automatic on ≥1024-token prefixes, retention minutes-to-24h by model family.
• Verify with usage fields (cache_read_input_tokens etc.); zero reads across identical requests means a silent invalidator.
• Break-even for a 5-min Anthropic cache is 2 requests (1.25 + 0.1 < 2 x 1.0); caching a prefix used once is a net loss.

Batch API mechanics. Submit up to ~tens of thousands of requests (Anthropic: 100k requests / 256 MB per batch) each with a custom_id; poll or webhook for completion; download results, which arrive in any order — always key by custom_id, never position. Most batches finish well under the 24h ceiling. Use for: evals, backfills, bulk classification/embedding-adjacent jobs, nightly report generation. Caching stacks with batching.

The flows

A worked example

Employee chat on the governed enterprise platform, turn 4 of a session:

Cost bug of the week: someone adds request_id and iso_timestamp to the system policy for "traceability." Cache reads go to zero across the fleet; the bill jumps toward the cold-path math on every turn. Traces belong in your logs and headers, not in the cached prompt bytes.

Common drill-downs

Why do you have to resend the whole conversation every turn? The API is stateless — no server-side conversation store (unless you opt into a stateful/responses-style API). The model needs full context in the window to condition on. This makes per-turn cost grow with history length and makes prompt caching + history pruning essential.

How does streaming work under the hood? Server-Sent Events over a held-open HTTP response: named events with JSON data — message start, block start/delta/stop per content block, message delta with stop reason and usage, message stop. Client accumulates deltas; tool arguments arrive as partial-JSON strings; errors can arrive as in-stream events.

Why does streaming matter beyond UX? (1) Perceived latency: users see first tokens in <1s instead of waiting for the full generation. (2) Reliability: long generations exceed HTTP timeouts unless streamed. (3) Pipelining: downstream steps (TTS, progressive rendering) can start early.

Explain prompt caching and when it fails. Server persists the KV-cache for a request's prefix; an exact-prefix match on a later request skips prefill — ~90% cheaper and much faster TTFT. Fails on any prefix byte change: timestamps or UUIDs in the system prompt, non-deterministic serialization, tool list changes, model switches. Design prompts stable-first and verify via usage fields.

When do you use the batch API? No-synchronous-answer workloads: nightly enrichment, eval runs, bulk extraction, re-embedding. 50% off, 24h window, results keyed by custom_id in arbitrary order. Combine with caching for shared prefixes.

How do you handle a 429? Exponential backoff with jitter, honor retry-after, cap retries, and fix the cause: client-side rate limiting/queuing sized to TPM+RPM, request smoothing, smaller contexts, or a tier increase. Also decide a degradation path (queue, fallback model, or user-visible backpressure).

A request costs 10x what you expected. Debug it. Read usage: input vs output vs cached split. Usual suspects: agent loop resending ballooning history (input-heavy), cache misses from a prefix invalidator (cache_read = 0), verbose output with no length constraint, or images/PDFs you forgot are thousands of tokens each.

Design cost controls for a chat product. Prompt caching on the system prompt + history; history compaction beyond a threshold; max_tokens caps per route; small-model routing for cheap intents; batch for offline jobs; per-user token budgets; usage logging + anomaly alerts.

Production concerns

• Timeout design. Long generations can run minutes; default HTTP client timeouts will kill them. Stream by default for anything above ~16k output tokens; set client timeouts deliberately; remember retries multiply wall-clock (timeout x (retries+1)). See latency.md and reliability.md.
• Error taxonomy and handling: 400 (malformed / context overflow — don't retry), 401/403 (auth — don't retry), 404 (bad model ID), 413 (too large), 429 (rate limit — backoff + retry-after), 500/529 (server/overload — retry with jitter, consider a fallback model). SDKs auto-retry the retryable class; know what they do before adding your own layer.
• Rate limits are multi-dimensional — RPM and input/output TPM per model. One 500k-token request can exhaust a minute of TPM. High-throughput systems need client-side token accounting, queuing, and load-shedding, not just retries.
• Multi-provider abstraction. Field names, streaming event shapes, tool-call formats, and cache semantics all differ. Wrap providers behind your own interface (or a gateway/proxy layer) so fallback and A/B are config changes; never scatter raw SDK calls across the codebase.
• Cost observability. Log usage per request with route/feature tags; alert on tokens-per-request drift (agent-loop history growth) and on cache hit-rate drops (someone touched the stable prefix). Cached-vs-uncached and batch-vs-live are 10x-level cost differences — architecture, not model choice, sets your bill. See cost.md and observability.md.
• Version pinning. Model aliases move; pin snapshot/version IDs in production and re-run evals before adopting a new one. See choosing-models.md.

Test yourself

Go deeper

• Anthropic docs: Streaming Messages — full SSE event taxonomy with wire-format examples, including tool-use and error events.
• Anthropic docs: Prompt caching — breakpoints, TTLs, exact pricing multipliers, invalidation rules.
• OpenAI docs: Prompt caching — automatic caching, 1024-token minimum, retention windows.
• OpenAI docs: Batch API — 50% discount, 24h window, file-based submission flow.
• Intro to Large Language Models — Andrej Karpathy — the mental model (LLM as a new kind of OS) that makes API design choices legible.

Where this connects

• Cost — token accounting, caching, batch, and routing as bill design.
• Latency — TTFT, streaming UX, and prefill vs decode.
• Reliability — retries, timeouts, fallbacks, circuit breakers around these endpoints.
• Design an LLM Gateway — governed multi-model access, quotas, and audit on top of the same API shape.