Production
Anyone can build a demo. This section is everything between a demo and a system.
Prerequisites
- LLM APIs — streaming, caching, and batch are the levers most of this section pulls.
- The RAG Pipeline and Agent Foundations — the two systems being productionized.
Why this section exists
An LLM feature has an unusual production profile, and every item below is a direct consequence:
| Property of LLM systems | What it forces on you |
|---|---|
| Responses take seconds, not milliseconds | Latency is a UX design problem (streaming, perceived speed), not just a p99 number |
| Cost is per token, and prompts grow | Cost is an architecture concern — caching, tiering, and context discipline |
| Providers rate-limit, time out, and deprecate | Reliability needs retries, fallbacks, circuit breakers, degradation paths |
| Output quality is invisible to HTTP status codes | Observability must trace and score, not just count 200s |
| The same input can produce different output | Evals & Testing replace assertions with measured distributions |
| Every input is untrusted, including retrieved text | Security has a whole new attack surface: prompt injection |
The unifying idea: a traditional service fails loudly; a language model fails plausibly. A broken database returns a 500. A broken RAG system returns a fluent, confident, wrong paragraph with a 200. Every practice in this section exists to make quality failures visible, bounded, and recoverable.
Why this matters
Everything here is invisible until the moment it isn't. Tutorials never mention token budgets, provider outages, judge drift, or what happens when retrieval returns nothing — so these are exactly the things you meet for the first time in production, with users watching.
What's in here
| Page | The question it answers |
|---|---|
| Latency | TTFT vs total time, streaming UX, parallel retrieval, semantic caching, routing |
| Cost | Token accounting, prompt caching, model tiering, truncation, batch processing |
| Reliability | Retries and backoff, timeouts, provider fallback, circuit breakers, idempotency |
| Observability | Tracing, token and quality metrics, drift detection, feedback loops |
| Evals & Testing | Testing non-deterministic systems, regression suites, CI for prompts, canaries |
| Security | Prompt injection and jailbreaks, RAG data leakage, moderation, adversarial testing |
Where this connects
- Production RAG and Agent Reliability — the same concerns, specialized.
- Choosing Models — most latency and cost levers are ultimately model-selection levers.
- System Design — operations is the part of a design that gets skipped, and the part that decides whether the system survives contact with users.