System Design
Everything above, assembled into one system, under constraints that conflict.
Prerequisites
Effectively the whole site — this section assembles it. At minimum:
- The RAG Pipeline and Production RAG
- Agent Foundations
- Production — especially cost and latency
Why this section exists
Designing an AI system is not a distributed-systems problem with an LLM sprinkled on top, and it isn't a matter of knowing the components either. It's holding five things in tension — quality, latency, cost, freshness, and safety — while staying explicit about what you'd measure.
Two failure modes dominate, and they're opposites:
- Naming without structure. Pinecone, LangGraph, and a reranker get chosen before a single requirement is stated. Impressive vocabulary, no engineering.
- Structure without commitment. Endless "it depends" and no decision. A design has to choose, defend the choice, and name what would change it.
How to reason about it gives the framework that avoids both: requirements → data → pipeline → serving → evaluation → operations. It's the thing that keeps an open-ended problem from stalling at the first step.
Key insight
Do the arithmetic explicitly. 10M documents × 4 chunks × 1536 dims × 4 bytes ≈ 245 GB of raw vectors — which doesn't fit one node's RAM, which forces sharding or quantization. Numbers are what turn a plausible-sounding architecture into a defensible one, and they usually decide the design for you before any preference does.
What's in here
| Page | The question it answers |
|---|---|
| How to reason about it | The repeatable framework — read this before any of the worked designs |
| Design: Enterprise RAG | RAG over 10M documents for 50k users — the canonical question |
| Design: Document Q&A at Scale | High-volume document QA with freshness and access control |
| Design: Semantic Code Search | Search over a large codebase — why code isn't prose, and what that changes |
| Design: Customer Support Agent | An agent that resolves tickets, with tools, escalation, and guardrails |
| Design: LLM Gateway | Governed multi-model access: quotas, audit, routing — the enterprise middleware question |
Where this connects
- Rapid Fire — the compressed recall layer over everything here.
- Every earlier section — when a design feels thin at some step, that step's page is the repair.