How to Answer AI System Design

A shallow design names boxes ("RAG with a vector DB"). A sound design sizes them, budgets latency and cost per stage, and says how you would know the system is working. Learn this skeleton once; reuse it on every problem in this folder.

The intuition

Treat the design like a construction budget, not a furniture catalogue. Naming "vector DB + reranker + GPT" is picking fixtures. Engineering asks how many people walk through the door, how fast each room must respond, what happens when a supplier is late, and how you detect failure before users do. In applied AI: size the load, budget latency and dollars per stage, pick mechanisms (not product names), and close with evaluation and operations. Quality is part of the architecture — availability and p95 alone do not prove answers are right.

Why it exists

Open-ended AI problems stall two ways: name-dropping stacks before any requirement, or saying "it depends" forever without committing. Without a skeleton people camp on retriever → LLM and never answer how stale the index can be, who may see which chunks, what a wrong answer costs, or how regressions get caught. Those decisions separate a demo from a system.

The core idea

Run every AI design through six phases. Time boxes pace a ~45-minute pass; phases matter more than the clock.

Leave a few minutes to restate the top two tradeoffs. The usual failure mode is never leaving phase 3 — eval and ops are where applied-AI design quality shows.

How it actually works

Phase 1 — Requirements. Ask 4–6 clarifying questions, each with a design consequence: scale (users, QPS, corpus → sharding; whether cost is first-order), latency (chat ~1–2s to first token, search under 500ms, or batch → model size, rerank depth, streaming), freshness (seconds vs a day → incremental vs batch indexing), access control (per-user data → ACL filtering at retrieval is core, not an add-on), risk (cost of wrong answers → guardrails, human-in-the-loop, citations), and cost envelope (budget → model tiering, caching). State what you are not building. "How many users?" is incomplete; "How many users — because that decides whether we shard" drives a decision.

Phase 2 — Capacity. Round aggressively. Four estimates: (1) Query load — users × DAU × queries/day → average QPS → ×5–10 peak. Illustrative: 50k × 10% × 5/day → ~0.3 QPS avg, ~3 QPS peak — quality/freshness/permissions usually beat serving throughput. (2) Index size — 1536-dim float32 ≈ 6 KB/vector; 50M chunks ≈ ~300 GB raw, often ~75 GB after int8 (single-node vs sharded vs quantized). (3) Token cost — illustrative mid-2026 mid-tier (~$3/M in, ~$15/M out): 4k+300 tokens ≈ $0.015–0.02/query. (4) Indexing — embedding dollars are often small; re-embedding on model change is the real cost. Within 2× with a stated assumption is enough.

Phase 3 — Architecture. Draw three planes: offline/ingestion (extract → chunk → embed → index), online/serving (query → retrieve → assemble → generate → post-process), and evaluation/feedback (log → label/judge → regression suite → redeploy). Narrate left to right, then deep-dive the riskiest components. Mechanism depth lives on sibling pages: ingestion and chunking, retrieval patterns, hybrid search, reliability, observability.

Phase 4 — Deep dives. The bar: could you implement it tomorrow? "We'll rerank" is vocabulary; "cross-encoder over top 50, ~60–80ms, so we cap at 50" is mechanism. Attach a number and a tradeoff to every choice. Dominant risk selects depth — see The flows.

Phase 5 — Evaluation. Three layers, cheapest first. Offline: recall@k / MRR on a golden set; judge-scored faithfulness and answer relevance on ~100–500 queries — in CI on every prompt/model change (RAG evaluation, evals and testing). Online: thumbs, deflection/resolution, p95 latency, cost/query. The loop: mine production failures into the golden set — the difference between a score and a process.

Phase 6 — Operations. Four LLM-specific modes: provider outage (fallback behind an abstraction, ideally cross-provider), cost blowout (per-user token budgets, $/query alerts), quality regression (version pin + eval gate), prompt injection / exfiltration (treat retrieved content as untrusted; restrict tools — security). Degrade in order: drop rerank → shrink k → smaller model → cached answers. Design for non-determinism: validation, grounding, version pins (sampling and determinism, hallucination).

The flows

The skeleton is fixed; phase-4 depth follows dominant risk:

What breaks the flow is uniform shallow coverage, or a full diagram with no numbers. Prefer a shallow pass everywhere, then two deep dives on the riskiest axes.

A worked example

On the governed enterprise platform, enterprise RAG at 50k employees, ~10% DAU, ~5 queries/user/day lands at ~0.3 QPS average and ~3 QPS peak (phase 2 arithmetic). Corpus at 10M docs is tens of millions of chunks and hundreds of GB float32 before quantization. Conclusion to say out loud: not a serving-throughput problem — spend complexity on ACLs, hybrid retrieval, freshness, and the eval loop.

Break it. Skip phase 2 for "Pinecone + LangChain + GPT" and you over-build serving while under-designing permission filters. Or chase low recall@10 by shopping embedding models before segmenting failures: identifiers need BM25, mid-table splits need chunking, vocabulary mismatch needs rewrite. Global metric moves without segmentation often help one slice and hurt another.

Production concerns

Patterns that sink designs under review and under traffic:

1. Jumping to architecture — diagram first; wrong system optimized.
2. No numbers — components named, nothing sized.
3. Ignoring eval and ops — ends at "the LLM generates." Halfway point in applied AI.
4. Framework name-dropping — tools instead of mechanisms.
5. Uniform depth — shallow everywhere; pick two risky deep dives instead.
6. LLM as deterministic — no validation, grounding, or version pin.

Assumption template: "I'll assume 5 queries/user/day; if 10×, the design changes at retrieval." Silent assumptions are fatal. Tradeoff template: "A is lower latency, B is better recall. Given a sub-500ms target, pick A and recover recall with a better embedding." Tie every pick to a phase-1 requirement. When corrected, update visibly — defending a wrong branch costs more than the mistake.

Test yourself

Go deeper

• Building A Generative AI Platform — Chip Huyen — three planes for phase 3.
• Patterns for Building LLM-based Systems & Products — Eugene Yan — evals, RAG, caching, guardrails.
• Your AI Product Needs Evals — Hamel Husain — three-level eval framework for phase 5.
• Design ChatGPT - System Design Mock Interview — Exponent — pacing requirements → architecture → deep dives.
• A Survey of Techniques for Maximizing LLM Performance — OpenAI — prompt engineering vs RAG vs fine-tuning.

Where this connects

• Design: Enterprise RAG — full six phases on quality and permissions at modest QPS.
• Design: LLM Gateway — cost attribution, routing, and kill switches for every other design.
• Production RAG — freshness, ACLs, cost, and degradation on the retrieval path.
• Evals and testing — phase 5 as a regression suite, not a one-off score.