How to Answer AI System Design

AI system design is less about naming components and more about sizing them, budgeting latency per stage, stating what it costs, and explaining how you would know the system is working. A shallow design names boxes ("we'll use RAG with a vector DB"); a sound design attaches numbers, tradeoffs, and a feedback loop. This file is the method the worked designs in this folder follow — internalize the skeleton once, then reuse it on every AI system design problem.

The six-phase skeleton

Run every AI design problem through the same six phases. The time boxes below are a useful pacing discipline when you have roughly 45 minutes to work a design end to end; the phases themselves matter more than the clock.

Leave a few minutes at the end to restate the top two tradeoffs you made and what you would do with more time. The single most common failure mode is spending almost all of the time in phase 3 and never reaching 5 and 6 — evaluation and operations are where applied-AI design quality actually shows, because anyone can draw retriever → LLM.

Phase 1: Requirements — ask before you draw

Ask 4–6 clarifying questions, each tied to a design consequence. The generic set for LLM systems:

• Scale: users, QPS, corpus size. (Decides sharding, index choice, whether cost is a first-order concern.)
• Latency target: interactive chat (~1–2s to first token), search (<500ms), or batch (minutes)? (Decides model size, rerank depth, streaming.)
• Freshness: how stale can the index be — seconds, minutes, a day? (Decides incremental vs batch indexing.)
• Access control: do different users see different data? (If yes, ACL filtering at retrieval is a core component, not an add-on.)
• Accuracy/risk tolerance: what happens when the system is wrong? (Decides guardrails, human-in-the-loop, citation requirements.)
• Cost envelope: is there a budget per query or per month? (Decides model tiering and caching aggressiveness.)

Then state out loud what you are not building ("I'll assume auth infrastructure exists; I won't design the ingestion connectors in detail"). Scoping is how you keep the design coherent instead of trying to solve the entire company at once.

Phase 2: Capacity estimates — the numbers that anchor everything

Do this on the board (or a scratch pad), rounding aggressively. The four estimates that matter for LLM systems:

1. Query load: users × active fraction × queries/day → average QPS → ×5–10 for peak. Worked arithmetic: 50,000 users × 10% daily active = 5,000 active users. × 5 queries/day = 25,000 queries/day. ÷ 86,400 s/day ≈ 0.29 QPS average, so call it 0.3 QPS average and ~3 QPS peak (10×). At that load, serving throughput is rarely the hard problem — quality, freshness, and permissions usually are.
2. Corpus/index size: docs × tokens/doc → chunks (÷ ~500 tokens/chunk) → vectors × bytes/vector. A 1536-dim float32 vector is 1536 × 4 bytes = 6,144 bytes ≈ 6 KB. So 50M chunks × 6 KB = 300,000,000 KB = ~300 GB raw; with int8 quantization (~1 byte/dim + HNSW graph overhead) you land near ~75 GB.
3. Token throughput: queries × (prompt + completion tokens). A RAG query with 4k prompt tokens and 300 completion tokens at ~$3/M input, ~$15/M output (mid-tier 2026 pricing): input cost = 4,000 / 1,000,000 × $3 = $0.012; output = 300 / 1,000,000 × $15 = $0.0045; total ≈ $0.015–0.02/query.
4. Indexing throughput: corpus churn/day → docs/sec through the embedding pipeline. Embedding at ~$0.02/M tokens means a 25B-token corpus costs 25,000 × $0.02 = ~$500 to embed once with a cheap model — cheap in dollars; re-embedding on every model change is the real cost (pipeline time, not dollars).

State prices as orders of magnitude with a hedge ("roughly, at mid-2026 pricing"). Being within 2× with a stated assumption beats refusing to estimate.

Phase 3: Architecture — always two planes

Every LLM application has an offline/ingestion plane (extract → chunk → embed → index) and an online/serving plane (query → retrieve → assemble context → generate → post-process). Draw both, plus the third loop that is easy to forget: the evaluation/feedback plane (log traces → label/judge → regression suite → redeploy). Narrate data flow left to right, then say which components you want to deep-dive and pick the two or three that carry the most risk for this problem.

Cross-links that almost always matter when you draw the planes:

• Ingestion and chunking: ingestion and chunking
• Retrieval mechanics: retrieval patterns, hybrid search
• Serving reliability: reliability, observability

Phase 4: Deep dives — mechanism, not vocabulary

In a deep dive, the bar is: could you implement it tomorrow? "We'll rerank" is vocabulary; "cross-encoder over the top 50 candidates, ~60–80ms, which is why we cap candidates at 50" is mechanism. Always attach a number (latency, size, cost) and a tradeoff (what you gave up) to every choice. When you don't know a number, derive it or bound it aloud.

Good deep-dive targets across the designs in this folder:

Phase 5: Evaluation — the applied-AI differentiator

Cover three layers, in order of increasing cost:

• Component metrics, offline: retrieval recall@k / MRR against a golden set; judge-scored faithfulness and answer relevance on ~100–500 curated queries. Runs in CI on every prompt/model change. See RAG evaluation and evals and testing.
• System metrics, online: thumbs up/down rate, deflection/resolution rate, latency p95, cost/query.
• The loop: production failures get mined from logs → added to the golden set → the eval suite grows to cover the actual failure distribution. That loop is the difference between "we have evals" and "we have an eval process."

Phase 6: Operations — LLM-specific failure modes

Generic reliability plus the four LLM-specific ones:

1. Provider outage/degradation — fallback model behind an abstraction, ideally cross-provider.
2. Cost blowout — per-user token budgets, alerts on $/query drift.
3. Quality regression — model version pinning + eval gate before upgrades.
4. Prompt injection / data exfiltration — treat retrieved content as untrusted input, restrict tool scopes. See security.

Name graceful degradation modes: drop reranking, shrink k, fall back to a smaller model, serve cached answers — in that order. A design that only has a happy path is incomplete.

Stating assumptions and tradeoffs aloud

• Assumption template: "I'll assume X — say, 5 queries per user per day; if it's 10× that, the design changes at the retrieval layer, and I'll flag where." Assumptions are free; silent assumptions are fatal.
• Tradeoff template: "Two options: A gives lower latency, B gives better recall. Given the stated <500ms target, I'd pick A and recover recall with a better embedding model." Always tie the pick back to a requirement from phase 1 — that's what makes it a decision instead of a preference.
• When corrected, update visibly: "Good point — at that scale the semantic cache hit rate won't pay for its complexity; I'd cut it." Defending a wrong branch costs more than the mistake did.

Failure patterns that sink designs

1. Jumping to architecture — drawing the RAG diagram immediately with no requirements. You then optimize the wrong system.
2. No numbers — every component named, nothing sized. Reads as blog-post knowledge rather than engineering judgment.
3. Ignoring eval and ops — the answer ends at "then the LLM generates the answer." In applied AI that's the halfway point.
4. Framework name-dropping — "we'll use LangChain and Pinecone" as a substitute for describing what the code does. Name mechanisms; mention tools only as implementations of them.
5. Uniform depth — shallow coverage of everything. Better: explicit shallow passes plus 2–3 deliberate deep dives on the riskiest components.
6. Treating the LLM as deterministic — no retry/validation story for malformed output, no hallucination mitigation, no version pinning. Non-determinism is a first-class constraint; design for it up front. See sampling and determinism and hallucination.

How the worked designs use this skeleton

Test yourself

Go deeper

• Building A Generative AI Platform — Chip Huyen — the canonical component-by-component architecture of an LLM application platform; the mental model behind phase 3.
• Patterns for Building LLM-based Systems & Products — Eugene Yan — evals, RAG, caching, guardrails, feedback: the shared vocabulary these designs are built from.
• Your AI Product Needs Evals — Hamel Husain — the three-level eval framework that phase 5 leans on.
• Design ChatGPT - System Design Mock Interview (with eBay EM) — Exponent — watch how a real candidate paces requirements → architecture → deep dives on an AI question.
• A Survey of Techniques for Maximizing LLM Performance — OpenAI — the prompt-engineering vs RAG vs fine-tuning decision framework, straight from the provider.

Where this connects

• Design: Enterprise RAG — apply the full six phases to the canonical quality-and-permissions problem at modest QPS.
• Design: LLM Gateway — the platform layer that makes cost attribution, routing, and kill switches real for every other design.
• Production RAG — freshness, ACLs, cost, and degradation in the retrieval path specifically.
• Evals and testing — how to operationalize phase 5 as a regression suite rather than a one-off score.