Design: Enterprise RAG
RAG over 10M documents for 50k users — the canonical question.
Prerequisites
- How to reason about AI system design — the six-phase skeleton this walkthrough follows.
- RAG pipeline — end-to-end retrieval + generation before enterprise constraints.
- Filtering and metadata — why ACL pre-filters matter inside ANN.
- Production RAG — freshness, ACLs, cost, and degradation patterns.
The problem
Design a company-wide knowledge assistant: natural-language Q&A with citations over roughly 10M internal documents for 50k employees. Interview variants ("document search at that scale," "productionize RAG for the enterprise") are the same system.
Generation is the easy 10%. Enterprise hardness is four constraints tutorial RAG never hits: ingestion at scale, incremental freshness, per-user access control, and retrieval quality you can measure. Miss any one and the product leaks, lies with confidence, or never ships.
In the governed enterprise platform, this is the entitlement-filtered retrieval layer employees use every day — and the layer that fails loudly if an M&A doc leaks or last week's policy is still what the assistant cites.
Why it is hard
Tutorial RAG is: chunk a folder of PDFs, embed, top-k, prompt. That shape collapses here.
Sources are heterogeneous (SharePoint, Confluence, Drive, tickets, scanned PDFs), so connectors and layout-aware parsing become a subsystem. Permissions are per-document and change — "the assistant leaked the M&A doc" kills the project — so ACL at retrieval is a core axis. Freshness rules out nightly batch when a 9:00 edit must be answerable at 9:05. Enterprise queries are full of exact-match tokens (error codes, codenames, acronyms) that pure dense retrieval loses, so hybrid search (dense ANN + BM25, fused by rank) is structural. Quality without measurement is theater: golden set, deploy gates, a loop that mines real failures. Cost is real but secondary at this QPS.
Alternatives that lose: pure keyword (no NL intent), pure dense without ACLs (compliance failure), nightly reindex (stale answers that look authoritative).
Requirements and scale
Clarifying questions (each tied to a design consequence)
- Sources and formats? Connector complexity; scanned PDFs need OCR.
- Per-user permissions? Almost always yes → ACL at retrieval is core.
- Freshness? Minutes → event-driven incremental indexing.
- Latency and mode? Streaming chat ~1s TTFT; search page <500ms total.
- Multilingual or structured? Embedding choice; scope text-to-SQL out if possible.
- Budget? Model tier and caching aggressiveness.
Requirements (assumed)
| Kind | Requirement |
|---|---|
| Functional | NL Q&A with citations over 10M docs; per-doc ACLs; edits visible within ~15 min |
| Non-functional | p95 TTFT < 1.5s; ≥90% recall@10; cite or refuse; ~$10k/month serving |
Capacity estimates
Illustrative arithmetic (prices as mid-2026 orders of magnitude):
| Quantity | Estimate | Derivation |
|---|---|---|
| Corpus / chunks | ~25B tokens → ~50M chunks | 10M × ~2.5k tok; ~512-tok chunks |
| Vector storage | ~300 GB fp32 → ~75 GB int8 | 1536-dim ≈ 6 KB/vec; int8 + HNSW |
| Embed once | ~$500–3,000 | 25B × ~$0.02–0.13/M |
| Query load | ~0.3 QPS avg, ~3 QPS peak | 50k × 10% DAU × 5 q/day; peak ~10× |
| LLM cost | ~$0.015–0.02/q → ~$9–13k/mo | |
| Index churn | ~1–2 docs/s | ~1% of corpus/day |
Takeaway: at 3 QPS peak this is not throughput — it is quality, freshness, and permissions.
The design
Two planes plus a loop: ingestion keeps the index honest, serving answers under ACL, eval turns failures into the next golden set. The loop is the deliverable.
Ingestion plane
Connectors emit create/update/delete events (webhooks or change-token polls) into a queue. Workers parse (layout-aware PDF for the hard 20% — ingestion and chunking), chunk at ~512 tokens on headings, then prepend a short document-context string before embedding. That is contextual retrieval: the chunk keeps which company, quarter, or policy it belongs to. Anthropic reported ~49% fewer top-20 retrieval failures with contextual embeddings + BM25, ~67% with a reranker (their evals — directional, not a guarantee on your corpus). Batch-embed and upsert into both indexes. Full backfill at ~100k tokens/s is ~3 days for 25B tokens. Re-embed with a blue/green index swap, never in place.
Index and hybrid retrieval
One HNSW vector index (int8, ~75 GB on 2–3 nodes with replication) plus BM25 over the same chunks. Fuse ranks with reciprocal rank fusion (RRF) — rank-based, so you need not calibrate BM25 against cosine. Hybrid exists because embeddings compress rare identifiers away; BM25 catches them. See hybrid search, ANN indexes, vector DB landscape, scaling and operations.
ACL enforcement — core, not optional
Store principal lists as chunk metadata, synced from source permission APIs. Expand the caller's groups once per session (5-min TTL). Enforce as a pre-filter inside the ANN query (filtered HNSW), not after top-k. Post-filter empties results when the authorized subset is sparse in embedding space — and a bug fails open (data leak). Deletes and revocations take a fast lane (~1 min vs ~15 min for content).
Common misconception
Post-filtering ACLs after top-k looks simpler but fails open under bugs and fails closed under sparse permissions. Pre-filter inside ANN is the enterprise default.
Serving path — latency budget
| Stage | p50 | p95 |
|---|---|---|
| Query rewrite (optional; skip short queries) | 150 ms | 300 ms |
| Embed query | 20 ms | 50 ms |
| Hybrid retrieve top 100 (parallel, ACL pre-filtered) | 30 ms | 80 ms |
| Cross-encoder 100 → 8 (rerankers) | 60 ms | 120 ms |
| Context assembly (~4k tokens) | 5 ms | 10 ms |
| LLM TTFT (streaming) | 400 ms | 900 ms |
| Total to first token | ~0.7 s | ~1.5 s |
p95 stack: 300+50+80+120+10+900 = 1,460 ms. A cross-encoder scores (query, passage) jointly — more accurate than bi-encoder cosine, too slow for the full corpus — so it only sees the hybrid top 100. Under load, rerank is the first stage you shed.
Evaluation
Golden set of 300–500 (query → relevant chunk → reference answer): seed synthetic, replace with mined real queries. Track recall@10 and MRR for retrieval separately from faithfulness (LLM-as-judge, human spot-check) — RAG evaluation. Gate every prompt/chunk/embed/model change: >2-point recall drop blocks deploy. Online metrics feed weekly triage into the set.
Key decisions and tradeoffs
| Decision | Choice | Why / tradeoff |
|---|---|---|
| Retrieval | Hybrid + RRF | Extra index; catches identifier queries dense loses |
| ACL | Pre-filter in ANN | Harder than post-filter; avoids empty results and fail-open |
| Chunks / rerank | ~512 tok + context headers; 100→8 | Headers cost embed tokens; +60–120 ms, first shed under load |
| Freshness | Event-driven, 15 min / 1 min ACL | More parts than nightly batch; correct for trust |
| Serving / re-embed | Few nodes; blue/green | No multi-region at 3 QPS; 2× storage at cutover |
| Grounding | Cite or refuse | More refusals; fewer silent hallucinations |
Key insight
At this scale the dominant surface is who can see what and whether the right chunk was retrieved — not LLM tokens per second.
The flows
Flow A — Online query
Auth question → optional rewrite → embed + expand groups → hybrid top 100 with ACL pre-filter → cross-encoder top 8 → ~4k context with citations → stream grounded answer or refuse → log full trace.
Breaks when: ACL metadata is stale, fusion is mis-weighted, or the supporting chunk never enters context.
Flow B — Incremental ingest
Webhook/poll → queue → parse/re-chunk → contextual headers → batch embed → upsert → update doc→chunk map. Target: answerable within ~15 min.
Breaks when: missed events, tables dropped by the parser, or upsert races a delete.
Flow C — Permission revoke / delete (fast lane)
Revoke or delete → priority queue → update/tombstone all chunks via doc→chunk map → invalidate group-cache on membership change. Next query within ~1 min. Semantic cache keys must include a permission-set hash so answers cannot cross ACL boundaries.
Breaks when: cache is not ACL-scoped, or group cache is not invalidated on membership events.
Flow D — Eval gate
CI golden set on every change; >2-point recall drop blocks deploy. Weekly online mining grows the set. Canary only after the gate passes.
Breaks when: synthetic-only goldens miss identifier queries, or mining never runs.
Failure modes and degradation
| Failure | Response |
|---|---|
| LLM outage | Fallback via gateway; extractive top passages |
| Index node loss | Replicas; rebuild from document store, not sources |
| Cost drift | Per-team metering; alert at 120% of forecast |
| Model upgrade regression | Pin versions; eval gate; canary ~5% |
| Poisoned documents | Treat chunks as untrusted (security); no write actions |
| Overload | Drop rerank → shrink k → skip rewrite → smaller model → extractive |
See reliability and latency.
Common drill-downs
User lost access to a doc. Permission event → fast lane → retag chunks via doc→chunk map → pre-filter excludes them. Invalidate that user's group cache. If semantic cache lacks a permission-set hash, a cached answer still leaks.
Why hybrid? Embeddings compress rare identifiers (SKU-4471, "Project Foxtrot") into one vector; BM25 catches them if you already keep a text index for filters.
Recall@10 is 70% — where first? Chunking bugs → missing keyword path → vocabulary mismatch (rewrite or contextual headers) → only then model shopping. Segment eval by query type first.
10M → 500M docs? Shard by department; prioritize backfill by access frequency; represent "everyone" groups specially. That scale is document Q&A at scale.
Stop stale or deleted answers? Fast-lane tombstones (~1 min). Prompt includes chunk last-modified and prefers recency on conflict. Citations are versioned links.
Test yourself
Peak load is ~3 QPS. A proposal adds multi-region active-active vector indexes 'for scale.' How do you respond?
Post-filter ACLs after top 100. A restricted user often gets empty answers. Diagnose.
Monthly LLM spend is ~$12k pre-cache against a $10k budget. Which levers, in order?
After switching embedding models, recall@10 drops 5 points on identifier queries only. What failed?
Full re-embed takes ~3 days. How do you cut over without a quality hole?
Go deeper
- Introducing Contextual Retrieval — Anthropic — chunk-context before embed/BM25; measured failure reductions (~49% / ~67% with rerank in their evals).
- Systematically Improving Your RAG — Jason Liu — synthetic evals first, hybrid search, feedback mining.
- Building A Generative AI Platform — Chip Huyen — where enterprise RAG sits in the full platform (gateway, caches, guardrails, observability).
- Building Production-Ready RAG Applications: Jerry Liu — AI Engineer — production concerns for this class of system.
Where this connects
- Document Q&A at scale — same retrieval under multi-tenant SaaS throughput.
- LLM gateway — model routing, budgets, and fallback — call this, do not hard-code a provider.
- Advanced RAG — agentic and graph patterns when single-shot hybrid is not enough.
- Observability — per-stage traces for recall and cost regressions.