Design: Enterprise RAG

The problem

Design a company-wide knowledge assistant over all internal documents — natural-language Q&A with citations over roughly 10M documents for 50k employees. Variants of the same problem: document search + Q&A at that scale, or "productionize RAG for an enterprise."

What makes this the canonical design is not the generator. The generation step is the easy 10%. The work is the four things that make enterprise RAG hard: ingestion at scale, incremental freshness, per-user access control, and retrieval quality you can measure.

In the governed enterprise platform, this is the retrieval layer over internal documents with entitlement-filtered search — the feature employees touch every day, and the feature that fails loudly if an M&A doc leaks or last week's policy is still what the assistant cites.

Why it is hard

Tutorial RAG is: chunk a folder of PDFs, embed, top-k, prompt. That shape collapses under enterprise constraints for specific reasons:

1. Sources are heterogeneous. SharePoint, Confluence, Drive, tickets, PDFs with tables and scans — connectors and layout-aware parsing become a subsystem, not a script.
2. Permissions are per-document and change. Different users see different data. "The assistant leaked the M&A doc" is the failure mode that kills the project. ACL enforcement at retrieval is a core axis, not an add-on.
3. Freshness is contractual in spirit even when not on paper. If a doc edited at 9:00 must be answerable at 9:05, nightly batch indexing is wrong; you need event-driven incremental indexing.
4. Enterprise queries are full of exact-match tokens. Error codes, project codenames, acronyms, people names — pure dense retrieval loses them. Hybrid search is structural, not optional.
5. Quality without measurement is theater. You need a golden set, gates on deploy, and a loop that mines real failures — not a demo that looked good on five hand-picked questions.
6. Cost is real but secondary to quality at this QPS. At a few queries per second peak, over- engineering serving is the wrong place to spend complexity.

Alternatives that lose: pure keyword search (no NL intent), pure dense RAG without ACLs (compliance failure), and a single nightly reindex (stale answers that look authoritative).

Requirements and scale

Clarifying questions (each tied to a design consequence)

1. What are the document sources and formats? SharePoint/Confluence/Drive/PDFs — decides connector complexity and whether parsing is a major subsystem. PDFs with tables/scans need OCR and layout-aware parsing; wikis are trivial.
2. Do users have different permissions on documents? Almost always yes in enterprise. ACL enforcement at retrieval time becomes a core design axis.
3. Freshness requirement? If a doc edited at 9:00 must be answerable at 9:05, you need event-driven incremental indexing, not nightly batch.
4. Latency target and interaction mode? Chat with streaming tolerates ~1s to first token; a search-results page wants <500ms total.
5. Multilingual? Structured data too? Embedding model choice; whether you also need text-to-SQL — scope it out if possible.
6. Budget sensitivity? Decides model tier and how aggressive caching must be.

Requirements (assumed after clarification)

Capacity estimates

Takeaway: at 3 QPS peak, this is not a throughput problem — it is a quality, freshness, and permissions problem. Do not over-engineer serving; spend complexity on retrieval and eval.

The design

Ingestion plane

Per-source connectors emit create/update/delete events (webhooks where the source supports them, else polling with change tokens) into a queue. Workers:

1. Parse — layout-aware PDF parsing for the hard 20%; simpler extractors for wikis and tickets. See ingestion and chunking.
2. Chunk at ~512 tokens respecting headings.
3. Prepend a 1–2 sentence document-context string to each chunk before embedding (contextual retrieval — cuts retrieval failure rate materially; see Anthropic's measured results in Go deeper).
4. Embed in batches (batch API is often ~50% cheaper than online embed).
5. Upsert into vector + keyword indexes.

Full corpus backfill: 25B tokens at ~5k tokens/s/worker with 20 workers:

• Throughput = 20 × 5,000 = 100,000 tokens/s
• Time = 25,000,000,000 / 100,000 = 250,000 s ≈ 250,000 / 86,400 ≈ 2.9 days ≈ 3 days

Plan re-embedding (new model) as a blue/green index swap, never in place.

Index

One vector index (HNSW, int8-quantized, ~75 GB — fits on 2–3 nodes with replication) plus a BM25/keyword index over the same chunks. Hybrid matters in enterprise because queries are full of exact-match tokens embeddings handle poorly: error codes, project codenames, acronyms, people. Merge with reciprocal rank fusion.

Index choice and scaling background: ANN indexes, vector DB landscape, scaling and operations.

ACL enforcement — core, not optional

Store principal lists (users/groups) as metadata on every chunk, synced from the source systems' permission APIs; expand the querying user's groups once per session (cached, 5-min TTL). Enforce as a pre-filter inside the ANN query (filtered HNSW), not post-retrieval — post-filtering can return 0 results for a user who matches few docs, and worse, a bug fails open.

Permission changes flow through the same event queue as content changes; deletes and permission revocations get a fast lane (target: enforced within 1 minute, vs 15 for content).

Serving path — latency budget

p95 stack check: 300 + 50 + 80 + 120 + 10 + 900 = 1,460 ms ≈ 1.5 s.

Evaluation

• Golden set: 300–500 (query → relevant-chunk → reference-answer) triples; seed with synthetic questions generated per document, then replace with mined real queries. Track recall@10 and MRR for retrieval separately from faithfulness/answer-quality (LLM-as-judge with a rubric, spot-checked by humans). Detail: RAG evaluation.
• Gate: eval suite runs on every prompt, chunking, embedding, or model change; a >2-point recall drop blocks deploy.
• Online: citation-click rate, thumbs ratio, refusal rate, p95 latency, $/query. Weekly triage of worst-rated answers feeds the golden set — the loop is the deliverable, not the score.

Key decisions and tradeoffs

The flows

Flow A — Online query (happy path)

1. User submits natural-language question (authenticated session).
2. Optional query rewrite for long/ambiguous questions; short keyword-ish queries skip rewrite.
3. Embed query; expand user's groups from session cache (5-min TTL).
4. Hybrid retrieve top 100 with ACL pre-filter on both vector and BM25 legs.
5. Cross-encoder rerank → top 8; assemble ~4k-token context with citations metadata.
6. LLM generates grounded answer with citations (or refuses if evidence insufficient).
7. Stream tokens; log full trace for eval mining.

Breaks when: ACL metadata stale, hybrid fusion mis-weighted, or context omits the one supporting chunk.

Flow B — Incremental ingest (content update)

1. Source emits create/update via webhook or change-token poll.
2. Event lands on ingest queue; worker parses and re-chunks the document.
3. Contextual headers applied; batch embed; upsert vectors + BM25 postings.
4. Doc→chunk mapping updated in document store.
5. Answerable within ~15 minutes target.

Breaks when: connector misses events, parser drops tables, or upsert races with a concurrent delete.

Flow C — Permission revoke / delete (fast lane)

1. Source emits permission change or delete.
2. Event takes fast lane (priority queue).
3. All chunks for that doc updated or tombstoned via doc→chunk map.
4. User's group-membership cache entry invalidated when membership changes.
5. Enforced on next query within ~1 minute; semantic cache keys include permission-set hash so cached answers cannot cross ACL boundaries.

Breaks when: semantic cache is not ACL-scoped, or group cache is not invalidated on membership events.

Flow D — Evaluation and regression gate

1. CI runs golden set on every prompt / chunk / embed / model change.
2. Recall@10 and faithfulness compared to baseline; >2-point recall drop blocks deploy.
3. Online worst-rated answers triaged weekly → new golden triples.
4. Canary or full deploy only after gate passes.

Breaks when: golden set is synthetic-only and misses real identifier queries, or online mining never runs.

Failure modes and degradation

Degradation order for latency pressure: drop reranking → shrink candidate k → skip query rewrite → smaller model → extractive passages only. Aligns with reliability and latency.

Security posture for retrieved content: security — prompt injection via documents is a real path; treat chunks as untrusted data.

Common drill-downs

A user just lost access to a document. Walk through what happens.

The source emits a permission event → fast-lane queue → metadata update on all chunks of that doc (doc→chunk mapping kept in the doc store) → enforced on next query via pre-filter; group-membership cache TTL (5 min) bounds staleness, and you can invalidate that user's cache entry on the event for ~immediate enforcement. Also: the semantic cache must be ACL-scoped, or a cached answer leaks across permission boundaries — cache key includes the permission-set hash.

Why hybrid retrieval? Wouldn't a better embedding model do?

Embeddings compress 512 tokens into one vector — rare exact identifiers (SKU-4471, "Project Foxtrot") are precisely what gets lost. BM25 catches them at ~zero marginal cost since you need a text index for filters anyway. Know the failure classes, not just the pattern name.

Your recall@10 is 70%. Where do you look first?

In order of measured payoff: chunking bugs (mid-table splits, lost headings) → missing keyword path for identifier queries → query/document vocabulary mismatch (fix with query rewrite or contextual chunk headers) → only then embedding model shopping. Segment the eval by query type before changing anything global.

10M docs becomes 500M. What breaks?

Vector index no longer fits a few nodes → shard by tenant/department (queries rarely span shards); embedding backfill becomes a weeks-long job → prioritize by access frequency; ACL metadata explodes for org-wide docs → represent "everyone" groups specially instead of materializing 50k principals per chunk; eval set must grow per domain.

How do you stop it answering from stale or deleted content?

Deletes get the fast lane (tombstone in index within 1 min); generation prompt includes each chunk's last-modified date and instructs recency preference on conflict; citations carry versioned links so a stale answer is at least auditable.

Test yourself

Go deeper

• Introducing Contextual Retrieval — Anthropic — the chunk-context technique above, with measured retrieval-failure reductions.
• Systematically Improving Your RAG — Jason Liu — the retrieval-first improvement loop: synthetic evals, hybrid search, feedback mining.
• Building A Generative AI Platform — Chip Huyen — where enterprise RAG sits inside the full platform: gateway, caches, guardrails, observability.
• Building Production-Ready RAG Applications: Jerry Liu — AI Engineer — the most-watched conference talk on exactly this question's production concerns.

Where this connects

• Document Q&A at scale — same retrieval ideas under multi-tenant SaaS throughput where QPS and isolation dominate.
• LLM gateway — the control plane for model routing, budgets, and fallback that enterprise RAG should call rather than hard-coding a provider.
• Advanced RAG — agentic and graph patterns when single-shot hybrid retrieval is not enough.
• Observability — tracing the serving path so recall and cost regressions are visible per stage.