Design: Document Q&A at Scale
High-volume document question-answering with freshness and access control.
Prerequisites
- How to reason about AI system design — capacity estimates and degradation ladders.
- Design: Enterprise RAG — quality and permissions at tiny QPS; this page is the throughput variant.
- Production RAG — freshness, ACLs, cost, latency under load.
- Cost and Latency — unit economics and tail control.
The problem
Design a document Q&A API that serves thousands of customers as a SaaS product — or, equivalently: your RAG demo works; make it serve 500 QPS across 2,000 tenants with a contractual SLA.
Where enterprise RAG is a quality-and-permissions problem at tiny QPS, this is the throughput and multi-tenancy variant. Isolation, cache design, tail latency, and what you shed under load matter more than "can we retrieve a good chunk in a demo."
In the governed enterprise platform, this is internal document Q&A turned into a multi-tenant product: many corpora, strict isolation, and a bill where the LLM API dwarfs infrastructure.
Why it is hard
Tenant size follows a power law — ~20 whales with millions of docs, a long tail with ~1k.
Shared everything creates noisy neighbors; dedicated everything is impossible for the tail. A
cross-tenant leak ends the company, so isolation must be structural at every layer (auth,
store queries, caches), not a single WHERE tenant_id = ?.
Contractual p95 forces engineered tails: hedges, admission control, per-tenant quotas. A 5-minute upload SLA dies if a whale dumps 5M docs into a shared queue without fair ingest. The LLM bill is ~10:1 over infra, so caches and model routing are the product. Query mix varies by tenant (support-KB repeats; discovery does not), so global cache policy fails.
Alternatives that lose: one shared unpartitioned index; a global semantic cache (leak + wrong-corpus answers); scaling the LLM tier without caching (burn the margin).
Requirements and scale
Clarifying questions (each tied to a design consequence)
- Tenant size distribution? Power law → tiered isolation (dedicated shards for whales, shared partitions for the tail).
- SLA — latency, availability, freshness? Contractual p95 forces hedges and per-tenant quotas.
- Upload-to-queryable target? Minutes vs hours is a 10× difference in indexing architecture.
- Query mix: repeated or unique? Support-KB is caching gold; discovery is not. Cache per-tenant.
- Logical or physical isolation? Dedicated infra or region pinning is a product tier.
- Sync or async? Bulk QA over 10k docs should be an async batch path (~50% cost).
Requirements (assumed)
| Kind | Requirement |
|---|---|
| Scale | 2,000 tenants, ~100M documents (top 20 hold ~50%) |
| Scale | 500 QPS peak, multi-region |
| Latency | p95 ≤ 2.5 s full answer; p95 generation TTFT ≤ 800 ms (~1.1 s end-to-end first token on miss path) |
| Freshness | Upload-to-queryable ≤ 5 min |
| Availability | 99.9% |
| Isolation | Strict tenant isolation — cross-tenant leak is catastrophic |
Capacity estimates
| Quantity | Estimate | Derivation |
|---|---|---|
| Corpus | ~250B tokens → ~500M chunks | 100M × ~2.5k tokens; ~500 tokens/chunk |
| Vector storage | ~750 GB int8 (1536-dim) | 500M × ~1.5 KB; ~12–15 shards + replicas |
| Query load | 500 QPS peak ≈ 20M/day sustained | 60% interactive, 40% API/batch |
| LLM cost | ~$0.015/query → ~$300k/mo uncached | 4k in / 300 out at ~$3/$15 per M (illustrative mid-2026); after cache+routing ~$120–150k/mo |
| Ingest | ~1M docs/day, 10× burst | ~12 docs/s sustained; whale onboarding is the burst |
| In-flight (miss path) | ~800 concurrent | Little's law: 500 QPS × 1.6 s mean — size pools for this |
Cache and routing are the unit-economics design: LLM spend dominates infra ~10:1.
The design
Multi-tenant isolation — three tiers, one code path
Tiered multitenancy pools small tenants for cost, silos large ones for noise isolation, and sells full physical isolation as a premium tier. Modern vector stores implement this directly — payload partitions for the long tail, named shards for whales, transparent tenant promotion from a shared fallback shard (Qdrant multitenancy, scaling and operations).
- Tail (~1,900): one collection per embedding model, partitioned by
tenant_idwith a mandatory filter. Thousands of collections is an anti-pattern. Index the tenant field as a tenant keyword (is_tenant-style) so vectors co-locate. - Whales (~20): dedicated shards. Promote when corpus size or sustained QPS crosses a share-of-pool threshold — a hot small tenant is still a noisy neighbor.
- Contractual isolation: dedicated cluster/region, priced as such.
Enforcement is structural: tenant_id from the verified JWT only, injected by middleware; the
repository refuses unfiltered queries; cache namespaces are tenant-scoped.
Common misconception
Filtering by tenant_id in the vector query is necessary but not sufficient. The classic leak
is an unscoped semantic cache serving tenant A's answer to tenant B's similar question.
Freshness pipeline
Upload → durable queue → parse/chunk/embed → shard upsert → cache invalidation for that tenant. Weighted fair queuing so a whale's 5M-doc backfill cannot blow the 5-min SLA for a tail tenant's single upload. Track upload-to-queryable as a per-tenant SLI. Deletes and permission changes take a fast lane (< 60 s). Chunking: ingestion and chunking.
Caching — four layers, cheapest first
| Layer | Hit rate (est.) | Saves | Key design point |
|---|---|---|---|
| L1 exact response | 10–20% (support-KB higher) | full pipeline | key = hash(tenant, normalized query, corpus version) |
| L2 semantic response | +5–10% | full pipeline | per-tenant only; similarity ≥ 0.97 + safety check |
| Provider prompt cache | 60–90% of stable prefix tokens | up to ~90% of prefix input cost | static system + tenant policy first; query and chunks last |
| Embedding cache | 20–40% of query embeds | ~20 ms + embed cost | trivial, do it |
A semantic cache keys answers by embedding similarity rather than exact string match. That raises hit rate on rephrases — and makes tenant scoping non-negotiable. Global caching is legitimate only for tenant-agnostic artifacts (common query embeddings), never for answers.
Generation and model routing
A cheap classifier (or heuristics: query length, retrieval score margin) routes ~60% of high-confidence factual lookups to a small model (illustrative ~$0.3/$2.5 per M); the rest to mid-tier. Router mistakes are caught by eval. Batch endpoints use provider batch APIs at roughly half price. Call the LLM gateway for failover, TPM, and cost attribution (choosing models).
Serving path — latency budget (cache-miss)
| Stage | p50 | p95 | Degradation lever |
|---|---|---|---|
| Admission + auth + cache probes | 8 ms | 20 ms | — |
| Query embed | 15 ms | 40 ms | cache |
| Hybrid retrieval (tenant-filtered) | 40 ms | 120 ms | shrink k, drop BM25 leg |
| Rerank 60 → 8 | 50 ms | 110 ms | first thing shed under load |
| Generation TTFT | 350 ms | 800 ms | route to small model |
| Generation total (300 tok streamed) | 1.2 s | 2.2 s | cap max_tokens |
| End-to-end | ~1.6 s | ≤ 2.5 s | ~10% headroom |
p95 pre-generation ≈ 290 ms + gen TTFT 800 ms ≈ 1.1 s end-to-end to first token on the miss path. The 800 ms number is generation TTFT; hit tighter end-to-end first token via cache hits or by shedding rerank. Tail control: hedge retrieval at a p90 timer; per-tenant concurrency caps; timeout per stage with its degradation lever.
Evaluation
Per-tenant quality floors — a global average hides one broken corpus. Nightly canaries track recall, faithfulness, and latency; alert on per-tenant deltas. Sample semantic-cache hits and tighten the threshold if false-hit rate exceeds ~1%. Online: thumbs ratio, $/query per tenant. See RAG evaluation and evals and testing.
Key decisions and tradeoffs
| Decision | Choice | Tradeoff |
|---|---|---|
| Isolation | Tiered: shared filtered / dedicated shard / dedicated cluster | Promotion complexity; correct economics |
| Semantic cache | Per-tenant only | Lower global hit rate; avoids leak and wrong-corpus answers |
| Rerank under load | First shed | Slight quality dip; protects TTFT |
| Model routing | ~60% small model | Router errors need eval catch; large cost/latency win |
| Ingest fairness | Weighted fair queue per tenant | Whale onboarding slower; protects everyone else's SLA |
| Batch path | Async + ~50% batch discount | Product complexity; keeps interactive path clean |
| Prompt cache layout | Static prefix first | Prompt discipline; largest cheap cost lever |
Key insight
At 500 QPS the design centers on isolation + tails + COGS. A perfect chunker that ignores tenant-scoped caches and fair ingest still produces incidents finance and security care about more than recall@10.
The flows
Flow A — Interactive query (cache miss)
Admission (per-tenant rate + interactive priority) → L1 then L2 (tenant-scoped) → on miss: embed → tenant-filtered hybrid retrieve → optional rerank → model router → generate → stream, trace, fill caches with tenant keys + corpus version.
Breaks when: tenant filter omitted, cache key drops tenant, or rerank saturates without shed.
Flow B — Interactive query (cache hit)
Same admission. L1 returns the full answer, or L2 after similarity ≥ 0.97 + safety check. Audit-log the hit for attribution.
Breaks when: ingest failed to invalidate after an update, or L2 false-positive.
Flow C — Document upload to queryable
Upload → fair-queued per tenant → parse/chunk/embed → upsert to correct shard tier → invalidate that tenant's cache entries. SLI: ≤ 5 min queryable; deletes/permission changes < 60 s fast lane.
Breaks when: a whale monopolizes the queue, or invalidation is best-effort.
Flow D — Overload degradation
Detect stage latency, provider errors, queue depth. Shed in order: rerank → force small model → relaxed cache threshold with "may be outdated" flag → retrieval-only (passages + citations) → admit paid interactive last, reject batch/free first. Two providers behind the gateway; monthly failover drills.
Breaks when: degradation is ad hoc, or batch is not distinguishable from interactive.
Failure modes and degradation
The ladder in Flow D is the production response — each step trades quality for staying up, including retrieval-only mode (good search, not a 503, under total LLM outage). Bad chunker deploys recover from the document store via shard-scoped reindex, blue/green. Align with reliability and LLM gateway.
Common drill-downs
Prove tenant A never sees tenant B's data. JWT-derived tenant_id (never client-supplied);
repository that cannot express unfiltered queries; tenant-scoped caches; adversarial cross-tenant
tests; periodic auditor sampling vector payloads against ownership. Design the cache leak first.
p95 is 2.5 s but p99 is 9 s. Decompose by stage: generation-length outliers (cap max_tokens, measure TTFT separately); ANN on the biggest shared shard (promote the whale); cold cache after deploys (warm L1 from top-1k per tenant); provider TTFT spikes (hedge high-tier tenants).
A tenant uploads 5M docs at 9 am. Without fairness every tenant misses the 5-min SLA. With it: fair queuing, autoscaled workers, async bulk-import with progress, and promotion to a dedicated shard before shared-tier search latency bloats.
Why per-tenant semantic cache? Same question, different corpus, different correct answer. A global answer cache is both a leak and a correctness bug.
Where does the money go? ~10:1 LLM vs infra. Levers in order: prompt-prefix caching → small-model routing (~60%) → response caches → only then infra tuning.
Test yourself
You find a semantic cache implementation keyed only by embedding(query). What is wrong, and how bad is it?
500 QPS peak, mean latency 1.6 s. Roughly how many in-flight requests?
Whale promotion threshold is 1M docs. A tenant has 200k docs but 100 QPS. Promote or not?
After a chunker deploy, one tenant's faithfulness collapses; global average barely moves. What was missing?
Prompt caching saves ~90% of prefix input cost. How do you structure the prompt so it hits?
Go deeper
- Multitenancy — Qdrant docs —
payload partitions,
is_tenantindexes, tiered shards, and transparent tenant promotion. - Secure multitenant RAG — Azure Architecture Center — store-per-tenant vs multitenant-store tradeoffs for RAG isolation.
- Prompt caching — Claude Docs — prefix rules, TTLs, and cache-hit pricing for the largest cheap cost lever.
- Building A Generative AI Platform — Chip Huyen — cache hierarchy, router, and gateway layers in full-platform context.
- Your AI Product Needs Evals — Hamel Husain — the eval process that per-tenant quality floors depend on.
Where this connects
- Enterprise RAG — quality/ACL-first at low QPS; contrast what changes when multi-tenancy and 500 QPS dominate.
- LLM gateway — provider failover, TPM limits, and cost attribution this path should not reimplement per product.
- Filtering and metadata — pre-filter mechanics tenant partitions rely on.
- Reliability — degradation ladders and timeout patterns in general form.