Design: Document Q&A at Scale

The problem

Design a document Q&A API that serves thousands of customers as a SaaS product — or, equivalently: your RAG demo works; make it serve 500 QPS across 2,000 tenants with a contractual SLA.

Where enterprise RAG is a quality-and-permissions problem at tiny QPS, this is the throughput and multi-tenancy variant. The design must survive scale: tenant isolation, cache design, per-stage latency engineering, and what you shed when overloaded. The pressure points are tail latency and noisy neighbors, not "can we retrieve a good chunk in a demo."

In the governed enterprise platform framing, this is what happens when internal document Q&A becomes a multi-tenant product: many corpora, strict isolation, and a bill where the LLM API dwarfs infrastructure.

Why it is hard

1. Tenant size is a power law. 2,000 tenants is never uniform — expect ~20 whales with millions of docs and a long tail with ~1k. Shared everything creates noisy-neighbor latency; dedicated everything is economically impossible for the tail.
2. A cross-tenant leak is a company-ending bug. Isolation must be structural at every layer — auth, store queries, caches, embeddings namespaces — not a single WHERE tenant_id = ? hope.
3. SLA turns "fast on average" into engineered tails. Contractual p95 needs hedged requests, admission control, per-tenant quotas, and stage timeouts with attached degradation levers.
4. Freshness SLAs fight bulk onboarding. "Searchable in 5 minutes" for a single upload is incompatible with a fair queue if a whale dumps 5M docs at 9am without isolation in the ingest path.
5. Unit economics are dominated by the LLM. At this QPS the API bill can be 10:1 over infra — caches and model routing are the product, not optimizations.
6. Query mix varies by tenant. Support-KB tenants repeat; legal-discovery tenants do not. Global cache assumptions fail; per-tenant policy is required.

Alternatives that lose: one shared unpartitioned index (noisy neighbors + leak risk), global semantic cache for higher hit rate (data leak + wrong answers), and "scale the LLM tier" without caching (burn the margin).

Requirements and scale

Clarifying questions (each tied to a design consequence)

1. Tenant size distribution? 2,000 tenants is never uniform — expect a power law: 20 tenants with 5M docs, a long tail with 1k. Decides tiered isolation: dedicated shards for whales, shared partitioned storage for the tail.
2. What's the SLA — latency, availability, freshness — and are they per-tenant? A contractual p95 changes the design from "fast on average" to "engineered tails": hedged requests, admission control, per-tenant quotas.
3. Freshness: what happens after a customer uploads/updates a document? "Searchable in 1 min" vs. "by tomorrow" is a 10× difference in indexing architecture.
4. Query mix: repeated or unique? Support-KB tenants see heavy repetition — caching gold; legal-discovery tenants see unique queries — caching useless. Probably both exist; cache per-tenant.
5. Data isolation requirements — logical or physical? Some enterprise customers contractually require dedicated infrastructure or region pinning; that's a product tier, not a per-request choice.
6. Sync answers or async jobs allowed? Bulk QA over 10k docs should be an async batch endpoint at 50% cost, keeping the interactive path clean.

Requirements (assumed)

Capacity estimates

Explicit arithmetic:

• Corpus: 100,000,000 docs × 2,500 tokens = 250,000,000,000 tokens (250B). At ~500 tokens effective per chunk, 250B / 500 = 500,000,000 chunks.
• Vector bytes: 500M × 1.5 KB = 750 GB int8 at 1536-dim with overhead; ~12–15 shards + replicas.
• Peak vs sustained queries: 500 QPS × 86,400 s ≈ 43.2M if peak held all day; the design uses ~20M/day as the sustained mixed load (60% interactive, 40% API/batch), not 24h flat peak.
• Per-query LLM cost: 4,000 / 1e6 × $3 = $0.012 input; 300 / 1e6 × $15 = $0.0045 output; total ≈ $0.0165 ≈ $0.015/query.
• Spend scale: 20M × $0.015 = $300,000 at that daily volume. The table's ~$300k/month uncached is the planning COGS line for this design; after caching + tiering ~$120–150k/month is the target once 25–30% response-cache hits, small-model routing, and prompt caching apply. The product 20M × $0.015 also equals $300k for a full day at that volume — which is why cache and routing are existential either way: LLM spend dominates infra ~10:1.
• Ingest: 1M docs/day / 86,400 ≈ 11.6 docs/s sustained; 10× burst ≈ 116 docs/s.
• LLM TPM headroom: ~2M tokens/min ≈ 2e6 / 60 ≈ 33k tokens/s provisioned across providers.

At this scale the LLM API bill dwarfs infrastructure — the cache and routing layers are not optimizations, they are the unit-economics design.

The design

Multi-tenant isolation — three tiers, one code path

• Tail tenants (~1,900): shared vector collections partitioned by tenant_id payload with mandatory filtering — one collection per embedding model, never per tenant (thousands of collections is an operational anti-pattern; per-tenant filtered partitions are the documented pattern in every major vector DB). See Qdrant multitenancy pattern in Go deeper, and scaling and operations.
• Whale tenants (~20): dedicated shards — isolates their index size from tail latency and gives noisy-neighbor containment; tenants get promoted when they cross ~1M docs or sustained QPS thresholds.
• Contractual-isolation tier: dedicated cluster/region — priced as such.

Enforcement is structural: tenant_id comes only from the verified JWT, injected by middleware into every store query; application code physically cannot omit the filter (a repository layer refuses unfiltered queries). Embedding cost/query and cache namespaces are tenant-scoped too — isolation is an every-layer property, not a database setting.

Freshness pipeline

Upload → durable queue → parse/chunk/embed → shard upsert → cache invalidation event for that tenant's affected entries. Per-tenant fairness in the queue (weighted round-robin) so a whale's 5M-doc backfill cannot blow the 5-min SLA for a tail tenant's single upload — the burst case is exactly why fairness is designed in, not added later. Track upload-to-queryable as a first-class SLI per tenant; deletes and permission changes take a fast lane (< 60 s) since serving deleted content is worse than serving slowly.

Chunking and ingest mechanics: ingestion and chunking.

Caching — four layers, cheapest first

Generation with a model router

Classifier (or heuristic: query length, retrieval score margin) routes 60% of queries — simple factual lookups with high-confidence retrieval — to a small model ($0.3/$2.5 per M, TTFT ~200 ms); the rest to mid-tier. Router mistakes are caught by the eval loop, not guessed at. Batch/async endpoints run on provider batch APIs at ~50% discount with minutes-level SLA.

Model selection context: choosing models.

Serving path — per-stage latency budget (cache-miss path)

p95 stack (to first token path critical pieces): 20 + 40 + 120 + 110 + 800 = 1,090 ms if generation total dominates the 2.5 s full-answer budget — full answer p95 ≤ 2.5 s with streaming UX measured on TTFT ≤ 800 ms.

Tail control: hedge retrieval reads (send to replica at p90 timer); per-tenant concurrency caps so one tenant's burst queues their own requests, not the region's; strict timeout per stage with the degradation lever attached to each.

Evaluation

Per-tenant quality floors, not one global average — a global metric hides one tenant's broken corpus. Nightly: canary query set per tier (synthetic + mined) through the full stack, tracking retrieval recall, judge-scored faithfulness, and end-to-end latency; alert on per-tenant deltas. The semantic cache gets its own eval: sampled cache hits re-answered fresh and compared — cache-staleness or false-hit rate above ~1% tightens the similarity threshold automatically. Online: thumbs ratio, $/query per tenant (both a COGS metric and an anomaly signal), deflection into support for the product's own docs.

See RAG evaluation and evals and testing.

Key decisions and tradeoffs

The flows

Flow A — Interactive query (cache miss)

1. Client calls API with tenant JWT.
2. Admission control: per-tenant rate + priority (interactive over batch).
3. L1 exact cache probe → L2 semantic cache probe (tenant-scoped).
4. On miss: embed query → tenant-filtered hybrid retrieval → optional rerank → model router → generate.
5. Stream response; write traces; fill caches with tenant keys + corpus version.

Breaks when: tenant filter omitted, cache key drops tenant, or rerank fleet saturates without shed.

Flow B — Interactive query (cache hit)

1. Same admission + auth.
2. L1 hit returns full answer; or L2 hit after similarity ≥ 0.97 + safety check.
3. No retrieval/generation cost; still audit-log the hit for attribution.

Breaks when: ingest failed to invalidate after doc update → stale answer; or L2 false-positive.

Flow C — Document upload to queryable

1. Upload API accepts doc; enqueue with tenant fairness weight.
2. Autoscaled workers parse/chunk/embed; upsert to correct shard tier.
3. Emit cache invalidation for affected tenant entries.
4. SLI: upload-to-queryable ≤ 5 min; deletes/permission changes < 60 s fast lane.

Breaks when: whale monopolizes queue (without fairness) or invalidation is best-effort.

Flow D — Overload degradation

1. Detect: stage latency, provider errors, queue depth.
2. Shed rerank → force small model → relaxed cache threshold with "may be outdated" flag → retrieval-only mode → admit paid interactive last, reject batch/free first.
3. Two providers behind gateway; monthly failover drills.

Breaks when: degradation is ad hoc (no ordered ladder) or batch is not distinguishable from interactive.

Failure modes and degradation

Explicit, ordered, and rehearsed; each step trades quality for staying up:

1. Shed reranking — answer quality dips slightly; saves ~100 ms and rerank fleet capacity.
2. Route all traffic to the small model — quality dips, cost and TTFT improve; survives mid-tier provider incident.
3. Serve L1/L2 cache with relaxed similarity threshold + "may be outdated" flag.
4. Retrieval-only mode — return top passages with citations, no generation (survives total LLM provider outage — the product degrades to good search, not to a 503).
5. Admission control rejects batch/free-tier first; paid interactive traffic last.

Provider strategy: two LLM providers behind a gateway with reserved/provisioned throughput on the primary at this volume; automatic failover drills monthly. Corpus-affecting incidents (bad chunker deploy) recover from the document store via shard-scoped reindex, blue/green.

Align with reliability and the platform LLM gateway.

Common drill-downs

Prove tenant A can never see tenant B's data.

Layers: JWT-derived tenant_id injected by middleware (never client-supplied), repository layer that cannot express an unfiltered query, tenant-scoped cache namespaces (the classic leak is the cache, not the DB — an unscoped semantic cache serves tenant A's answer to tenant B's similar question), integration tests that adversarially query cross-tenant, and a periodic auditor job sampling stored vectors' payloads against ownership records. The cache leak is the subtle one — design for it explicitly, not as an afterthought.

Your p95 is 2.5 s but p99 is 9 s. Walk through fixing the tail.

Decompose by stage from traces: usual culprits are (a) generation-length outliers — cap max_tokens, stream, and measure TTFT separately from completion; (b) ANN searches on the biggest shared shard — promote the whale causing it to a dedicated shard; (c) cold cache after deploys — warm L1 from the top-1k queries per tenant; (d) provider TTFT spikes — hedge to the secondary at a p95-based timer for high-tier tenants.

A tenant uploads 5M documents at 9 am. What breaks?

Without design: the shared ingest queue backs up and every tenant misses the 5-min freshness SLA. With it: per-tenant fair queuing caps their share, autoscaled workers absorb what fairness allows, the tenant gets an async bulk-import path with progress reporting instead of the interactive pipeline, and their promotion to a dedicated shard is triggered before the index bloats shared-tier search latency.

Why is your semantic cache per-tenant? You'd get a much higher hit rate globally.

Because a 0.97-similar query from another tenant must be answered from that tenant's corpus — a global answer cache is both a data leak and a correctness bug (same question, different tenant docs, different correct answer). Global caching is only legitimate for tenant-agnostic artifacts (query embeddings of common phrasings). Correctness and isolation beat a higher global hit rate.

Where does the money actually go, and what's your first cost lever?

~10:1 LLM API vs. infra. First lever: prompt-prefix caching (near-free, mechanical, up to ~90% off repeated input tokens). Second: small-model routing for the easy 60%. Third: response caches. Only then infra tuning — quantization and shard right-sizing are real but an order of magnitude smaller.

Test yourself

Go deeper

• Multitenancy — Qdrant docs — partition-by-payload vs. tiered shard isolation and tenant promotion; the vendor-documented version of this design's isolation tiers.
• Prompt caching — Claude Docs — mechanics of the biggest cost lever: prefix rules, TTLs, cache-hit pricing.
• Building A Generative AI Platform — Chip Huyen — the cache hierarchy, router, and gateway layers in full-platform context.
• Your AI Product Needs Evals — Hamel Husain — building the eval process that per-tenant quality floors depend on.
• A Survey of Techniques for Maximizing LLM Performance — OpenAI — the optimization decision framework behind the model-routing and quality levers here.

Where this connects

• Enterprise RAG — the quality/ACL-first design at low QPS; contrast what changes when multi-tenancy and 500 QPS dominate.
• LLM gateway — provider failover, TPM limits, and cost attribution that this serving path should not reimplement per product.
• Filtering and metadata — pre-filter mechanics that tenant partitions rely on.
• Reliability — the degradation ladder and timeout patterns in general form.