Scaling & Operations

The intuition

A vector store is a database with two twists: the hot path wants the index in memory, and the graph index was not designed for frequent hard deletes. Everything hard about operating one at scale follows from those two facts.

Think of a city's road map printed on a huge poster on the wall (the HNSW graph in RAM). Adding a street is easy — draw it in. Removing a neighborhood is not: you tape over intersections so drivers skip them (tombstones), but the poster still takes the same wall space until a night crew reprints whole districts (compaction). If half the city is taped over, navigation gets worse and the wall is still full. Sometimes you print a fresh poster in the back room and swap it in (blue/green rebuild).

Scale-out is then ordinary distributed systems: hang more posters (shards), keep copies (replicas), decide whether each tenant gets their own poster or a sticker on a shared map.

Why it exists

Getting ANN search working on a laptop is easy. Running it under continuous upserts, deletes, multi-tenant isolation, and disaster recovery is a different job.

Hard constraints:

1. RAM bounds the graph. Quantize, spill to disk, or shard — not swap.
2. Deletes are soft. Tombstones until compaction; high churn degrades search while memory stays high.
3. The index is derived data. Snapshots for speed; re-embed for authority.
4. Dual-write is not a transaction. CDC/outbox and reconciliation required.
5. Multi-tenancy is a product requirement. Isolation vs cost vs tenant-count ceiling forces an explicit pattern.

Alternatives lose differently. Ignore compaction → silent recall and latency death. Shard first → expensive machines before quantization would have fit. Snapshot-only DR without a rebuild path → embed-model bugs leave you unable to regenerate. Metadata-filter tenancy without server-side injection → one missed filter is a breach.

The core idea

Operating a vector store is ordinary database work plus two vector-specific twists: the index lives in RAM, and the index hates mutation.

Memory first. Size with the industry rule of thumb (Qdrant's public formula): raw vectors = count × dimensions × 4 bytes (float32), provisioned RAM ≈ raw × 1.5 to cover graph links, metadata indexes, and temporary segments during optimization. That number drives node size, shard count, and whether you quantize (int8 ≈ 4× reduction, binary up to ~32× with rescoring) or move cold data to disk before buying more machines.

Mutation second. HNSW inserts are incremental; deletes are not. Engines soft-delete with a tombstone — a bit that marks the vector dead so queries skip it — then a background optimizer (segment merge, vacuum, compaction) rewrites segments without the dead entries. Memory does not come back on DELETE. Past roughly a 10–20% tombstone ratio (engine defaults often trigger vacuum near 20%, e.g. Qdrant's deleted_threshold), latency and recall sag until compaction or a full rebuild runs.

Scale-out, multi-tenancy, and backup sit on top of that — and the sections below unpack each.

How it actually works

Write path, segments, and what search sees:

Segments and the write path. Writes land in a WAL, then a mutable in-memory segment searched by brute force — that is why fresh points are findable before indexing. When the segment reaches a size threshold it seals and an HNSW graph is built. A background optimizer merges small sealed segments into larger ones and vacuums segments whose deleted fraction is too high. Search unions every segment (indexed sealed ones plus the fresh mutable one), so the engine delivers near-real-time freshness without rebuilding a monolithic index per write. The bill comes due as segment count and tombstone ratio climb without optimizer capacity — and under continuous write load, that optimizer competes with queries for CPU and I/O.

Deletes, updates, rebuilds. Delete = tombstone consulted during traversal; space is reclaimed only at merge/compaction. Update = delete + reinsert (vectors are immutable in the graph). Metadata-only updates are cheap where payload lives separately (Qdrant) and expensive where a row rewrite creates MVCC dead tuples (pgvector → autovacuum, eventually REINDEX CONCURRENTLY). When tombstones or segment sprawl dominate, a blue/green rebuild — build a fresh index beside the live one, dual-write, verify recall, swap — is cheaper than living with a degraded graph.

Sharding. A collection splits into shards (by ID hash, or by tenant/partition key). Query path: scatter to all shards (unless a partition key prunes), each returns local top-k, coordinator merges by score — correct because similarity scores are globally comparable. Size each shard so its graph fits in node RAM with merge headroom. p95 is the slowest shard.

Multi-tenancy. Three patterns cover almost every product. Namespace per tenant (Pinecone-style) physically partitions data: no noisy neighbors, instant offboarding by namespace delete, query cost tracks namespace size. Common plan territory is ~100k namespaces; larger counts need vendor headroom. Collection or native tenant shard (Weaviate) is strongest isolation: separate index, per-tenant hot / inactive / offloaded-to-object-storage states, scaling toward ~1M+ with cold offload — right for enterprise compliance and per-tenant SLAs. Shared collection + tenant metadata filter is cheapest but isolation is only logical: every query is filtered, one missed filter leaks data, and some serverless pricing models bill as if you scanned all tenants. Use it only for huge counts of tiny tenants, with server-side filter injection. The axis is per-tenant isolation vs memory amortization.

Backup and consistency. Engines offer collection snapshots (Qdrant) or object-storage-native backup (Milvus); pgvector inherits Postgres PITR. Treat the vector DB as derived: source docs + embedding model + pipeline can regenerate it. Mature teams use re-embedding as the authoritative recovery path and snapshots as a restore-time optimization. Dual-write drift is the chronic consistency failure — primary write succeeds, vector upsert or delete fails, search silently diverges. Fix with outbox/CDC, idempotent deterministic IDs (doc_id + chunk_id), and a reconciliation job that diffs source IDs against index IDs.

The flows

On the governed enterprise platform, personal upload stores often use short-TTL namespaces with scheduled purge, while org-wide corpora use sharded collections with entitlement filters, CDC from the document system of record, and a tested rebuild path for model upgrades.

A worked example

Goal. Size and operate 10M chunks × 1536-dim float32 embeddings, steady ingest, moderate delete churn (~5% of points deleted over a quarter), multi-tenant SaaS with ~50k customers.

Memory and cost levers (illustrative; re-measure on your engine). Raw vectors: 10M × 1536 × 4 B ≈ 61.4 GB. Provisioned HNSW rule of thumb: ×1.5 ≈ ~92 GB before OS and query headroom. Int8 first cuts vector storage ~4× → ~23 GB class for vectors (validate recall; graph overhead remains). Sketch 2–3 shards so each graph fits with merge headroom. The wrong first move is buying ten memory-optimized nodes before trying int8 or disk offload.

At 50k customers, namespace or native tenant shard per customer is the fit. Naive collection-per-tenant without engine support dies from per-index overhead. Shared collection + tenant filter is only for millions of tiny tenants, and only with server-side filter injection.

Churn incident. Over two months, delete/re-ingest cycles push tombstone ratio toward the 10–20% danger band. p95 search rises ~3× with flat traffic. Investigate in order: RAM headroom (page faults on graph walk) → deleted-ratio / segment stats → optimizer lag → on pgvector, vacuum/bloat history. Remediate: schedule compaction off-peak; if still degraded, blue/green rebuild; fix dual-write so deletes reach the index. Consistency failure looks the same way: primary write succeeds, vector delete fails, stale chunks still retrieve in RAG — outbox/CDC with deterministic doc_id+chunk_id, idempotent upsert/delete, and scheduled reconciliation. Freshness SLO example: "searchable ≤60s after write" measured end-to-end.

1. Create new collection with target M / ef_construct / quantization
2. Dual-write all upserts/deletes to old and new
3. Backfill from source with deterministic IDs
4. Sample queries: recall@k new vs old vs brute-force subset
5. Atomic alias swap; keep old for rollback window
6. Drop old after soak; stop dual-write

Production concerns

• Watch tombstone ratio and segment counts, not just QPS. Rising deleted-fraction predicts latency and recall sag. Alert on optimizer lag. Wire into observability alongside retrieval quality.
• Compaction is a resource event. Merges burn CPU and can double disk footprint transiently; under continuous write load they compete with search for cores and I/O. Schedule heavy optimization off-peak; on Postgres, tune autovacuum for the vector table.
• Rebuild without downtime. Build new index alongside, dual-write, verify recall parity, atomically swap, keep the old one for rollback. Embedding-model migrations change every vector and are always a full rebuild.
• Cold start / restore time. Loading hundreds of GB into RAM takes minutes to hours; replicas exist partly so restarts do not take the service down. Test restore time, not just backup success. See reliability.
• Cost levers in order: quantize (int8 is often near-free in recall; binary needs rescoring), offload cold tenants/segments, then shard. Sharding first is the expensive mistake. See cost.
• Multi-tenant safety. Inject the tenant filter in one server-side choke point; contract-test that cross-tenant queries return zero; set per-tenant quotas so one tenant's ingest storm cannot starve compaction. See security.
• Consistency SLO. Define index freshness explicitly (e.g. "document searchable ≤60s after write") and measure it end-to-end. CDC lag and indexing-queue depth are leading indicators. Product coupling: production RAG.

Common drill-downs

Why does the optimizer fight the query path under heavy writes? Sealing, HNSW build, merge, and vacuum compete with search for CPU and I/O. High continuous ingest can leave large unindexed segments that every query must brute-force. Engines expose knobs for this (segment size caps, indexing thresholds, "indexed only" query modes) with explicit freshness tradeoffs — know which side your product prioritizes before you turn them.

Incremental upsert vs full rebuild — when each? Incremental for steady low churn. Full blue/green when tombstone ratio is high, recall has decayed, index parameters change, or the embedding model changes. Serve from the old index during the build; swap after recall parity.

Latency rose 3× over two months with no traffic change. Suspects in order? Working set outgrew RAM → tombstone accumulation → segment-count growth outpacing the optimizer → on pgvector, MVCC index bloat. Check those before touching query code or efSearch.

Test yourself

Go deeper

• Qdrant capacity planning — memory formula (vectors × dims × 4 B × 1.5) and sizing guidance.
• Qdrant optimizer — vacuum, merge, and indexing optimizers; deleted_threshold defaults.
• Great Algorithms Are Not Enough — Pinecone blog — why HNSW under churn forces tombstones and blue-green rebuilds (architecture details are dated; the mutation problem is not).
• Weaviate multi-tenancy architecture explained — shard-per-tenant, ACTIVE/INACTIVE/OFFLOADED, object-storage offload.
• Implement multi-tenancy — Pinecone docs — namespace-per-tenant recommended; case against metadata-filter tenancy for isolation and cost.
• Weaviate at CMU Database Group — segments, HNSW ops, and production tradeoffs from a co-founder.

Where this connects

• Filtering and metadata — shared-collection tenancy turns isolation into filtered search.
• Vector database landscape — managed vs self-hosted determines which of these ops knobs you own.
• Production RAG — freshness SLOs, ACL, and degradation when the index is part of a live retrieval product.
• Reliability — outbox/CDC and failure modes that keep the index consistent with the source of truth.