Scaling & Operations

The intuition

A vector store is a database with two twists: the hot path wants the index in memory, and the graph index was not designed for frequent hard deletes. Everything hard about operating one at scale is a consequence of those two facts.

Think of a city's road map printed on a huge poster on the wall (the HNSW graph in RAM). Adding a new street is easy — draw it in. Removing a neighborhood is not: you tape over intersections (tombstones) so drivers skip them, but the poster still takes the same wall space until a night crew reprints whole districts (compaction). If half the city is taped over, navigation gets worse and the wall is still full. Sometimes you print a fresh poster in the back room and swap it in (blue/green rebuild).

Scale-out is then ordinary distributed systems: hang more posters (shards), keep copies (replicas), decide whether each tenant gets their own poster or a sticker on a shared map.

Why it exists

Getting ANN search working on a laptop is easy. Running it for a product with continuous upserts, deletes, multi-tenant isolation, and disaster recovery is a different job.

Hard constraints:

1. RAM bounds the graph. Float32 vectors plus HNSW links dominate cost; past a node's capacity you must quantize, spill to disk, or shard — not just "add a bit of swap."
2. Deletes are soft. Hard removal from HNSW can sever paths; engines tombstone and compact later. High churn without optimizer headroom degrades search while memory stays high.
3. The index is derived data. Source documents and the embedding pipeline can rebuild it — but re-embedding 10M chunks costs real money and hours; snapshots are the fast path, rebuild is the correctness path.
4. Dual-write is not a transaction. App DB and vector index drift unless CDC/outbox and reconciliation exist.
5. Multi-tenancy is a product requirement. Isolation vs cost vs tenant-count ceiling forces an explicit pattern, not a default shared collection.

Alternatives lose differently. Ignore compaction → silent recall/latency death. Shard first → expensive machines before quantization would have fit. Snapshot-only DR without rebuild → embed model or pipeline bugs leave you unable to regenerate. Metadata-filter tenancy without server-side injection → one missed filter is a breach. Operations practice exists to keep memory, mutation, consistency, and isolation under control as N and churn grow.

The core idea

Operating a vector store is mostly ordinary database operations plus two vector-specific twists: the index lives in RAM, and the index hates mutation.

Memory first. The sizing rule of thumb: raw vectors = count × dimensions × 4 bytes (float32), and total RAM ≈ raw × 1.5 to cover graph links, metadata indexes, and temporary segments during optimization (Qdrant's official formula). 10M OpenAI-sized 1536-dim vectors = ~61 GB raw, ~92 GB provisioned. That number drives everything: node size, shard count, and whether you reach for quantization (int8 = 4× reduction, binary = up to 32×) or disk-backed indexes before you reach for more machines.

Mutation second. HNSW inserts are incremental, but deletes are not: removing a node can sever graph paths, so every engine soft-deletes — a tombstone marks the vector, queries skip it, and a background optimizer (segment merge/compaction/vacuum) later rewrites segments without the dead entries. Consequence: a high-churn collection silently accumulates tombstones, memory doesn't come back on delete, and recall/latency degrade until compaction runs. Full index rebuild is the escape hatch — build a fresh index in the background while serving from the old one (blue/green), then swap.

Scale-out is standard sharding: hash vectors across shards, fan out the query, merge per-shard top-k; replication gives HA and read throughput. Multi-tenancy has three patterns — namespace per tenant, collection per tenant, or shared collection with a tenant filter — trading isolation against cost and tenant-count ceiling. And backup has a twist: the most reliable recovery for a vector index is often not a snapshot but re-embedding from the source-of-truth store, so the rebuild pipeline is part of your disaster-recovery story.

How it actually works

Write path, segments, and what search sees:

Sharding and replication. A collection is split into shards (by ID hash, or explicitly by tenant/partition key); each shard holds its own segments and HNSW graphs. Query path: scatter to all shards (unless a partition key prunes), each returns its local top-k with scores, coordinator merges — correct because similarity scores are globally comparable. Writes route by key. Replication is per-shard (replica count N); consensus (e.g., Raft) coordinates cluster metadata, while data replication is usually eventually consistent with tunable write/read consistency factors. More shards = faster parallel search but higher fan-out overhead and more merge work; shards sized so each fits comfortably in its node's RAM.

Segments and the write path. Writes land in a WAL, then a mutable in-memory segment (searched by brute force — this is why fresh points are findable before indexing); when a segment reaches threshold size it's sealed and an HNSW graph is built for it; the optimizer merges small sealed segments into bigger ones in the background. Search = union over all segments (indexed and unindexed), which is how vector DBs deliver near-real-time freshness without rebuilding a monolithic index per write.

Deletes, tombstones, updates.

• Delete = tombstone in a bitset consulted during traversal; space reclaimed only at merge/compaction ("vacuum").
• Update = delete + reinsert (vectors are immutable in the graph); metadata-only updates are cheap in engines that store payload separately (Qdrant), expensive where a row rewrite is needed (pgvector: MVCC dead tuple + index bloat → autovacuum tuning matters; heavy churn eventually wants REINDEX CONCURRENTLY).
• Rebuild vs incremental: incremental upsert is fine at low churn; past roughly 10–20% tombstone ratio (engine-dependent) query latency and recall sag, and a compaction or full rebuild is cheaper than living with the degraded graph. Pinecone's "Great Algorithms Are Not Enough" post is the canonical description of why bolted-on HNSW forces periodic full rebuilds and blue-green swaps.

Multi-tenancy patterns:

The real tradeoff axis: per-tenant index quality and isolation vs memory amortization. Dedicated per-tenant indexes give unfiltered fast searches and hard boundaries but multiply index overhead; shared-index filtering amortizes memory but makes every query a filtered query and every bug a potential cross-tenant leak.

Backup and consistency.

• Engines offer per-collection snapshots (Qdrant) or object-storage backups (Milvus, whose storage already lives on S3); pgvector inherits Postgres PITR/WAL archiving — a genuinely better story than most dedicated engines.
• The vector DB is a derived store: source documents + embedding model + pipeline can regenerate it. Mature teams treat re-embedding from source as the authoritative recovery path and snapshots as a restore-time optimization (re-embedding 10M chunks costs real money and hours — snapshot restore is minutes).
• Dual-write drift is the chronic consistency problem: app writes doc to primary DB, then upserts the vector — the second write fails and search silently misses the doc. Fixes: outbox/CDC-driven indexing with idempotent deterministic IDs, plus a periodic reconciliation job diffing source IDs against index IDs.

Memory sizing worked examples (float32, HNSW, 1.5× rule):

HNSW link overhead specifically ≈ M × 2 × 4–8 bytes per vector (M=16 default → ~128–256 B/vector) — included in the 1.5× rule. Also: pgvector HNSW builds need the graph in maintenance_work_mem; an index build OOM-ing or spilling is the classic first scaling incident.

The flows

On the governed enterprise platform, personal upload stores often use short-TTL namespaces or collections with scheduled purge, while org-wide corpora use sharded collections with entitlement filters, CDC from the document system of record, and a tested rebuild path for embedding-model upgrades.

A worked example

Goal. Size and operate a collection for 10M chunks × 1536-dim float32 embeddings, steady ingest, moderate delete churn (~5% of points deleted over a quarter), multi-tenant SaaS with ~50k customers.

Memory and cost levers.

Multi-tenancy at 50k customers.

Churn incident. Over two months delete/re-ingest cycles push tombstone ratio toward the 10–20% danger band. p95 search rises ~3× with flat traffic. Order of investigation: RAM headroom (page faults on graph walk) → deleted-ratio / segment stats → optimizer lag → on pgvector, vacuum/bloat history. Remediation: schedule compaction off-peak; if still degraded, blue/green full rebuild; fix upstream dual-write so deletes actually reach the index.

Consistency. Document write to primary succeeds; vector delete fails. Stale chunks still retrieve in RAG. Fix: outbox/CDC with deterministic doc_id+chunk_id, idempotent upsert/delete, reconciliation job diffing source IDs vs index IDs on a schedule. Freshness SLO example: "searchable ≤60s after write" measured end-to-end, with CDC lag and queue depth as leading indicators.

1. Create new collection with target M / ef_construct / quantization
2. Dual-write all upserts/deletes to old and new
3. Backfill from source with deterministic IDs
4. Sample queries: recall@k new vs old vs brute-force subset
5. Atomic alias swap; keep old for rollback window
6. Drop old after soak; stop dual-write

Common drill-downs

These are the questions engineers ask one level down when operating vector search at scale.

How much RAM for 10M 1536-dim embeddings? 10M × 1536 × 4 B ≈ 61 GB raw; ×1.5 for graph links, metadata index, and merge headroom ≈ 92 GB. Int8 quantization drops vector storage 4× (~23 GB provisioned); binary up to 32× with a rescoring pass. Always state the formula before the number.

How do vector DBs handle deletes? Tombstones: the point is flagged, traversal skips it, storage and graph links persist until a background compaction/merge rewrites the segment. Hard-deleting from HNSW directly can disconnect the graph. Implication: memory isn't freed on delete, and high churn degrades queries until compaction or rebuild.

Incremental upsert vs full rebuild — when each? Incremental for steady low churn; HNSW inserts are native. Full (blue/green) rebuild when tombstone ratio is high, when recall has measurably decayed, when changing index parameters (M, ef_construct), and always when changing the embedding model. Serve from the old index during the build; swap after verifying recall parity.

Design multi-tenancy for a RAG SaaS with 50k customers. Namespace-per-tenant (or native tenant shards): physical scoping, no noisy neighbors, instant offboarding by namespace delete, cold tenants offloadable. Collection-per-tenant dies at this count from per-index overhead unless the engine has purpose-built tenant management; metadata-filter tenancy is the fallback for millions of tiny tenants but makes isolation a query-correctness property. Enforce tenant scoping server-side either way.

How does a distributed vector search execute? Route by partition key if present, else scatter to all shards; each shard searches its segments (indexed sealed ones plus brute-force over the fresh in-memory one), returns local top-k with scores; coordinator merge-sorts to global top-k. Scores are globally comparable, so the merge is trivial; the cost is fan-out latency = slowest shard.

How do you back up a vector database? Engine snapshots to object storage for fast restore, plus the real answer: the index is derived data — keep source docs + pipeline able to re-embed from scratch, with deterministic IDs so the rebuild is idempotent. Measure restore time; a backup that takes 12 hours to reload into RAM is a 12-hour outage.

Your index and your primary DB disagree — deleted docs still surface in RAG. Root cause? Dual-write without a transaction: primary write succeeded, index delete failed (or arrived out of order). Fix with CDC/outbox-driven indexing, idempotent upserts/deletes keyed by doc ID + version, and a reconciliation job that diffs IDs between source and index on a schedule.

Search latency slowly rose 3× over two months with no traffic change. Suspects? In order: working set outgrew RAM (corpus growth → page faults on graph traversal), tombstone accumulation from churn, segment-count growth outpacing the optimizer, and — on pgvector — index bloat from MVCC dead tuples. Check memory headroom, deleted-ratio, segment stats, and vacuum history before touching query code.

Production concerns

• Watch tombstone ratio and segment counts, not just QPS: rising deleted-fraction predicts the latency/recall sag before users notice. Alert on optimizer/compaction lag. Wire these into observability alongside retrieval quality metrics.
• Compaction is a resource event: segment merges burn CPU and can double a collection's disk footprint transiently. Schedule for off-peak, cap concurrency; on Postgres, tune autovacuum for the vector table instead of accepting defaults.
• Rebuild without downtime: build new index/collection alongside, dual-write during the build, verify recall parity on a sample query set, atomically swap (alias/DNS/collection rename), keep the old one for rollback. Same play for embedding-model migrations — which change every vector and are therefore always a full rebuild.
• Cold start / restore time: loading hundreds of GB of index from disk into RAM takes minutes; replicas exist partly so restarts don't take the service down. Test restore time, not just backup success. See reliability.
• Cost levers in order: quantize (int8 is nearly free in recall for most embedding models, binary needs rescoring), offload cold tenants/segments to disk or object storage, then shard. Sharding first is the expensive mistake. Tie spend to cost.
• Multi-tenant safety: tenant filter injected in one server-side choke point, contract tests that assert cross-tenant queries return zero, and per-tenant quotas so one tenant's ingest storm can't starve compaction for everyone else. Security framing: security.
• Consistency SLO: define index freshness explicitly (e.g., "document searchable ≤60s after write") and measure it end-to-end — it's the metric users actually feel, and CDC lag, WAL backlog, or indexing-queue depth are its leading indicators. Product-level coupling: production RAG.

Test yourself

Go deeper

• Qdrant capacity planning — the memory formula (vectors × dims × 4 B × 1.5), disk vs RAM tradeoffs, and scaling guidance.
• Great Algorithms Are Not Enough — Pinecone blog — why HNSW under churn forces tombstones, rebuilds, and blue-green swaps; the clearest public write-up of the mutation problem.
• Weaviate multi-tenancy architecture explained — shard-per-tenant design, ACTIVE/INACTIVE/OFFLOADED tenant states, and scaling to ~1M tenants with S3 offloading.
• Implement multi-tenancy — Pinecone docs — namespace-per-tenant as the recommended pattern and the explicit case against metadata-filter tenancy.
• Weaviate: The Vector Database Your Parents Wished They Had! — CMU Database Group — a vector-DB co-founder walking database researchers through real architecture: segments, HNSW ops, and production tradeoffs.
• What is a Vector Database? — IBM Technology — foundational context if the operational material above is your entry point.

Where this connects

• Filtering and metadata — shared-collection tenancy turns isolation into filtered search; read this before treating tenant_id as "just another payload field."
• Vector database landscape — managed vs self-hosted and engine choice determine which of these ops knobs you own.
• Production RAG — freshness SLOs, ACL, and degradation when the index is part of a live retrieval product.
• Reliability — outbox/CDC, retries, and failure modes that keep the vector index consistent with the source of truth.