Filtering & Metadata

The intuition

Nearest-neighbor search answers: "which vectors are closest to this query?" Product retrieval almost always adds: "…among documents this user may see, from these sources, newer than this date." That second clause is a structured filter — a predicate on metadata beside the embedding, not inside it.

Picture an open-plan office where everyone sits near people who do similar work (the HNSW graph). You want the ten people most similar to a visitor who also hold a Floor-7 badge. Grab the ten nearest in the whole building, then check badges, and you may get zero Floor-7 employees — the real matches sat just outside the unfiltered top-10. List Floor 7 first, then walk only among them, and the "similar work" paths break when most neighbors live on other floors. Modern engines add extra hallways and a planner so the walk stays connected under the badge constraint.

Why it exists

Unfiltered top-k is a research default, not a product default. Enterprise RAG, multi-tenant SaaS, and any system with ACLs need structured predicates on every query.

Authorization is not a ranking preference — returning a forbidden neighbor is a security bug, so tenant and ACL checks must run inside the retrieval path. Business scopes (source, doc type, language, region, recency) are cheap predicates; stuffing them into the embedding is the wrong tool. The ANN graph was built without tomorrow's WHERE clause, so masking nodes severs assumed paths.

Alternatives lose differently. Post-filter only collapses recall under selective ACLs. Always brute-force the allow-list dies when match sets are large. Encode filters into the vector fails open on security. Filter only after retrieval leaks snippets into logs, rerankers, and prompts. Dedicated strategies exist so recall, latency, and security can hold together.

The core idea

Real retrieval is "nearest neighbors among documents that pass predicates." The ANN index and the filter do not naturally compose, so three strategies compete.

Post-filtering runs vector search first — retrieve top-k, then drop non-matches. Expected survivors ≈ k × selectivity: at 1%, top-100 yields ~1 survivor on average, often zero. Relevant chunks exist; they were never in the unfiltered shortlist. Overfetching only helps mildly — at 0.1% you need ~k × 1000 candidates, and ANN work approaches brute force on the filtered subset. No constant overfetch is both cheap and safe for arbitrary filters.

Pre-filtering resolves the predicate first — matching IDs via an inverted index over metadata, then search only those. Recall against the filtered set is correct. Small match sets brute-force exactly; large ones walk the graph under a mask. HNSW links were built without the filter, so mid-selectivity masks (~1–20% of the corpus) disconnect the graph and greedy search dies early in a pocket.

Filter-aware search is what production engines add: extra graph links or multi-hop expansion so connectivity holds, plus a query planner that estimates cardinality and switches between exact scan (few matches) and filtered graph traversal (many matches). Know which strategy your engine uses — it decides whether filtered queries lose recall, lose latency, or neither.

How it actually works

Payload indexes make the allow-list cheap (keyword → posting list; range structures for numerics/dates). Without them, filtered queries degrade toward full payload scans, and planners cannot estimate cardinality.

ACORN (Stanford, 2024) frames predicate-agnostic HNSW with multi-hop expansion over the filtered subgraph. Weaviate's strategy is named after it (custom implementation inspired by the paper).

Metadata on every chunk. Deterministic doc_id / chunk_id; low-cardinality routing fields (source, doc_type, language) — index them; updated_at as numeric epoch; acl group tags with array-contains at query time; version / embedding_model for shadow reindexes; display fields (title, url) returned but not filtered on. Keep payloads lean and flat-typed — nested JSON fights inverted indexes and cardinality estimates.

The flows

On the governed enterprise platform, every retrieval path injects tenant and entitlement filters server-side. Personal vs org-wide stores often isolate physically (namespace or collection), with finer ACL arrays on chunks for group-level security inside a tenant.

A worked example

Setup. 1M chunks. User may only see groups {G_finance, G_audit} — about 1% match (10,000 chunks). Product wants k = 10.

Post-filter (illustrative): top-100 unfiltered → expected survivors ≈ 1. Overfetch for ~10 survivors needs ~1000 candidates (k / selectivity); at 0.1% that is ~10,000. Outcome: often 0–2 chunks and "RAG found nothing."

Filter-aware (illustrative): resolve acl CONTAINS ANY (...) → ~10k IDs; planner chooses exact scan or filtered graph; return true nearest neighbors among the 10k. Measure recall against filtered ground truth, not global top-k.

pgvector. Iterative scans off: HNSW returns ef_search candidates (say 40), WHERE drops most, LIMIT 10 returns ~1 row. Fix order: enable iterative scans → partial index on hot doc_type → partition by tenant_id. Even with iterative scans on, a very selective filter can hit max_scan_tuples and undershoot k — a latency safety valve, not silent success.

{
  "doc_id": "policy-441",
  "chunk_id": "policy-441#7",
  "source": "sharepoint",
  "doc_type": "policy",
  "language": "en",
  "updated_at": 1717200000,
  "acl": ["G_finance", "G_legal"],
  "embedding_model": "text-embedding-3-large",
  "version": 3,
  "title": "Travel and expense policy",
  "url": "https://intranet.example/policies/441"
}

Production concerns

ACL filtering is a correctness and security requirement — enforce it in the store query, inject the tenant filter server-side on every path. One miss is a breach (scaling and operations; security).

Watch filtered recall separately. Global recall@k hides empty sets for sparse-ACL users. Track "returned < k" and recall by selectivity bucket; mid-selectivity is where graph engines quietly underperform (RAG evaluation, evals and testing).

Latency is often bimodal on planner-based engines (exact-scan vs graph). A filter that flips the plan can move p95 sharply — pin selectivity in load tests (latency). On Qdrant, payload indexes created after HNSW was built leave you without filter-aware edges until rebuild.

Cardinality and cost. Cap unbounded user tags. Pinecone's 10k values-per-$in stops naive huge ACL lists — group tags or physical isolation scale better. Serverless filters do not necessarily cut billed reads; isolation you want billed cheaply must be a namespace (cost).

Freshness (last 90 days) is a range filter on updated_at, not a similarity trick. Filter first; optional recency boost only among already-eligible hits (production RAG).

Common drill-downs

How does Qdrant's filterable HNSW stay connected? Payload indexes resolve the filter; cardinality routes exact scan vs graph. At build time — with payload indexes already present — Qdrant adds extra edges so filtered subgraphs stay navigable. Index after the fact and you lack connectivity repair until rebuild.

pgvector returns 3 rows for LIMIT 10 with a WHERE. Fix ladder? HNSW produced ef_search candidates; WHERE discarded most; scan stopped. Enable iterative scans (strict_order / relaxed_order); raise ef_search if needed; partial HNSW for a hot predicate; partition for tenant. Watch max_scan_tuples.

Partition vs metadata filter? Mandatory low-cardinality scopes (tenant, region, environment) belong in partitions/namespaces — smaller graphs and hard isolation. Metadata filters are for ad-hoc combinable predicates. Soft-filter tenancy alone is a security review item.

Document-level security without re-embedding on permission change? Resolve user → groups at request time; push acl contains_any(groups) as a mandatory store filter. Permission changes update the ACL payload. Short-TTL cache on group membership so revokes land quickly.

Test yourself

Go deeper

• A Complete Guide to Filtering in Vector Search — Qdrant — pre/post/filterable-HNSW, payload indexes, cardinality planning.
• Filtered search (pre-filtering) — Weaviate docs — allow-list, ACORN vs sweeping, flat-search cutoff.
• pgvector README — filtering and iterative scans — post-filter behavior, scan modes, partial indexes, partitions, bounds.
• ACORN paper (arXiv:2403.04871) — predicate-agnostic filtered HNSW; problem framing and throughput-at-fixed-recall results.
• Filter by metadata — Pinecone docs — single-stage filtered query and $in / $nin limits (10k values).

Where this connects

• Scaling and operations — namespace vs shared collection + filter: physical isolation vs query-correctness.
• Vector database landscape — which engines are post-filter, planner-based, or ACORN-style.
• Production RAG — ACLs, freshness, and degradation in a full pipeline.
• Ingestion and chunking — metadata design starts at ingest; bad payloads cannot be fixed only at query time.