Filtering & Metadata

The intuition

Nearest-neighbor search answers: "which vectors are closest to this query?" Product retrieval almost always adds: "…among documents this user is allowed to see, from these sources, newer than this date." That second clause is a structured filter. The ANN index and the filter do not naturally compose — they are two different systems glued at query time.

Picture a huge open-plan office where everyone sits near people who do similar work (the HNSW graph). You want the ten people most similar to a visitor who also have a security badge for Floor 7. If you first grab the ten nearest people in the whole building, then check badges, you may get zero Floor-7 employees — the real matches sat just outside the unfiltered top-10. If you first list everyone on Floor 7, then walk the "similar work" paths only among them, the walk can break when most neighbors are on other floors. Modern engines add extra hallways and a planner so the walk stays connected under the badge constraint.

Why it exists

Pure "top-k over the whole corpus" is a research default, not a product default. Enterprise RAG, multi-tenant SaaS, and any system with ACLs need structured predicates on every query.

Hard constraints:

1. Authorization is not optional. Returning a neighbor the user must not see is a security bug, not a ranking bug. Filters for tenant and ACL must run inside the retrieval path, not as a hope in application code after the fact.
2. Business scopes are structured. Source system, document type, language, region, and recency are cheap predicates — encoding them into the embedding is the wrong tool.
3. ANN indexes optimize unfiltered neighborhoods. HNSW links were built without knowing tomorrow's WHERE clause; masking nodes severs paths the algorithm assumed existed.
4. Latency budgets still apply. Brute-forcing millions of matching IDs is correct but can blow the budget; over-fetching post-filter is wrong and slow at high selectivity.
5. Metrics lie if unfiltered. Global recall@k can look healthy while filtered queries return empty result sets for the users who matter.

Alternatives lose differently. Post-filter only collapses recall under selective ACLs. Always brute-force the allow-list dies when match sets are large. Encode filters into the vector ("append tenant id to text") fails open on security and confuses similarity. Filter only in the LLM layer after retrieval leaks snippets into logs, rerankers, and prompts. Dedicated filter strategies exist so recall, latency, and security can hold together.

The core idea

Real retrieval is almost never "nearest neighbors over everything" — it's "nearest neighbors among documents this user may see, from this source, newer than this date." That combination of ANN search and structured predicates is where naive implementations break, because the ANN index and the filter don't naturally compose.

Post-filtering runs the vector search first — retrieve top-k, then drop results failing the filter. The failure is arithmetic: if the filter matches 1% of the corpus, top-100 retrieval yields on average ~1 surviving result, and often zero. You asked for 10 relevant chunks and got none, not because they don't exist but because they weren't in the unfiltered top-k. Recall collapses in proportion to filter selectivity.

Pre-filtering resolves the predicate first — build the set of matching IDs, then search only those. Recall is now correct, but there are two costs: computing the allow-list (an inverted index over metadata makes this cheap), and the fact that an HNSW graph doesn't restrict well — if you traverse the graph while ignoring non-matching nodes, a selective filter disconnects the graph and the search gets stuck; if you brute-force the allow-list instead, cost is linear in matches.

Modern engines therefore do filter-aware search: Qdrant's filterable HNSW adds extra graph links and uses a query planner that switches between exact scan (selective filters) and filtered graph traversal (broad filters) based on estimated cardinality. Weaviate's ACORN (default since v1.34) traverses with multi-hop expansion so the graph stays connected under the filter. pgvector post-filters after the index scan, but iterative index scans (0.8.0+) keep pulling from the index until k matching rows are found — patching recall at a latency cost. The practical takeaway: know which strategy your engine uses, because it decides whether filtered queries lose recall, lose latency, or neither.

How it actually works

Three strategies and where they fail:

The post-filtering math. Expected survivors ≈ k × selectivity. Overfetching (retrieve 10×k and filter) helps only mildly: for a 0.1% filter you'd need k×1000, at which point the ANN search is slower than brute-force over the filtered subset. There is no overfetch factor that is both cheap and safe for arbitrary filters — which is why post-filtering can't be fixed with a constant.

Pre-filtering mechanics. Metadata fields get inverted indexes (keyword → posting list of IDs). The filter expression is resolved to an ID set / bitmask first. Then:

• Small match set → exact (brute-force) distance computation over just those vectors. Perfectly accurate, and fast when matches number in the low thousands.
• Large match set → graph search with the mask applied. The danger: HNSW links were built without knowledge of the filter; masking out most nodes severs paths, and greedy search terminates early in a disconnected region — recall drops precisely when the filter is mid-selectivity.

Filter-aware strategies per engine:

The research reference is ACORN (Stanford, 2024): predicate-agnostic HNSW traversal achieving 2–1,000× higher throughput than prior filtered-search methods at fixed recall — it's the idea Weaviate's implementation is named after.

pgvector specifics worth quoting: without iterative scans, ORDER BY embedding <=> $q LIMIT 10 with a WHERE clause returns fewer than 10 rows when the scan's ef_search candidates get filtered out — a classic production bug. Fixes, in order: enable iterative scans; create a partial index (WHERE doc_type = 'kb') for known hot predicates; partition the table by the filter column (the tenant pattern); or accept exact search on small filtered subsets.

Metadata schema design for RAG. Store on every chunk:

• doc_id, chunk_id — identity; deterministic IDs make upserts idempotent and deletes targetable.
• source / doc_type / language — routing and scoping filters; low-cardinality, index them.
• created_at / updated_at (numeric epoch) — range filters for freshness and incremental sync.
• acl — array of group/role tags checked with array-contains at query time (document-level security); resolve the user's groups before the query, never post-hoc in the LLM layer.
• version / embedding_model — enables shadow reindexing and safe model migrations.
• Display fields (title, url) — returned to the app, never filtered on, kept small.

Rules: filter fields must be indexed (unindexed payload filters degrade to full scans); keep payloads lean because they ride along in every result; prefer flat typed fields over nested JSON — inverted indexes and cardinality estimation work on the former.

The flows

On the governed enterprise platform, every retrieval path injects tenant and entitlement filters server-side before the vector store is called. Personal vs org-wide stores often use physical isolation (namespace or collection) for the hard boundary, with finer ACL arrays on chunks for group-level document security inside a tenant.

A worked example

Corpus. 1M chunks in a shared collection. User may only see documents tagged with groups in {G_finance, G_audit} — about 1% of the corpus match (10,000 chunks). Product asks for k = 10 grounded chunks.

Post-filtering arithmetic.

Filter-aware path (illustrative).

1. Resolve acl CONTAINS ANY (G_finance, G_audit) via inverted index → ~10k IDs.
2. Planner: 10k is large enough that pure brute-force may be acceptable or borderline; for broader filters (e.g. 20% of corpus) it would choose filtered graph traversal with extra links / ACORN expansion.
3. Return true nearest neighbors among the 10k — recall measured against filtered ground truth, not global top-k.

pgvector failure mode. Same 1% filter, iterative scans off: HNSW returns ef_search candidates (say 40), WHERE drops 39, query returns 1 row for LIMIT 10. Fix order: enable iterative scans → partial index on hot doc_type → partition by tenant_id for mandatory scopes.

{
  "doc_id": "policy-441",
  "chunk_id": "policy-441#7",
  "source": "sharepoint",
  "doc_type": "policy",
  "language": "en",
  "updated_at": 1717200000,
  "acl": ["G_finance", "G_legal"],
  "embedding_model": "text-embedding-3-large",
  "version": 3,
  "title": "Travel and expense policy",
  "url": "https://intranet.example/policies/441"
}

Common drill-downs

These are the questions engineers ask one level down when filtered retrieval misbehaves.

Why does post-filtering break recall? Expected surviving results ≈ k × filter selectivity. A 1% filter on top-100 leaves ~1 result; relevant items outside the unfiltered top-k are unrecoverable. Overfetching scales inversely with selectivity, so it can't fix arbitrary filters.

So why not always pre-filter? Pre-filtering needs the match set first. Tiny match set → brute-force is fine. Big match set → you must search the graph under a mask, and HNSW built without filter knowledge disconnects — recall drops or you fall back to linear scan. Hence planner-based and filter-aware-graph designs.

How does Qdrant's filterable HNSW work? Payload inverted indexes resolve the filter; a cardinality estimate routes the query — exact scan when few match, graph traversal when many do; the HNSW graph itself is built with extra links so commonly-filtered subgraphs remain navigable. Recall holds across selectivity, latency stays bounded.

You use pgvector; queries with WHERE return 3 rows instead of 10. Why, and fix? HNSW scan returned ef_search candidates, the WHERE clause discarded most, and the scan stopped. Enable iterative index scans (0.8.0+) so it keeps scanning until k matches; for a known hot filter use a partial index; for tenant filters, partition.

Design chunk metadata for an enterprise RAG system. doc_id/chunk_id (deterministic), source, doc_type, language, updated_at epoch, acl group array, embedding_model/version, plus display-only title/url. Index the filter fields, enforce ACL and tenant in the query server-side, use version for shadow reindexes.

How do you do document-level security in RAG? Resolve user → groups at request time; query with acl-contains-any(groups) as a mandatory filter inside the vector store. Never filter after retrieval in app code. Cache group membership; on permission change, update the chunk's acl payload — cheaper than re-embedding.

When is a filter better served by partitioning than by metadata? When it's mandatory on every query and low-cardinality — tenant, region, environment. Partitions (namespaces, Postgres partitions, Milvus partitions) prune data physically, giving smaller graphs and hard isolation; metadata filters are for ad-hoc, combinable predicates.

Does filtering make vector search faster or slower? Both, engine-dependent. Filter-aware engines get faster with selective filters (smaller effective search space, or cheap exact scan). Post-filtering engines get slower and worse (overfetch to compensate). This asymmetry is a top reason to graduate off naive stacks.

Production concerns

• ACL filtering is a correctness/security requirement, not an optimization — post-filtering ACLs in application code risks leaking snippets through logs, rerankers, or bugs; enforce in the retrieval query. In multi-tenant systems, inject the tenant filter server-side on every query path — one missed filter is a data breach (see scaling and operations for tenancy patterns). Cross-link: security for RAG data leakage and injection adjacent risks.
• Watch filtered recall separately. Global recall metrics hide filtered-query failures. Track "returned < k results" rates and recall by filter selectivity bucket; the mid-selectivity band (~1–20%) is where graph-based engines quietly underperform. Tie eval design to RAG evaluation and evals and testing.
• Latency shape: filtered queries have bimodal latency in planner-based engines (cheap brute-force path vs graph path). A filter whose cardinality estimate flips the plan can move p95 by an order of magnitude — pin selectivity assumptions in load tests. See latency.
• Cardinality explosions: filters like user-generated tags create unbounded inverted indexes; cap tag vocabularies. Pinecone caps values-per-operator (10k) — high-cardinality "id IN (...)" ACL patterns hit it.
• Cost: in serverless pricing, filters do not necessarily reduce billed read units — Pinecone bills by namespace size scanned, so isolation you want billed cheaply must be a namespace, not a metadata filter. See cost.
• Time-based freshness (only retrieve last 90 days) is a filter, not a similarity concern — don't try to encode recency into embeddings; filter on timestamp and, if needed, re-rank with a recency boost in application code. Broader freshness design: production RAG.

Test yourself

Go deeper

• A Complete Guide to Filtering in Vector Search — Qdrant — the best single write-up of pre/post/filterable-HNSW mechanics, with failure examples.
• Filtered search (pre-filtering) concepts — Weaviate docs — allow-list construction, ACORN vs sweeping strategies, and when each wins.
• pgvector README — filtering section — iterative index scans, strict vs relaxed ordering, partial indexes, partitioning.
• ACORN: Performant and Predicate-Agnostic Search Over Vector Embeddings and Structured Data — the research behind filter-aware graph traversal; skim for the problem framing and the throughput-at-fixed-recall results.
• Vector databases are so hot right now. WTF are they? — Fireship — context refresher on where filtering sits in the vector search stack.

Where this connects

• Scaling and operations — multi-tenancy patterns (namespace vs shared collection + filter) decide whether isolation is physical or a query-correctness property.
• Vector database landscape — which engines are post-filter, planner-based, or ACORN-style when you choose a store.
• Production RAG — ACLs, freshness, and degradation when filtered retrieval is part of a full pipeline.
• Ingestion and chunking — metadata design starts at ingest; bad chunk payloads cannot be fixed only at query time.