Embeddings Beyond RAG

The intuition

RAG made embeddings famous, but the underlying move is older and wider: turn text into a point where distance means similarity, then run ordinary algorithms on the points.

Once every ticket, article, or product description is a vector, you can:

• train a tiny classifier instead of calling an LLM on every row
• treat "related items" as nearest neighbors
• flag outliers as points far from everything
• collapse near-duplicates above a similarity threshold
• cluster free text into themes without reading 20k rows by hand

Think of embeddings as a universal semantic feature layer. The LLM is still useful at the edges — labeling seed data, naming clusters, adjudicating borderline cases — but the high-volume middle is classical ML and nearest-neighbor search.

Why it exists

Calling a large model for every repetitive judgment does not scale.

Hard constraints:

1. Volume. Routing 100k tickets/day through an LLM is slow and expensive compared with one embed + a matrix multiply.
2. Consistency. Prompted classifications drift with prompt and model revisions; a fitted head on fixed vectors is stable until you retrain.
3. Latency. Online recs and semantic cache lookups need milliseconds, not multi-second generations.
4. Structure. Many tasks are literally "find similar," "find far," or "group near" — geometry already is the answer.

Alternatives lose on the high-volume path. LLM-per-item wins for open-ended reasoning and constantly shifting label schemas, but loses on cost and determinism. Hand-built keyword rules miss paraphrase. Training a large task-specific neural net from scratch is usually unnecessary once a strong general embedding exists. Embeddings + small models / ANN sit in the productive middle.

The core idea

Embeddings are a general-purpose semantic feature extractor, and RAG is just their most famous consumer. Once text becomes a vector where distance means similarity, a whole family of classic ML problems becomes cheap: classification (train a tiny classifier on embeddings instead of prompting an LLM per item), recommendations ("similar items" = nearest neighbors), anomaly detection (outliers = points far from everything), deduplication (near-duplicates = pairs above a similarity threshold), and clustering (k-means over embeddings to discover themes).

The engineering argument: for high-volume, repetitive judgments, embeddings + a small classical model beat calling an LLM per item by 10–1000× on cost and latency, with more consistent outputs — you pay the embedding once (fractions of a cent per thousand items) and inference is a matrix multiply. The LLM is still useful in these pipelines, but at the edges: labeling training examples, naming discovered clusters, adjudicating borderline cases. Knowing when a problem needs an LLM call versus a nearest-neighbor lookup is exactly the judgment that separates efficient systems from expensive ones.

How it actually works

Each pattern is "embed, then apply a decades-old algorithm," and each has one design decision that matters. Shared skeleton:

Classification. Embed the text, train logistic regression / a small MLP / kNN on labeled examples.

• Mini-example: support-ticket routing to 12 teams. Embed 1,500 historical tickets (already labeled by which team resolved them), fit logistic regression — millisecond inference, consistent, retrainable in seconds. Zero-shot variant when you have no labels: embed each label description ("billing issues: charges, refunds, invoices…") and assign each ticket to the nearest label vector.
• Key decision: embeddings + LR needs hundreds of examples and gives you a real, thresholdable probability; LLM prompting needs zero examples but costs per call and drifts with prompt/model changes. Common path: LLM labels a seed set → classifier serves production traffic.

Recommendations ("similar items"). Embed item content (title + description); serve k-NN from a vector index.

• Mini-example: "related articles" on a docs site — embed every article, and at render time return the 5 nearest neighbors to the current page (excluding itself), filtered to published status. Cold-start-proof for new items, since it's content-based, not behavior-based.
• Key decision: content embeddings capture "about the same thing," not "users who liked X liked Y" — for behavioral signals you blend with interaction data (e.g. a user vector = mean of embeddings of their recently viewed items, dot product to rank candidates; norms can carry popularity, which is why recommenders favor dot product over cosine).

Anomaly / outlier detection. Score each point by distance to its k-th nearest neighbor or to its cluster centroid; large distance = unlike anything seen.

• Mini-example: monitoring a support inbox for novel incidents — embed each incoming ticket, compute mean distance to its 10 nearest historical tickets; alert when a burst of tickets appears that is far from all history and close to each other (a new, correlated failure mode, before anyone has written a category for it).
• Key decision: it flags semantic novelty, not badness — expect benign novelty (new product launch) and route flags to triage, not to automated action.

Deduplication / near-duplicate detection. Pairs with similarity above a threshold (typically ~0.9–0.97, calibrated on your data) are duplicate candidates.

• Mini-example: dedupe a scraped job-postings corpus where the same role appears on 4 boards with different boilerplate — exact hashing misses them; embedding similarity at ~0.95 catches them. Directly relevant to RAG hygiene too: duplicated chunks waste context and crowd out diverse evidence in top-k.
• Key decision: naive all-pairs is O(N²) — at scale, query each item against the ANN index for neighbors above the threshold (this is sentence-transformers' "paraphrase mining" pattern), and always human-spot-check the threshold: 0.93 can be a paraphrase in one corpus and merely-related in another.

Clustering / topic discovery. k-means or HDBSCAN over embeddings (often after UMAP dimensionality reduction); then label each cluster.

• Mini-example: 20k pieces of free-text NPS feedback → embed → cluster → sample 10 items per cluster and have an LLM name each ("checkout errors on mobile", "pricing confusion") → sized, named themes with example quotes, instead of an intern reading for a week.
• Key decision: k-means forces every point into one of exactly k spherical clusters; HDBSCAN finds a variable number and leaves noise unassigned — better for messy feedback, at the cost of tuning.

Two more worth naming: semantic caching (embed incoming LLM queries; if a cached query is within threshold, serve its stored answer — cuts cost/latency on repetitive traffic, with false-hit risk to manage) and semantic routing (nearest intent-exemplar decides which model/pipeline handles the request).

The flows

On the governed enterprise platform, the same embedding service that powers RAG also feeds ticket routing classifiers, near-duplicate detection on uploaded session docs, and semantic cache for frequent policy questions — each with its own calibrated thresholds and entitlement rules.

A worked example

Task: categorize ~100k support tickets/month into 15 queues on the platform.

Zero-shot bootstrap (week 0): embed label descriptions for each queue; assign by nearest label vector; use that to pre-sort for human correction; then fit LR.

Contrast — LLM every ticket: flexible when queues change daily and need reasoning; slower, costlier, and prompt-sensitive. Use LLM when categories are unstable or each item needs multi-step judgment; use embeddings+LR when the taxonomy is stable and volume is high.

Common drill-downs

These are the questions a careful engineer asks one level down when embeddings leave the RAG path.

You need to categorize 100k support tickets into 15 categories. LLM or embeddings? Embeddings + logistic regression: label a seed set (possibly LLM-assisted), train, then classify at ~zero marginal cost with consistent, thresholdable outputs. Per-ticket LLM prompting costs orders of magnitude more, is slower, and drifts with prompts. LLM wins only when categories change constantly or need reasoning per item.

How do you do zero-shot classification with embeddings alone? Embed each label's description, embed the item, assign to the most similar label. Works surprisingly well when labels are semantically distinct; upgrade path is replacing label vectors with centroids of a few labeled examples per class.

How would you find near-duplicates in 10M documents without O(N²) comparisons? Embed everything into an ANN index; for each doc, query its neighbors and keep pairs above a calibrated threshold (~0.95); union-find the pairs into duplicate groups. ANN turns all-pairs into N cheap top-k queries.

Recommendations from embeddings — what's the limitation vs collaborative filtering? Content embeddings capture topical similarity and handle cold-start, but not taste ("users who bought X buy Y"). Production blends both: content vectors for new/long-tail items, behavioral signals (interaction-derived vectors, dot product with popularity in the norm) for the head.

How do embeddings power anomaly detection, and what's the failure mode? Score by distance to k-th nearest historical neighbor or cluster centroid; far = semantically novel. Failure mode: novelty ≠ problem — benign new topics flag too, so it's a triage signal; and the baseline must roll forward or it goes stale.

How would you turn 20k pieces of free-text feedback into themes? Embed → (optionally UMAP) → cluster (HDBSCAN for messy data, k-means for a fixed k) → LLM names each cluster from sampled members → report cluster sizes, names, exemplar quotes. Embeddings do the heavy lifting; the LLM only summarizes ~dozens of samples.

What is semantic caching and its main risk? Serve a stored response when a new query embeds within threshold of a cached one. Risk: false hits — similar-looking queries with different intent ("cancel my order" vs "cancel my subscription") — so thresholds must be strict, entries TTL'd, and cache keys should include user/context where answers differ.

What breaks across all these systems when you change embedding model? Everything downstream of the vectors: stored vectors are invalid (full re-embed), all thresholds (dedupe, cache, anomaly) need recalibration, and classifiers need retraining — vectors from different models share no geometry. Treat the model ID as part of every vector's schema.

Production concerns

• Thresholds don't transfer. Every model has its own similarity distribution — recalibrate dedupe/cache/anomaly thresholds per model, per corpus, and re-derive them after any model swap (which also invalidates every stored vector — same re-embed tax as RAG). See similarity metrics.
• Drift: classifiers on embeddings drift with vocabulary and traffic mix; monitor prediction distributions and refresh training data. Anomaly baselines need rolling windows or yesterday's novelty is forever anomalous. See observability.
• Cost asymmetry is the point: embedding ~1M short items costs on the order of single-digit dollars with small API models (and less self-hosted); classifying 1M items with an LLM costs orders of magnitude more and runs far slower. State the ratio explicitly in system designs. See cost.
• Batch vs online: clustering and dedupe are naturally offline batch jobs; classification/recs/caching serve online — different infra, same vectors. Reuse one embedding store where possible.
• Privacy: embeddings are derived data — they leak content (inversion attacks recover substantial text) and must inherit the source data's access controls and retention rules. On the platform, personal vector stores and org-wide indexes both inherit IAM entitlements. See security.
• Latency: online classify/recs should stay in the embed + ANN budget (tens of ms), not the generation budget. See latency.

Test yourself

Go deeper

• OpenAI Embeddings guide — official walk-through of search, classification, clustering, recommendations, and anomaly-detection use cases with code.
• Sentence Transformers documentation — usage docs and runnable examples for clustering, paraphrase mining (scalable dedupe), and semantic search.
• Embeddings — Google Machine Learning Crash Course — the general ML framing of embeddings beyond text retrieval.
• The Illustrated Word2vec — Jay Alammar — foundational intuition for why vector arithmetic on meaning works, including the recommendation analogy.
• Word Embedding and Word2Vec, Clearly Explained!!! — StatQuest with Josh Starmer — canonical video grounding for the embedding-space intuition all five patterns rely on.

Where this connects

• Embeddings — the shared model, versioning, and re-embed economics every pattern inherits.
• ANN indexes — the lookup engine behind recs, dedupe, anomaly k-NN, and semantic cache.
• The RAG pipeline — still the primary consumer; these patterns reuse the same embed+index investment.
• Cost — where the 10–1000× gap between embed-once and LLM-per-item shows up in real budgets.