Embeddings Beyond RAG

The intuition

RAG made embeddings famous, but the underlying move is older and wider: turn text into a point where distance means similarity, then run ordinary algorithms on the points.

Once every ticket, article, or product is a vector, "related" means nearest neighbors, outliers sit far from everything, near-duplicates cross a similarity threshold, free text clusters into themes, and a tiny classifier can replace an LLM call per row.

Think of embeddings as a universal semantic feature layer. The LLM stays useful at the edges — labeling seed data, naming clusters, adjudicating borderlines — but the high-volume middle is classical ML and nearest-neighbor search.

Why it exists

Calling a large model for every repetitive judgment does not scale.

Volume: routing 100k tickets a day through an LLM is slow and expensive next to one embed plus a matrix multiply. Consistency: prompted classifications drift when prompts or models change; a fitted head on fixed vectors stays stable until you retrain. Latency: online recs and semantic cache lookups need milliseconds, not multi-second generations. Structure: many tasks are literally "find similar," "find far," or "group near" — geometry already is the answer.

LLM-per-item wins for open-ended reasoning and shifting schemas, but loses on cost and determinism. Keyword rules miss paraphrase. Training a large task net from scratch is usually unnecessary once a strong general embedding exists. Embeddings plus small models or ANN sit in the middle.

The core idea

Embeddings are a general-purpose semantic feature extractor. RAG is their most famous consumer, not their only one. Once text is a vector where distance tracks similarity, classic problems become cheap: classification (tiny head on frozen vectors), recommendations (nearest neighbors), anomaly detection (distance to the local neighborhood), deduplication (pairs above a calibrated threshold), and clustering (k-means or density-based groups).

For high-volume, repetitive judgments, that stack beats an LLM per item by orders of magnitude on cost and latency, with more consistent outputs. Pay the embed once; inference is a matrix multiply. The LLM stays at the edges, not on every row.

How it actually works

Each pattern is "embed, then apply a decades-old algorithm." One design decision dominates each path:

Classification. Embed the text; train logistic regression, a small MLP, or kNN on labeled examples. Hundreds of labels buy thresholdable probabilities and millisecond inference. With no labels, embed each label description and assign to the nearest (zero-shot classification). Usual path: LLM-label a seed → fit a head → serve with that head.

Recommendations. Embed item content; serve k-NN. That is content-based similarity ("about the same thing"), cold-start-proof — not collaborative filtering ("users who liked X liked Y"). Production blends both. Prefer dot product when norms carry popularity.

Anomaly detection. Score by distance to the k-th nearest neighbor or cluster centroid. Large distance means semantically novel, not bad. A burst far from history and close together can surface a new failure mode early — product launches look novel too.

Deduplication. Pairs above a calibrated threshold (illustratively ~0.9–0.97; right number is corpus-specific) are candidates. All-pairs is O(N²); at scale, query ANN for neighbors above τ — paraphrase mining — then union-find. Spot-check τ: 0.93 can be a paraphrase in one corpus and merely-related in another. Same hygiene for RAG: duplicate chunks waste context and crowd diverse evidence out of top-k.

Clustering. k-means or HDBSCAN over embeddings (often after UMAP); sample members; LLM names each cluster. k-means forces every point into exactly k spherical clusters. HDBSCAN finds a variable number and leaves noise unassigned — better for messy feedback, at the cost of tuning.

Two more patterns reuse the same geometry. Semantic caching serves a stored LLM answer when a new query embeds within threshold and context keys match — cost wins on repetitive traffic, false-hit risk on similar-looking intents. Semantic routing picks model or pipeline by nearest intent exemplar.

The flows

On the governed enterprise platform, the same embedding service that powers RAG also feeds ticket routing, near-duplicate detection on session docs, and a semantic cache for frequent policy questions — each with its own thresholds and entitlement rules.

A worked example

Task: categorize ~100k support tickets/month into 15 queues.

Seed 2,000 historical tickets already labeled by resolvers (LLM-assist thin classes to ~100+ each). Embed with the same production model as RAG; store model version on every row. Fit logistic regression — seconds on CPU — for 15-way probabilities.

Online: embed (~20–40 ms, illustrative) + predict (~1 ms). Route if max probability ≥ 0.55; else human triage. Embedding 100k short tickets typically costs orders of magnitude less than LLM-classifying them. Weekly: sample errors, refresh training data, watch prediction drift.

Week 0: embed each queue description, assign by nearest label vector, correct with humans, then fit LR. Keep LLM-per-ticket only when categories change daily or each item needs multi-step reasoning.

Production concerns

Thresholds do not transfer. Recalibrate dedupe, cache, and anomaly thresholds per model and corpus — and after any model swap, which also invalidates every stored vector. See similarity metrics.

Drift is continuous. Classifiers drift with vocabulary and traffic mix; monitor prediction distributions and refresh training data. Anomaly baselines need rolling windows. See observability.

Cost asymmetry is the point. Embedding on the order of a million short items costs single-digit dollars with small API models (less self-hosted). LLM classification of the same volume costs orders of magnitude more. State the ratio in designs. See cost.

Batch vs online. Clustering and dedupe are offline; classification, recs, and caching serve online. Different infra, same vectors — reuse one store. Online paths stay in the embed + ANN budget (tens of ms), not generation. See latency.

Privacy. Embeddings are derived data. Embedding inversion can recover substantial original text from vectors, so they inherit source access controls and retention. Personal and org-wide indexes both inherit IAM. See security.

Semantic cache false hits are silent. Hit rate alone is a vanity metric. "Cancel my order" and "cancel my subscription" can land near each other; the wrong answer looks like success until CSAT drops. Track false-hit rate on a labeled paraphrase-vs-confusion set, shadow-mode before cutover, keep TTL, and put tenant/product/user in the key when answers differ.

Common drill-downs

LLM or embeddings for 100k tickets into 15 stable categories? Embeddings + logistic regression: LLM-assist a seed once, train, serve with thresholds. Per-ticket LLM wins only when categories change constantly or need multi-step reasoning per item.

Zero-shot classification without labels? Embed each label description, embed the item, assign to the nearest. Upgrade: replace label vectors with centroids of a few labeled examples per class.

Near-duplicates in 10M docs without O(N²)? ANN + neighbors above calibrated τ + union-find. That is paraphrase mining at scale.

Content embeddings vs collaborative filtering? Content captures topical similarity and cold-start; collaborative captures taste. Production blends both. Prefer dot product when popularity lives in the norm.

What breaks when you change the embedding model? Everything downstream: full re-embed, recalibrate all thresholds, retrain every classifier. Different models share no geometry. Store embedding_model_id on every vector; refuse to score foreign spaces.

Test yourself

Go deeper

• OpenAI Embeddings guide — official walk-through of search, classification, clustering, recommendations, anomaly detection, and zero-shot label matching.
• Sentence Transformers — Paraphrase Mining — the scalable near-duplicate pattern (chunked search over all-pairs).
• Sentence Transformers — Clustering — k-means, agglomerative, and fast community clustering on embeddings.
• The Illustrated Word2vec — Jay Alammar — foundational geometry intuition, including the recommendation analogy.
• There and Back Again: An Embedding Attack Journey — IronCore Labs — why embeddings must inherit source access controls (inversion recovers meaning from vectors).

Where this connects

• Embeddings — the shared model, versioning, and re-embed economics every pattern inherits.
• ANN indexes — the lookup engine behind recs, dedupe, anomaly k-NN, and semantic cache.
• The RAG pipeline — still the primary consumer; these patterns reuse the same embed+index investment.
• Cost — where the gap between embed-once and LLM-per-item shows up in real budgets.