Embeddings

The intuition

Imagine every document, paragraph, and query as a point on a map — not a flat map of cities, but a map with hundreds or thousands of axes. Similar meaning places points near each other. "Reset my password" sits close to "credential recovery procedure"; both sit far from "Q3 revenue forecast."

You do not draw the map by hand. An embedding model learns where each piece of text belongs so that distance approximates relevance. Finding the right documents stops being a string-matching problem and becomes a geometry problem: which stored points are nearest this query point?

That single move — meaning → fixed-length vector → nearest-neighbor lookup — is the foundation of modern retrieval, recommendations, deduplication, and a long list of "embed then classical ML" patterns.

Why it exists

Keyword search answers "which documents share these tokens?" That fails the moment users paraphrase, use synonyms, or ask in natural language while the corpus uses different words for the same idea.

Three hard constraints make learned vectors necessary:

1. Exact tokens are a fragile match signal. "PTO policy" and "vacation leave rules" share almost no surface form, yet they are the same intent for retrieval.
2. You need a representation computers can compare at scale. A fixed-length float vector turns every document and every query into the same kind of object, so one distance function ranks the whole corpus.
3. You cannot hand-craft features for every domain. Contrastive training on millions of pairs learns the geometry from data instead of from rules you invent and maintain.

The alternatives lose on different axes. Pure keyword (BM25) is cheap and excellent on exact IDs, but blind to paraphrase. Hand-written taxonomies need constant curation and never cover freestyle questions. Stuffing the full corpus into an LLM context does not scale past what fits, costs tokens on every query, and still does not give you a reusable index. Embeddings are the representation layer that makes semantic search, RAG, and "similar items" product features operational.

The core idea

An embedding is a fixed-length vector of floats — typically 384 to 3072 dimensions — that represents the meaning of a piece of text as a point in high-dimensional space. The model is trained so that semantically similar texts land close together and unrelated texts land far apart, which turns "find relevant documents" into "find nearby vectors" — a geometry problem computers solve fast.

The models that produce them are transformer encoders trained with contrastive learning: the training data is millions of pairs that should match (question ↔ answer, title ↔ body, paraphrase ↔ paraphrase). The loss pulls matching pairs together and pushes non-matching pairs (other examples in the same batch) apart. The result is a space where distance approximates relevance.

Three practical facts complete the picture. First, the whole input is compressed into one vector, so a 500-token chunk and a 5-word query both become the same-sized vector — that lossy compression is why retrieval quality depends heavily on chunking. Second, embeddings from different models are not comparable: you must embed queries and documents with the same model and version, and changing models means re-embedding the entire corpus. Third, choosing a model is an empirical decision — start from the MTEB leaderboard as a shortlist, then evaluate on your own data, because leaderboard rank does not guarantee rank on your domain.

How it actually works

From raw text to a searchable vector, the path is short to describe and full of production traps:

Architecture. A transformer encoder (BERT-style, or increasingly an LLM decoder backbone) produces one contextual vector per token. A pooling step collapses these into a single vector: mean pooling over tokens, the [CLS] token, or last-token pooling for decoder-based models. The output is usually L2-normalized to unit length. (How that normalization interacts with similarity metrics matters when you choose cosine vs dot product.)

Contrastive training, one level down. The dominant loss is InfoNCE / MultipleNegativesRankingLoss: for a batch of (query, positive) pairs, every other positive in the batch serves as a negative. The model maximizes similarity of true pairs relative to all in-batch negatives — which is why large batch sizes help (more, harder negatives). Training typically runs in two stages:

1. Weakly-supervised pretraining on billions of naturally-occurring pairs (title/body, question/answer from forums, citation pairs).
2. Supervised fine-tuning on labeled relevance data with hard negatives — documents that look topically similar but are wrong — mined by running retrieval and picking high-ranking non-answers. Hard negatives are what teach the model fine distinctions.

Asymmetry and instructions. Queries and documents differ in length and style, so many models train with prefixes (query: / passage:) or free-text task instructions. Omitting the required prefix at inference silently degrades quality — a classic production bug.

Dimensionality tradeoffs. More dimensions = more capacity, but cost scales linearly everywhere:

Storage = dims × 4 bytes × count (before index overhead); distance computation and index memory scale with dims too. Quality gains flatten well before 4096 — dimension count is a cost knob more than a quality knob. At large N, that cost is why ANN indexes and quantization become non-optional.

Matryoshka embeddings (MRL). Trained with the loss applied simultaneously at several prefix lengths (e.g. 768, 512, 256, 128, 64), forcing the most important information into the earliest dimensions. You can then truncate a stored embedding to the first N dims (and re-normalize) and keep most quality — an MRL model at ~8% of full size retains ~98% of performance. OpenAI's text-embedding-3 models expose this as the dimensions API parameter; text-embedding-3-large truncated to 256 dims still beats the older ada-002 at 1536. Enables adaptive retrieval: coarse search over short vectors, rescore top candidates with full vectors.

Choosing a model — the actual procedure.

1. Filter MTEB by hard constraints: language(s), max sequence length (512 vs 8k+ tokens changes your chunking strategy), dimensions, license (self-host vs API), price.
2. Shortlist 2–4: as of 2026 that usually means one API model (Gemini embedding, OpenAI text-embedding-3, Cohere embed-v4, Voyage) and one open model (Qwen3-Embedding, bge/e5 family). Note MTEB was revised (v2) — scores across versions are not comparable, and top labs partly train on retrieval data similar to the benchmark, so treat rankings as a shortlist, not a verdict.
3. Evaluate on your own golden set: 50–200 real queries with known relevant chunks; measure recall@k and nDCG@k per model. Domain shift (legal, medical, code, internal jargon) routinely reorders the leaderboard.

The flows

On the governed enterprise platform, offline ingest is a scheduled pipeline over internal wikis and policy PDFs; online query embed is a single hop on the request path before entitlement-filtered vector search. Model migrations are treated as versioned dual-write jobs, not silent API upgrades.

A worked example

Consider a platform where employees ask policy questions over ~10k internal documents. Trace one chunk and one query.

Document side (offline). A 1,200-token expense-policy section is chunked to ~400 tokens with overlap. One chunk reads, roughly: "International travel requires pre-approval above the threshold; per-diem rates update annually…" (~90 words, ~120 tokens after tokenization).

Query side (online). Employee asks: "Does international travel still need pre-approval?"

Scores are model-specific; a cosine of 0.82 is not "universally good" — it is only meaningful relative to this model's distribution on this corpus. (See similarity metrics.)

Common drill-downs

These are the questions a careful engineer asks one level down once the core idea is clear.

What is an embedding, in one sentence? A learned fixed-length vector representation of content where geometric proximity approximates semantic similarity, enabling meaning-based search via nearest-neighbor lookup.

How are embedding models trained? Contrastive learning on massive pair datasets: pull true (query, positive) pairs together, push in-batch and mined hard negatives apart (InfoNCE-style loss). Usually weakly-supervised pretraining on web-scale pairs, then fine-tuning with hard negatives.

Can you compare vectors from two different embedding models? No — each model defines its own space; even same-dimension vectors from different models (or versions) are geometrically unrelated. Same model, same version, same preprocessing for both sides of every comparison.

What are Matryoshka embeddings and why do they matter? Models trained with loss at multiple prefix dimensions, so truncating a vector to its first N dims keeps most quality. They let you cut storage and search cost several-fold post-hoc (e.g. OpenAI's dimensions parameter) and enable coarse-search-then-rescore retrieval.

How would you choose an embedding model for a new project? Filter MTEB by language, context length, license, and cost; shortlist one API and one open model; then benchmark recall@k/nDCG on a golden set of your own queries — MTEB rank is a prior, not a decision.

Why does a bigger embedding dimension not always help? Quality gains saturate while storage, memory, and distance-computation cost grow linearly with dims; past ~1–3k dims you mostly buy cost, not recall. Measured recall on your data decides.

What happens if you embed a document longer than the model's context window? The tail is truncated — silently, in most stacks — so its content becomes unsearchable. That's why chunking to within the model's limit is enforced upstream of embedding.

Your retrieval quality dropped after a "routine" model upgrade. What happened? Query and document embeddings now come from different spaces — the corpus was embedded with the old model. Any model change requires re-embedding everything and cutting over atomically (or dual-indexing during migration).

Production concerns

• Model lock-in: vectors are useless without the exact model that made them. A model swap or provider deprecation forces a full corpus re-embed — budget for it (tokens × price) and design the pipeline to support versioned, dual-write re-indexing with cutover. (Related: reliability for cutover and fallback patterns.)
• Cost: API embedding is cheap per call (order of $0.02–$0.13 per 1M tokens for OpenAI's text-embedding-3 tier) but re-embedding 100M chunks is not; self-hosting shifts cost to GPUs and ops. See cost.
• Latency: query-time embedding adds one network hop (typically tens of ms via API); batch document embedding is throughput-bound — batch requests, parallelize, respect rate limits. See latency.
• Input limits: text beyond the model's max tokens is truncated silently — enforce chunk sizes upstream or you index half-empty meaning. Chunking strategy is covered in ingestion and chunking.
• Normalization and prefixes must be identical between indexing and querying; drift here produces quietly-bad retrieval that no error log will show.
• Storage math compounds: dims × 4 bytes × chunks × replicas, plus index overhead. Matryoshka truncation and quantization (see ann-indexes.md) are the levers.
• Observability: track embedding model ID and version as first-class fields on every vector and every query log so you can detect space mismatch. See observability.

Test yourself

Go deeper

• MTEB Leaderboard — Hugging Face — the standard embedding benchmark; filter, don't just sort.
• Matryoshka Representation Learning — Kusupati et al. — the MRL paper.
• Introduction to Matryoshka Embedding Models — Hugging Face blog — training mechanics plus measured truncation results.
• OpenAI Embeddings guide — the dimensions parameter and canonical use cases.
• Sentence Transformers documentation — the reference open-source stack for training and serving embedding models.
• The Illustrated Word2vec — Jay Alammar — the classic visual intuition for embedding spaces.
• Word Embedding and Word2Vec, Clearly Explained!!! — StatQuest with Josh Starmer — the canonical beginner-to-solid video on how embedding training shapes the space.

Where this connects

• Similarity metrics — once you have vectors, which distance you use (and whether they are normalized) decides ranking.
• ANN indexes — exact scan does not scale; HNSW/IVF/quantization make "find nearby" fast.
• The RAG pipeline — embeddings are the representation layer inside offline ingest and online retrieve.
• Embeddings beyond RAG — classification, recsys, dedupe, and anomaly detection reuse the same vectors.