Similarity Metrics
Cosine vs dot product vs euclidean; normalization; when the choice matters.
Prerequisites
- Embeddings — what the vectors are and why models often L2-normalize them.
- How LLMs Work — enough geometry of high-dimensional spaces to follow ranking arguments.
The intuition
You have two points on a map. "How similar are they?" can mean three different things:
- Do they point the same way? (ignore how far each is from the origin) — cosine
- Are they aligned and large? (direction and length both count) — dot product
- How far apart are the points as locations? — euclidean (L2) distance
Most text embedding search cares about direction: a short query and a long paragraph about the same topic should match. Cosine is built for that. Dot product is cosine's cheaper cousin once vectors sit on the unit sphere. Euclidean is the natural language of "how far as points," which shows up more in clustering and some image/scientific features.
The load-bearing fact: if every vector has length 1, all three order neighbors the same way. That is why so many production stacks "choose cosine" and then implement it as normalized dot product.
Key insight
Metric choice is usually a geometry contract with the embedding model, not a free performance knob. Match the metric the model was trained with; take the free dot-product shortcut only when vectors are truly unit-normalized.
Why it exists
Once text is a vector, you still need a rule that turns two vectors into a ranking. Without an explicit metric, "nearest neighbor" is undefined.
Hard constraints that force the choice into the open:
- Indexes encode a metric. HNSW edges and IVF clusters assume one neighborhood structure. Build with L2 and query with unnormalized dot product and you are searching the wrong graph.
- Magnitude may or may not be signal. In pure semantic text search you usually want to ignore document length in the vector; in recommenders, item norm often is popularity.
- Score APIs disagree on polarity. Some stores return similarity (higher better); others return distance (lower better). Threshold logic written for one convention silently inverts under the other.
- Training shaped the space for one objective. A model optimized under cosine loss was not guaranteed to rank correctly under a different geometry.
Alternatives that skip a careful choice — "just use whatever the DB defaults to" or "always cosine for everything" — work until the first model card that says inner product, the first recommender that needs MIPS, or the first absolute threshold that was calibrated under a different range.
The core idea
Three metrics cover almost all vector search. Dot product (a·b = Σ aᵢbᵢ) measures alignment scaled by both vectors' magnitudes. Cosine similarity (a·b / (‖a‖‖b‖)) is the dot product with magnitude divided out — it compares direction only, ranging −1 to 1. Euclidean (L2) distance (‖a−b‖) is straight-line distance between the points.
The key fact: on unit-normalized vectors the three are equivalent for ranking. If ‖a‖ = ‖b‖ = 1, then cosine equals dot product exactly, and squared L2 distance is 2 − 2·cos(a,b) — a monotonic transformation, so all three return identical nearest-neighbor orderings. Most modern embedding APIs (OpenAI included) return normalized vectors, so in practice the "choice" is often a no-op, and dot product is preferred because it skips computing norms.
When it does matter: use the metric the embedding model was trained with — the model's loss shaped the geometry, and querying with a different metric queries a geometry the model never optimized. And when magnitude carries signal — recommendation systems often encode popularity or confidence in vector length — dot product preserves that signal while cosine deliberately discards it.
How it actually works
How the three metrics relate once vectors leave the embedding model:
Definitions and properties.
| Metric | Formula | Range | Sensitive to magnitude? | Cost per comparison |
|---|---|---|---|---|
| Dot product | Σ aᵢbᵢ | −∞ … ∞ | Yes | d mults + d−1 adds (cheapest) |
| Cosine | a·b / (‖a‖‖b‖) | −1 … 1 | No | dot + 2 norms + 1 div |
| Euclidean (L2) | √Σ(aᵢ−bᵢ)² | 0 … ∞ | Yes | d subs + d mults + sqrt |
The normalization identity. For unit vectors: ‖a−b‖² = ‖a‖² + ‖b‖² − 2a·b = 2 − 2a·b. Smaller L2 ⇔ larger dot ⇔ larger cosine. This is why vector DBs implement cosine as "normalize at write and query time, then use dot product" — same ranking, cheaper math, and dot product SIMD-vectorizes cleanly.
Key insight
"Cosine" in a vector DB is often a write-time contract, not a different formula at query time: normalize once, then run pure dot products forever. If you write unnormalized vectors into a "cosine" collection that assumes unit length, you corrupt rankings without throwing an error.
What each metric "means" geometrically.
- Cosine asks: do these vectors point the same way? A short and a long vector in the same direction are identical to cosine. Good default for text similarity, where document length shouldn't dominate relevance.
- Dot product asks: aligned and large? A high-magnitude vector scores high against many queries — a "hubness" effect. In recommenders this is a feature: item norm learns popularity (matrix factorization scores are literally user·item dot products, i.e. MIPS — maximum inner product search).
- Euclidean asks: how far apart as points? Natural for data where absolute position matters — k-means centroids, image descriptors, scientific measurements.
Distance vs similarity conventions. Databases report either similarity (higher = closer) or distance (lower = closer). Weaviate's cosine distance is 1 − cosine similarity (range 0–2); FAISS METRIC_INNER_PRODUCT returns similarities while METRIC_L2 returns squared distances (it skips the sqrt — monotonic, so ranking is unchanged). Know which convention your store uses before writing threshold logic.
Index interaction. HNSW and IVF are built for a metric — the graph edges / cluster assignments encode that metric's neighborhood structure. You cannot build with L2 and query with dot product. Pure MIPS (unnormalized dot product) is not a proper metric (no triangle inequality), so libraries either use specialized MIPS support or reduce it to cosine/L2 via normalization or an extra-dimension transform.
Common misconception
Switching the index metric "because L2 is faster" is almost never a free win. Metric math is O(d) with SIMD for all three; the real cost is mismatch with training and with absolute thresholds. If vectors are normalized, rankings may look unchanged while every score-based rule in the application quietly breaks.
The flows
| Flow | Sequence | When it applies | What breaks it |
|---|---|---|---|
| Normalized text retrieval | embed → L2-normalize (if API did not) → index with cosine or IP → query with same metric | Default semantic search / RAG over text | Unnormalized writes; double-assuming normalization that is not there |
| Cosine-as-dot optimization | normalize at ingest and query → store/search with inner product | High-QPS text search when unit vectors are guaranteed | Matryoshka truncation without re-normalize; partial pipelines that skip query-side norm |
| MIPS recommender path | keep item (and maybe user) magnitudes → max inner product search → rank by score | Content or collaborative recs where norm ≈ popularity/confidence | Forcing cosine and wiping the popularity signal; using a pure metric index without MIPS support |
| Threshold / filter path | retrieve top-k → apply absolute score cutoff → keep survivors | "Only answer if confident" product rules | Cross-model thresholds; confusing distance with similarity; uncalibrated 0.8 cutoffs |
On the governed enterprise platform, policy RAG uses unit-normalized vectors and cosine-equivalent IP ranking; absolute cosine cutoffs are avoided in favor of a reranker when precision matters. A separate "similar tickets" feature that encodes volume in the vector would deliberately stay on dot product.
A worked example
Three 2-D toy vectors (easy to compute by hand; the same algebra holds at 1536-d):
| Vector | Components | L2 norm | Meaning in the story |
|---|---|---|---|
| q (query) | (0.6, 0.8) | 1.0 | "international travel pre-approval" |
| d₁ (right chunk) | (0.6, 0.8) | 1.0 | same direction, unit length |
| d₂ (popular hub) | (1.2, 1.6) | 2.0 | same direction, double magnitude |
| d₃ (wrong topic) | (0.8, −0.6) | 1.0 | orthogonal-ish direction |
| Pair | Dot | Cosine | L2 distance |
|---|---|---|---|
| q · d₁ | 1.0 | 1.0 | 0 |
| q · d₂ | 2.0 | 1.0 | 1.0 |
| q · d₃ | 0.0 | 0.0 | √2 ≈ 1.41 |
Ranking under cosine: d₁ and d₂ tie (both perfect direction), then d₃.
Ranking under dot product: d₂ first (hub wins), then d₁, then d₃.
If you L2-normalize d₂ first to (0.6, 0.8): all three metrics agree again — d₁/d₂ tie, d₃ last.
Now the production-shaped version on the platform: OpenAI-style API returns unit vectors; the index is configured for cosine but stores IP after normalize-on-write. For a real query, top hits might show cosine similarities of 0.78, 0.74, 0.71 — not transferable as thresholds to a different model whose scores all sit above 0.85.
What each omission looks like in production
- Model trained with cosine, index set to unnormalized L2 on raw vectors → neighborhood structure wrong; recall drops without a hard error.
- Assume OpenAI-style normalization on a model that does not normalize → IP ≠ cosine; long vectors dominate.
- Truncate Matryoshka dims and skip re-normalize → "unit sphere" identities fail; IP ranking drifts.
- Threshold
score > 0.8copied from a blog → one model returns everything, another returns nothing. - Switch metric from cosine to L2 on normalized data and keep similarity thresholds → ranking may stay OK; every absolute-score branch is now wrong polarity/scale.
Common drill-downs
These are the questions a careful engineer asks one level down once the formulas are clear.
Cosine vs dot product — what's the actual difference? Cosine = dot product ÷ both magnitudes: direction only, range −1..1. Dot product also rewards magnitude, so longer vectors score higher. On unit vectors they're identical.
When are cosine, dot product, and euclidean equivalent? When vectors are L2-normalized: cosine ≡ dot, and L2² = 2 − 2cos, a monotonic map — all three produce the same nearest-neighbor ranking.
Why do vector databases often use dot product internally even when you configure "cosine"? They normalize vectors at ingest/query, after which dot product equals cosine but is cheaper — no norm computation per comparison and clean SIMD.
When would you deliberately prefer dot product over cosine? When magnitude encodes signal — recommender item embeddings where norm captures popularity/confidence (MIPS), or any model explicitly trained with inner-product loss.
How do you decide which metric to use for a given embedding model? Use the metric the model was trained with (model card / docs). The training loss shaped the space for that metric; anything else queries an unoptimized geometry.
Is a cosine similarity of 0.75 "good"? Unanswerable without the model — score distributions differ wildly across models, and many models compress all pairs into a narrow high band. Thresholds must be calibrated on your data, per model.
A teammate switched the index metric from cosine to L2 and nothing broke — why, and is it safe? If vectors are normalized, rankings are identical (L2² = 2 − 2cos), so retrieval is unchanged — but any absolute-score thresholds and "similarity" semantics in code are now wrong (distances, lower-is-better). Safe for ranking, hazardous for score logic.
Production concerns
- Model–metric mismatch is a silent quality bug: retrieval still returns results, they're just worse. Check the model card ("trained with cosine similarity") and configure the index to match.
- Normalize once, at the right place. If the store expects normalized vectors for its dot-product-as-cosine trick, un-normalized writes corrupt rankings. If the API already normalizes (OpenAI does), double-normalizing is harmless; assuming normalization that isn't there is not. Note: truncating Matryoshka embeddings destroys unit norm — re-normalize after truncation. See embeddings.
- Score thresholds are metric- and model-specific. "Filter results below 0.8 cosine" is meaningless across models — different models occupy different similarity ranges (many never produce scores below ~0.6 for any pair). Calibrate thresholds empirically per model; better, avoid absolute-score cutoffs and use a reranker.
- Cost: metric choice is a rounding error next to index choice — all three are O(d) with SIMD. Choose for correctness (match training), not speed; take the free dot-product shortcut only when vectors are truly normalized. Latency budgets live in latency.
- Hubness at high dimension: with unnormalized dot product, a few high-norm vectors appear in everyone's top-k. If you see the same documents dominating unrelated queries, check norms.
- Filtering + scores: entitlement filters on the platform change the candidate set; do not interpret raw cosine bands as calibrated confidence across ACL slices. See filtering and metadata.
Test yourself
You configure a collection as cosine. The embedding API returns unnormalized vectors. You never normalize on write. What ranking do users actually get?
Why is pure MIPS not a metric, and why does that matter for graph indexes?
After truncating text-embedding-3-large from 3072 to 256 dimensions, top-k order looks slightly off vs full vectors even though the model card allows truncation. Name two distinct causes.
Weaviate returns cosine *distance* 0.15 for a hit. A dashboard alerts when 'similarity < 0.8'. Is this hit good or bad under that rule?
When would you keep absolute score thresholds instead of 'always top-k + rerank'?
Go deeper
- Vector Similarity Explained — Pinecone — the three metrics with visuals and use-case guidance.
- Distance metrics — Weaviate docs — how a real vector DB defines cosine/dot/L2 distances, ranges, and conventions.
- Cosine Similarity, Clearly Explained!!! — StatQuest with Josh Starmer — the canonical short video on the geometry.
Where this connects
- Embeddings — where unit-length vectors and Matryoshka truncation come from.
- ANN indexes — the index is built for one metric; recall tuning assumes that geometry.
- Embeddings beyond RAG — recommenders often want dot product so magnitude can carry popularity.
- Rerankers — when absolute bi-encoder scores are untrustworthy, rescore pairs instead of thresholding cosine.