Tradeoff One-Liners

How to use this page

Each line is a decision with its deciding axis. The value isn't the verdict — it's the because clause. If you can state the axis, you can re-derive the verdict for a situation these lines don't cover, which is the situation you'll usually be in.

RAG vs fine-tuning — RAG for knowledge (updatable, citable, per-user access control); fine-tuning for behavior (format, style, domain reflexes). If the facts change or differ per user, it must be RAG.

Agent vs workflow — If you can write down the steps, build a workflow: cheaper, faster, testable. Use an agent only when the path is genuinely unpredictable and the outcome justifies token cost and error risk.

Hybrid vs dense-only retrieval — Add BM25 whenever queries contain identifiers, names, codes, or jargon that must match exactly; dense-only is acceptable for purely conceptual search. Hybrid with RRF is cheap insurance, so it's the production default.

Self-host vs API — Self-host only with sustained high utilization, hard data-residency constraints, or heavy fine-tune serving; otherwise the API wins on cost, capability, and zero ops. Idle GPUs bill the same as busy ones.

pgvector vs dedicated vector DB — pgvector while vectors fit one Postgres node (≤ a few million, moderate QPS) and you want joins, transactions, and one system to run. Move when scale, QPS, or filtered-search performance outgrow it.

Big model vs small + routing — One frontier model to prove the feature and simplify ops; add routing when volume makes cost/latency hurt — send easy traffic to a small model, escalate hard cases. Route only after evals define "easy."

Streaming vs batch — Anything a user watches streams (perceived latency = TTFT); anything offline goes to the batch API for ~50% off within 24h. The deciding question: is a human waiting?

Semantic cache vs none — Worth it for high-traffic repetitive queries (support, FAQ) where near-duplicates dominate; skip it for personalized, stateful, or long-tail traffic where false hits serve wrong answers. Always scope per tenant.

Cross-encoder rerank vs more top-k — Raising k adds noise for the LLM to wade through; reranking raises precision of what you keep. If the right chunk is retrieved but drowned, rerank; if it's not retrieved at all, fix recall first — reranking can't recover what isn't there.

Graph RAG vs vector RAG — Vector RAG answers "find passages about X"; graph RAG answers "how are X and Y connected" and corpus-level themes via entity graphs. Pay graph RAG's heavy indexing cost only when multi-hop relational questions are the actual workload.

Long context vs RAG — Long context wins for one bounded document set you interrogate repeatedly (with prompt caching); RAG wins for large, changing, access-controlled corpora — cheaper per query, citable, and fresh. A million tokens still isn't your whole knowledge base.

JSON mode vs function calling — Formatting an answer? Structured output/JSON schema. Deciding to trigger your code with typed arguments? Function calling. The criterion is whether the model chooses an action, not just a shape.

Multi-agent vs single agent — Go multi-agent only for context isolation: parallel independent explorations or tool sets too big for one prompt. It multiplies token cost and adds coordination failures, so a single agent that fits is always preferable.

Harness work vs orchestration work — Failures clustering inside a step (wrong tool picked, window blown, unactionable errors) are harness problems and a better graph won't touch them. Failures clustering between steps (lost work on crash, hours-long approvals, replayed side effects) are orchestration problems and a better prompt won't touch them.

Harness investment vs model upgrade — Fix the harness when the bottleneck is expression: how the model states an edit, reads a result, or navigates a catalogue — cheap, immediate, and measurable. Wait for the model when the bottleneck is long-horizon capability; scaffolding doesn't move that ceiling, and it goes stale on the next release.

Compaction vs a bigger context window — Compact. Quality degrades well before the window fills, and irrelevant context actively hurts, so a focused prompt beats a long one containing the same facts. A bigger window buys headroom, not reliability.

Gate every tool vs gate the irreversible ones — Gate narrowly. Users approve the overwhelming majority of prompts they're shown, so a gate on everything is a rubber stamp with extra latency; auto-approve reads, hard-gate irreversible external effects, and treat rejection rate as a health metric.

Prompt engineering vs fine-tuning for format — Prompting plus schema-constrained output handles almost all formatting needs at zero training cost; fine-tune only when a strict format must hold at massive scale on a small model where saved prompt tokens pay for training.

Managed vs self-hosted vector DB — Managed (Pinecone-class) buys you zero ops and elastic scale at a margin and data-egress dependency; self-hosted (Qdrant/Milvus) buys control and lower unit cost at the price of running it. Decide by whether you have — and want to spend — infra engineers.

---

These compress decisions argued in full across the site — start with RAG, Agents, and Model Landscape for the reasoning behind them.