Fine-Tuning

The intuition

Think of a generalist contractor who already knows how to build houses. Fine-tuning is teaching them your firm's house style — where the light switches go, how you write change orders, the exact tone of a client email. After enough supervised examples, they stop needing a five-page checklist every morning; the style is in their hands. RAG is handing them today's blueprints and the building code binder for this job — facts that change, must be citable, and should never be "memorized" into muscle memory because yesterday's code amendment would become a liability.

You do not teach the contractor the entire municipal code by repetition drills (fine-tuning-for-knowledge fails). You do not staple a style guide to every single text message if you can train the habit once (prompting-only-at-scale fails). Production systems usually need both: retrieval for living knowledge, fine-tuning (or distillation) when behavior must hold cheaply across millions of calls.

Why it exists

Prompting and RAG cover a huge fraction of applied work, but four hard constraints still create a fine-tuning shaped hole:

1. Behavior that will not stick. Strict JSON schemas, house tone, domain jargon, or multi-step house rules drift across millions of calls no matter how carefully you prompt — especially when context is crowded with retrieved docs.
2. Cost and latency at narrow high volume. A frontier model already solves ticket triage or classification, but paying frontier prices (and latency) on every call is wasteful. Distilling that behavior into a small model is often a ~10× cost and ~2–5× latency win on that route.
3. Facts vs weights. Teams reach for fine-tuning to "teach the model our docs" and discover gradient updates do not make a reliable, citable knowledge base. That pain is why the RAG-vs-fine-tune distinction must be sharp.
4. Full fine-tunes are operationally obsolete for most teams. Updating all N billion weights needs optimizer state ~2–4× model size and produces a full copy per task. LoRA/QLoRA exist so adapters are small, swappable, and trainable on one GPU.

The alternatives lose on the wrong failure mode. More prompt engineering alone loses when the behavior must be automatic and short. RAG alone loses when the model formats or prioritizes wrong even with perfect chunks. Fine-tuning for fresh facts loses on staleness, attribution, and capacity. Training without baselines loses because "it feels better" is not a ship gate. Fine-tuning exists as a deliberate stage in a sequence — not as the default first move when an LLM answer disappoints.

The core idea

The clean framing: fine-tuning changes behavior, RAG changes knowledge. Fine-tuning adjusts the model's weights so it reliably produces a particular style, format, or skill — it is terrible at injecting facts (limited capacity, no source attribution, stale the day training ends). RAG injects facts at inference time from an updatable index — it is terrible at changing how the model behaves. They are not competitors; they compose: a common production system is a fine-tuned small model (behavior) sitting on top of a retrieval pipeline (knowledge).

The decision sequence is: prompt → RAG → fine-tune → distill. Exhaust prompting first (cheapest iteration loop), add RAG when the failure is missing/stale knowledge, and reach for fine-tuning only when prompting demonstrably can't hold the behavior — or when you want to compress a big model's behavior on a narrow task into a small, cheap, fast model. Fine-tuning wins on: consistent output format/tone at scale, narrow repeated tasks, domain vocabulary/jargon, and cost/latency reduction via distillation. It loses on: fresh or changing facts, anything needing auditability/citations, small datasets (under a few hundred good examples you'll overfit), and broad open-ended tasks where the base model's generality is the point.

Mechanically, almost nobody full-fine-tunes anymore: LoRA trains tiny low-rank adapter matrices (~0.1–1% of parameters) on a frozen base model, and QLoRA does the same over a 4-bit-quantized base — which is how a 70B-class fine-tune fits on a single GPU. The work is 80% data curation and evals, 20% training.

How it actually works

The decision table

Why fine-tuning fails at knowledge, mechanically: gradient updates on a few thousand examples can shift response distributions but can't reliably store retrievable facts the way a search index does; new facts also start going stale immediately, and each refresh is a training run instead of an index upsert. Deeper RAG mechanics: the RAG pipeline. Caching before fine-tuning for prompt length: LLM APIs, cost.

LoRA at survival depth

Full fine-tuning updates all N billion weights — needs optimizer state ~2–4x model size in GPU memory, produces a full model copy per task. LoRA (Low-Rank Adaptation) instead freezes the base weights W and learns an additive update ΔW = B·A, where A and B are two small matrices of rank r (typically 8–64) injected into attention/MLP layers. The insight from the paper: task adaptations are intrinsically low-rank, so a tiny ΔW captures them.

Why it's cheap, concretely:

• Trainable parameters drop to ~0.1–1% of the model → optimizer memory shrinks proportionally; training runs on one GPU instead of a cluster.
• The artifact is an adapter of tens–hundreds of MB, not a multi-GB model copy — you can store one per task/customer and hot-swap them on a shared base (vLLM serves multiple LoRA adapters concurrently).
• At inference you can merge B·A into W — zero added latency — or keep adapters separate for swapping.

QLoRA adds: quantize the frozen base to 4-bit (NF4), keep the LoRA adapters in higher precision, backpropagate through the quantized weights. Memory for a 65–70B fine-tune drops to a single ~48GB GPU with near-parity quality. This is the default recipe in 2026: 4-bit base + LoRA rank ~16.

The practical pipeline

1. Data prep — this dominates. 500–2,000 curated prompt→response examples in chat format typically beat 50K scraped ones. Sources: real production traffic (best), frontier-model outputs on your prompts (distillation), hand-written gold examples. Deduplicate, filter failures, hold out a test split before looking at anything.
2. Baseline evals first. Score the base model (and the prompted frontier model) on your eval set — otherwise "the fine-tune helped" is vibes. Practice: evals and testing.
3. Train. QLoRA via HF peft/TRL, Unsloth, Axolotl, or a provider's hosted fine-tuning API (OpenAI, Gemini offer hosted tuning; you trade control for zero infra). Typical run: under an hour on one GPU for a small model. Few epochs; watch for the loss going to zero (memorization).
4. Eval after — same set. Check the target behavior improved and general capability didn't regress (catastrophic forgetting shows up as degraded off-task performance; low LoRA rank and few epochs mitigate).
5. Ship + monitor. Version the adapter with its dataset hash; re-run evals when the base model or data changes. Retraining cadence is your "knowledge staleness" analog.

Distillation (mention level)

Distillation = transferring a large model's behavior on a task into a small model, in practice by generating training pairs with the teacher and fine-tuning the student on them. It's the strongest commercial case for fine-tuning in 2026: cut inference cost by ~10x and latency by ~2–5x on a narrow route the frontier model already handles — e.g. frontier-quality ticket triage served by a fine-tuned 8B model. Check provider ToS: some restrict using outputs to train competing models.

The flows

A worked example

Consider the governed enterprise platform. Two failures show up in the same product:

Failure A — knowledge. Employees ask about policy versions that change quarterly. A prompted frontier model invents plausible thresholds when retrieval is empty. Fix: RAG, not fine-tuning. Upsert the new PDF chunks; citations appear; last quarter's numbers leave the index.

Failure B — behavior at volume. Ticket triage must emit a strict JSON object: {priority, queue, pii_flags[], summary} with house rules for priority. The frontier model does this well with a long prompt, but the route is high volume (~tens of thousands of tickets/day at typical enterprise scale) and latency/cost hurt. Prompting a small base model drifts on edge cases.

Distillation + LoRA path for B:

Economics sketch (order of magnitude): if frontier triage costs ~$0.01–0.03/ticket and the student is ~10× cheaper on that narrow task, at 20k tickets/day the route-level savings dominate training cost (often tens of dollars of GPU time with QLoRA). Training is cheap; ongoing cost is data refresh, eval maintenance, and retrain when queues or priority rules change.

Compose with RAG: triage summary may still attach retrieved account notes — eval the composed system. A model fine-tuned for terse JSON can get worse at long retrieved context; that shows up only in the full pipeline.

Common drill-downs

RAG or fine-tuning? Wrong question as posed — they solve different failures. Missing/stale facts → RAG (updatable, citable). Unreliable style/format/skill → fine-tune (weights hold behavior). Production systems compose both. Sequence: prompt → RAG → fine-tune → distill.

Why can't fine-tuning teach the model our product docs? A few thousand gradient steps shift behavior distributions but don't reliably store retrievable facts; there's no attribution, updates require retraining, and hallucination risk stays. An index handles facts; that's RAG's job.

Explain LoRA to me. Freeze the base weights; learn a low-rank additive update ΔW = B·A (rank ~8–64) on attention/MLP layers — about 0.1–1% of parameters. Near-full-fine-tune quality at a fraction of memory/compute; artifact is a small swappable adapter that can be merged for zero inference overhead.

What does QLoRA add? Quantizes the frozen base to 4-bit (NF4) and trains LoRA adapters on top, backpropagating through the quantized weights. Cuts memory ~4x further — a 65–70B fine-tune fits one 48GB GPU with near-parity quality. The default recipe today.

When does fine-tuning actually beat prompting? When the behavior must hold across millions of calls where prompts drift (strict format/tone), when the task is narrow and repeated (classification with house rules, domain extraction), and when distilling a frontier model into a small one for 10x cost/latency wins. Only after prompting has been genuinely exhausted — prompt iteration is minutes, a fine-tune cycle is days.

How much data do you need? Quality over quantity: ~500–2,000 curated, deduplicated examples for style/format tasks; more for skill acquisition. Below a few hundred, few-shot prompting usually wins. Always split a held-out test set before training.

How do you know the fine-tune worked? Same eval set before and after: target-task metrics up, off-task/general metrics not down (catastrophic forgetting check), plus a live canary. No baseline, no claim.

What's distillation and when do you use it? Generate training pairs with a big teacher model, fine-tune a small student on them. Use when a narrow, high-volume route is proven on the frontier model and you want ~10x cheaper, faster serving at equivalent on-task quality. Mind provider terms on training from outputs.

Production concerns

• You own the model lifecycle now: a fine-tune pins you to a base-model snapshot. When the provider deprecates it (or the open-weight base is superseded), you re-run the pipeline — so the pipeline must be reproducible (versioned data, config, eval set), not a one-off notebook. Ties to deprecation discipline in choosing models.
• Regression surface widens: every retrain can silently regress an untested behavior. Gate deploys on the eval suite; canary the new adapter against live traffic. See evals and testing, reliability.
• Serving: merged adapters = zero latency overhead, one artifact per task; unmerged multi-LoRA serving (vLLM) = one base model in memory, per-request adapter choice — the economical path for many tenants/tasks. See open source & self-hosting.
• Cost shape: training is cheap (often tens of dollars of GPU time with QLoRA); the recurring costs are data curation, eval maintenance, and retrains. Hosted fine-tuning APIs also add a per-token inference premium over the base model. Broader levers: cost.
• Drift: the fine-tuned behavior is frozen while your domain moves. Monitor output-quality metrics in prod; schedule data refresh + retrain rather than waiting for user complaints. See observability.
• Composition gotcha: a model fine-tuned for terse structured output may get worse at consuming long RAG context or verbose reasoning. Eval the composed system (fine-tune + retrieval + prompt), not the model in isolation.

Test yourself

Go deeper

• LoRA: Low-Rank Adaptation of Large Language Models (arXiv 2106.09685) — the original paper; §4 has the method in two equations.
• QLoRA: Efficient Finetuning of Quantized LLMs (arXiv 2305.14314) — 4-bit NF4 + paged optimizers; why 65B fits on one GPU.
• Hugging Face PEFT documentation — the standard library for LoRA/QLoRA adapters in practice.
• Unsloth docs — fastest practical path to a local QLoRA fine-tune (~2x faster, ~70% less VRAM).
• LoRA & QLoRA Fine-tuning Explained In-Depth — Entry Point AI — the go-to conceptual explainer, including rank/alpha intuition.
• LoRA: Low-Rank Adaptation of Large Language Models — Explained visually + PyTorch code from scratch — Umar Jamil — the math and a from-scratch implementation.

Where this connects

• Choosing models — distillation targets a cheaper rung on the same ladder; route the student only where evals clear.
• Open source & self-hosting — adapters need a serving stack; multi-LoRA on vLLM is the multi-task pattern.
• The RAG pipeline — knowledge stays in the index; fine-tunes supply behavior on top of retrieval.
• Evals and testing — baseline → post-train → canary is the only honest proof the fine-tune worked.