Fine-Tuning

The intuition

Think of a generalist contractor who already knows how to build houses. Fine-tuning is teaching them your firm's house style — where the light switches go, how you write change orders, the tone of a client email. After enough supervised examples, they stop needing a five-page checklist every morning. RAG is handing them today's blueprints and the building code for this job — facts that change, must be citable, and should never be drilled into muscle memory.

You do not teach the municipal code by repetition. You also do not staple a style guide to every message if you can train the habit once. Production usually needs both: retrieval for living knowledge, fine-tuning (or distillation) when behavior must hold cheaply at high volume.

Why it exists

Prompting and RAG cover most applied work, but four constraints still open a fine-tuning-shaped hole.

Behavior that will not stick. Strict JSON, house tone, domain jargon, or multi-step rules drift across millions of calls — especially when context is full of retrieved docs. More prompting alone loses when the behavior must be automatic and short.

Cost and latency on a narrow high-volume route. A frontier model already solves triage, but paying frontier prices on every call is wasteful. Distilling into a small model is often roughly a 10× cost win and a 2–5× latency win on that route.

Facts vs weights. Teams try to "teach the model our docs" with gradients and still lack citations, ACLs, and cheap refreshes. That failure belongs to RAG, not weights.

Full fine-tunes are operationally obsolete for most teams. Updating all N billion weights needs optimizer state roughly 2–4× model size and a full copy per task. LoRA and QLoRA keep adapters small, swappable, and trainable on one GPU.

Training without baselines also loses: "it feels better" is not a ship gate. Fine-tuning is a deliberate stage, not the default first move when an answer disappoints.

The core idea

Fine-tuning adjusts weights so the model reliably produces a style, format, or skill. It is a poor knowledge store: limited capacity, no attribution, stale the day training ends. RAG injects facts from an updatable index; it does not change how the model behaves. They compose — often a fine-tuned small model on top of retrieval.

Climb only as far as the failure requires: prompt → RAG → fine-tune → distill. Fine-tuning wins on format and tone at scale, narrow repeated tasks, domain vocabulary, and distillation for cost and latency. It loses on fresh facts, auditability, tiny datasets (under a few hundred good examples), and broad open-ended work where base generality is the point.

Almost nobody full-fine-tunes for applied task adaptation. LoRA trains tiny low-rank adapters on a frozen base; QLoRA does the same over a 4-bit base. The work is still about 80% data and evals, 20% training.

How it actually works

When to reach for what

A few thousand gradient steps can shift response distributions, but they do not store retrievable facts the way an index does. Deeper retrieval: the RAG pipeline. For long repeated prompts, try prompt caching before training; see also cost.

LoRA at survival depth

Full fine-tuning updates all N billion weights — multi-GPU memory and a full copy per task.

LoRA (Low-Rank Adaptation) freezes base weights W and learns ΔW = BA, where A and B are small matrices of rank r (typically 8–64) injected into attention and MLP layers. Task adaptations are intrinsically low-rank, so a tiny Δ W captures them. Libraries also expose lora_alpha: the update is scaled by α / r. Rank is capacity; alpha is strength. Common starts: α = r or 2r; then let holdout metrics decide.

Trainable parameters drop to roughly 0.1–1% of the model; the artifact is tens to hundreds of MB. Merge BA into W for zero inference overhead, or keep adapters separate for hot-swap — vLLM serves multiple LoRAs on one base.

QLoRA quantizes the frozen base to 4-bit (NF4 — NormalFloat4), keeps adapters in higher precision, and backpropagates through the quantized weights. A 65–70B-class fine-tune fits a single ~48GB GPU with near full-precision PEFT quality. Default self-serve recipe: 4-bit base + rank ~16.

The family name is PEFT (parameter-efficient fine-tuning). Hugging Face peft is the library surface most stacks wrap (TRL, Axolotl, Unsloth).

The practical pipeline

Data prep dominates. About 500–2,000 curated pairs typically beat 50K scraped ones. Prefer production traffic, teacher outputs, and hand-written gold. Deduplicate; hold out a test split before training.

Baseline first. Score base and prompted frontier on that holdout. Without it, "the fine-tune helped" is vibes (evals and testing).

Train. QLoRA via peft/TRL, Unsloth, Axolotl, or a hosted fine-tuning API when you trade control for zero training infra. Hosted surfaces change — as of 2026 some major APIs are winding fine-tuning platforms down. Few epochs; if train loss collapses while holdout stalls, you are memorizing.

Eval after on the same set. Target metrics up, off-task not collapsed. Catastrophic forgetting is degraded general behavior after a task-specific train — only measurement proves it. Version the adapter with its dataset hash; retrain when base or data changes.

Distillation

Distillation means generating pairs with a large teacher and fine-tuning a small student on them. Strongest commercial case: roughly 10× cheaper and 2–5× faster on a narrow route the frontier already handles. Check provider terms — some ban training on outputs.

The flows

A worked example

On the governed enterprise platform, two failures appear together.

Failure A — knowledge. Policy thresholds change quarterly. Without retrieval the model invents plausible numbers. Fix: RAG. Upsert new PDF chunks; old numbers leave the index; citations appear for auditors.

Failure B — behavior at volume. Ticket triage must emit {priority, queue, pii_flags[], summary} with house rules. Frontier does this with a long prompt, but volume is high (illustrative: tens of thousands of tickets/day). A small prompted model drifts on edges.

Distillation + LoRA path (illustrative numbers):

1. Teacher + rubric on 2,000 tickets → ~96% schema-valid, ~91% priority agreement with humans.
2. Generate pairs on 3,000 sanitized tickets; drop bad JSON; hold out 400 before training.
3. Untuned 8B + long prompt on the 400 → ~70% schema-valid, ~62% priority.
4. QLoRA rank ~16, few epochs; stop before loss → 0. Often under an hour on one GPU-class machine.
5. Same 400 + off-task suite: aim ≥90% schema-valid, ≥85% priority, off-task not collapsed.
6. Serve merged for one route, or multi-LoRA via vLLM. Gateway triage → student; policy Q&A stays RAG + workhorse.

If frontier triage is roughly $0.01–0.03/ticket and the student is ~10× cheaper, at 20k tickets/day route savings dominate training (often tens of dollars of GPU time). Ongoing cost is data, evals, and retrain. If triage also attaches retrieved notes, eval the composed system — a terse-JSON fine-tune can worsen long-context RAG.

Production concerns

You own a model lifecycle. A fine-tune pins a base snapshot; when it is deprecated, re-run a reproducible pipeline (data, config, eval set), not a notebook (choosing models). Every retrain widens the regression surface — gate on evals and canary live traffic (evals, reliability).

Serving is a product choice. Merged adapters: zero overhead, one artifact. Multi-LoRA (vLLM): one base in memory, adapter per request — right for many tenants or tasks (open source & self-hosting).

Cost is inverted. Training is often tens of dollars with QLoRA; recurring cost is curation, evals, and retrains (cost). Behavior freezes while the domain moves — schedule refresh + retrain (observability). Always eval the composed system: structured-output fine-tunes can harm long RAG context.

Common drill-downs

Rank vs alpha — which first? Rank sets capacity; alpha scales the update (α / r). Barely moving → raise alpha or train longer before jumping rank. Overfit or off-task drop → lower rank/alpha, fewer epochs, more diverse data. Start r ∈ {8,16,32}, α ≈ r or 2r.

Merge or multi-LoRA? Merge for one stable route that owns the GPU. Multi-LoRA when many tones or tenants share a base. Merging early means multiple full models or reload thrashing the moment a second adapter appears.

When is full fine-tuning justified? Rarely. Only if PEFT plateaus on a skill that needs high-rank updates across most layers and you accept a full checkpoint per task. Try higher rank and more target modules first.

How do you catch catastrophic forgetting cheaply? Keep a fixed off-task pack (general instructions, unrelated tasks, shared routes). Run before and after every train. A large drop blocks ship even if the target task looks perfect.

Hosted API vs self-serve LoRA? Hosted: zero GPU ops, less control, provider deprecation calendar. Self-serve open weights + PEFT: portable adapters and multi-LoRA, you own ops. The decision is lock-in, not science.

Test yourself

Go deeper

• LoRA (arXiv 2106.09685) — low-rank updates and merge-at-inference.
• QLoRA (arXiv 2305.14314) — NF4, double quantization, paged optimizers; 65B on one 48GB GPU.
• Hugging Face PEFT — standard LoRA/QLoRA library surface.
• vLLM LoRA adapters — multi-adapter serving and per-request selection.
• Unsloth docs — practical local QLoRA (claims ~2× faster / ~70% less VRAM; re-measure on your hardware).
• LoRA & QLoRA Explained — Entry Point AI — rank/alpha intuition without the papers first.

Where this connects

• Choosing models — distillation targets a cheaper rung; route the student only where evals clear.
• Open source & self-hosting — adapters need a serving stack; multi-LoRA on vLLM is the multi-task pattern.
• The RAG pipeline — knowledge stays in the index; fine-tunes supply behavior on top.
• Evals and testing — baseline → post-train → canary is the only honest proof.