Choosing Models

The intuition

Picking a model is less like buying the "best" laptop and more like staffing a help desk. You would not put your most expensive senior on every password-reset ticket — and you would not leave a junior alone on a multi-system incident. You staff by difficulty of the work, not by prestige of the title.

Frontier models clear the hardest tasks but cost more and are wasted on easy traffic. Mid-tier and cheap/fast models handle the bulk once the task is in bounds. The durable skill is building a small eval set, proving the task is solvable, walking down the price ladder until quality drops below your bar, and routing each request to the cheapest model that still clears it.

Why it exists

Without a framework, teams settle into bad equilibria. Always frontier burns a multi-fold premium on easy traffic. Always cheapest fails fluently on multi-step tool use and hard extraction. Chase the leaderboard ships on a public score that does not predict your schema adherence, RAG faithfulness, or tool-loop reliability.

Four constraints force a real process. Capability is task-specific. Price and latency are multi-dimensional — input vs output rates diverge, reasoning modes bill thinking tokens (internal chain-of-thought) at output rates, long-context surcharges kick in above ~200K on some providers, and cache or batch can change the bill by roughly 10× or 2×. The landscape rots weekly, so the process must survive: eval → walk down → route → re-run on version bumps. Hard constraints (residency, retention, quotas, SLA) can veto a model that wins every benchmark. Choosing models exists so cost, latency, and quality are deliberate per route.

The core idea

Evaluate along six axes: capability (on your eval set), cost (input vs output per million tokens), latency (TTFT and tokens/sec; reasoning adds seconds-to-minutes before visible output), context length, modality (vision, audio, PDF), and constraints (residency, retention, quotas, SLA).

The sequence is fixed: prove solvability on a strong model → build a small eval set from real traffic → walk down the price ladder until quality drops below the bar. Most production traffic lands on mid-tier models because the frontier premium only pays for the hardest slice — typically the top 10–20%. That is why model routing exists: send each request to the cheapest tier that still clears quality. Two facts change implementation: public benchmarks mislead (contamination, task mismatch), and reasoning / extended thinking is a per-request dial — enable high effort only where multi-step correctness moves your metrics.

How it actually works

Every provider lineup is a three-tier ladder — frontier, workhorse, cheap/fast — plus optional open-weight APIs that undercut proprietary tiers. Names and dollars change; the ladder and the process do not.

The landscape as of August 2026

Prices are illustrative, per million tokens (input/output), from provider docs around August 2026. Re-check before any budget decision; this table ages in weeks.

Mechanics beyond sticker price. Output usually costs several times input (cost). Cached input is ~10× cheaper; batch APIs roughly half off (LLM APIs). Long-context is not flat-priced: Gemini 3.1 Pro doubles above 200K input; OpenAI flagships use higher long-context rates (older GPT-5.x still documents a ~272K threshold). Rankings shift every release — defend the framework, not a snapshot.

Reasoning / extended thinking

The model generates a (usually hidden or summarized) chain of thought, billed at output-token rates, then emits the visible answer. Cost is dominated by tokens you never show; latency grows from hundreds of ms to seconds or minutes; quality improves on multi-step work (math, debugging, planning, agent loops) — not lookups or classification.

APIs expose this as a dial: Anthropic adaptive thinking + effort, OpenAI reasoning_effort (none through max), Gemini thinking budgets. Worth it for agentic coding, complex extraction, plan-then-execute. Not for classification, simple RAG, or latency-sensitive chat. A/B on your eval set, per route (latency).

Why leaderboards mislead

Public scores shortlist; they do not ship. Contamination (benchmark items leak into training data), task mismatch (MMLU does not predict your JSON extraction or agent loops), and Arena rankings (pleasing single-turn chat, not domain correctness) all mislead. Instead: 50–200 examples from real traffic, graded rubrics, re-run on every candidate and version bump (evals and testing).

Model routing in practice

Three patterns cover almost all systems. Static routing maps request type to model ID — zero added latency. Dynamic routing scores difficulty with a small classifier (or cheap LLM call) and escalates the hard tail (~100ms). Cascade / fallback tries cheap first, validates (schema, judge, confidence), and retries stronger only on failure — only when failure is cheap to detect. Fluent wrong answers need better retrieval or prompts, not infinite escalation. FrugalGPT-style cascades formalize this: cheaper first, stop when quality is enough.

Keep the provider abstraction thin so a routing-table change is not a code change. Caches are per-model: bouncing between models never warms a shared cache. Gateway: design an LLM gateway. Self-hosted tiers: open source & self-hosting.

The flows

A worked example

Consider the governed enterprise platform: policy Q&A, multi-step tool use, and agentic writes with human approval. Scale: ~10–15k daily users, mixed difficulty.

Prove solvability. Run ~80 real tickets on a strong model with full tools. Suppose 92% pass a graded rubric (correct citation, correct action proposal, no policy invention). Solvable; now optimize.

Build the eval set. Hold out ~120 examples: 70 FAQ, 30 multi-doc synthesis, 20 tool-use / write-path. Exact match on structured fields; LLM-as-judge on grounded prose. Never reuse this split for prompt tuning.

Walk down (illustrative mid-2026 price classes — re-check before budgeting):

Router. Static map by intent (FAQ → cheap; "needs tools" → frontier). Optional cascade: if cheap fails schema or citation check, retry once on workhorse. Log route, model_id, tokens, cascade_hit.

Bill shape. At 10k requests/day, all-workhorse (~$3/$15 class, ~8k in + 400 out) is roughly ~$0.03/request → ~$300/day. Routing ~70% to cheap and thinking only on ~10% often cuts blended cost well below half — if evals stay above bar. Luna-class is far cheaper than Haiku-class; the win shape is routing + selective thinking.

Production concerns

Deprecations are your problem. Providers retire models on roughly 6–18 month cycles. Pin exact model IDs, subscribe to notices, keep the eval set ready so migration is a re-run — not a research project.

Version drift is silent. Same-name updates shift verbosity, tool-call rate, refusals. Re-run evals on every change; never point prod at an auto-updating alias without a canary (evals and testing).

Measure cost or it does not move. Log input, output, cached, and (when exposed) thinking tokens per request and route. The common bug is a bloated prompt found only on the invoice (cost, observability). Quotas are per-model, so cheaper routes are also different rate-limit pools — useful overflow during spikes (reliability). Multi-provider fallback is not an SDK swap: keep a second provider eval-verified and budget prompt re-tuning. Residency can veto capability regardless of eval score (security, open source & self-hosting).

Common drill-downs

How would you choose a model for a new feature? Prove feasibility, build an eval set from real examples, walk down until quality drops. Choose per-route, not per-app; re-evaluate when the landscape shifts.

When is extended thinking worth it? Multi-step, correctness-sensitive work only. Never for classification or simple RAG — thinking tokens bill at output rates.

How do you cut cost without hurting quality? Route easy traffic cheaper first, then cache stable prefixes, trim prompts, batch async work, then distill a narrow high-volume route (fine-tuning).

What breaks on a model swap? Tone, tool-call eagerness, JSON quirks, refusals, token counts (different tokenizers), cache invalidation. Treat it as a deploy: eval + canary.

What does "1M context" buy you? Ability, not advisability. Mid-window recall degrades; cost and prefill latency grow; some providers surcharge past ~200K. RAG over a curated subset usually wins; reserve long context for holistic tasks.

Test yourself

Go deeper

• Anthropic — Models overview — Claude tiers, adaptive thinking.
• OpenAI — Models — Sol / Terra / Luna and reasoning-effort.
• OpenAI — Pricing — input/output, cache, batch, long-context.
• Google — Gemini API models — stable/preview lineup.
• Google — Gemini API pricing — thinking tokens and long-context tiers.
• Artificial Analysis — Models — independent intelligence, price, speed comparison.
• Arena (formerly LMArena) — pairwise rankings; shortlist only.
• FrugalGPT (Chen, Zaharia, Zou) — cascade: cheaper first, stop when good enough.
• [1hr Talk] Intro to Large Language Models — Andrej Karpathy — mental model for these systems.

Where this connects

• Open source & self-hosting — when the "model" is a GPU you operate; the same routing table can span API and self-hosted tiers.
• Fine-tuning — after routing and caching, distillation is the next cost/latency lever for narrow high-volume routes.
• Cost — tiering, caching, and batch make model choice show up on the bill.
• Design an LLM gateway — multi-model access, quotas, audit, and routing as a platform concern.