Choosing Models

The intuition

Picking a model is less like buying the "best" laptop and more like staffing a help desk. You would not put your most expensive senior engineer on every password-reset ticket — and you would not put a junior alone on a multi-system incident that needs judgment under ambiguity. You staff by difficulty of the work, not by prestige of the title.

Frontier models are the seniors: they clear the hardest tasks, but they cost more, take longer, and are wasted on easy traffic. Mid-tier and cheap/fast models are the rest of the desk: they handle the bulk of volume once you know the task is in bounds. The durable skill is not memorizing this month's product names — it is building a small eval set, proving the task is solvable, then walking down the price ladder until quality drops below your bar, and routing each request to the cheapest model that still clears it.

Why it exists

Without a decision framework, teams default to one of three bad equilibria:

1. Always use the frontier model. Quality feels safe, but easy traffic pays a 2–10× premium and often worse latency. Unit economics break as volume grows.
2. Always use the cheapest model. Classification and FAQ look fine until multi-step tool use, hard extraction, or agent loops quietly fail — and failures look fluent.
3. Chase the leaderboard. Public scores measure memorization and chat style more than your JSON schema adherence, RAG faithfulness, or tool-loop reliability. You ship on a number that does not predict production.

The constraints that force a real process:

1. Capability is task-specific. A model that leads coding agents may not lead scientific reasoning or terse structured extraction. "Best model" without a task is meaningless.
2. Price and latency are multi-dimensional. Input vs output pricing diverges; reasoning modes bill thinking tokens at output rates; long-context surcharges kick in above ~200K on some providers; cache and batch change the bill by ~10× and ~2×.
3. The landscape rots weekly. Names, prices, and context windows in any static table age in weeks. What must survive is the process: eval set → walk down the ladder → route by difficulty → re-run on every version bump.
4. Constraints can veto capability. Data residency, retention policy, rate limits, and provider SLA can eliminate a model that wins every benchmark.

The alternatives lose. Picking once and forgetting fails at the next deprecation. A/B without a rubric optimizes for vibe. Routing without validation sends hard tickets to a model that cannot finish them. Choosing models exists as engineering so cost, latency, and quality are deliberate per route — not accidental side effects of whoever set the default string last quarter.

The core idea

Model choice is a requirements problem, not a leaderboard problem. Evaluate along six axes: capability (can it do the task at all — verified on your own eval set, not benchmarks), cost (input/output per million tokens, which diverge 100x between the cheapest and priciest mainstream models), latency (time-to-first-token and tokens/sec — reasoning models add seconds-to-minutes of thinking before output), context length (do you need 1M tokens, or is 200K plenty), modality (vision, audio, PDF ingestion), and constraints (data residency, retention policy, rate limits, provider SLA).

The practical process: start with a frontier model to prove the task is solvable, build a small eval set from real traffic, then walk down the price ladder until quality degrades below the bar. Most production traffic ends up on mid-tier models (Claude Sonnet, GPT mid-tier, Gemini Flash) because the frontier premium — 2–10x on price, plus latency — only pays for the hardest 10–20% of requests. That insight leads directly to routing: classify requests by difficulty and send each to the cheapest model that clears the quality bar.

Two more points that matter in practice: benchmarks mislead (contamination, task mismatch — trust your own evals), and reasoning/extended-thinking modes are a dial, not a model family — you pay thinking tokens at output rates and add latency, so you enable them per-request only where multi-step correctness matters.

How it actually works

The durable shape of every provider lineup is a three-tier ladder — frontier, workhorse, cheap/fast — plus optional open-weight APIs that undercut proprietary tiers on price. Names and dollars change; the ladder and the selection process do not.

The landscape as of July 2026

Each major provider ships a three-tier ladder: a frontier model, a balanced workhorse, and a cheap/fast model. Prices are per million tokens (input/output), from provider docs as of July 2026 — always re-check, this table ages in weeks:

Mechanics that matter beyond the sticker price:

• Output tokens cost 3–6x input tokens — a summarization task (big input, small output) and a generation task (small input, big output) have very different cost profiles on the same model. See cost.
• Cached input is ~10x cheaper (all three providers offer prompt/context caching); batch APIs give a flat ~50% discount for async workloads. Mechanics in LLM APIs.
• Long-context surcharges exist: Gemini 3.1 Pro doubles input price above 200K tokens; OpenAI applies a multiplier above ~272K. A "1M context window" is not flat-priced.
• Rough capability picture (July 2026, per cross-benchmark aggregators): Claude leads frontier coding/agentic work (Fable 5 ~95% SWE-bench Verified), Gemini leads scientific/abstract reasoning benchmarks, GPT leads computer use, DeepSeek leads price-performance. This shifts every release cycle — the framework is what you defend, not the ranking.

Reasoning / extended-thinking models

Reasoning models spend tokens "thinking" before answering. Mechanism: the model generates a (usually hidden or summarized) chain of thought, billed at output-token rates, then produces the visible answer. Three consequences: cost is dominated by thinking tokens you never see, latency grows from hundreds of ms to seconds or minutes, and quality improves specifically on multi-step tasks (math, debugging, planning, agentic tool loops) — not on lookups or classification.

Modern APIs expose this as a dial rather than a separate model: Anthropic's adaptive thinking with an effort parameter (low → max), OpenAI's reasoning-effort settings, Gemini's thinking budgets. Worth it when: agentic coding, complex extraction, plan-then-execute loops. Not worth it when: classification, simple RAG answers, latency-sensitive chat — there you pay 2–10x for no measurable gain. Decide with an A/B on your eval set, per route. Latency interaction: latency.

Why leaderboards mislead, and what to do instead

• Contamination: benchmark questions leak into training data (they're on the public internet), so scores measure memorization, not generalization. Vendors also tune for headline benchmarks.
• Task mismatch: MMLU-style multiple choice predicts nothing about your JSON-extraction accuracy or agent-loop reliability. SWE-bench predicts coding-agent skill, not summarization tone.
• Arena-style rankings (crowd pairwise votes) measure pleasing single-turn chat answers — style, length, formatting — not correctness on your domain.
• What to do: build a 50–200 example eval set from real production traffic, with graded rubrics (exact-match where possible, LLM-as-judge where not). Re-run it on every candidate model and every provider version bump. This is a few hours of work that replaces every leaderboard argument. Use leaderboards only for shortlisting candidates. Full practice: evals and testing.

Model routing in practice

• Static routing (most common): route by request type — classification → cheap model, agent loop → frontier model. Configured in code; zero added latency.
• Dynamic routing: a small classifier (or the cheap LLM itself) scores request difficulty, escalates hard ones. Adds one cheap call (~100ms) but captures long-tail difficulty.
• Cascade / fallback: try the cheap model, validate the output (schema check, judge, confidence heuristic), retry on a stronger model on failure. Works when validation is cheap and failure is detectable.
• Keep provider abstraction thin (a gateway or your own adapter layer) so a routing table change doesn't mean a code change. Note routing interacts with prompt caching: caches are per-model, so a request bouncing between models never gets warm-cache pricing. Gateway-scale design: design an LLM gateway. Self-hosted tiers in the same router: open source & self-hosting.

The flows

A worked example

Consider the governed enterprise platform: employees ask internal policy questions, some escalate into multi-step tool use (ticket create, access request), and a fraction need agentic actions with human approval on writes. Typical scale: ~10–15k daily users, mixed difficulty.

Step 1 — Prove solvability. Take 80 real tickets from last month. Run the frontier workhorse-or-above tier with full tools. Suppose 92% pass a graded rubric (correct citation, correct action proposal, no policy invention). The task is solvable; now optimize.

Step 2 — Build the eval set. Hold out 120 examples: 70 FAQ-style, 30 multi-doc synthesis, 20 tool-use / write-path. Rubrics: exact match on structured fields where possible; LLM-as-judge on grounded prose. Split never used for prompt tuning.

Step 3 — Walk down the ladder (illustrative mid-2026 workhorse vs cheap numbers from the table above — re-check before budgeting):

Step 4 — Router design. Static map by intent first (FAQ tag → cheap; "needs tools" → frontier). Optional cascade: if cheap path fails JSON schema or citation-presence check, retry once on workhorse. Log route, model_id, tokens, cascade_hit for offline eval.

Step 5 — Numbers that change the bill. At 10k requests/day, if all traffic sat on a $3/$15 workhorse with ~8k input + 400 output tokens: rough order ~$0.03/request → ~$300/day. After routing ~70% to a $1/$5 class model and enabling thinking only on ~10%: blended cost often drops well below half, with FAQ latency improved (no thinking wait) — provided evals stay above bar.

Common drill-downs

How would you choose a model for a new feature? Prove feasibility on a frontier model, build an eval set from real examples, then step down the price ladder until quality drops below the bar. Choose per-route, not per-app — and re-evaluate on a schedule because the landscape shifts monthly.

When is a reasoning/extended-thinking model worth it? When the task is multi-step and correctness-sensitive: agentic coding, planning, hard extraction. Thinking tokens bill at output rates and add seconds of latency, so an A/B on your evals must show a gain that justifies 2–10x cost. Never for classification or simple RAG.

Why not just pick the top model on the leaderboard? Contamination and task mismatch: public benchmarks leak into training data, and MMLU/arena scores don't predict your task. Leaderboards shortlist candidates; your own evals decide.

How do you cut LLM costs without hurting quality? In order of leverage: route easy traffic to cheaper models, cache stable prompt prefixes (~90% input discount), trim prompts (history windowing, tighter RAG top-k), batch async work (50% off), and only then consider fine-tuning a small model on the distilled task.

What breaks when you swap model A for model B? Tone/verbosity, tool-calling eagerness, JSON escaping quirks, refusal boundaries, token counts (tokenizers differ — same text, different bill), and prompt-cache invalidation. Treat a model swap as a deploy: eval run + canary.

Design a router for a support chatbot. Cheap model handles the ~80% FAQ tier; a lightweight intent/difficulty classifier escalates multi-step or high-stakes tickets to the workhorse tier; agentic actions (refunds, account changes) go to the frontier tier with tool use. Validate outputs and cascade upward on schema/judge failure; log route decisions for offline eval.

What does "1M context window" actually buy you? The ability, not the advisability: long-context recall degrades in the middle, input cost scales linearly (with surcharges past ~200K on some providers), and latency grows. RAG over a curated subset usually beats stuffing the window; use long context for genuinely holistic tasks (whole-repo reasoning, long transcripts).

Production concerns

• Model deprecations are your problem: providers retire models on ~6–18 month cycles (pinned snapshots → deprecation notice → 404). Pin exact model IDs, subscribe to deprecation notices, and keep the eval set ready so a forced migration is a re-run, not a research project.
• Version drift: even a same-name model update can shift behavior (verbosity, tool-call rate, refusals). Re-run evals on every model change; never point production at an auto-updating alias without a canary. Wire into evals and testing.
• Cost observability: log input/output/cached token counts per request and per route. The most common cost bug is a bloated prompt (unbounded history, duplicated context) discovered only on the invoice. Full treatment: cost, observability.
• Rate limits and quotas are per-model and per-tier. Routing to a cheaper model also moves you to a different quota pool — useful as an overflow valve during traffic spikes or provider incidents. See reliability.
• Multi-provider fallback: for uptime-critical paths, keep a second provider integrated and eval-verified. Prompts don't transfer perfectly — budget for prompt re-tuning, not just an SDK swap.
• Data residency/retention: some frontier models require minimum data-retention terms; regulated workloads may need region-pinned inference or zero-retention agreements — this can eliminate a model regardless of capability. Related: security, open source & self-hosting.

Test yourself

Go deeper

• Anthropic — Models overview — current Claude tiers, pricing, context/output limits.
• OpenAI — Models — current GPT tiers and pricing.
• Google — Gemini API models — current Gemini stable/preview lineup.
• Artificial Analysis — Models — independent cross-model comparison of intelligence, price, speed, latency, and context.
• Arena (formerly LMArena) — crowdsourced pairwise model rankings; useful for shortlisting, read with the caveats above.
• [1hr Talk] Intro to Large Language Models — Andrej Karpathy — the canonical mental model for what these systems are and where they're headed.

Where this connects

• Open source & self-hosting — when the "model" is a GPU you operate; the same routing table can span API tiers and self-hosted small models.
• Fine-tuning — after routing and caching, distillation into a small model is the next cost/latency lever for narrow high-volume routes.
• Cost — tiering, caching, and batch are the structural levers that make model choice show up on the bill.
• Design an LLM gateway — multi-model access, quotas, audit, and routing as a platform concern.