Cost

The intuition

Running an LLM feature is less like paying a flat SaaS subscription and more like running a metered taxi: every mile (token) has a price, the return trip (output) costs more than the outbound (input), and a loop that keeps the meter running without a destination will bankrupt the trip.

You do not get a single monthly line item that magically stays stable. You get input_tokens × input_price + output_tokens × output_price on every call, summed across retrieval helpers, routers, judges, and agent steps. A "small" prompt change that makes answers twice as long is not a style tweak — it is a cost regression on the expensive side of the bill.

The engineering job is to see cost per successful task, not the aggregate invoice, and to stack structural discounts (cache, tier, batch, trim) the way a taxi trip stacks shorter routes and off-peak rates.

Why it exists

LLM products have a real marginal cost per use. That creates hard constraints classical free-tier backends do not:

1. Every token is billed, including the ones you re-send for free. System prompts, tool schemas, and full chat history ride along on every turn. Agents re-send a growing transcript each step.
2. Output is the expensive class. Pricing mirrors physics: prefill is one parallel pass; decode is one forward pass per token. Verbose prompts and uncapped max_tokens hit the expensive side first.
3. Traffic shape decides which lever wins. RAG and agents are often input-heavy (large context, short answer). Drafting and code products are output-heavy. One playbook does not fit both.
4. Aggregate spend without a denominator is meaningless. $30k/month can be excellent or disastrous depending on successful tasks, users, and value per task.

The alternatives lose. Ignoring cost until the invoice doubles produces emergency rewrites. Always using the frontier model leaves 5–25× on the table for easy traffic. Skipping batch for offline work pays full price for work that could wait hours. Stuffing a 1M-token window "because it fits" buys latency, "lost in the middle," and a linear bill — not free knowledge.

Cost engineering exists so unit economics are visible per feature and per tenant, and so structural levers ship with quality checks — not so someone argues about the monthly total in finance after the fact.

The core idea

LLM cost is input_tokens × input_price + output_tokens × output_price, summed over every call in the pipeline — and the two prices are asymmetric: output tokens cost roughly 4–5x input tokens on every major provider (Claude's lineup is exactly 5x: Sonnet-class $3/$15 per million, Haiku 4.5 $1/$5, Opus-class $5/$25, mid-2026). The asymmetry reflects physics: input is one parallel prefill pass; output is generated serially, one forward pass per token.

That means the cost profile depends on workload shape. RAG and agents are input-heavy — a 20k-token stuffed context producing a 300-token answer is dominated by input, so the levers are prompt caching, context trimming, and retrieval precision. Generation products (drafting, code) are output-heavy, so the levers are concise-output instructions and max_tokens caps.

The four structural levers, in the order to apply them: prompt caching (repeated prefix at ~10% of input price on Anthropic, automatic ≥1024 tokens on OpenAI), model tiering (route easy traffic to a model 5–25x cheaper), batch APIs (50% off both input and output for anything asynchronous), and context discipline (send fewer tokens: fewer/better chunks, summarized history, trimmed tool definitions). Underneath it all sits unit economics: track cost per request, per user, per feature — an LLM feature has a real marginal cost per use, so "cost per successful task vs value per task" decides whether the feature survives, not the monthly API bill in aggregate.

How it actually works

Where tokens hide. Billed input includes the system prompt, tool/JSON schemas (tool definitions are re-sent every call), the entire conversation history (chat is quadratic-ish: every turn re-sends all prior turns), retrieved chunks, and tool results. Agents multiply this: a 10-step loop re-sends the growing transcript 10 times. Providers return exact usage per response (usage.input_tokens / output_tokens, plus cache read/write counts) — log them; never estimate what the API reports. Connects to tokens and context windows and agent foundations.

Prompt caching economics (Anthropic, explicit cache_control): cache write = 1.25x input price (5-minute TTL) or 2x (1-hour TTL); cache read = 0.1x. Break-even after a single read on the 5-minute cache, two reads for 1-hour. Reads reset the TTL, so steady traffic pays the write cost once. Worked example: 50k tokens of system prompt + tools + policy docs at $3/MTok = $0.15 per request uncached; at 90% cache-hit it's ~$0.017 — a ~9x cut on that component. Bonus: cached reads don't count toward Anthropic ITPM rate limits, so caching also raises effective throughput. OpenAI: caching is automatic for prompts ≥1024 tokens sharing a stable prefix; cached input bills at a discounted rate with ~30-minute retention (newer models add explicit breakpoints and 1.25x write pricing). Both schemes require prefix stability: static content first, variable content (user query, timestamp) last — a timestamp at the top of the system prompt silently kills every cache hit.

Model tiering/routing. Price gaps between tiers are 5–25x, and frontier-model price-performance keeps falling — re-benchmark quarterly; last year's "only the big model can do this" tasks routinely fit a small model today. Route by task type (classification, extraction, routing itself → small model; synthesis, reasoning → large), or cascade: small model answers, escalate on low confidence or failed validation. Sub-LLM tasks (dedup, filtering, keyword match) shouldn't hit an LLM at all. See choosing models and latency.

Batch APIs. Both OpenAI and Anthropic run asynchronous batch endpoints at a flat 50% discount on input and output, with a 24-hour completion window (usually much faster) and separate, larger rate limits. Anything not user-facing belongs there: eval runs, backfills, document summarization/embedding-adjacent enrichment, nightly report generation, judge scoring of sampled traffic. Discounts stack: batch + prompt caching combine.

Context trimming/summarization. Retrieve fewer, better chunks (precision beats recall for cost); summarize conversation history beyond a sliding window of recent turns; strip boilerplate from documents at ingestion; prune unused tool definitions per request. Diminishing returns and real risk: over-aggressive trimming trades a cheap request for a wrong answer and a retry — evaluate quality alongside cost (evals and testing, production RAG).

Attribution. Tag every call with request ID, user/tenant, feature, model, and prompt version; emit cost = f(usage, price_table) as a metric. That enables cost per feature (kill or fix the unprofitable one), per tenant (pricing and abuse detection), and per prompt version (a prompt change that doubles output length shows up as a cost regression the same day, not on the invoice). Wire tags into observability.

The flows

A worked example

Consider the governed enterprise platform serving internal policy Q&A for roughly ~10–15k daily users. A typical grounded answer:

Pricing (illustrative Sonnet-class $3 / $15 per MTok):

At 10k such requests/day uncached: ~$440/day on this feature alone. With prompt caching + tiering on easy FAQs: often under ~$120/day for comparable traffic — before semantic cache and context trimming.

Stack one more lever: if 25% of questions are near-duplicates and semantic cache returns them without an LLM call, those requests cost milliseconds of embedding search only.

Common drill-downs

Your LLM bill doubled month-over-month with flat traffic. Diagnose. Check per-dimension cost metrics: cache hit rate collapse (prompt reordering), output-length drift from a prompt change, tier-mix shift (router sending more to the big model), agent iteration creep, retry storms, or a model/pricing change. With flat requests, cost per request rose — the dimensions above are the only places it can hide.

Why do output tokens cost more than input tokens? Input is processed in one parallel prefill pass; output is autoregressive — a full forward pass per token, memory-bandwidth-bound, consuming far more accelerator time per token. Pricing mirrors the compute asymmetry (commonly 4–5x).

When does prompt caching pay off, and when doesn't it? Pays off with a large stable prefix hit repeatedly within the TTL: system prompt + tools + shared documents, multi-turn chat, agent loops. Break-even is one read (5-min Anthropic cache). Doesn't pay: unique-per-request contexts, prefixes below the minimum cacheable length, traffic too sparse to hit the TTL, or prompts with dynamic content near the top breaking prefix identity.

How would you cut costs 10x without hurting quality? Measure first (per-feature attribution). Then stack: route the easy 70% to a small model (~5x on that traffic), prompt-cache the static prefix (~90% off cached input), move async work to batch (50%), trim context and cap outputs, semantic-cache repeated questions. Verify with evals at each step — the constraint is "without hurting quality," so every lever ships with a regression check.

What belongs on the Batch API? Anything tolerant of hours of delay: evals and judge scoring, backfills, enrichment, summarization pipelines, report generation, dataset labeling. 50% off input+output, separate rate limits, 24h window. Anti-pattern: putting latency-tolerant work on the synchronous API at full price because the code path already existed.

How do you think about unit economics for an LLM feature? Cost per successful task (including retries and failed attempts) vs value per task. Aggregate spend is meaningless without a denominator: $30k/month serving 10M requests at $0.003 each may be excellent; $30k for 20k power-user requests may demand routing, caching, or a price change. Per-user cost distributions also expose abuse and inform rate limits and pricing tiers.

Context window is 1M tokens — should you fill it? No. You pay per token regardless; prefill latency grows with input; retrieval quality ("lost in the middle") degrades with stuffing. Long context replaces engineering effort, not the bill — retrieve precisely and send what's relevant.

Production concerns

• The agent multiplier. Loops re-send growing context every iteration; a runaway agent is a cost incident. Enforce max iterations, per-request token budgets, and per-user/tenant spend caps. This is OWASP LLM10 (Unbounded Consumption) as an economics problem. See agent reliability and security.
• Cache-hit-rate regressions. A deploy that reorders the prompt or injects a dynamic value up top drops the hit rate and can multiply input spend overnight. Alert on cache hit rate and cost-per-request, not just totals (observability).
• Silent output-length drift. Prompt tweaks ("be thorough") lengthen outputs; output is the expensive token class. Track mean output tokens per request per prompt version.
• Retries and fallbacks double-spend. A timeout that retries after 60s of decode pays for both attempts. Prefer streaming with idle-timeouts and cap retries (reliability, latency).
• Pricing is a moving target. Prices, tiers, and even tokenizers change (newer Claude tokenizers emit ~30% more tokens for the same text — a per-token price cut can be partially offset by token inflation). Keep the price table in config, dated, and re-verified — never hardcoded in five places.
• Forecasting: model cost as (requests/day) × (avg tokens) × price, then stress the assumptions: cache hit rate, tier mix, agent iteration count. The variance is in those three, not in the price table.

Test yourself

Go deeper

• Pricing — Anthropic docs — full price table with cache multipliers, batch rates, tool-use token overheads, and worked examples.
• Prompt caching — Anthropic docs — TTLs, breakpoints, minimum cacheable lengths, best practices.
• Prompt caching — OpenAI docs — automatic ≥1024-token caching, retention, cached-input pricing.
• Batch API — OpenAI docs — JSONL format, 24-hour window, 50% discount mechanics.

Where this connects

• Latency — caching, routing, and shorter outputs improve both spend and wait time; measure them together.
• Reliability — retries and fallbacks are correctness tools that double-spend if unbounded.
• Observability — cost metrics, cache hit rates, and per-version attribution live in the tracing layer.
• Evals & testing — every cost lever needs a quality gate so cheaper does not mean worse.