Cost
Token accounting, caching, model tiering, truncation strategies, batch processing.
Prerequisites
- Tokens & Context Windows — what gets billed, how tokenizers differ, and why output is expensive.
- LLM APIs — usage fields, batch endpoints, and cache-related request shape.
- Latency — most cost levers also change TTFT and total time; the two are coupled.
- Agent foundations — agent loops re-send growing context and multiply spend.
The intuition
Running an LLM feature is less like a flat SaaS fee and more like a metered taxi. Every mile (token) has a price. The return trip (output) costs more than the outbound (input). A loop that keeps the meter running without a destination will bankrupt the trip.
You do not get one stable monthly line item. You get input_tokens × input_price + output_tokens × output_price on every call, summed across helpers, routers, judges, and agent steps. A prompt change that doubles answer length is a cost regression on the expensive side of the bill. The job is to track cost per successful task, not the aggregate invoice, and stack structural discounts the way a taxi stacks shorter routes and off-peak rates.
Key insight
Output tokens cost roughly 4–5× input tokens because they are generated serially. Workloads that look cheap on input still explode if answers or agent transcripts grow without a cap. Separate input-heavy from output-heavy shapes before you pick levers.
Why it exists
LLM products have a real marginal cost per use. Every token is billed — including system prompts, tool schemas, and history re-sent every turn. Agents re-send a growing transcript each step. Output is the expensive class because pricing mirrors physics: prefill is one parallel pass; decode is one forward pass per token. Traffic shape decides which lever wins: RAG and agents are often input-heavy; drafting and code are often output-heavy. Aggregate spend without a denominator is meaningless — $30k/month can be excellent or disastrous depending on successful tasks and value per task.
Ignoring cost until the invoice doubles produces emergency rewrites. Always using the frontier model leaves large multiples on the table for easy traffic. Skipping batch for offline work pays full price for jobs that could wait hours. Stuffing a long context window "because it fits" buys latency, lost-in-the-middle quality loss, and a linear bill. Cost engineering exists so unit economics are visible per feature and tenant, and so levers ship with quality checks.
The core idea
Bill = Σ (input_tokens × input_price + output_tokens × output_price) over every call in the pipeline. Prices are asymmetric: output is commonly about 4–5× input (Claude's Sonnet-class line is exactly 5× at the standard $3 / $15 per million tokens; Haiku-class is $1 / $5 — treat dollar figures as dated, the ratio is the mechanism). That is why workload shape picks the lever. Input-heavy paths win on prompt caching, context trimming, and retrieval precision. Output-heavy paths win on concise instructions and max_tokens caps.
Four structural levers, in order: prompt caching (repeated exact prefixes at a steep input discount), model tiering (route easy traffic cheaper), batch APIs (~50% off for anything asynchronous), and context discipline (send fewer tokens). Underneath sits unit economics: cost per successful task vs value per task decides whether the feature survives.
How it actually works
Where tokens hide. Billed input includes the system prompt, tool schemas (re-sent every call), conversation history, retrieved chunks, and tool results. Agents multiply this: a 10-step loop re-sends the growing transcript ten times. Providers return exact usage — input, output, cache read/write. Log those fields; never estimate what the API reports. See tokens and context and agent foundations.
Prompt caching reuses the provider's computed KV state for an exact prompt prefix. Both major providers require prefix stability: static content first (tools, system, shared docs), variable content last (user query, timestamps). One dynamic field at the top silently kills every cache hit.
Anthropic bills cache writes at 1.25× base input (5-minute TTL) or 2× (1-hour TTL), and reads at 0.1× — break-even after one subsequent read on the 5-minute cache. Reads refresh the TTL. Enable with top-level automatic cache_control or up to four explicit breakpoints; minimum length is model-dependent (often ~1k–4k tokens). Shape: 50k tokens at $3/MTok is $0.15 uncached; at ~90% cache-hit that component is roughly an order of magnitude cheaper. Cache hits also spare Anthropic's input-token rate-limit budget.
OpenAI caches automatically for prompts ≥ ~1024 tokens sharing a stable prefix. Cached input is discounted (commonly half on older families, deeper on newer ones — check the live table). Older families charged no write premium; newer ones add 1.25× write pricing and optional explicit breakpoints. Retention ranges from tens of minutes to longer extended policies.
Key insight
Prefix order is a cost control. Static system prompt, tools, and shared documents first; user query and timestamps last. One misplaced dynamic field at the top can zero the cache hit rate overnight.
Model tiering. Price gaps between tiers are often 5–25×. Re-benchmark quarterly. Route by task type (classification, extraction, routing → small; synthesis and hard reasoning → large), or cascade: small first, escalate on low confidence or failed validation. Sub-LLM work (dedup, filtering, keyword match) should not hit an LLM. See choosing models and latency.
Batch APIs. Both major providers offer async batch at ~50% off input and output, with a 24-hour window (often faster) and separate, larger rate limits. Use for evals, backfills, enrichment, nightly reports, and judge scoring. Batch and prompt caching stack.
Context trimming. Retrieve fewer, better chunks; summarize history beyond a recent-turn window; strip boilerplate at ingestion; prune unused tools per request. Over-trimming trades a cheap request for a wrong answer and a retry — gate with evals and production RAG.
Attribution. Tag every call with request ID, user/tenant, feature, model, and prompt version; emit cost = f(usage, price_table). That unlocks cost per feature, tenant, and prompt version — wire tags into observability.
Common misconception
"We have a 1M-token context window, so we should fill it" is not a cost strategy. You pay per token regardless; prefill latency grows with input; retrieval quality degrades with stuffing. Long context replaces some engineering effort — not the bill.
The flows
| Flow | Sequence | When it applies | What breaks it |
|---|---|---|---|
| Sync user-facing call | Tag → static prefix first → optional cache → generate → log usage/cost | Interactive chat, RAG, agents | Dynamic content at top; missing attribution tags |
| Prompt-cache warm path | Write prefix once → reads at ~0.1× → TTL refreshed by traffic | Large stable system/tools/docs | Prompt reorder on deploy; traffic too sparse for TTL |
| Model tier / cascade | Try small model → escalate on failure/low confidence → large only when needed | Mixed difficulty traffic | Router bias toward frontier; no quality gate on the cheap path |
| Batch offline path | Enqueue JSONL → 50% price → results within 24h | Evals, judges, backfills, enrichment | Latency-tolerant work left on the sync API out of habit |
| Agent loop cost path | Each step re-sends transcript + tools → multiply by iterations | Multi-step agents | No max iterations / token budget; runaway tool loops (agent reliability) |
| Attribution & alert path | usage → price table → cost by feature/tenant/version → alert | All production traffic | Hardcoded prices; no per-request cost metric |
A worked example
Consider the governed enterprise platform serving internal policy Q&A for roughly 10–15k daily users. A typical grounded answer (token counts illustrative):
| Component | Tokens | Notes |
|---|---|---|
| System prompt + safety | 1 200 | Stable across requests |
| Tool / schema definitions | 2 800 | Stable; good cache prefix |
| Shared policy excerpts always injected | 6 000 | Stable within a release |
| Retrieved chunks (top 5) | 2 500 | Variable |
| User question + history tail | 400 | Variable |
| Input total | ~12 900 | ~10k cacheable if ordered correctly |
| Output answer | 350 | Mid-length cited answer |
Pricing (illustrative Sonnet-class $3 / $15 per MTok, mid-2026): fully uncached ≈ $0.044/request; with 90% of the 10k static prefix cache-hit at 0.1× ≈ $0.014; with 70% of traffic also on Haiku-class $1/$5, blended ≈ $0.008–0.012; nightly judge scoring on batch is half of that judge path.
At 10k such requests/day uncached: roughly $440/day on this feature alone. With prompt caching plus tiering on easy FAQs: often under ~$120/day — before semantic cache and context trimming. If 25% of questions are near-duplicates and semantic cache returns them without an LLM call, those requests cost only embedding search.
What each omitted stage looks like in production
- No usage logging → the invoice is the first metric; you cannot tell which feature or tenant spiked.
- No prompt caching / unstable prefix → every request pays full price for the same 10k-token policy block; TTFT stays high (latency).
- Everything on the frontier model → easy classification and FAQ traffic pay 5–25× needlessly.
- Evals and judges on the sync API → offline work competes for rate limits and pays 2× batch price.
- No agent iteration / spend caps → a stuck loop is a cost incident (OWASP LLM10, Unbounded Consumption), not just a quality bug.
- Prompt tweak "be thorough" without tracking mean output tokens → silent length drift on the expensive token class.
Production concerns
The agent multiplier. Loops re-send growing context every iteration; cost is superlinear in step count. Enforce max iterations, per-request token budgets, and per-user/tenant spend caps. This is OWASP LLM10 (Unbounded Consumption) as an economics problem — see agent reliability and security.
Cache-hit-rate regressions and output drift. A deploy that reorders the prompt or injects a dynamic value up top can multiply input spend overnight. Prompt tweaks ("be thorough") lengthen outputs on the expensive class. Alert on cache hit rate, mean output tokens per prompt version, and cost-per-request — not just totals (observability).
Retries double-spend. A timeout that retries after 60s of decode pays for both attempts. Prefer streaming with idle timeouts and hard retry caps (reliability, latency).
Pricing moves. Prices, tiers, and tokenizers change — newer Claude tokenizers can emit roughly ~30% more tokens for the same text, so a per-token price cut can be partially offset by token inflation. Keep the price table in config, dated, re-verified. Forecast as (requests/day) × (avg tokens) × price, then stress cache hit rate, tier mix, and agent iteration count — the variance is in those three.
Common drill-downs
When does prompt caching pay off? When a large stable prefix is hit repeatedly within the TTL: system + tools + shared documents, multi-turn chat, agent loops. Anthropic's 5-minute break-even is roughly one read after the write. It fails for unique-per-request contexts, prefixes below the minimum length, sparse traffic that never re-hits the TTL, or dynamic content near the top.
Prompt cache vs semantic cache — which saves what? Prompt caching is provider-side KV reuse of an identical prefix: the model still generates; you save prefill cost and TTFT. Semantic caching is app-side reuse of a full response for a similar query: zero LLM call, but wrong-answer and staleness risk. Use prompt cache for shared system/tools/docs; semantic cache for high-repeat FAQ heads with a tight similarity threshold and TTLs.
What belongs on the Batch API? Anything tolerant of hours of delay: evals, judges, backfills, enrichment, reports, dataset labeling. Anti-pattern: leaving that work on the sync API at full price because the code path already existed.
How do you think about unit economics for an LLM feature? Cost per successful task (including retries and failed attempts) versus value per task. Aggregate spend is meaningless without a denominator. Per-user distributions expose abuse and inform rate limits and pricing tiers.
Test yourself
Traffic is flat but cost per request rose 40% in a week. What three metrics do you check first, and why?
Why is break-even for Anthropic's 5-minute prompt cache often a single subsequent read?
An agent averages 8 tool steps. You cut average steps to 5 without changing the model. Roughly what happens to cost, and what else must you watch?
Finance wants a large cost cut and product refuses any quality drop. How do you structure the work?
You moved judge scoring to the Batch API and the finance dashboard still shows high spend on 'evals'. What might be wrong?
Go deeper
- Pricing — Anthropic docs — full price table with cache multipliers, batch rates, tool-use overheads, and worked examples.
- Prompt caching — Anthropic docs — automatic and explicit caching, TTLs, breakpoints, minimum lengths, best practices.
- Prompt caching — OpenAI docs — automatic ≥1024-token caching, retention, write pricing on newer families, explicit breakpoints.
- Batch API — OpenAI docs — JSONL format, 24-hour window, 50% discount mechanics.
- OWASP LLM10: Unbounded Consumption — denial-of-wallet and uncontrolled inference as a security class.
Where this connects
- Latency — caching, routing, and shorter outputs improve both spend and wait time; measure them together.
- Reliability — retries and fallbacks are correctness tools that double-spend if unbounded.
- Observability — cost metrics, cache hit rates, and per-version attribution live in the tracing layer.
- Evals & testing — every cost lever needs a quality gate so cheaper does not mean worse.