Agent Reliability

The intuition

A single coin flip that lands heads 95% of the time feels reliable. Flip it twenty times in a row and demand all heads: you get about 36%. Agent runs are that chain of flips. Each tool call, each intermediate conclusion, each retrieval is another flip that conditions the next.

Reliability engineering is therefore not "make the model perfect." It is raise per-step odds and — more importantly — break the multiplication: verification, recovery observations, checkpoints, and human gates so one bad step does not silently doom the rest.

Why it exists

Open loops fail in ways single-shot chat does not.

1. Multiplicative error. 0.95^10 ≈ 60%; 0.95^20 ≈ 36%. Long trajectories need recovery, not hope.
2. Unproductive loops. Models retry the same failing call, oscillate, or chase unreachable goals unless the harness enforces stop and progress checks.
3. Tool hallucination and unsafe intent. Invented names/args and schema-valid but unauthorized actions are routine without constrained decoding, validation, and permission tiers.
4. Silent corruption. Plausible wrong results (empty search as "confirmed absent") poison every later step more often than hard exceptions do.
5. Side effects in the real world. Shell, email, money, and delete tools turn a confused agent into an incident — sandboxing and human gates are mandatory for high blast radius.

Alternatives — shorter workflows, deterministic stages, fewer tools — remain the first reliability lever (agents vs workflows). When you do need a loop, this page is the harness catalog.

The core idea

Agent reliability is dominated by one piece of arithmetic: errors compound multiplicatively. A step that succeeds 95% of the time gives you 0.95^10 ≈ 60% over ten steps and 0.95^20 ≈ 36% over twenty. So reliability engineering for agents is two campaigns: push per-step reliability up (strict schemas, validated tools, good context) and — more importantly — break the multiplication with recovery mechanisms, verification steps, and checkpoints, so one bad step doesn't doom the run.

The failure catalog to design against, with the standard mitigation for each: unproductive loops — the agent repeats a failing action or oscillates without progress → hard iteration caps, token/cost budgets, and progress detection; tool hallucination — calling tools that don't exist or with invented arguments → strict schema enforcement (constrained decoding) plus runtime validation; error propagation — a wrong intermediate result silently becomes ground truth for every later step → verification steps and errors surfaced to the model as observations it can react to; and unsafe actions — the agent doing something irreversible or unauthorized → permission tiers, human-in-the-loop gates on irreversible operations, and sandboxed execution. The framing that holds in production: treat the model as an unreliable component in a system you make reliable — the same posture distributed systems take toward the network.

How it actually works

Loop containment. Layered stop conditions, all enforced by the harness (never trust the model to stop itself): max iterations (typical 10–50 by task class), token/cost budget per run, wall-clock timeout, and progress checks — hash recent (tool, args) pairs and trip after N identical calls; or a cheap "is this trajectory advancing?" classifier on the last few steps. On trip: fail gracefully — summarize what was accomplished and escalate, don't just kill the process. Root causes of loops are usually upstream: an unactionable error message the model can't recover from, a missing tool it keeps approximating, or a goal with no reachable success state.

Tool hallucination. Three layers: (1) prevention — strict/constrained decoding (strict: true on OpenAI and Claude) makes schema-invalid arguments impossible to emit; (2) validation — runtime checks for semantics schemas can't express (does this ID exist? is this path inside the sandbox? is this caller authorized?); (3) recovery — reject with an instructive error as the tool result ("Unknown tool search_db. Available: query_orders, query_users") so the model self-corrects; models typically retry 2–3 times with corrections. Unknown-tool-name calls are handled the same way: error-as-result, not a crash.

Error propagation. The insidious version isn't the thrown exception — it's the plausible wrong result (an empty search treated as "confirmed absent", a scraper returning a cookie banner treated as page content). Mitigations: tools return rich, honest signals ("0 results — index may be stale; try X" instead of []); verification nodes after critical steps (run the tests, check the claim, validate against schema); prefer verifiable feedback loops — compilers, test suites, type checkers — because they reset the error chain: a caught mistake becomes a recoverable observation instead of silent corruption. This is the deep reason coding agents work better than most: the environment refutes bad steps cheaply.

Human-in-the-loop gates. Classify every tool by blast radius, then gate by tier:

Mechanically the gate is an interrupt: the framework checkpoints state before the tool executes and resumes on approval (LangGraph interrupt() → Command(resume=...)), so approvals can take hours without holding compute. Two design rules: gate at the tool boundary in the harness (a prompt saying "ask before deleting" is a suggestion, not a control), and fight approval fatigue — over-gating trains humans to click yes, which is worse than a well-scoped allowlist.

Sandboxing. Anything that executes model-influenced code or commands runs in a container/VM jail: filesystem scoped to a workspace, network egress denied by default or allowlisted, resource limits (CPU/memory/time), least-privilege credentials — scoped, short-lived tokens, never root secrets in the agent's environment. Sandbox also mitigates prompt injection: tool results carrying external content (web pages, emails) can contain instructions that hijack the agent; treat tool output as untrusted data, and cap the blast radius so a hijacked agent still can't reach beyond its jail. Rule of thumb: an agent with untrusted input + secrets + egress is an exfiltration machine — remove one leg of that triangle.

State recovery. Checkpoint after every step (transcript or graph state) so crashes resume instead of restart; make mutating tools idempotent (idempotency keys) because the model will retry after ambiguous outcomes; wrap multi-step external mutations in compensation logic (sagas) where possible.

On the governed enterprise platform, agent mode is only shippable because write tools are tiered, MCP tools run with least privilege, approvals use durable interrupts, and every run has iteration/cost caps with full per-step traces.

The flows

A worked example

Agent on the governed enterprise platform tasked with:

"Find vendor contract V-2201 and archive the expired draft if a signed version exists."

Tools: search_contracts, get_document, archive_document (irreversible write), cap 15 iterations, cost budget illustrative $2/run.

Arithmetic sketch: suppose each of search/get/archive-decision is ~95% correct without verification. Three dependent steps → ~0.95³ ≈ 86%. Add one verification on document status at 99% detection of bad fetches → you break the silent path even if raw step accuracy is imperfect.

Loop scenario: archive API returns "error" with no detail; model retries identical call 20 times. Harness: after 3 identical (tool,args) hashes → trip breaker, return partial summary ("found signed+draft; archive failed with opaque error; escalated").

Common drill-downs

The failure-mode questions that sit under the arithmetic — diagnose, mitigate, measure.

Your agent loops forever retrying a failing tool. Diagnose and fix. Immediate: iteration cap + duplicate-call detection (same tool+args N times → trip breaker, fail gracefully with a summary). Root cause: usually an unactionable error message — return errors that say what to do differently ("retry after 60s", "param must be ISO date"); if the model retries identically anyway, the task may need a tool or permission it doesn't have — surface that instead of burning budget.

How do you stop tool hallucination? Prevention: strict schema enforcement via constrained decoding — invalid shapes become impossible. Validation: runtime authorization and semantic checks on every call regardless. Recovery: unknown tools or bad args come back as instructive error results the model can correct. Then reduce the cause: fewer, clearer, non-overlapping tools.

Why do errors compound in agents, and what actually breaks the chain? Each step conditions on prior outputs, so a wrong step contaminates everything downstream: 0.95^20 ≈ 36%. Chain-breakers: verifiable feedback (tests/compilers refute bad steps immediately), explicit verification nodes after critical steps, errors-as-observations so the model self-corrects, and checkpoints so recovery restarts from the last good state, not zero.

Which actions get a human gate, and how do you decide? Score tools on reversibility × blast radius × auditability. Irreversible + external (money, deletion, customer comms, prod changes) → hard gate via checkpointed interrupt. Reversible → auto with audit trail. Enforce in the harness at the tool boundary, never via prompt. And watch rejection rates: near-zero rejections means the gate is fatigue theater — narrow it.

How would you sandbox a code-executing agent? Ephemeral container per session: workspace-scoped filesystem, default-deny egress with an allowlist, CPU/memory/time limits, no ambient credentials — short-lived scoped tokens injected per call. Assume prompt injection will happen; the sandbox bounds what a hijacked agent can reach. Destroy on completion; artifacts out via a reviewed channel.

The agent crashed at step 8 of 12 — some steps had side effects. Now what? Resume from the step-8 checkpoint rather than replay (replay re-executes side effects). This is why mutating tools must be idempotent — after ambiguous failures you can't know if step 8 applied; an idempotency key makes retry safe. For multi-step external mutations, compensating actions (saga pattern) undo cleanly on abort.

How do you measure agent reliability before shipping? Eval suite of representative tasks with programmatic success checks; N runs per task; track pass@1, pass^k, cost and steps per success, and failure taxonomy (loop, wrong answer, unsafe attempt, gave up). Gate releases on the suite; replay real production failures into it so the suite ratchets.

Prompt injection via tool results — what's the defense in depth? (1) Provenance: keep untrusted content inside tool-result blocks, never promoted to system/user instructions. (2) Detection is unreliable — assume bypass. (3) Capability containment: least-privilege tools, egress controls, no secrets in context. (4) Human gates on consequential actions. The mindset: you can't fully stop the injection; you cap what obeying it can do.

Production concerns

• Measure reliability as a distribution: run each eval task N times; report pass^k (probability all k runs succeed) alongside pass@1 — an agent that's right 4/5 runs is a 20% pager load, and pass^k exposes flakiness that averages hide. See evals and testing.
• Observability is the prerequisite: per-step traces (prompt, tool, args, result, tokens, latency) with run-level ids; dashboards on loop-trip rate, per-tool error rate, human-rejection rate at gates, cost per completed task. Per-tool error rate tells you which schema or description to fix next. See observability.
• Cost of containment: verification steps and retries add tokens and latency — a verify-after-every-step agent can double cost. Place verification at high-leverage points (before irreversible actions, before returning to user), not uniformly. See cost and latency.
• Long chains argue for decomposition: if end-to-end reliability over 30 steps is unacceptable, restructure into checkpointed workflow stages with agentic subtasks of 5–10 steps — each stage verifiable, each restartable. This is the reliability case for workflows over agents. See agents vs workflows.
• Degradation policy: every run needs a defined failure output — partial results + summary + escalation path. "Agent gave up cleanly with a report" is a feature; a silent half-completed mutation is the incident. See reliability.
• The 12-factor lens: the widely-cited 12-Factor Agents principles (own your context window, small focused agents, contact humans via tool calls, stateless-reducer design) are all reliability arguments — production agent code ends up mostly deterministic software wrapped around small, contained LLM decisions.
• Security coupling: injection, data leakage, and tool auth are first-class reliability failures, not a separate brochure. See security.

Test yourself

Go deeper

• Building effective agents — Anthropic — guardrails, stopping conditions, and "add complexity only when it demonstrably improves outcomes."
• 12-Factor Agents — HumanLayer (GitHub) — 24k-star principles repo: own your context window, small agents, humans as tool calls, stateless reducers.
• Handle tool calls — Claude docs — is_error mechanics, instructive error messages, retry behavior, injection warning on tool results.
• LangGraph interrupts — LangChain docs — the checkpointed pause/resume mechanics behind approval gates.
• 12-Factor Agents: Patterns of reliable LLM applications — Dex Horthy, HumanLayer — AI Engineer — the widely-watched talk distilling ~100 production agent teams' lessons.

Where this connects

• Tool calling — strict schemas, parallel calls, and error-as-result as the first reliability layer.
• The agent harness — where caps, permission tiers, and sandboxing are actually implemented, and the measured cost of gating too much.
• Orchestration — durable checkpoints and interrupts that implement gates and crash resume.
• Agents vs workflows — shortening the chain is often the highest-leverage reliability fix.
• Security — prompt injection, leakage, and least privilege around tools and RAG.