Tool Calling
Function-calling mechanics, tool schemas, parallel calls, tool-error handling.
Prerequisites
- Agent Foundations — tool calling is the action half of the agent loop.
- Structured Output — strict tool args use the same constrained-decoding idea as JSON mode.
- LLM APIs — request/response shapes, streaming, and where tool blocks sit in the message list.
The intuition
Imagine a capable contractor who cannot touch your systems. They only fill out work orders: named forms with typed fields. You read each form, do the work (or refuse it), and hand back a receipt. They never hold the keys — only intent. That is tool calling.
Form design is the whole game. A vague form produces invented fields; a precise one — enums,
examples, "when to use this" notes — produces clean calls. A useful rejection slip ("retry after 60
seconds") lets them recover; a stamp that says "failed" does not.
Key insight
The model never executes anything. It emits structured intent. Auth, validation, side effects, and safety all live in your runtime — and tool definitions are prompts: the model chooses tools by reading names and descriptions the same way it follows any other instruction.
Why it exists
Without structured tool calling, connecting a model to the world is fragile glue.
Free-form text is a terrible API. Prompting Action: search\nInput: … and parsing with regex
breaks on wording drift. Native tool-call tokens and JSON arguments are a contract the model is
trained to emit.
Side effects must stay outside generation. Databases, payments, and shells cannot run "inside" a completion. Model proposes, runtime disposes — that boundary holds authz, sandboxing, and audit.
Schemas beat hope. Descriptions, enums, and constrained decoding cut malformed arguments. Semantic mistakes remain; syntax failures become engineering problems with mechanical fixes.
The agent loop needs a standard observation channel. A tool_result (or role: tool) message is
how the world answers so the next turn can plan.
Alternatives — pure chat with copy-pasted data, or brittle string protocols — lose on reliability and security.
The core idea
Tool calling (also function calling — same feature, two vendor names) is the round trip that lets an LLM interact with the world: you send tool definitions (name, description, JSON Schema); the model returns a structured call; your runtime validates, authorizes, and executes; you append a tool result keyed by call id and call the API again so the model can answer or issue another call.
Two production facts follow. The security boundary is entirely in your runtime — the model only produces intent. And tool definitions are prompts: vague descriptions produce wrong picks the same way a vague system prompt produces bad answers. Strict schema enforcement removes malformed-call bugs; it does not remove semantic or authorization errors.
How it actually works
Wire format. On Claude, a tool-wanting response has stop_reason: "tool_use" and one or more
tool_use blocks {id, name, input}. You reply with a user message of tool_result blocks that
reference each id. OpenAI's shape is equivalent (tool_calls / Responses function_call items,
results keyed by call id). On Claude, ordering is strict: tool_result blocks must come first in
the next user message and immediately follow their tool_use — violating that is a 400.
Client vs server tools. Client tools are yours to define and execute. Server tools (web search, code execution, tool search, and similar) run on the vendor. This page is the client-tool contract; server tools share selection ideas but not always the same error path.
How the model chooses. Definitions enter context as input tokens; the model is trained to emit
call tokens when a request maps to a tool's capability. tool_choice overrides that: auto,
any/required, a specific tool name, or none.
Schema design that moves accuracy. Write descriptions like docs for a new hire — what it does,
when to use it, what it returns. Prefer unambiguous names with examples
(location: "City and state, e.g. San Francisco, CA" beats loc: string). Use enum for closed
sets. Namespace related tools (jira_search, jira_create). Prefer few high-level tools over near-
duplicates. Design outputs with equal care: pagination, field allowlists, and
concise/detailed formats so results do not flood the window.
Strict mode. OpenAI strict: true (needs additionalProperties: false and every property in
required) and Claude's strict tool use both use constrained decoding so arguments cannot leave the
schema. That kills malformed JSON and missing parameters — not semantic errors (valid shape, wrong
values).
Common misconception
Strict mode guarantees syntax, not semantics or safety. delete_user(id=42) can be
schema-perfect while unauthorized. Enforce authz and business rules in the runtime.
Parallel tool calls. The model may emit several tool_use blocks in one turn. Execute them
concurrently and return all results together. Disable via disable_parallel_tool_use: true
(Claude) or parallel_tool_calls: false (OpenAI) when tools have ordering dependencies or
non-idempotent side effects.
Error handling. Return failures as the tool result (Claude: is_error: true) instead of
crashing the loop. "Rate limit exceeded. Retry after 60 seconds." is actionable; "failed" is
not. Cap retries in the harness so a permanent outage cannot burn the budget.
Scaling the tool count. Every definition costs context and dilutes selection. Keep the active
set small (tens, not hundreds). Beyond that, use dynamic discovery — Anthropic's Tool Search
Tool and OpenAI's tool_search load definitions on demand (defer_loading) — or route to
sub-agents with focused toolsets. Programmatic tool calling (Claude) lets the
model orchestrate tools from code so bulk intermediate data never enters the model context.
On the governed enterprise platform, internal tools often come through MCP. MCP standardizes discovery and transport; it does not replace this call contract.
The flows
| Flow | Sequence | When | What breaks it |
|---|---|---|---|
| Happy path | Schemas → tool_use → validate → execute → tool_result → answer or next call | Needs data or side effects | Vague descriptions; missing fields without strict mode |
| Tool-error recovery | Failure → is_error result with guidance → retry or pivot | Recoverable tool failures | Crash-the-loop; bare "failed" |
| Parallel batch | Multiple tool_use → concurrent execute → all results | Independent reads | Order-dependent writes; B needs A's output |
| Forced tool use | tool_choice: required or a named tool | Must not free-answer | Wrong forced tool; no refuse path |
| Injection-aware observe | External content only inside tool_result | Web/email/document tools | Promoting tool text into system instructions |
A worked example
Employee on the governed enterprise platform:
"What's the weather impact on our London warehouse SLA today, and open Sev tickets there?"
Tools (abbreviated): get_weather(city, units) and
search_tickets(site, severity, status) with enums on units/severity/status.
| Turn | Model emits | Runtime | Into context |
|---|---|---|---|
| 1 | Parallel: weather for London + open Sev1 tickets at LON-WH | Concurrent (~200 ms tools) | Weather ~150 tokens; tickets ~400 tokens (illustrative) |
| 2 | Final prose synthesizing both | — | One answer |
Without parallel calls this is 3 model turns; with parallel it is 2. Schemas plus scaffolding might add ~400–800 input tokens every request (illustrative). With prompt caching those definitions sit in the cached prefix after the first hit.
Break it deliberately: rename to get_wx(loc: string) with description "weather". Typical
failures: model passes "UK" instead of a city; invents get_london_weather; dumps a full forecast
payload and crowds out turn 2.
What omitted stages look like in production
- No runtime validation — schema-valid
site: "*"becomes a table scan or auth bypass. - Errors crash the loop — one 503 kills the whole answer; weather data is wasted.
- Untruncated tool output — tens of KB of JSON crowd out reasoning.
- Parallel left on for debit-then-credit — racey double side effects.
- No per-tool metrics — you never see
search_ticketshas 18% error rate.
Production concerns
Latency. Each sequential tool is a full model call — four sequential tools ≈ five turns. Use parallel calls, faster models for tool-heavy steps, prompt caching of stable definitions, and programmatic orchestration when bulk intermediate data is the bottleneck. See latency.
Cost. Schemas plus the tool-use system prompt (a few hundred tokens, model-dependent) bill on every request; call/result pairs grow the transcript. Trim outputs; prefer dynamic discovery over shipping hundreds of schemas each turn. See cost.
Security. Treat every argument as untrusted — validate types, ranges, and authz server-side even
with strict mode. Tool results carrying web pages or email are the main indirect prompt injection
vector: keep them inside tool_result and treat instructions inside as data. See
security.
Failure modes. Hallucinated names/args (strict + validation + error-as-result). Calling when it
shouldn't (tighter descriptions, tool_choice). Not calling when it should (system-prompt nudge).
Infinite retry of a failing tool (harness caps). See agent reliability.
Idempotency. The model may repeat a call after an ambiguous result or timeout. Make mutating tools idempotent (idempotency keys) or gate them behind confirmation.
Observability. Log name, arguments, result size, latency, and error rate per tool — the best signal for which schema needs fixing. See observability.
Common drill-downs
Wrong tool or invented parameters? Strict mode, clearer "when to use / when not" descriptions, enums and input examples, merge overlaps. Measure selection error before and after.
Disable parallel when? Order-dependent ops, non-idempotent side effects, rate-limited backends, or when B's args depend on A's result. Otherwise leave it on.
200 tools? Context bloat and selection collapse. Dynamic discovery, namespacing, parameterized tools, or sub-agents with focused toolsets.
Tool calling vs MCP? Tool calling is the model capability. MCP standardizes discovery and transport for external tool servers. Your runtime still presents MCP tools through this mechanism.
Test yourself
A tool returns HTTP 429. Your runtime currently raises and the agent UI shows a generic failure. What should change, and what should the error string look like?
Strict mode is on, yet users still report 'wrong tool' and 'deleted the wrong record.' Is strict mode broken?
You need read-then-write on the same ticket. The model emits both calls in one parallel turn. What goes wrong, and how do you prevent it?
Why do tool *outputs* need as much design attention as tool *schemas*?
When would you move from 'all tools always in the request' to dynamic tool discovery?
Go deeper
- Tool use overview — Claude docs — mechanics, client vs server tools, strict mode, pricing.
- Handle tool calls — Claude docs —
tool_resultformatting,is_error, injection warning. - Function calling — OpenAI docs —
tools,strict,parallel_tool_calls, tool search. - Writing effective tools for agents — Anthropic — schema design from real agent evals: namespacing, token-efficient outputs.
- Advanced tool use — Anthropic — Tool Search Tool, programmatic tool calling, tool-use examples.
Where this connects
- Agent foundations — the loop that issues tool calls and consumes observations.
- MCP protocol — external servers that expose tools your runtime still presents through this mechanism.
- The agent harness — dispatch, result shaping, and permission tiers.
- Agent reliability — tool hallucination, retry storms, injection via results.
- Security — treating arguments and tool outputs as untrusted data.