Tool Calling
Function-calling mechanics, tool schemas, parallel calls, tool-error handling.
Prerequisites
- Agent Foundations — tool calling is the action half of the agent loop.
- Structured Output — strict tool args use the same constrained-decoding idea as JSON mode.
- LLM APIs — request/response shapes, streaming, and where tool blocks sit in the message list.
The intuition
Imagine you hire a capable contractor who cannot touch your systems directly. They can only fill out work orders: named forms with typed fields. You read the form, do the work (or refuse it), and hand back a receipt. They never hold the keys — they only express intent. That is tool calling.
The quality of the whole arrangement hangs on the form design. A vague form ("do something with
orders") produces vague or invented fields. A precise form with enums, examples, and "when to use
this" notes produces clean calls. And when a form is rejected, a useful rejection slip ("retry
after 60 seconds") lets them recover; a stamp that says "failed" does not.
Key insight
The model never executes anything. It emits structured intent. Auth, validation, side effects, and safety all live in your runtime — and tool definitions are prompts: the model chooses tools by reading names and descriptions the same way it follows any other instruction.
Why it exists
Without structured tool calling, connecting a model to the world is fragile glue.
- Free-form text is a terrible API. Prompting "Reply with
Action: search\nInput: …" and parsing with regex breaks on slight wording drift. Native tool-call tokens and JSON arguments replace that contract with something the model is trained to emit. - Side effects must stay outside the model. Databases, payments, and shells cannot run "inside" generation. The split — model proposes, runtime disposes — is the only place to put authz, sandboxing, and audit.
- Schemas beat hope. Descriptions, enums, and constrained decoding cut malformed arguments. Semantic mistakes remain, but syntax and missing-parameter failures become engineering problems with mechanical fixes.
- The agent loop needs a standard observation channel.
tool_result(orrole: tool) is how the world answers back so the next model turn can plan. Without it you are stuck in one-shot completion land.
Alternatives — pure chat with copy-pasted data, or brittle string protocols — lose on reliability and security. Tool calling is the industrial interface between language models and software.
The core idea
Tool calling (function calling — same feature, two names) is the mechanism that lets an LLM interact with the world: instead of answering in prose, the model emits a structured request to invoke a function you defined, your runtime executes it, and the result goes back into the conversation so the model can continue. The full round trip: (1) you send tool definitions — name, description, JSON Schema for parameters — alongside the messages; (2) the model decides a tool is needed and returns a structured call with the tool name and JSON arguments instead of (or alongside) text; (3) your code validates the arguments, runs the actual function, and appends the output as a tool-result message; (4) you call the API again and the model either answers using the result or calls another tool.
Two points matter in production. First: the model never executes anything — it produces intent; execution, auth, validation, and side effects are entirely your runtime's job. That boundary is where all the security and reliability engineering lives. Second: tool definitions are prompts. The model chooses tools by reading names and descriptions, so schema design is prompt engineering — a vague description produces wrong tool choices and hallucinated arguments just like a vague prompt produces bad answers. Both major APIs now also offer strict schema enforcement (constrained decoding), which guarantees the emitted arguments parse against your schema — eliminating malformed-call bugs entirely.
How it actually works
Wire format (Claude API): a response that wants a tool has stop_reason: "tool_use" and one or more tool_use content blocks: {id, name, input}. You reply with a user message containing a tool_result block referencing that id. OpenAI's shape is equivalent (tool_calls on the assistant message, results as role: "tool" messages keyed by tool_call_id). Ordering is strict on Claude: tool_result blocks must come first in the next user message and immediately follow their tool_use — violating this is a 400 error.
How the model "chooses": tool definitions are injected into the context (they cost input tokens — a few hundred for the tool-use system prompt plus your schemas), and the model is trained to emit call tokens when a request maps to a tool's described capability. tool_choice overrides this: auto (model decides), any/required (must call some tool), a specific tool name (must call that one), none.
Schema design rules that actually move accuracy:
| Rule | Why |
|---|---|
| Descriptions written like docs for a new hire | The description is the model's only knowledge of semantics — state what it does, when to use it, what it returns, edge cases |
| Unambiguous parameter names + per-parameter descriptions | location: "City and state, e.g. San Francisco, CA" beats loc: string — the example steers the format |
enum for closed sets, formats/constraints in the schema | Constrains generation instead of hoping; with strict mode, guarantees it |
Namespace related tools (jira_search, jira_create) | Helps selection when tool count grows; keeps similar tools distinct |
| Few, distinct, high-level tools | Overlapping tools cause wrong picks; one search_orders(filters) beats five near-duplicates |
| Return errors with guidance, paginate/truncate outputs | Tool outputs are context too — a 50 KB JSON dump per call burns the window |
Strict mode: OpenAI strict: true (requires additionalProperties: false, all fields listed in required) and Claude's strict tool use both use constrained decoding so arguments cannot deviate from the schema. This removes malformed JSON and missing-parameter failures — but not semantic errors (syntactically valid, wrong values).
Common misconception
Strict mode does not make validation optional. It guarantees syntax (parseable,
schema-conformant), not semantics or safety. delete_user(id=42) can be schema-perfect while
unauthorized. Always enforce authorization and business rules in the runtime.
Parallel tool calls: by default the model may emit several tool_use blocks in one turn (fetch weather for three cities at once). You execute them concurrently and return all results in the next message. Disable via disable_parallel_tool_use: true (Claude) / parallel_tool_calls: false (OpenAI) when tools have ordering dependencies or side effects that must be sequential. Parallel calls cut wall-clock latency and round trips, but only for independent operations.
Error handling — the key design decision: when a tool fails, return the error as the tool result (Claude: is_error: true with a message) instead of crashing the loop. The model reads it and recovers — retries with corrected arguments, tries another tool, or tells the user. Claude will retry an invalid call 2–3 times with corrections before giving up. Write instructive errors: "Rate limit exceeded. Retry after 60 seconds." is actionable; "failed" forces guessing. Reserve hard loop failure for your own infrastructure errors, not tool-level ones.
Scaling the tool count: every tool definition consumes context and dilutes selection accuracy. Beyond a few dozen tools, use dynamic discovery (Anthropic's tool search tool loads definitions on demand) or route to sub-agents with small toolsets. Anthropic's 2026 advanced tool use additions go further: programmatic tool calling lets the model orchestrate tools from code instead of one API round trip per call.
On the governed enterprise platform, internal tools are often exposed through MCP: servers advertise tools; the host presents them to the model via the same tool-calling mechanism. MCP standardizes discovery and transport; it does not replace the model↔runtime call contract described here.
The flows
| Flow | Sequence | When it applies | What breaks it |
|---|---|---|---|
| Happy path | Schemas in request → tool_use → validate → execute → tool_result → model answers or next call | Model needs external data or side effects | Vague descriptions; missing required fields without strict mode |
| Tool-error recovery | Execute throws / validation fails → is_error result with guidance → model retries or pivots | Recoverable tool failures | Crash-the-loop on tool errors; "failed" with no next step |
| Parallel batch | Multiple tool_use in one turn → concurrent execute → all results in one user message | Independent reads | Order-dependent writes; B needs A's output |
| Forced tool use | tool_choice: required or a specific tool name → model must call | Extraction/routing steps that must not free-answer | Wrong forced tool; no path to refuse unsafe ops |
| Injection-aware observe | External content lands only inside tool_result → treated as data | Web/email/document tools | Promoting tool text into system instructions |
A worked example
Employee on the governed enterprise platform:
"What's the weather impact on our London warehouse SLA today, and open Sev tickets there?"
Tools (abbreviated schemas):
get_weather(city: string, units: enum[celsius,fahrenheit])
search_tickets(site: string, severity: enum[Sev1,Sev2,Sev3], status: enum[open,closed])| Turn | What the model emits | Runtime | Result into context |
|---|---|---|---|
| 1 | Parallel: get_weather({city:"London", units:"celsius"}) and search_tickets({site:"LON-WH", severity:"Sev1", status:"open"}) | Both run concurrently (~200 ms tools) | Weather JSON ~150 tokens; tickets list ~400 tokens |
| 2 | Final prose synthesizing both | — | User sees one answer |
Without parallel calls this is 3 model turns (weather → tickets → answer). With parallel it is 2 model turns. Tool schemas + tool-use scaffolding might add ~400–800 input tokens on every request; with prompt caching those definitions sit in the cached prefix after the first hit.
Break it deliberately — bad schema: rename to get_wx(loc: string) with description "weather".
Typical failures: model passes "UK" instead of a city; invents get_london_weather; dumps a full
forecast API payload (8k tokens) and blows the window on turn 2.
What omitted stages look like in production
- No runtime validation — schema-valid
site: "*"becomes a full table scan or auth bypass. - Errors crash the loop — one 503 from tickets kills the whole answer; weather data is wasted.
- Untruncated tool output — 50 KB JSON per call; mid-conversation context fills with noise.
- Parallel left on for debit-then-credit — racey double application of side effects.
- No per-tool metrics — you never see that
search_ticketshas 18% error rate and a bad enum.
Common drill-downs
Questions that sit one level under the mechanism — the ones that show up when the happy path is no longer enough.
Walk me through one tool call end to end.
Request carries tool schemas + messages → model returns stop_reason: "tool_use" with {id, name, input} → runtime validates args and executes the function → result appended as tool_result keyed by the call id → second API call → model answers with the data or calls the next tool. Two model invocations minimum per tool used.
The model calls the wrong tool or invents parameters. Fix it? In order: enable strict mode (kills schema violations mechanically); rewrite descriptions — state when to use each tool and disambiguate overlapping ones; add enums/examples to parameters; reduce or merge overlapping tools; add few-shot examples of correct calls. Measure per-tool error rate before and after.
How do you handle a tool that throws?
Catch it, return the error text as the tool result with is_error: true, and let the model decide: retry corrected, use an alternative, or surface to the user. Failing the whole loop on a recoverable error throws away the agent's ability to self-correct. Cap retries in the harness to stop pathological loops.
When would you disable parallel tool calls? Order-dependent operations (read-then-write on the same resource), non-idempotent side effects, rate-limited backends, or when call B's arguments depend on call A's result. Otherwise leave parallel on — it saves a round trip per batch of independent calls.
Does strict mode make validation unnecessary?
No. It guarantees syntax (parseable, schema-conformant), not semantics or safety. delete_user(id=42) is schema-perfect; whether the caller may delete user 42 is authorization — always enforced in your runtime.
You have 200 tools. What breaks and what do you do? Context bloat (200 schemas per request) and selection accuracy collapse (similar tools confuse ranking). Fixes: dynamic tool discovery/search so only relevant definitions load, namespacing, consolidating variants into parameterized tools, or splitting into sub-agents with focused toolsets.
Tool calling vs MCP — relationship? Tool calling is the model capability (emit structured calls). MCP is a protocol standardizing how runtimes discover and invoke tools from external servers — MCP servers expose tools that your runtime presents to the model through the same tool-calling mechanism. MCP replaces per-integration glue code, not the calling mechanics.
How do tool definitions affect cost? They're input tokens on every request — schemas plus a tool-use system prompt (~300–700 tokens by model/mode). With prompt caching, definitions sit in the cached prefix so you pay full price once and the cached rate thereafter. Uncached, 30 verbose tools can add thousands of tokens per call.
Production concerns
- Latency: each tool round trip is a full model call. An answer needing 4 sequential tools ≈ 5 model turns. Levers: parallel calls, faster models for tool-heavy steps, prompt caching (tool definitions are a stable prefix — cache them), programmatic/code-based tool orchestration to collapse round trips. See latency.
- Cost: tool schemas + the tool-use system prompt (roughly 300–700 tokens depending on model) are billed on every request, and every call/result pair grows the transcript. Trim tool outputs aggressively; prefer
concise/detailedresponse-format parameters on chatty tools. See cost. - Security: treat every argument as untrusted input — validate types, ranges, and authorization server-side even with strict mode (schema-valid ≠ safe). Tool results carrying external content (web pages, emails) are the prompt-injection vector: keep them inside
tool_resultblocks and treat any instructions in them as data, not commands. See security. - Failure modes to design for: hallucinated tool names or arguments (strict schemas + validation + error-as-result), model calling tools when it shouldn't (tighten descriptions, adjust
tool_choice), model not calling tools when it should (system-prompt nudge: "use tools to investigate before answering"), infinite retry of a failing tool (cap retries in the harness, not just the model). See agent reliability and reliability. - Idempotency: the model may repeat a call after an ambiguous result or timeout. Make mutating tools idempotent (idempotency keys) or gate them behind confirmation.
- Observability: log every call — name, arguments, result size, latency, error rate per tool. Per-tool error rate is your best signal for which description or schema needs fixing. See observability.
Test yourself
A tool returns HTTP 429. Your runtime currently raises and the agent UI shows a generic failure. What should change, and what should the error string look like?
Strict mode is on, yet users still report 'wrong tool' and 'deleted the wrong record.' Is strict mode broken?
You need read-then-write on the same ticket. The model emits both calls in one parallel turn. What goes wrong, and how do you prevent it?
Why do tool *outputs* need as much design attention as tool *schemas*?
When would you move from 'all tools always in the request' to dynamic tool discovery?
Go deeper
- Tool use overview — Claude docs — full mechanics, strict tool use, server vs client tools, pricing.
- Handle tool calls — Claude docs —
tool_resultformatting rules,is_error, retry behavior, injection warning. - Function calling — OpenAI docs — the OpenAI-side equivalent:
tools,strict,parallel_tool_calls. - Writing effective tools for agents — Anthropic — schema and description design measured against real agent evals: namespacing, token-efficient outputs, response formats.
- What is Tool Calling? Connecting LLMs to Your Data — IBM Technology — short canonical explainer of the mechanism.
Where this connects
- Agent foundations — the loop that repeatedly issues tool calls and consumes observations.
- MCP protocol — how external servers expose tools that your runtime still presents through this same calling mechanism.
- The agent harness — the runtime that dispatches these calls, shapes their results, and decides what a tool is even allowed to do.
- Agent reliability — tool hallucination, retry storms, and injection via tool results.
- Security — treating arguments and tool outputs as untrusted data.