---
kind: concept
slug: tool-use
title: Tool use
summary: >-
  Let a language model emit a call to an external tool, run it outside the model, and feed the result back into the context. Twenty-seven wikillm cards cover prompt-only, few-shot, planned, structured-loop, learned, trained, gated, guarded, versioned and API-writing variants. Nineteen report measured results, most mixing tool use with other changes.
tags: [agents, tools, prompting]
papers: ["2303.12712", "2505.10468", "2210.02875", "2306.03314", "2604.15034", "2306.02224", "2311.05437", "2509.21842", "2201.08239", "2210.03629", "2308.08155", "2308.11432", "2211.10435", "2304.09842", "2505.11792", "2607.23929", "2607.26148", "2309.16609", "2305.15334", "2406.12793", "2503.23037", "2504.19678", "2506.04133", "2412.10402", "2404.02831", "2505.15062", "2507.14267"]
related: [prompt-engineering, chain-of-thought-prompting, retrieval-augmented-generation, in-context-learning, llm-agent, multi-agent-systems, neural-symbolic-integration, self-consistency, self-refinement, imitation-learning, policy-gradient, multimodal-language-model, visual-instruction-tuning, hallucination-mitigation, dialogue-generation, dialogue-evaluation, task-decomposition, llm-planning, table-question-answering, reinforcement-learning-verifiable-rewards, agent-memory, benchmark-validity, knowledge-graph-reasoning, llm-as-a-judge]
status: draft
updated_at: 2026-11-01
---

## Overview

Tool use lets a language model act on the outside world or fetch information it
does not have. The model writes a call in plain text or as structured
arguments, such as a search query or an arithmetic expression. In the common
loop, generation pauses, the tool runs, and its output is pasted into the
context before generation continues [arxiv:2303.12712 §5.1] [fig. 5.2]
[arxiv:2505.10468 §2.1.2]. No weights change in that loop: the protocol lives
in the prompt and in the code that parses the model's output. Some variants
do train the model to make calls, as described below.

Twenty-seven cards on wikillm cover this topic, and they differ in kind. Two are a
probe and a review of prompt-level tool use. The probe is of a pre-release,
text-only GPT-4 with a handful of tool transcripts and no tool-use accuracy
figure [arxiv:2303.12712 §5.1]. The review is narrative and reports no
experiments [arxiv:2505.10468 §1.1]. A survey puts tool use in its taxonomy of
agent actions, where the action space is either external tools or the model's
own abilities [arxiv:2308.11432 §2.1.4]. It reports no experiments of its own.
The plugin framework proposes a design and reports no experiments either
[arxiv:2306.03314 §2.2] [arxiv:2306.03314 §5.1]. Three more surveys and
reviews from 2025 and 2026 also report no experiments. A 2024 perspective on
biomedical AI agents lists tool use as one of four modules and reports no
experiments [arxiv:2404.02831 §5.2]. One lists tool calling among agent actions [arxiv:2503.23037 §3.2]. One compares agent protocols [arxiv:2504.19678 §IV.C]. One lists tool misuse among its threats [arxiv:2506.04133 §3]. The other nineteen report measured results. Gorilla fine-tunes a 7B model to write API calls and is scored on the authors' own benchmark [arxiv:2305.15334 tab. 1]. ChatGLM's All Tools is a deployed chat model that picks its own built-in tools [arxiv:2406.12793 §2]. TANGO has an LLM write a short program over navigation and perception skills, and reports zero-shot results on three embodied benchmarks [arxiv:2412.10402 §3] [arxiv:2412.10402 tab. 1]. ReAct interleaves free-text thoughts with lookups and
reports question-answering and interactive-task scores [arxiv:2210.03629 tab. 1]
[arxiv:2210.03629 tab. 3]. LaMDA is a dialogue model fine-tuned to query a
search tool, a calculator and a translator [arxiv:2201.08239 §6.2]. AutoGen
runs model-written code in a conversation [arxiv:2308.08155 §2.1]. PAL has the
model write Python as its reasoning, and an interpreter, not the model,
produces the answer [arxiv:2211.10435 §3]. Binder is a
training-free method for table question answering in which the model's program
calls the model again [arxiv:2210.02875 §2.1]. Auto-GPT-style agents run a tool
loop on two simulated decision tasks [arxiv:2306.02224 §2.2]. LLaVA-Plus is a
multimodal model fine-tuned to emit tool calls [arxiv:2311.05437 §2.1].
Qwen's chat models mix about 2,000 tool-use samples into a general
fine-tuning set [arxiv:2309.16609 §3.4]. DeepTravel is a travel-planning agent trained with reinforcement learning
against a cached tool sandbox [arxiv:2509.21842 §3.1]. SAKE trains a Qwen2.5 model with reinforcement learning to call two knowledge-graph tools in a fixed order [arxiv:2505.15062 §3.2]. DREAMS puts a guard in front of each tool call that a materials-simulation agent makes [arxiv:2507.14267 Methods: Safety guard]. Autogenesis registers
tools as versioned resources and reports agent gains that cannot be split by
tool [arxiv:2604.15034 §5.2]. Chameleon has a planner write a module sequence over a fixed inventory, and
reports 86.54 on ScienceQA and 98.78 on TabMWP with GPT-4 [arxiv:2304.09842
tab. 3] [arxiv:2304.09842 tab. 4]. SIRL trains a model to write solver code,
with the solver's result as the reward [arxiv:2505.11792 §3.3]. MemTX blocks
irreversible tool calls while uncommitted memory exists [arxiv:2607.23929
§3.3]. The navigation study gives a model two tools and nothing else
[arxiv:2607.26148 §3.2]. Each of these mixes tool use with other changes,
apart from the PAL runner ablation [arxiv:2211.10435 §6] and Chameleon's
single-module removals [arxiv:2304.09842 tab. 5]. The navigation study's
waypoint arm also changes the observation [arxiv:2607.26148 app. A].
Binder's numbers depend on a retired base model [arxiv:2210.02875 §3.1]. PAL's
numbers use code-davinci-002, whose shutdown the authors' code notes
anticipates [arxiv:2211.10435 §4]. The Auto-GPT, LLaVA-Plus, Qwen, DeepTravel, Chameleon, SIRL, MemTX, TANGO and
navigation results are cited in their variants below.
Gorilla and ChatGLM compare whole models, so their gains cannot be credited to tool access alone. No card gives a general tool-use success rate, so treat the claims below as
observations of one model at one point in time, as results on one task family,
or as proposals.

The shared invariant across variants: the model proposes the call, but the
harness executes it and decides what text goes back into the context.

Other tool-use work is not summarized here.

## Variants

### Prompt-described tools with a text call format

The prompt lists each tool with its name and argument format, for example
`SEARCH("query")`, `CALC(expression)` and `CHARACTER(string, index)`. The
prompt has no worked examples of tool use. The paper's loop reads the model's
output up to the next tool call, runs the tool and appends the result
[arxiv:2303.12712 §5.1] [fig. 5.2]. Card: [Sparks of Artificial General
Intelligence: Early experiments with GPT-4](/papers/2303.12712).

### Interleaved thoughts and actions in a few-shot trace

ReAct keeps a text call format but adds worked examples. Each example is a
sequence of thought, action and observation lines from one solved task
[arxiv:2210.03629 §2] [arxiv:2210.03629 app. C]. A thought is free text. It
changes nothing in the environment and returns no observation, but later steps
can read it [arxiv:2210.03629 §2]. The actions come from a fixed set. For
question answering they are a Wikipedia search, a lookup within the current
page and a finish step [arxiv:2210.03629 §3.1]. The prompt holds one to six
examples per task, and the authors say more examples did not help on HotpotQA
[arxiv:2210.03629 §3.2].

With a frozen PaLM-540B, ReAct scores 27.4 exact match on HotpotQA, below the
29.4 of chain-of-thought, and 60.9 accuracy on FEVER, above the 56.3 of
chain-of-thought [arxiv:2210.03629 tab. 1]. Combining it with chain-of-thought
self-consistency gives the best prompting result on both tasks, at 35.1 and
62.0 [arxiv:2210.03629 tab. 1]. On ALFWorld, the best of six prompt sets reaches
71% success, against 45% for the same trajectories with thoughts removed
[arxiv:2210.03629 tab. 3]. The average over the six sets is 57%
[arxiv:2210.03629 tab. 3]. On WebShop, ReAct reaches 40.0 success against 30.1
for the action-only version [arxiv:2210.03629 tab. 4]. The failures change
shape. In a hand-labelled sample of 200 HotpotQA items, reasoning errors,
including repeated steps, make up 47% of ReAct's failures. Invented facts make
up 56% of chain-of-thought's failures and none of ReAct's [arxiv:2210.03629
tab. 2].

The main results need PaLM-540B, which the paper's reproducibility statement
says is not openly accessible. The public check uses GPT-3 text-davinci-002 and
reports ALFWorld success of 78.4 against 70.9 for PaLM [arxiv:2210.03629 app. A.1]
[arxiv:2210.03629 tab. 5]. Search results come from a live Wikipedia, which
can return newer facts, so repeat runs need a frozen snapshot
[arxiv:2210.03629 app. A.2]. Card: [ReAct: Synergizing Reasoning and Acting in
Language Models](/papers/2210.03629).

### Chained multi-tool sessions

The same mechanism is used with several tools in one session. The paper shows a
command-line "zoo manager" task, an email and calendar task, and web browsing
[arxiv:2303.12712 §5.1.1] [fig. 5.3] [fig. 5.4] [fig. 5.5]. Each is a single
transcript, so success rates across runs are not known.

### Model-written code run by an external executor

For arithmetic, the authors suggest that the model write code which a separate
executor runs, with the result appended to the prompt. They give one
proof-of-concept prompt and say they did not evaluate it systematically
[arxiv:2303.12712 app. D.4]. Treat this as a proposal, not a measured result.

AutoGen makes the executor an agent. The code-writing agent sends code to a
user-proxy agent that runs it and returns the output as a message, so the run
is a conversation rather than a fixed pipeline [arxiv:2308.08155 §2.1]
[arxiv:2308.08155 §3]. On the full MATH test set of 5,000 problems, the
authors report 69.48% accuracy for GPT-4 with this setup, against 55.18% for
vanilla GPT-4 [arxiv:2308.08155 app. A1]. That is the only full-set number in
the paper. The smaller 120-problem comparisons appear only in a figure
[arxiv:2308.08155 app. A1] [fig. 4a]. The paper warns that agents which run code
or call functions can change the environment, for example by installing
packages [arxiv:2308.08155 app. Ethics statement]. Card: [AutoGen: Enabling
Next-Gen LLM Applications via Multi-Agent Conversation](/papers/2308.08155).

PAL is a training-free version of this idea. Each few-shot example is a
question followed by a Python program, with a comment on each line that gives a
step in words. The program's value is the answer, and no final answer is
written in the examples [arxiv:2211.10435 §3]. The main results run one program
per question after generation, in a standard interpreter [arxiv:2211.10435 §3].
With code-davinci-002 and greedy decoding, PAL solves 72.0% of GSM8K against
65.6% for chain-of-thought with the same model [arxiv:2211.10435 §5.1]. Sampling
40 programs and taking the most common answer gives 80.4%, against 78.0% for
chain-of-thought with the same vote [arxiv:2211.10435 §5.1]. On five symbolic
and algorithmic tasks, PAL is ahead of chain-of-thought on each. Penguins goes
from 79.2% to 93.3% [arxiv:2211.10435 §5.2]. The one ablation that keeps the
program fixed and changes only who runs it on GSM8K is the closest this set of
cards has to a tool-only test. When the model writes the answer after reading
its own program, the score falls to 23.2% [arxiv:2211.10435 §6]. Card:
[PAL: Program-aided Language Models](/papers/2211.10435).

### Planner-chosen module sequence from a fixed inventory

Chameleon gives an LLM planner an inventory of thirteen modules. They cover LLM
prompts, an image captioner, OCR, web search, a Python program generator,
verifier and executor, and rule-based table helpers [arxiv:2304.09842 §4.1]
[arxiv:2304.09842 tab. 2]. The planner receives a task instruction, the module
descriptions with their constraints and a few demonstrations. It returns a list
of module names once, and each name is bound to a module by string matching
[arxiv:2304.09842 §3] [eq. 1]. Each module reads the current input and a cache,
and update functions prepare the input and cache for the next module
[eq. 2] [eq. 3] [eq. 4]. The last module's output is the answer [eq. 5]. The
planner is not called again during the run [arxiv:2304.09842 §3].

Each task has its own check on the plan. On ScienceQA a valid plan ends with
Solution Generator and then Answer Generator, and that pair is also the
fallback. On TabMWP a valid plan includes Answer Generator and puts Program
Generator before the verifier and executor. Otherwise a program-of-thought
sequence with verification runs [arxiv:2304.09842 §4.2] [arxiv:2304.09842 §4.3].

With GPT-4 as the LLM, the authors report 86.54% on ScienceQA, against 83.99%
for GPT-4 chain-of-thought, and 98.78% on TabMWP, against 96.93% for GPT-4
program-of-thought [arxiv:2304.09842 tab. 3] [arxiv:2304.09842 tab. 4]. With
ChatGPT the figures are 79.93% and 93.28% [arxiv:2304.09842 tab. 3]
[arxiv:2304.09842 tab. 4]. The fine-tuned MM-COT model scores higher on
ScienceQA, at 91.68% [arxiv:2304.09842 tab. 3]. Plans vary with the question:
GPT-4 wrote 11 distinct module sequences on ScienceQA and 19 on TabMWP, against
14 and 28 for ChatGPT [arxiv:2304.09842 tab. 7]. In an ablation on 500 test
examples with ChatGPT, removing one module at a time costs 7.4 points for Bing
search on ScienceQA, 8.4 for the text detector and 7.4 for the program generator
on TabMWP [arxiv:2304.09842 §5.2] [arxiv:2304.09842 tab. 5]. Card:
[Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models](/papers/2304.09842).

### Few-shot program over pretrained embodied skills

TANGO applies the program-composition idea to embodied navigation. An LLM
receives an instruction and a fixed prompt of 15 in-context examples, and
outputs a step-by-step program with comments [arxiv:2412.10402 §3] [fig. 2].
Each line calls one module, such as `detect`, `navigate_to` or `explore_scene`,
and each module writes its output to a named variable [arxiv:2412.10402 §3]
[fig. 3]. The paper describes one program per task. The navigation core is a
PointGoal policy pretrained on HM3D and not trained on the test scenes
[arxiv:2412.10402 §4.1]. Exploration follows VLFM, with a per-pixel feature
memory added so that several targets can be remembered [arxiv:2412.10402 §3].
The method is training-free for the task itself [arxiv:2412.10402 §1].

Author-reported zero-shot results. On HM3D-OVON val unseen, TANGO has 35.5
success and 19.5 SPL, against 35.2 and 19.6 for VLFM [arxiv:2412.10402 tab. 1].
DAgRL+OD is higher on both measures there [arxiv:2412.10402 tab. 1]. On
GOAT-Bench val unseen, success is 32.1 against 24.9 for the best prior row,
but SPL is 16.5 against 17.2 [arxiv:2412.10402 tab. 3]. On OpenEQA the score is
37.2 against 38.1 for Socratic LLMs with frame captions [arxiv:2412.10402
tab. 2]. The text calls the OVON result on par with VLFM and the OpenEQA result
second [arxiv:2412.10402 §4.2]. In a failure analysis of EQA episodes, the
`detect` module is the main cause, and the LLM wrote incorrect pseudo-code in
about 18% of failures. The sample size is not stated [arxiv:2412.10402 §4.3].
Card: [TANGO: Training-free Embodied AI Agents for Open-world
Tasks](/papers/2412.10402).

### Structured calls in a reason-act-observe loop

The review describes an agent cycle. The model reasons, emits a tool invocation
with structured arguments, such as JSON, the result is integrated into the
context, and the next step follows [arxiv:2505.10468 §2.1.2]. It lists
tool-augmented reasoning and this reason-act-observe loop among its ten
mitigation mechanisms [arxiv:2505.10468 §6.1]. It does not state a stop rule,
a step budget or a memory format [arxiv:2505.10468 §6.1]. The review extends
the same loop to several specialised agents with shared memory and an
orchestrator [arxiv:2505.10468 §3.1.2], and gives no evaluation of that design.
Card: [AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and
Challenges](/papers/2505.10468).

Auto-GPT-style agents run this loop with a JSON reply at each step. The reply
holds a `thoughts` object and a `command` with a name and arguments. The named
tool runs, and its observation goes back into the history [arxiv:2306.02224
§2.2]. The authors found that tool instructions without examples performed much
worse, so they add one to three demonstrations per tool [arxiv:2306.02224
§2.2]. The loop ends on a `finish` command or at a step limit, set to 20 steps
on WebShop and 35 on ALFWorld [arxiv:2306.02224 §3.2.1] [arxiv:2306.02224
§3.2.2]. On both tasks, GPT-4 with this loop scores above the imitation
baseline, but only by small margins on WebShop, which has 50 test instructions.
Claude and GPT-3.5 score below it on both tasks, and Vicuna scores zero
[arxiv:2306.02224 tab. 1] [arxiv:2306.02224 tab. 2]. The loop can also take a
supervised model's suggested action as a reference in the prompt, with no
fine-tuning. For GPT-4 this lifts WebShop success from 0.240 to 0.300 with one
suggestion and to 0.320 with five, and ALFWorld success from 0.485 to 0.515
with one [arxiv:2306.02224 tab. 1] [arxiv:2306.02224 tab. 2]. The test sets are
small, and the authors' own caution about gaps of a few tasks applies
[arxiv:2306.02224 §3.1.1] [arxiv:2306.02224 §3.1.2]. Card: [Auto-GPT for Online
Decision Making: Benchmarks and Additional Opinions](/papers/2306.02224).

### Minimal observe and step primitives in an embodied loop

In a zero-shot navigation study, the model gets two tools and nothing else
[arxiv:2607.26148 §3.2]. `observe()` returns one 512 x 512 RGB view with a 90
degree field of view and changes nothing. `step()` runs up to 50 primitives in
order: stop, forward 0.25 m, or turn 15 degrees left or right. The reply gives
how many primitives ran and the remaining budget, with no image and no collision
signal [arxiv:2607.26148 app. A]. The model chooses every call, the harness
only runs it, and the episode ends on a STOP or at a 500-step budget
[arxiv:2607.26148 §3.1] [arxiv:2607.26148 app. A]. Success needs the agent's
own STOP within 3 m of the goal [arxiv:2607.26148 app. A]. The agent has no
pose, map, depth or memory [arxiv:2607.26148 §3.2].

On 100 R2R-CE episodes the authors report, with the Claude Agent SDK at default
effort, 70.7 success for opus-5, a mean of three runs, and 68.3 for fable-5
[arxiv:2607.26148 tab. 2]. fable-5 at max effort reaches 78 in one run
[arxiv:2607.26148 tab. 2]. Across models in the mini-swe-agent harness, success
ranges from 5 to 72 [arxiv:2607.26148 tab. 2]. The paper treats harness gaps of
1.7 to 7.3 points as descriptive [arxiv:2607.26148 §4.3]. An optional trained
waypoint tool, added beside the primitives, reaches 76.7 as a mean of three runs
for fable-5, with a median of 48 steps per episode against 87 for primitives
alone [arxiv:2607.26148 tab. 5]. The waypoint module is trained, so that cell is
not zero-shot [arxiv:2607.26148 app. A]. Card:
[Embodied Agents Take Control: Minimal-Interface Zero-Shot Agents Rival
Industrial-Scale Policies in Vision-and-Language Navigation](/papers/2607.26148).

### Tool calls learned through fine-tuning

LLaVA-Plus trains a multimodal model on dialogues that contain tool calls. The
model then chooses tools from its own training, not from a tool list in the
prompt. Each assistant turn has three fields: `thoughts`, `actions` (a JSON
list of `API_name` and `API_params`) and `value` [arxiv:2311.05437 §2.1]. A
tool's output is written as text and sent back as a new human turn, followed by
the original question [arxiv:2311.05437 §2.2]. The skill set covers detection,
segmentation, tagging, captioning, OCR, retrieval, image generation and editing
[arxiv:2311.05437 tab. 2]. Most training labels come from GPT-4
[arxiv:2311.05437 §2.2]. On LLaVA-Bench In-the-Wild, the 7B model with the Fly
tool set scores 59.1 against 57.1 for LLaVA. Removing the thoughts field drops
it to 48.7 [arxiv:2311.05437 tab. 4]. The All Tools setting writes tool outputs
into the image context before the model answers, and it scores 69.5 on the same
benchmark. That setting does not test whether the model chooses tools
[arxiv:2311.05437 §2.3]. Card: [LLaVA-Plus: Learning to Use Tools for Creating
Multimodal Agents](/papers/2311.05437).

LaMDA applies the same idea to grounding in dialogue. Crowdworkers checked
factual claims with a search tool, a calculator and a translator, and edited
replies to cite their sources [arxiv:2201.08239 §5] [arxiv:2201.08239 §6.2].
The fine-tuned model first writes a query from a draft reply. The snippet the
tool returns goes back into the context, and the model then writes either a
grounded reply or another query. The first word of its output picks the
recipient, the tool or the user [arxiv:2201.08239 §6.2]. The worked example
caps this loop at four steps, and the paper does not say what is returned when
the cap is reached [arxiv:2201.08239 §6.2]. If a query cannot be parsed, the
tool returns nothing and no snippet is added [arxiv:2201.08239 §6.2]. At 137B
parameters, the authors report 73.2% groundedness and 65% citation accuracy
[arxiv:2201.08239 app. F] [arxiv:2201.08239 §7]. The model before this
groundedness stage scores 67.9 [arxiv:2201.08239 app. F]. That stage also
trains on crowd-labelled groundedness data, so the gap cannot be credited to
tool access alone. Crowdworkers using the same tools still score higher on
groundedness and informativeness [arxiv:2201.08239 §7]. Card: [LaMDA: Language
Models for Dialog Applications](/papers/2201.08239).

Qwen's chat models learn tool use as one skill among many. The authors write
about 2,000 self-instructed tool-use and agent samples, use the ReAct prompt
format, and add them to the general fine-tuning mix rather than a separate
stage [arxiv:2309.16609 §3.4]. Their tool benchmark leaves out plugins seen in
training and scores the choice among up to five tools, parameter plausibility
and false positives [arxiv:2309.16609 §3.4] [arxiv:2309.16609 tab. 6]. On tool
selection, Qwen-Chat 14B scores 98 against 95 for GPT-4 [arxiv:2309.16609
tab. 6]. The benchmark is self-built and Chinese-focused, and the authors
attribute part of GPT-3.5's low score to that [arxiv:2309.16609 §3.4]. The card
lists no ablation of the tool samples alone [arxiv:2309.16609 §3.4]. Card:
[Qwen Technical Report](/papers/2309.16609).

### Tool use trained by reinforcement learning against a cached sandbox

DeepTravel trains a travel-planning agent end to end with reinforcement
learning. Each turn has a thought, then either a tool call or a final answer,
and the policy reads the tool's response [arxiv:2509.21842 §2]. Six tools sit
behind one interface: flight, train, hotel, route, place and web search
[arxiv:2509.21842 §3.1.1]. Responses are cached, so replays of a trajectory see
the same outputs, and the cache is refreshed every 24 hours
[arxiv:2509.21842 §3.1.3]. The reward is 1 only if a trajectory-level check of
feasibility passes and a turn-level check finds the final plan consistent with
every tool response. Otherwise it is 0 [arxiv:2509.21842 §3.2.3]. Queries that
no sampled trajectory solves go into a replay buffer [arxiv:2509.21842 §3.3.2].
The 32B model reports an 82.58% overall pass rate on 6,224 production queries
[arxiv:2509.21842 tab. 4] [arxiv:2509.21842 §5.2]. Those scores come from the
same LLM reward system used in training, and the authors hand-check 200 of
them, reporting 82% agreement [arxiv:2509.21842 §5.2.1] [arxiv:2509.21842
tab. 3]. In the 8B ablation, removing the turn-level verifier lowers the
constrained medium offline pass rate from 41.89 to 14.24
[arxiv:2509.21842 tab. 6]. Card: [DeepTravel: An End-to-End Agentic
Reinforcement Learning Framework for Autonomous Travel Planning
Agents](/papers/2509.21842).

### Knowledge-graph tools in a fixed three-turn rollout trained with GRPO

SAKE trains Qwen2.5 instruct models with GRPO, with no supervised stage first
[arxiv:2505.15062 §3.4] [§3.5]. A rollout has three model turns and two tool
outputs [arxiv:2505.15062 §3.2]. In the first turn the model names the key
concepts in the query. The first tool returns, for each concept, the concept
plus the most similar knowledge-graph concepts under a pre-trained encoder
[arxiv:2505.15062 §3.1] [eq. 6]. In the second turn the model keeps some of
these groups. The second tool returns every triplet that links two kept groups
[eq. 7]. In the third turn the model reasons by analogy over the triplets and
writes the answer. Both tools are deterministic and carry no gradient, and
their text is masked out of the loss [arxiv:2505.15062 §3.1] [alg. 1]. The
reward adds a format term, which checks that the closing tags are present, to
an exact-match term on the answer. At 7B the weights are 0.2 and 0.8
[arxiv:2505.15062 §3.5] [eq. 9]. The UMLS graph has 135 nodes and 5,877
triplets [arxiv:2505.15062 §4.1].

At 7B the authors report a weighted average of 75.4 on the 224 UMLS test
questions. That is against 70.1 for GPT-3.5-Turbo with GIVE, and 56.7 for GRPO
without the knowledge-graph tools [arxiv:2505.15062 tab. 1]. The GRPO
comparison is a difference in method as well as in tools, and the card cannot
tell whether the rollout format matches, because the paper gives few training
values [arxiv:2505.15062 §3.4]. Replacing agentic retrieval with precomputed
retrieval costs 5.0, 7.1 and 5.0 points on PubMedQA, BioASQ and ProcessBank
[arxiv:2505.15062 §4.3]. The repository has no code path for that ablation
[arxiv:2505.15062 Paper vs code]. Token use is 639 to 1,156 per question on the
three UMLS sets, against 7,010 to 19,461 for ToG and GIVE as the authors ran
them [arxiv:2505.15062 tab. 2]. The weighted average moves by about 0.45 points
per question on 224 questions [arxiv:2505.15062 Pitfalls].

The paper says tool calls are triggered by special tokens, but the shipped
configs stop generation at plain-text closing tags [arxiv:2505.15062 Paper vs
code]. Card: [SAKE: Structured Agentic Knowledge Extrapolation for Complex LLM
Reasoning via Reinforcement Learning](/papers/2505.15062).

### Solver-checked code as the reinforcement learning reward

SIRL fine-tunes Qwen2.5 instruct models with reinforcement learning, without a
supervised stage first [arxiv:2505.11792 §3.2]. Each answer has three tagged
parts: reasoning, a mathematical model and Gurobi Python code [arxiv:2505.11792
§3.2]. The policy does not choose whether to call the solver. Every answer is
code that the solver runs, and the objective it returns is scored [arxiv:2505.11792
§3.2] [arxiv:2505.11792 §3.3].

The reward has a format term of 0.5 when the tags appear in order, an execution
term of 1 when the code runs, and an accuracy term of 2 when the objective
matches the reference. Stage two adds a bonus for advanced modelling forms
[arxiv:2505.11792 §10] [eq. 6]. The KL penalty applies only to the model and
code sections, not to the reasoning [arxiv:2505.11792 §3.2] [eq. 5]. On MAMO
Complex, partial KL gives 51.7% accuracy, against 48.3% for full KL and 47.3%
with no KL [arxiv:2505.11792 tab. 3].

The authors report macro averages over five benchmark columns of 60.6 for the
7B model and 68.3 for the 32B model [arxiv:2505.11792 tab. 1]. The 7B average is
below the 61.1 that the offline OptMATH-Qwen2.5-32B baseline reports in the same
table. The paper gives the training setup only for the 7B model, on 8 H100 GPUs
for about 24 hours per stage [arxiv:2505.11792 §11]. Card:
[Solver-Informed RL: Grounding Large Language Models for Authentic Optimization
Modeling](/papers/2505.11792).

### Plugins with declared functions and usage limits

The multi-agent framework defines a plugin as a tuple of the functions agents
may call, its init-time configuration, and usage constraints such as limits on
compute, input types or output [arxiv:2306.03314 §2.2]. Agents and plugins are
nodes in a graph, and a plugin may also serve as a shared store
[arxiv:2306.03314 §2.3]. The paper proposes a stateless oracle agent to screen
sensitive actions, but does not say how the screen decides or how it was tested
[arxiv:2306.03314 §4.1.2]. It gives no values for the limits and reports no
experiments [arxiv:2306.03314 §5.1]. Card: [Multi-Agent Collaboration:
Harnessing the Power of Intelligent LLM Agents](/papers/2306.03314).

### Action gate before irreversible calls

The memory framework MemTX splits domain tools into reversible and irreversible
sets. The gate runs before every irreversible call [arxiv:2607.23929 §3.3]. It
blocks the call while any tentative memory record that the transaction did not
read exists in the store, including the caller's own uncommitted writes. For the
external-action tier it also needs an action-safe record in the snapshot
[arxiv:2607.23929 §3.3]. A blocked call returns a structured refusal, and the
agent may commit or abort and then retry [arxiv:2607.23929 §3.3]. Reversible
tools are not gated [arxiv:2607.23929 §3.3].

When a belief is withdrawn, the protocol repairs by type. Beliefs are revoked,
derived views are held back for rebuilding, and tool actions are compensated if
reversible or logged as leaked effects if not [arxiv:2607.23929 §3.4]. Because
the gate waits on any tentative record in the store, a stalled agent can hold up
other agents' irreversible calls [arxiv:2607.23929 §3.3].

On the authors' 90-case main suite with Qwen3-8B, task success is 0.956, against
0.822 for the strongest baseline [arxiv:2607.23929 tab. 2]. Realised harm on 24
refund episodes is 0.000 on every backbone, against 0.250 for the baseline on
Qwen3-8B [arxiv:2607.23929 tab. 11]. With GPT-5.5, MemTX scores 0.756 against
0.778, which the authors call a tie [arxiv:2607.23929 tab. 2] [arxiv:2607.23929
§5]. Removing the action gate costs nothing on the main suite, and costs only the
hardened families it covers [arxiv:2607.23929 fig. 3]. Card:
[MemTX: Transactional Belief Commit for Stateful Agent Memory](/papers/2607.23929).

### Guard before each tool call, with a provenance registry

DREAMS is a set of LLM agents that set up and run Quantum ESPRESSO jobs on a
cluster. Each tool output is stored with a provenance record. A guarded call
passes up to three checks before it runs [arxiv:2507.14267 Methods: Safety
guard] [fig. 2]. First, a deterministic check confirms that each referenced
value exists in the registry and equals the recorded one. Second, an LLM judge
checks each parameter against the reason the agent gave. Third, for convergence
selection and equation-of-state fitting, a second judge checks the parameters
together. A failed check stops the tool. A warning registers the value with the
concern in its metadata, and an infrastructure error in the guard does not
block the run [arxiv:2507.14267 Methods: Safety guard]. After a report is
written, a report judge checks each claim against its cited artifact and walks
the provenance graph. That verdict decides acceptance [arxiv:2507.14267 Methods:
Safety guard] [fig. 5].

On CO/Pt(111), the authors report that the guarded run executes and succeeds
on 100% of steps, against 100% executed and 81% succeeded without the guard.
The adsorption-energy MAPE is 0.1% with the guard and 0.2% without
[arxiv:2507.14267 tab. 6]. On the BCC Li lattice-constant task both runs show
0.55% MAPE [arxiv:2507.14267 tab. 6]. The guarded run uses about 13 times the
input tokens, 23.9 million against 1.8 million on CO/Pt(111) [arxiv:2507.14267
tab. 6] [tab. 7]. The production judge, Opus 4.8, makes 2 false positives and
5 false negatives on a 145-scenario adversarial suite [arxiv:2507.14267 fig. 8].
All judges are Anthropic models, and the rules were tuned on CO/Pt(111)-derived
scenarios plus one other chemistry, so other domains are untested
[arxiv:2507.14267 Methods: Judge implementation]. The list of sensitive
parameters is written by experts, and the guard does not find them itself
[arxiv:2507.14267 Discussion]. The call-time judge does not walk upstream
provenance. In one vacuum case it accepted a reference that the report judge
rejected [arxiv:2507.14267 Methods: Safety guard]. Card: [DREAMS: Density
Functional Theory Based Research Engine for Agentic Materials
Simulation](/papers/2507.14267).

### Tools registered as versioned resources

The protocol treats each tool as a registered resource with a name, a
description, an invocation mapping, a version and an evolvability marker. A
tool may be local code, an MCP-connected tool or a skill [arxiv:2604.15034
§3.1]. A change to it is proposed, assessed and committed, and a rejected
candidate leaves the active tool as it was [arxiv:2604.15034 §3.2]
[arxiv:2604.15034 app. G]. The evolving system can edit prompts, tools and
code together, so the tool's share of the reported GAIA gain is not separated
[arxiv:2604.15034 §5.2] [arxiv:2604.15034 tab. 2]. Restoring an earlier version
is defined, but the paper does not show recovery after a bad version goes live
[arxiv:2604.15034 §3.2] [arxiv:2604.15034 §6]. Card: [Autogenesis: A
Self-Evolving Agent Protocol](/papers/2604.15034).

### Fine-tuned API-call writer, with optional retrieved documentation

Gorilla fine-tunes LLaMA-7B on pairs of a plain-language request and the API
call that answers it, built from TorchHub, TensorFlow Hub and HuggingFace model
cards [arxiv:2305.15334 §3.1] [arxiv:2305.15334 §3.2]. The model writes one
call per request. The paper scores that text against the reference APIs and
never runs it, so a deployment must add the run step itself
[arxiv:2305.15334 §3.3]. The retrieval variant appends the top-1 document to
the request. The paper trains one model without documentation and one with it
[arxiv:2305.15334 §3.2] [arxiv:2305.15334 §4].

Zero-shot, Gorilla scores 59.13, 71.68 and 83.79 overall on TorchHub,
HuggingFace and TensorFlow Hub. Zero-shot GPT-4 scores 38.70, 19.80 and 18.20
[arxiv:2305.15334 tab. 1]. Its hallucination rate is 6.98, 10.95 and 5.40,
against 36.55, 37.16 and 78.65 for GPT-4 [arxiv:2305.15334 tab. 1]. These are
the authors' figures on their own benchmark, and no error bars are given
[arxiv:2305.15334 §4].

Documentation at test time helps only in some setups. A model trained without
documentation scores lower with BM25 retrieval on all three hubs, at 37.63,
11.28 and 34.30 [arxiv:2305.15334 tab. 2]. A model trained with the correct
document scores 0 overall zero-shot on every hub, so it needs a retriever at
inference [arxiv:2305.15334 tab. 2]. Oracle retrieval is an upper bound that a
deployment cannot reach [arxiv:2305.15334 §4]. Card: [Gorilla: Large Language
Model Connected with Massive APIs](/papers/2305.15334).

### Built-in tools chosen by a deployed chat model

GLM-4 All Tools is a chat model that, during a conversation, can call a web
browser, a Python interpreter, a text-to-image model or a developer's own
function [arxiv:2406.12793 §2]. The paper says the model plans, calls one or
more tools and uses their results [arxiv:2406.12793 §2]. The training method
for this choice is not stated. The public repository holds only the
client-side dispatcher for the built-in tools, so the learned choice cannot be
checked there [code: demo/composite_demo/src/tools/tool_registry.py#L22-L25].

Author-reported scores put the Python-based tasks close to GPT-4 with the same
tools: GSM8K 91.59 against 92.72, MATH 63.60 against 65.00, and Math23K 88.50
against 88.40 [arxiv:2406.12793 tab. 9]. On the browser task, labelled
Information Seeking, it scores 78.08 against 67.12. The paper does not define
that task or give its size [arxiv:2406.12793 tab. 9] [arxiv:2406.12793 §3.8].
The paper gives no step limit, stop rule or error handling for tool calls
[arxiv:2406.12793 §2]. The Berkeley Function Call Leaderboard table is for the
base chat model, not All Tools: GLM-4 (0520) scores 81.76 overall, against
81.24 for GPT-4 Turbo and 82.94 for GPT-4o [arxiv:2406.12793 tab. 7]. Card:
[ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All
Tools](/papers/2406.12793).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Model API without native tool calling, and you control the loop | Prompt-described tools with a text call format | The paper's only tested method for this setting. Tools are described in the prompt with no demonstrations [arxiv:2303.12712 §5.1] |
| Few worked examples are available, the task is interactive or lookup-based, and a readable trace matters | Interleaved thoughts and actions in a few-shot trace (ReAct) | On ALFWorld, 71% with the best of six prompt sets against 45% for the action-only version [arxiv:2210.03629 tab. 3]. On WebShop, 40.0 against 30.1 [arxiv:2210.03629 tab. 4]. The main results need PaLM-540B, which is not openly accessible [arxiv:2210.03629 app. A.1] |
| Arithmetic or facts the model may get stale or wrong | Enable a calculator or search tool, and say in the prompt that tools are allowed | Without a prompt that allows tools, the model gave stale or miscalculated answers [arxiv:2303.12712 fig. 5.1] |
| Multi-step tasks across several systems | Chained tools, one call per turn | The paper demonstrates this only in single transcripts [arxiv:2303.12712 §5.1.1] |
| Agent with many steps, and you write the loop yourself | Structured calls in a reason-act-observe loop, with your own stop rule and step budget | The review names this loop as a mitigation but specifies no stop rule or budget [arxiv:2505.10468 §6.1]. The Auto-GPT loop uses demonstrations and step limits, but a GPT-4-class model was needed to beat the baseline [arxiv:2306.02224 tab. 1] |
| Text-action environment, a strong chat model, and a supervised expert you can run per step | Structured loop with an expert suggestion written into the prompt | Adds 0.060 to 0.080 WebShop success for GPT-4 on 50 tasks [arxiv:2306.02224 tab. 1]. Test sets are small and no cost is reported [arxiv:2306.02224 §3.1.1] |
| Multimodal model that must pick vision tools, and you can fine-tune on tool dialogues | Tool calls learned through fine-tuning, in a fixed three-field format | The Fly variant beats LLaVA by 2 points on a 60-question benchmark, and the thoughts field matters on that benchmark [arxiv:2311.05437 tab. 4]. The benchmark is the authors' own, and the tool-choice test is the Fly variant, not All Tools [arxiv:2311.05437 §4.1] |
| Dialogue system that must cite external facts, and you can fine-tune on tool-use dialogues | Tool calls learned through fine-tuning, with a search tool and a calculator | Groundedness is 73.2 at 137B, against 67.9 before the groundedness stage [arxiv:2201.08239 app. F]. Citation accuracy is 65% [arxiv:2201.08239 §7]. The stage also changes its training data, so the gain is not tool-only |
| Chat model that must call tools alongside many other skills, and you can add tool samples to its fine-tuning mix | Self-instructed tool samples in the general SFT mix (Qwen) | Author-reported tool selection of 98 for Qwen-Chat 14B against 95 for GPT-4, on a self-built Chinese-focused benchmark [arxiv:2309.16609 tab. 6]. The card lists no ablation of the tool samples alone [arxiv:2309.16609 §3.4] |
| Math or code tasks where a checked answer matters, and you can sandbox execution | Executor agent that runs model-written code in a conversation | Author-reported 69.48% against 55.18% for vanilla GPT-4 on the full MATH test set [arxiv:2308.08155 app. A1]. The paper warns that running code can change the environment [arxiv:2308.08155 app. Ethics statement] |
| Domain agent with live APIs, checkable constraints, and a multi-GPU RL budget | RL against a cached tool sandbox, with a trajectory check and a turn check | The 8B model's offline human-evaluation score rises from 58.00 to 70.00 with RL [arxiv:2509.21842 tab. 5]. The authors use about 32 H800 GPUs for the 32B model [arxiv:2509.21842 §4]. Online pass rates are author-reported |
| Table QA where the query language cannot express a per-row judgement | Model calls inside a generated program | Binder reaches 64.6 on WikiTQ test against 61.1 for Codex SQL without calls [arxiv:2210.02875 §3.2]. Author-reported, and the model used is retired (see Contradictions) |
| Several specialised agents sharing a goal | Multi-agent orchestration, tested on your own tasks first | The review describes the design [arxiv:2505.10468 §3.1.2] and lists scaling, debugging and trust as open challenges [arxiv:2505.10468 §5.0.2] |
| Tools change over time and may need rollback | Versioned tool registry with assess, commit and restore | The protocol defines these operations but does not demonstrate rollback [arxiv:2604.15034 §3.2] [arxiv:2604.15034 §6] |
| Tools with side effects or cost limits | Plugins with declared usage constraints, plus a screen on sensitive calls | The framework names the limits and the screen but gives no values and no screening method [arxiv:2306.03314 §2.2] [arxiv:2306.03314 §4.1.2] |
| Planning over many tools, where each question needs a different set, and you can supply per-task default plans | Planner-chosen module sequence from a fixed inventory (Chameleon) | Author-reported 86.54 on ScienceQA against 83.99 for GPT-4 chain-of-thought, and 98.78 on TabMWP against 96.93 for GPT-4 program-of-thought [arxiv:2304.09842 tab. 3] [arxiv:2304.09842 tab. 4]. The plan is written once, with no re-planning during the run [arxiv:2304.09842 §3] |
| Camera-only agent in a simulator, with no map or memory, and a model you can run for hundreds of steps | Minimal observe and step primitives | The mean of three runs gives opus-5 70.7 success at default effort, and fable-5 reaches 78 at max effort in one run [arxiv:2607.26148 tab. 2]. Only the default-effort Claude cells have repeats, and the authors treat differences of a few points as descriptive [arxiv:2607.26148 §4.1] |
| Shared agent memory that drives irreversible actions, such as refunds | Action gate before irreversible calls, with typed repair (MemTX) | On the authors' 90-case suite, Qwen3-8B reaches 0.956 against 0.822 for the best baseline [arxiv:2607.23929 tab. 2]. Realised harm is 0.000 on 24 refund episodes, against 0.250 for the baseline on Qwen3-8B [arxiv:2607.23929 tab. 11]. With GPT-5.5 the result is a tie [arxiv:2607.23929 §5] |
| Embodied agent that must reach or find objects with pretrained skills, no task training, and an LLM you can run per task | Few-shot program over pretrained skills (TANGO) | Author-reported zero-shot OVON success of 35.5 against 35.2 for VLFM [arxiv:2412.10402 tab. 1]. GOAT-Bench success of 32.1 against 24.9, with SPL 16.5 against 17.2 [arxiv:2412.10402 tab. 3]. The released evaluator runs fixed programs, not the LLM, so test the planner yourself [arxiv:2412.10402 §3] |
| Math or optimisation modelling with a checkable objective, and a multi-GPU RL budget | Solver-checked code as the RL reward (SIRL) | Author-reported macro average of 60.6 for 7B and 68.3 for 32B over five benchmark columns [arxiv:2505.11792 tab. 1]. The paper gives the training setup only for 7B, on 8 H100 GPUs for about 24 hours per stage [arxiv:2505.11792 §11] |
| Trained agent that answers questions over a knowledge graph you can query deterministically, with two 48 GB GPUs for 7B | Knowledge-graph tools in a fixed three-turn GRPO rollout (SAKE) | Author-reported 75.4 weighted average on UMLS QA at 7B, against 70.1 for GPT-3.5-Turbo with GIVE [arxiv:2505.15062 tab. 1]. Token use is 639 to 1,156 per question, against 7,010 to 19,461 for ToG and GIVE [arxiv:2505.15062 tab. 2]. The GRPO baseline without tools scores 56.7, but differs in method too, and the UMLS test set has 224 questions |
| Scientific simulation agent where a wrong input can pass unnoticed, and you can pay about 13 times the input tokens | Guard before each tool call, with a provenance registry (DREAMS) | On CO/Pt(111), 100% of steps succeeded with the guard against 81% without it, at about 13 times the input tokens [arxiv:2507.14267 tab. 6]. The judges are Anthropic models and were tuned on CO/Pt(111)-derived scenarios [arxiv:2507.14267 Methods: Judge implementation] |
| Unfamiliar tool semantics | Return clear errors as tool results and expect retries | The model misused an unusual tool until it got feedback [arxiv:2303.12712 fig. 5.7] |
| Calls that must be correct on every request | Add a check on each call, outside the model | The model's choice of when to use a tool is weak: it still searched for the capital of France [arxiv:2303.12712 §5.1.2]. The review lists reliability and safety as agent limitations [arxiv:2505.10468 §5.0.1] |
| Heavy arithmetic or exact counting, with a code-capable model and a way to run code | Model-written code run by an interpreter (PAL) | Author-reported 72.0% against 65.6% for chain-of-thought on GSM8K, same model [arxiv:2211.10435 §5.1]. The GPT-4 probe proposed this without evaluation [arxiv:2303.12712 app. D.4]. PAL's gain depends on the model, see Contradictions. Test it on your own tasks first |
| Small model that must write correct calls to a fixed set of documented APIs, and you can fine-tune | Fine-tuned API-call writer (Gorilla), used zero-shot unless you run a retriever you trust | Zero-shot Gorilla scores 59.13, 71.68 and 83.79 against 38.70, 19.80 and 18.20 for GPT-4 on the authors' APIBench [arxiv:2305.15334 tab. 1]. A model trained with documentation scores 0 without it, and BM25 at test time lowers a model trained without documentation [arxiv:2305.15334 tab. 2] |
| Chat model with built-in browser, Python and image tools, and no way to train your own | Built-in tool set chosen by the deployed model (GLM-4 All Tools) | Author-reported Python scores within 1.5 points of GPT-4 with the same tools on GSM8K, MATH and Math23K [arxiv:2406.12793 tab. 9]. The browser task is not defined, and the paper states no step limit [arxiv:2406.12793 §2] |
| Agents that must call tools exposed by different servers | Compare MCP, ACP and A2A on your own messages before choosing | The survey describes each protocol and gives no measured comparison. MCP uses JSON-RPC 2.0 messages [arxiv:2504.19678 §IV.C] [arxiv:2504.19678 tab. 12] |
| Before deployment, a score for how well an agent uses its tools | Proposed Tool Utilization Efficacy score, with weights you set | The score is linear over selection, argument validity, execution, outcome and efficiency. Its weights and efficiency normalisation are not stated, and it is untested [arxiv:2506.04133 §5] [eq. 5] |

## Implementation sketch

The loop below follows the paper's description. The paper gives no iteration
cap, so the `max_steps` guard is our addition. The `<|endofprompt|>` separator
appears in the paper's prompts, but its role is not explained [fig. 5.2].

The structured-call variants run the same loop. The difference is that the call
is a structured object, parsed from JSON or a similar format, rather than text
matched by a pattern [arxiv:2505.10468 §2.1.2]. The review gives no stop rule
or memory format, so those are choices for the implementer [arxiv:2505.10468
§6.1]. The ReAct variant also runs this loop, but its call is a text action in
a fixed trace format, and each thought is written into the trace without
returning an observation [arxiv:2210.03629 §2]. The learned variants keep this
loop at serving time, but the choice of tool comes from training: LLaVA-Plus
learns it from tool-call dialogues [arxiv:2311.05437 §2.1], LaMDA learns it
from its tool-use data [arxiv:2201.08239 §6.2], and DeepTravel learns it from
the reward [arxiv:2509.21842 §3.2.3]. SAKE fixes the order of its two tools and
learns which entities to name and which groups to keep [arxiv:2505.15062 §3.2].
The All Tools loop in ChatGLM follows the
same shape, with each tool result added to the conversation before the next
call. The card gives no step limit or error rule [arxiv:2406.12793 §2].

The model-calls-inside-a-program variant does not use this loop. It generates
several programs in one pass, runs each, and votes on the answers
[arxiv:2210.02875 §2.2] [arxiv:2210.02875 app. A.3]. PAL does not loop either.
It generates one program per question for its main results, runs it once, and
returns the value, and it votes over sampled programs only for its 80.4% result
[arxiv:2211.10435 §3] [arxiv:2211.10435 §5.1].

Chameleon also plans once. Its planner writes a module sequence, the modules
run in order, and the result is the last output. An invalid plan is replaced by
the task's default sequence [arxiv:2304.09842 §3] [arxiv:2304.09842 §4.2]. Parse
the plan against a whitelist of module names. The released code evaluates the
plan text directly, which the [Chameleon card](/papers/2304.09842) flags.

```pseudocode
# Describe the tools in the prompt, with no demonstrations [§5.1]
prompt = tool_instructions + "<|endofprompt|>" + question
for step in range(max_steps):        # our guard; no cap in the paper
    # Generate until a tool call or the end of the answer [fig. 5.2]
    out = generate(prompt)
    if not ends_with_tool_call(out):
        return out
    # Run the tool outside the model [§5.1]
    result = execute(parse_call(out))
    # Append the call and its result, then continue generating
    prompt = prompt + out + "\n> " + result
raise StepLimitExceeded
```

MemTX puts a gate in front of the irreversible calls inside this loop
[arxiv:2607.23929 §3.3]. A call that the gate refuses returns its reason to the
model, and the model may commit or abort before retrying.

```pseudocode
# Gate before one irreversible tool call  [§3.3]
def gate_allows(tx, tool, store):
    # Reversible tools are never gated  [§3.3]
    if tool.reversible:
        return True
    # Any tentative record the transaction did not read blocks the call  [§3.3]
    for record in store.records:
        if record.state == "tentative" and record.id not in tx.snapshot:
            return False
    # The external-action tier also needs an action-safe record  [§3.3]
    if tx.risk_tier == "external_action":
        return any(store.is_action_safe(rid) for rid in tx.snapshot)
    return True
```

## Contradictions

The cards disagree in several places, and none of the disagreements is resolved
here.

The GPT-4 probe reports that the model often failed to decide when a tool was
needed [arxiv:2303.12712 §5.1.2]. The navigation study finds a similar gap at
the stop decision: 23 of 30 failures in one run claim success while 3.1 to 34.1
m from the goal [arxiv:2607.26148 §4.6] [arxiv:2607.26148 tab. 7]. The review presents tool-augmented reasoning
as a mitigation for agent limitations [arxiv:2505.10468 §6.1], and it gives no
measurement that would test that claim. Neither result settles the other. Qwen's benchmark gives Qwen-Chat 14B 98
against 95 for GPT-4 on tool selection [arxiv:2309.16609 tab. 6]. That score
covers which tool is picked from a set, not whether a call is needed, so it
does not settle the probe's point.
Binder avoids the run-time choice by writing its calls into the program before
it runs [arxiv:2210.02875 §2.1]. Its error analysis still lists missing API use
among the causes of failure [arxiv:2210.02875 §4.2], so the paper does not show
that writing calls up front removes the problem.

Worked examples are the first disagreement. The GPT-4 probe's tool prompt has
none [arxiv:2303.12712 §5.1]. ReAct relies on one to six examples per task
[arxiv:2210.03629 §3.2]. The Auto-GPT authors found that tool instructions
without examples performed much worse [arxiv:2306.02224 §2.2]. The models and
tasks differ, and no card compares examples with no examples for the same model.

The second is ReAct's ALFWorld score. AutoGen reports 54 for ReAct, averaged
over three attempts on the 134 unseen tasks, and 66 for its best attempt
[arxiv:2308.08155 app. A3]. The ReAct paper reports 57 averaged over six prompt
sets and 71 for the best set [arxiv:2210.03629 tab. 3]. The averaging differs,
so the two sets of numbers are not interchangeable.

Whether a tool helps depends on the model and the benchmark in the measured
cards. In Auto-GPT, the tool loop beats the imitation baseline only with GPT-4
[arxiv:2306.02224 tab. 1]. PAL shows the same dependence on the model. With
text-davinci-001 it scores 8.6% on GSM8K against 26.5% for chain-of-thought,
and with text-davinci-002 it scores 65.8% against 46.9% [arxiv:2211.10435 §6]. The navigation study shows the same pattern. Forcing
waypoints raised success by 17 to 38 points for the Qwen and GPT rows, by 8.7 to
9.3 for sonnet-5 and opus-4.8, and by 0.7 to 1.3 for fable-5 and opus-5
[arxiv:2607.26148 tab. 4]. In LLaVA-Plus, giving base LLaVA the tool outputs at
test time lowers its scores on LLaVA-Bench, from 81.2 to 59.1 on COCO and from
57.1 to 47.5 on In-the-Wild, but raises its SEED-Bench average from 44.45 to
51.54 [arxiv:2311.05437 tab. 4]. In ReAct, scores fall below chain-of-thought
on HotpotQA, at 27.4 against 29.4, and rise above it on FEVER, at 60.9 against
56.3 [arxiv:2210.03629 tab. 1]. Its chain-of-thought
baseline uses no tools, so that comparison changes the tools and the trace
together. The cards do not explain these directions. Only PAL's runner
ablation and Chameleon's module removals change one tool alone. Each covers
one model, and PAL's covers one benchmark.

TANGO's numbers do not test its LLM step. The paper says the LLM writes the
program [arxiv:2412.10402 §3]. The released evaluator builds the generator in
a debug mode that returns a fixed program per task, and the task templates
insert the episode's target name [code: habitat-baselines/habitat_baselines/rl/ppo/code_interpreter/code_generator.py#L92-L109]
[code: habitat-baselines/habitat_baselines/rl/ppo/utils/utils.py#L354-L355].
The paper does not say whether the agent receives the target label. Its
results therefore measure the modules and a fixed program, not the planner.

Binder's numbers come from code-davinci-002, which the card records as retired,
and the official evaluator changes its scores by 1 to 2.5 points
[arxiv:2210.02875 §3.1] [arxiv:2210.02875 app. C.2]. Reproducing them needs a
substitute model, so they do not transfer directly.

The measured cards do not test the same thing as each other. Binder tests table
QA. Auto-GPT, ReAct, AutoGen, DeepTravel, MemTX, TANGO and the navigation study
compare whole agents with baselines. Chameleon and SIRL compare whole methods with
theirs.
LaMDA compares its groundedness stage with the model before it, but that stage
also changes the training data. LLaVA-Plus trains a model, and the protocol
tests whole agent systems with several edits at once. PAL compares code with
text reasoning, so its main comparison changes the prompt format as well as the
executor. Only its runner ablation on GSM8K isolates the executor. Gorilla scores a
call without running it, and ChatGLM's All Tools results come from a whole
model with its tools [arxiv:2305.15334 §3.3] [arxiv:2406.12793 tab. 9]. None of
the other cards isolates a tool-only effect.

Two cards compare a learned model with GPT-4 on choosing a tool or an API, and
the margins differ. Gorilla's fine-tuned 7B model beats zero-shot GPT-4 by 20 to
66 points on its own API benchmark [arxiv:2305.15334 tab. 1]. Qwen-Chat 14B
scores 98 on its own tool-selection benchmark, against 95 for GPT-4
[arxiv:2309.16609 tab. 6]. The benchmarks differ in task and author, so this is
not one test with two answers. Compare numbers only within each card.

## Papers

- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](/papers/2303.12712): an early GPT-4 study that describes prompt-only tool use, shows chained tool sessions, and reports that tool use needs prompting and that the model's judgement about when to call tools is weak.
- [PAL: Program-aided Language Models](/papers/2211.10435): few-shot prompts in which the model writes Python steps with comments and an interpreter produces the answer. It reports author-measured results on math and symbolic tasks with code-davinci-002, and an ablation that changes only the runner.
- [AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges](/papers/2505.10468): a narrative review that describes the reason-act-observe tool loop, its multi-agent extension and the open challenges. It reports no experiments.
- [A Survey on Large Language Model based Autonomous Agents](/papers/2308.11432): a 2023 survey whose action module separates external tools from the model's own abilities. It reports no experiments and covers work up to August 2023.
- [Binding Language Models in Symbolic Languages](/papers/2210.02875): Binder, a training-free method in which a generated SQL or Python program calls the same model for steps the language cannot express. It reports author-measured table QA accuracy.
- [Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents](/papers/2306.03314): a conceptual framework that defines plugins as tuples of callable functions, configuration and usage constraints. It reports no experiments.
- [Autogenesis: A Self-Evolving Agent Protocol](/papers/2604.15034): a protocol that registers tools, prompts, agents, environments and memory as versioned resources, updated through propose, assess and commit. Its agent gains are author-reported and do not isolate tool changes.
- [Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models](/papers/2304.09842): an LLM planner writes a module sequence over thirteen module types, run in order. It reports author-measured ScienceQA and TabMWP accuracy with GPT-4 and ChatGPT, and module-removal ablations.
- [Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling](/papers/2505.11792): an RL method in which the reward comes from running generated solver code. It reports author-measured accuracy on five benchmark columns for 7B and 32B models.
- [MemTX: Transactional Belief Commit for Stateful Agent Memory](/papers/2607.23929): a commit protocol for agent memory with an action gate before irreversible tool calls and typed repair. It reports author-measured results on the authors' own 90-case suite.
- [Embodied Agents Take Control: Minimal-Interface Zero-Shot Agents Rival Industrial-Scale Policies in Vision-and-Language Navigation](/papers/2607.26148): a zero-shot navigation study in which a coding-agent harness calls observe and step tools on R2R-CE. It reports author-measured success rates on 100 episodes, with an optional trained waypoint tool.
- [TANGO: Training-free Embodied AI Agents for Open-world Tasks](/papers/2412.10402): an LLM writes a short program over a pretrained navigation policy, a memory-augmented value map and detectors. It reports author-measured zero-shot results on three embodied benchmarks, and its released evaluator does not call the LLM.
- [Qwen Technical Report](/papers/2309.16609): a family of base and chat models whose chat variants mix about 2,000 tool-use samples into fine-tuning. It reports author-measured tool-selection results on its own benchmark, among many other benchmarks.
- [ReAct: Synergizing Reasoning and Acting in Language Models](/papers/2210.03629): few-shot prompts that interleave free-text thoughts with Wikipedia lookups and environment actions. It reports author-measured results on two question-answering tasks and two interactive tasks, using PaLM-540B.
- [LaMDA: Language Models for Dialog Applications](/papers/2201.08239): a dialogue model fine-tuned to query a search tool, a calculator and a translator before replying. It reports author-measured groundedness and citation accuracy from crowd labels.
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](/papers/2308.08155): a framework in which agents, some of which run code, exchange messages. It reports author-measured MATH and ALFWorld results for its agent designs.
- [Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions](/papers/2306.02224): an Auto-GPT-style JSON tool loop on WebShop and ALFWorld, with an optional supervised suggestion in the prompt. It reports author-measured success rates on small test sets.
- [LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents](/papers/2311.05437): a multimodal model fine-tuned on tool-call dialogues in a three-field format. It reports author-measured benchmark scores, and its strongest setting pre-computes tool outputs.
- [DeepTravel: An End-to-End Agentic Reinforcement Learning Framework for Autonomous Travel Planning Agents](/papers/2509.21842): an RL-trained travel agent with a cached tool sandbox and a two-stage reward. It reports author-measured online and offline pass rates, graded by an LLM judge.
- [Gorilla: Large Language Model Connected with Massive APIs](/papers/2305.15334): a LLaMA-7B model fine-tuned to write one API call per request, with an optional retrieved document in the prompt. It reports author-measured accuracy and hallucination rates on its own APIBench.
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](/papers/2406.12793): a model family whose GLM-4 All Tools variant chooses among a browser, Python, an image model and user functions. It reports author-measured benchmark results and gives few training details.
- [Agentic Large Language Models, a survey](/papers/2503.23037): a 2025 survey that places tool calling in its acting category. It reports no experiments of its own.
- [From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review](/papers/2504.19678): a 2025 review that describes an agent loop and compares the MCP, ACP and A2A protocols. It reports no experiments of its own.
- [TRiSM for Agentic AI](/papers/2506.04133): a 2025 review of trust and security for multi-agent LLM systems. Its threat list includes tool misuse, and it proposes a Tool Utilization Efficacy score. It reports no experiments.
- [Empowering Biomedical Discovery with AI Agents](/papers/2404.02831): a 2024 perspective that defines four autonomy levels for biomedical agents and lists tool use among four modules. It reports no experiments.
- [SAKE: Structured Agentic Knowledge Extrapolation for Complex LLM Reasoning via Reinforcement Learning](/papers/2505.15062): a GRPO-trained Qwen2.5 model that calls two deterministic knowledge-graph tools in a fixed three-turn rollout. It reports author-measured QA accuracy and token use on UMLS and ConceptNet.
- [DREAMS: Density Functional Theory Based Research Engine for Agentic Materials Simulation](/papers/2507.14267): a multi-agent system for periodic DFT in which each tool call passes a provenance and judge guard. It reports author-measured lattice-constant and adsorption-energy results, and the token cost of the guard.