Tool use

Overview

Tool use lets a language model act on the outside world or fetch information it does not have. The model writes a call in plain text or as structured arguments, such as a search query or an arithmetic expression. In the common loop, generation pauses, the tool runs, and its output is pasted into the context before generation continues 2303.12712 §5.1 fig. 5.2 2505.10468 §2.1.2. No weights change in that loop: the protocol lives in the prompt and in the code that parses the model's output. Some variants do train the model to make calls, as described below.

Twenty-seven cards on wikillm cover this topic, and they differ in kind. Two are a probe and a review of prompt-level tool use. The probe is of a pre-release, text-only GPT-4 with a handful of tool transcripts and no tool-use accuracy figure 2303.12712 §5.1. The review is narrative and reports no experiments 2505.10468 §1.1. A survey puts tool use in its taxonomy of agent actions, where the action space is either external tools or the model's own abilities 2308.11432 §2.1.4. It reports no experiments of its own. The plugin framework proposes a design and reports no experiments either 2306.03314 §2.2 2306.03314 §5.1. Three more surveys and reviews from 2025 and 2026 also report no experiments. A 2024 perspective on biomedical AI agents lists tool use as one of four modules and reports no experiments 2404.02831 §5.2. One lists tool calling among agent actions 2503.23037 §3.2. One compares agent protocols 2504.19678 §IV.C. One lists tool misuse among its threats 2506.04133 §3. The other nineteen report measured results. Gorilla fine-tunes a 7B model to write API calls and is scored on the authors' own benchmark 2305.15334 tab. 1. ChatGLM's All Tools is a deployed chat model that picks its own built-in tools 2406.12793 §2. TANGO has an LLM write a short program over navigation and perception skills, and reports zero-shot results on three embodied benchmarks 2412.10402 §3 2412.10402 tab. 1. ReAct interleaves free-text thoughts with lookups and reports question-answering and interactive-task scores 2210.03629 tab. 1 2210.03629 tab. 3. LaMDA is a dialogue model fine-tuned to query a search tool, a calculator and a translator 2201.08239 §6.2. AutoGen runs model-written code in a conversation 2308.08155 §2.1. PAL has the model write Python as its reasoning, and an interpreter, not the model, produces the answer 2211.10435 §3. Binder is a training-free method for table question answering in which the model's program calls the model again 2210.02875 §2.1. Auto-GPT-style agents run a tool loop on two simulated decision tasks 2306.02224 §2.2. LLaVA-Plus is a multimodal model fine-tuned to emit tool calls 2311.05437 §2.1. Qwen's chat models mix about 2,000 tool-use samples into a general fine-tuning set 2309.16609 §3.4. DeepTravel is a travel-planning agent trained with reinforcement learning against a cached tool sandbox 2509.21842 §3.1. SAKE trains a Qwen2.5 model with reinforcement learning to call two knowledge-graph tools in a fixed order 2505.15062 §3.2. DREAMS puts a guard in front of each tool call that a materials-simulation agent makes 2507.14267 Methods: Safety guard. Autogenesis registers tools as versioned resources and reports agent gains that cannot be split by tool 2604.15034 §5.2. Chameleon has a planner write a module sequence over a fixed inventory, and reports 86.54 on ScienceQA and 98.78 on TabMWP with GPT-4 [arxiv:2304.09842 tab. 3] 2304.09842 tab. 4. SIRL trains a model to write solver code, with the solver's result as the reward 2505.11792 §3.3. MemTX blocks irreversible tool calls while uncommitted memory exists [arxiv:2607.23929 §3.3]. The navigation study gives a model two tools and nothing else 2607.26148 §3.2. Each of these mixes tool use with other changes, apart from the PAL runner ablation 2211.10435 §6 and Chameleon's single-module removals 2304.09842 tab. 5. The navigation study's waypoint arm also changes the observation 2607.26148 app. A. Binder's numbers depend on a retired base model 2210.02875 §3.1. PAL's numbers use code-davinci-002, whose shutdown the authors' code notes anticipates 2211.10435 §4. The Auto-GPT, LLaVA-Plus, Qwen, DeepTravel, Chameleon, SIRL, MemTX, TANGO and navigation results are cited in their variants below. Gorilla and ChatGLM compare whole models, so their gains cannot be credited to tool access alone. No card gives a general tool-use success rate, so treat the claims below as observations of one model at one point in time, as results on one task family, or as proposals.

The shared invariant across variants: the model proposes the call, but the harness executes it and decides what text goes back into the context.

Other tool-use work is not summarized here.

Variants

Prompt-described tools with a text call format

The prompt lists each tool with its name and argument format, for example SEARCH("query"), CALC(expression) and CHARACTER(string, index). The prompt has no worked examples of tool use. The paper's loop reads the model's output up to the next tool call, runs the tool and appends the result 2303.12712 §5.1 fig. 5.2. Card: Sparks of Artificial General Intelligence: Early experiments with GPT-4.

Interleaved thoughts and actions in a few-shot trace

ReAct keeps a text call format but adds worked examples. Each example is a sequence of thought, action and observation lines from one solved task 2210.03629 §2 2210.03629 app. C. A thought is free text. It changes nothing in the environment and returns no observation, but later steps can read it 2210.03629 §2. The actions come from a fixed set. For question answering they are a Wikipedia search, a lookup within the current page and a finish step 2210.03629 §3.1. The prompt holds one to six examples per task, and the authors say more examples did not help on HotpotQA 2210.03629 §3.2.

With a frozen PaLM-540B, ReAct scores 27.4 exact match on HotpotQA, below the 29.4 of chain-of-thought, and 60.9 accuracy on FEVER, above the 56.3 of chain-of-thought 2210.03629 tab. 1. Combining it with chain-of-thought self-consistency gives the best prompting result on both tasks, at 35.1 and 62.0 2210.03629 tab. 1. On ALFWorld, the best of six prompt sets reaches 71% success, against 45% for the same trajectories with thoughts removed 2210.03629 tab. 3. The average over the six sets is 57% 2210.03629 tab. 3. On WebShop, ReAct reaches 40.0 success against 30.1 for the action-only version 2210.03629 tab. 4. The failures change shape. In a hand-labelled sample of 200 HotpotQA items, reasoning errors, including repeated steps, make up 47% of ReAct's failures. Invented facts make up 56% of chain-of-thought's failures and none of ReAct's [arxiv:2210.03629 tab. 2].

The main results need PaLM-540B, which the paper's reproducibility statement says is not openly accessible. The public check uses GPT-3 text-davinci-002 and reports ALFWorld success of 78.4 against 70.9 for PaLM 2210.03629 app. A.1 2210.03629 tab. 5. Search results come from a live Wikipedia, which can return newer facts, so repeat runs need a frozen snapshot 2210.03629 app. A.2. Card: ReAct: Synergizing Reasoning and Acting in Language Models.

Chained multi-tool sessions

The same mechanism is used with several tools in one session. The paper shows a command-line "zoo manager" task, an email and calendar task, and web browsing 2303.12712 §5.1.1 fig. 5.3 fig. 5.4 fig. 5.5. Each is a single transcript, so success rates across runs are not known.

Model-written code run by an external executor

For arithmetic, the authors suggest that the model write code which a separate executor runs, with the result appended to the prompt. They give one proof-of-concept prompt and say they did not evaluate it systematically 2303.12712 app. D.4. Treat this as a proposal, not a measured result.

AutoGen makes the executor an agent. The code-writing agent sends code to a user-proxy agent that runs it and returns the output as a message, so the run is a conversation rather than a fixed pipeline 2308.08155 §2.1 2308.08155 §3. On the full MATH test set of 5,000 problems, the authors report 69.48% accuracy for GPT-4 with this setup, against 55.18% for vanilla GPT-4 2308.08155 app. A1. That is the only full-set number in the paper. The smaller 120-problem comparisons appear only in a figure 2308.08155 app. A1 fig. 4a. The paper warns that agents which run code or call functions can change the environment, for example by installing packages 2308.08155 app. Ethics statement. Card: AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation.

PAL is a training-free version of this idea. Each few-shot example is a question followed by a Python program, with a comment on each line that gives a step in words. The program's value is the answer, and no final answer is written in the examples 2211.10435 §3. The main results run one program per question after generation, in a standard interpreter 2211.10435 §3. With code-davinci-002 and greedy decoding, PAL solves 72.0% of GSM8K against 65.6% for chain-of-thought with the same model 2211.10435 §5.1. Sampling 40 programs and taking the most common answer gives 80.4%, against 78.0% for chain-of-thought with the same vote 2211.10435 §5.1. On five symbolic and algorithmic tasks, PAL is ahead of chain-of-thought on each. Penguins goes from 79.2% to 93.3% 2211.10435 §5.2. The one ablation that keeps the program fixed and changes only who runs it on GSM8K is the closest this set of cards has to a tool-only test. When the model writes the answer after reading its own program, the score falls to 23.2% 2211.10435 §6. Card: PAL: Program-aided Language Models.

Planner-chosen module sequence from a fixed inventory

Chameleon gives an LLM planner an inventory of thirteen modules. They cover LLM prompts, an image captioner, OCR, web search, a Python program generator, verifier and executor, and rule-based table helpers 2304.09842 §4.1 2304.09842 tab. 2. The planner receives a task instruction, the module descriptions with their constraints and a few demonstrations. It returns a list of module names once, and each name is bound to a module by string matching 2304.09842 §3 eq. 1. Each module reads the current input and a cache, and update functions prepare the input and cache for the next module eq. 2 eq. 3 eq. 4. The last module's output is the answer eq. 5. The planner is not called again during the run 2304.09842 §3.

Each task has its own check on the plan. On ScienceQA a valid plan ends with Solution Generator and then Answer Generator, and that pair is also the fallback. On TabMWP a valid plan includes Answer Generator and puts Program Generator before the verifier and executor. Otherwise a program-of-thought sequence with verification runs 2304.09842 §4.2 2304.09842 §4.3.

With GPT-4 as the LLM, the authors report 86.54% on ScienceQA, against 83.99% for GPT-4 chain-of-thought, and 98.78% on TabMWP, against 96.93% for GPT-4 program-of-thought 2304.09842 tab. 3 2304.09842 tab. 4. With ChatGPT the figures are 79.93% and 93.28% 2304.09842 tab. 3 2304.09842 tab. 4. The fine-tuned MM-COT model scores higher on ScienceQA, at 91.68% 2304.09842 tab. 3. Plans vary with the question: GPT-4 wrote 11 distinct module sequences on ScienceQA and 19 on TabMWP, against 14 and 28 for ChatGPT 2304.09842 tab. 7. In an ablation on 500 test examples with ChatGPT, removing one module at a time costs 7.4 points for Bing search on ScienceQA, 8.4 for the text detector and 7.4 for the program generator on TabMWP 2304.09842 §5.2 2304.09842 tab. 5. Card: Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models.

Few-shot program over pretrained embodied skills

TANGO applies the program-composition idea to embodied navigation. An LLM receives an instruction and a fixed prompt of 15 in-context examples, and outputs a step-by-step program with comments 2412.10402 §3 fig. 2. Each line calls one module, such as detect, navigate_to or explore_scene, and each module writes its output to a named variable 2412.10402 §3 fig. 3. The paper describes one program per task. The navigation core is a PointGoal policy pretrained on HM3D and not trained on the test scenes 2412.10402 §4.1. Exploration follows VLFM, with a per-pixel feature memory added so that several targets can be remembered 2412.10402 §3. The method is training-free for the task itself 2412.10402 §1.

Author-reported zero-shot results. On HM3D-OVON val unseen, TANGO has 35.5 success and 19.5 SPL, against 35.2 and 19.6 for VLFM 2412.10402 tab. 1. DAgRL+OD is higher on both measures there 2412.10402 tab. 1. On GOAT-Bench val unseen, success is 32.1 against 24.9 for the best prior row, but SPL is 16.5 against 17.2 2412.10402 tab. 3. On OpenEQA the score is 37.2 against 38.1 for Socratic LLMs with frame captions [arxiv:2412.10402 tab. 2]. The text calls the OVON result on par with VLFM and the OpenEQA result second 2412.10402 §4.2. In a failure analysis of EQA episodes, the detect module is the main cause, and the LLM wrote incorrect pseudo-code in about 18% of failures. The sample size is not stated 2412.10402 §4.3. Card: TANGO: Training-free Embodied AI Agents for Open-world Tasks.

Structured calls in a reason-act-observe loop

The review describes an agent cycle. The model reasons, emits a tool invocation with structured arguments, such as JSON, the result is integrated into the context, and the next step follows 2505.10468 §2.1.2. It lists tool-augmented reasoning and this reason-act-observe loop among its ten mitigation mechanisms 2505.10468 §6.1. It does not state a stop rule, a step budget or a memory format 2505.10468 §6.1. The review extends the same loop to several specialised agents with shared memory and an orchestrator 2505.10468 §3.1.2, and gives no evaluation of that design. Card: AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges.

Auto-GPT-style agents run this loop with a JSON reply at each step. The reply holds a thoughts object and a command with a name and arguments. The named tool runs, and its observation goes back into the history [arxiv:2306.02224 §2.2]. The authors found that tool instructions without examples performed much worse, so they add one to three demonstrations per tool [arxiv:2306.02224 §2.2]. The loop ends on a finish command or at a step limit, set to 20 steps on WebShop and 35 on ALFWorld 2306.02224 §3.2.1 [arxiv:2306.02224 §3.2.2]. On both tasks, GPT-4 with this loop scores above the imitation baseline, but only by small margins on WebShop, which has 50 test instructions. Claude and GPT-3.5 score below it on both tasks, and Vicuna scores zero 2306.02224 tab. 1 2306.02224 tab. 2. The loop can also take a supervised model's suggested action as a reference in the prompt, with no fine-tuning. For GPT-4 this lifts WebShop success from 0.240 to 0.300 with one suggestion and to 0.320 with five, and ALFWorld success from 0.485 to 0.515 with one 2306.02224 tab. 1 2306.02224 tab. 2. The test sets are small, and the authors' own caution about gaps of a few tasks applies 2306.02224 §3.1.1 2306.02224 §3.1.2. Card: Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions.

Minimal observe and step primitives in an embodied loop

In a zero-shot navigation study, the model gets two tools and nothing else 2607.26148 §3.2. observe() returns one 512 x 512 RGB view with a 90 degree field of view and changes nothing. step() runs up to 50 primitives in order: stop, forward 0.25 m, or turn 15 degrees left or right. The reply gives how many primitives ran and the remaining budget, with no image and no collision signal 2607.26148 app. A. The model chooses every call, the harness only runs it, and the episode ends on a STOP or at a 500-step budget 2607.26148 §3.1 2607.26148 app. A. Success needs the agent's own STOP within 3 m of the goal 2607.26148 app. A. The agent has no pose, map, depth or memory 2607.26148 §3.2.

On 100 R2R-CE episodes the authors report, with the Claude Agent SDK at default effort, 70.7 success for opus-5, a mean of three runs, and 68.3 for fable-5 2607.26148 tab. 2. fable-5 at max effort reaches 78 in one run 2607.26148 tab. 2. Across models in the mini-swe-agent harness, success ranges from 5 to 72 2607.26148 tab. 2. The paper treats harness gaps of 1.7 to 7.3 points as descriptive 2607.26148 §4.3. An optional trained waypoint tool, added beside the primitives, reaches 76.7 as a mean of three runs for fable-5, with a median of 48 steps per episode against 87 for primitives alone 2607.26148 tab. 5. The waypoint module is trained, so that cell is not zero-shot 2607.26148 app. A. Card: Embodied Agents Take Control: Minimal-Interface Zero-Shot Agents Rival Industrial-Scale Policies in Vision-and-Language Navigation.

Tool calls learned through fine-tuning

LLaVA-Plus trains a multimodal model on dialogues that contain tool calls. The model then chooses tools from its own training, not from a tool list in the prompt. Each assistant turn has three fields: thoughts, actions (a JSON list of API_name and API_params) and value 2311.05437 §2.1. A tool's output is written as text and sent back as a new human turn, followed by the original question 2311.05437 §2.2. The skill set covers detection, segmentation, tagging, captioning, OCR, retrieval, image generation and editing 2311.05437 tab. 2. Most training labels come from GPT-4 2311.05437 §2.2. On LLaVA-Bench In-the-Wild, the 7B model with the Fly tool set scores 59.1 against 57.1 for LLaVA. Removing the thoughts field drops it to 48.7 2311.05437 tab. 4. The All Tools setting writes tool outputs into the image context before the model answers, and it scores 69.5 on the same benchmark. That setting does not test whether the model chooses tools 2311.05437 §2.3. Card: LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents.

LaMDA applies the same idea to grounding in dialogue. Crowdworkers checked factual claims with a search tool, a calculator and a translator, and edited replies to cite their sources 2201.08239 §5 2201.08239 §6.2. The fine-tuned model first writes a query from a draft reply. The snippet the tool returns goes back into the context, and the model then writes either a grounded reply or another query. The first word of its output picks the recipient, the tool or the user 2201.08239 §6.2. The worked example caps this loop at four steps, and the paper does not say what is returned when the cap is reached 2201.08239 §6.2. If a query cannot be parsed, the tool returns nothing and no snippet is added 2201.08239 §6.2. At 137B parameters, the authors report 73.2% groundedness and 65% citation accuracy 2201.08239 app. F 2201.08239 §7. The model before this groundedness stage scores 67.9 2201.08239 app. F. That stage also trains on crowd-labelled groundedness data, so the gap cannot be credited to tool access alone. Crowdworkers using the same tools still score higher on groundedness and informativeness 2201.08239 §7. Card: LaMDA: Language Models for Dialog Applications.

Qwen's chat models learn tool use as one skill among many. The authors write about 2,000 self-instructed tool-use and agent samples, use the ReAct prompt format, and add them to the general fine-tuning mix rather than a separate stage 2309.16609 §3.4. Their tool benchmark leaves out plugins seen in training and scores the choice among up to five tools, parameter plausibility and false positives 2309.16609 §3.4 2309.16609 tab. 6. On tool selection, Qwen-Chat 14B scores 98 against 95 for GPT-4 [arxiv:2309.16609 tab. 6]. The benchmark is self-built and Chinese-focused, and the authors attribute part of GPT-3.5's low score to that 2309.16609 §3.4. The card lists no ablation of the tool samples alone 2309.16609 §3.4. Card: Qwen Technical Report.

Tool use trained by reinforcement learning against a cached sandbox

DeepTravel trains a travel-planning agent end to end with reinforcement learning. Each turn has a thought, then either a tool call or a final answer, and the policy reads the tool's response 2509.21842 §2. Six tools sit behind one interface: flight, train, hotel, route, place and web search 2509.21842 §3.1.1. Responses are cached, so replays of a trajectory see the same outputs, and the cache is refreshed every 24 hours 2509.21842 §3.1.3. The reward is 1 only if a trajectory-level check of feasibility passes and a turn-level check finds the final plan consistent with every tool response. Otherwise it is 0 2509.21842 §3.2.3. Queries that no sampled trajectory solves go into a replay buffer 2509.21842 §3.3.2. The 32B model reports an 82.58% overall pass rate on 6,224 production queries 2509.21842 tab. 4 2509.21842 §5.2. Those scores come from the same LLM reward system used in training, and the authors hand-check 200 of them, reporting 82% agreement 2509.21842 §5.2.1 [arxiv:2509.21842 tab. 3]. In the 8B ablation, removing the turn-level verifier lowers the constrained medium offline pass rate from 41.89 to 14.24 2509.21842 tab. 6. Card: DeepTravel: An End-to-End Agentic Reinforcement Learning Framework for Autonomous Travel Planning Agents.

Knowledge-graph tools in a fixed three-turn rollout trained with GRPO

SAKE trains Qwen2.5 instruct models with GRPO, with no supervised stage first 2505.15062 §3.4 §3.5. A rollout has three model turns and two tool outputs 2505.15062 §3.2. In the first turn the model names the key concepts in the query. The first tool returns, for each concept, the concept plus the most similar knowledge-graph concepts under a pre-trained encoder 2505.15062 §3.1 eq. 6. In the second turn the model keeps some of these groups. The second tool returns every triplet that links two kept groups eq. 7. In the third turn the model reasons by analogy over the triplets and writes the answer. Both tools are deterministic and carry no gradient, and their text is masked out of the loss 2505.15062 §3.1 alg. 1. The reward adds a format term, which checks that the closing tags are present, to an exact-match term on the answer. At 7B the weights are 0.2 and 0.8 2505.15062 §3.5 eq. 9. The UMLS graph has 135 nodes and 5,877 triplets 2505.15062 §4.1.

At 7B the authors report a weighted average of 75.4 on the 224 UMLS test questions. That is against 70.1 for GPT-3.5-Turbo with GIVE, and 56.7 for GRPO without the knowledge-graph tools 2505.15062 tab. 1. The GRPO comparison is a difference in method as well as in tools, and the card cannot tell whether the rollout format matches, because the paper gives few training values 2505.15062 §3.4. Replacing agentic retrieval with precomputed retrieval costs 5.0, 7.1 and 5.0 points on PubMedQA, BioASQ and ProcessBank 2505.15062 §4.3. The repository has no code path for that ablation 2505.15062 Paper vs code. Token use is 639 to 1,156 per question on the three UMLS sets, against 7,010 to 19,461 for ToG and GIVE as the authors ran them 2505.15062 tab. 2. The weighted average moves by about 0.45 points per question on 224 questions 2505.15062 Pitfalls.

The paper says tool calls are triggered by special tokens, but the shipped configs stop generation at plain-text closing tags [arxiv:2505.15062 Paper vs code]. Card: SAKE: Structured Agentic Knowledge Extrapolation for Complex LLM Reasoning via Reinforcement Learning.

Solver-checked code as the reinforcement learning reward

SIRL fine-tunes Qwen2.5 instruct models with reinforcement learning, without a supervised stage first 2505.11792 §3.2. Each answer has three tagged parts: reasoning, a mathematical model and Gurobi Python code [arxiv:2505.11792 §3.2]. The policy does not choose whether to call the solver. Every answer is code that the solver runs, and the objective it returns is scored [arxiv:2505.11792 §3.2] 2505.11792 §3.3.

The reward has a format term of 0.5 when the tags appear in order, an execution term of 1 when the code runs, and an accuracy term of 2 when the objective matches the reference. Stage two adds a bonus for advanced modelling forms 2505.11792 §10 eq. 6. The KL penalty applies only to the model and code sections, not to the reasoning 2505.11792 §3.2 eq. 5. On MAMO Complex, partial KL gives 51.7% accuracy, against 48.3% for full KL and 47.3% with no KL 2505.11792 tab. 3.

The authors report macro averages over five benchmark columns of 60.6 for the 7B model and 68.3 for the 32B model 2505.11792 tab. 1. The 7B average is below the 61.1 that the offline OptMATH-Qwen2.5-32B baseline reports in the same table. The paper gives the training setup only for the 7B model, on 8 H100 GPUs for about 24 hours per stage 2505.11792 §11. Card: Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling.

Plugins with declared functions and usage limits

The multi-agent framework defines a plugin as a tuple of the functions agents may call, its init-time configuration, and usage constraints such as limits on compute, input types or output 2306.03314 §2.2. Agents and plugins are nodes in a graph, and a plugin may also serve as a shared store 2306.03314 §2.3. The paper proposes a stateless oracle agent to screen sensitive actions, but does not say how the screen decides or how it was tested 2306.03314 §4.1.2. It gives no values for the limits and reports no experiments 2306.03314 §5.1. Card: Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents.

Action gate before irreversible calls

The memory framework MemTX splits domain tools into reversible and irreversible sets. The gate runs before every irreversible call 2607.23929 §3.3. It blocks the call while any tentative memory record that the transaction did not read exists in the store, including the caller's own uncommitted writes. For the external-action tier it also needs an action-safe record in the snapshot 2607.23929 §3.3. A blocked call returns a structured refusal, and the agent may commit or abort and then retry 2607.23929 §3.3. Reversible tools are not gated 2607.23929 §3.3.

When a belief is withdrawn, the protocol repairs by type. Beliefs are revoked, derived views are held back for rebuilding, and tool actions are compensated if reversible or logged as leaked effects if not 2607.23929 §3.4. Because the gate waits on any tentative record in the store, a stalled agent can hold up other agents' irreversible calls 2607.23929 §3.3.

On the authors' 90-case main suite with Qwen3-8B, task success is 0.956, against 0.822 for the strongest baseline 2607.23929 tab. 2. Realised harm on 24 refund episodes is 0.000 on every backbone, against 0.250 for the baseline on Qwen3-8B 2607.23929 tab. 11. With GPT-5.5, MemTX scores 0.756 against 0.778, which the authors call a tie 2607.23929 tab. 2 [arxiv:2607.23929 §5]. Removing the action gate costs nothing on the main suite, and costs only the hardened families it covers 2607.23929 fig. 3. Card: MemTX: Transactional Belief Commit for Stateful Agent Memory.

Guard before each tool call, with a provenance registry

DREAMS is a set of LLM agents that set up and run Quantum ESPRESSO jobs on a cluster. Each tool output is stored with a provenance record. A guarded call passes up to three checks before it runs [arxiv:2507.14267 Methods: Safety guard] fig. 2. First, a deterministic check confirms that each referenced value exists in the registry and equals the recorded one. Second, an LLM judge checks each parameter against the reason the agent gave. Third, for convergence selection and equation-of-state fitting, a second judge checks the parameters together. A failed check stops the tool. A warning registers the value with the concern in its metadata, and an infrastructure error in the guard does not block the run 2507.14267 Methods: Safety guard. After a report is written, a report judge checks each claim against its cited artifact and walks the provenance graph. That verdict decides acceptance [arxiv:2507.14267 Methods: Safety guard] fig. 5.

On CO/Pt(111), the authors report that the guarded run executes and succeeds on 100% of steps, against 100% executed and 81% succeeded without the guard. The adsorption-energy MAPE is 0.1% with the guard and 0.2% without 2507.14267 tab. 6. On the BCC Li lattice-constant task both runs show 0.55% MAPE 2507.14267 tab. 6. The guarded run uses about 13 times the input tokens, 23.9 million against 1.8 million on CO/Pt(111) [arxiv:2507.14267 tab. 6] tab. 7. The production judge, Opus 4.8, makes 2 false positives and 5 false negatives on a 145-scenario adversarial suite 2507.14267 fig. 8. All judges are Anthropic models, and the rules were tuned on CO/Pt(111)-derived scenarios plus one other chemistry, so other domains are untested 2507.14267 Methods: Judge implementation. The list of sensitive parameters is written by experts, and the guard does not find them itself 2507.14267 Discussion. The call-time judge does not walk upstream provenance. In one vacuum case it accepted a reference that the report judge rejected 2507.14267 Methods: Safety guard. Card: DREAMS: Density Functional Theory Based Research Engine for Agentic Materials Simulation.

Tools registered as versioned resources

The protocol treats each tool as a registered resource with a name, a description, an invocation mapping, a version and an evolvability marker. A tool may be local code, an MCP-connected tool or a skill [arxiv:2604.15034 §3.1]. A change to it is proposed, assessed and committed, and a rejected candidate leaves the active tool as it was 2604.15034 §3.2 2604.15034 app. G. The evolving system can edit prompts, tools and code together, so the tool's share of the reported GAIA gain is not separated 2604.15034 §5.2 2604.15034 tab. 2. Restoring an earlier version is defined, but the paper does not show recovery after a bad version goes live 2604.15034 §3.2 2604.15034 §6. Card: Autogenesis: A Self-Evolving Agent Protocol.

Fine-tuned API-call writer, with optional retrieved documentation

Gorilla fine-tunes LLaMA-7B on pairs of a plain-language request and the API call that answers it, built from TorchHub, TensorFlow Hub and HuggingFace model cards 2305.15334 §3.1 2305.15334 §3.2. The model writes one call per request. The paper scores that text against the reference APIs and never runs it, so a deployment must add the run step itself 2305.15334 §3.3. The retrieval variant appends the top-1 document to the request. The paper trains one model without documentation and one with it 2305.15334 §3.2 2305.15334 §4.

Zero-shot, Gorilla scores 59.13, 71.68 and 83.79 overall on TorchHub, HuggingFace and TensorFlow Hub. Zero-shot GPT-4 scores 38.70, 19.80 and 18.20 2305.15334 tab. 1. Its hallucination rate is 6.98, 10.95 and 5.40, against 36.55, 37.16 and 78.65 for GPT-4 2305.15334 tab. 1. These are the authors' figures on their own benchmark, and no error bars are given 2305.15334 §4.

Documentation at test time helps only in some setups. A model trained without documentation scores lower with BM25 retrieval on all three hubs, at 37.63, 11.28 and 34.30 2305.15334 tab. 2. A model trained with the correct document scores 0 overall zero-shot on every hub, so it needs a retriever at inference 2305.15334 tab. 2. Oracle retrieval is an upper bound that a deployment cannot reach 2305.15334 §4. Card: Gorilla: Large Language Model Connected with Massive APIs.

Built-in tools chosen by a deployed chat model

GLM-4 All Tools is a chat model that, during a conversation, can call a web browser, a Python interpreter, a text-to-image model or a developer's own function 2406.12793 §2. The paper says the model plans, calls one or more tools and uses their results 2406.12793 §2. The training method for this choice is not stated. The public repository holds only the client-side dispatcher for the built-in tools, so the learned choice cannot be checked there tool_registry.py.

Author-reported scores put the Python-based tasks close to GPT-4 with the same tools: GSM8K 91.59 against 92.72, MATH 63.60 against 65.00, and Math23K 88.50 against 88.40 2406.12793 tab. 9. On the browser task, labelled Information Seeking, it scores 78.08 against 67.12. The paper does not define that task or give its size 2406.12793 tab. 9 2406.12793 §3.8. The paper gives no step limit, stop rule or error handling for tool calls 2406.12793 §2. The Berkeley Function Call Leaderboard table is for the base chat model, not All Tools: GLM-4 (0520) scores 81.76 overall, against 81.24 for GPT-4 Turbo and 82.94 for GPT-4o 2406.12793 tab. 7. Card: ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools.

When to use which

Implementation sketch

The loop below follows the paper's description. The paper gives no iteration cap, so the max_steps guard is our addition. The <|endofprompt|> separator appears in the paper's prompts, but its role is not explained fig. 5.2.

The structured-call variants run the same loop. The difference is that the call is a structured object, parsed from JSON or a similar format, rather than text matched by a pattern 2505.10468 §2.1.2. The review gives no stop rule or memory format, so those are choices for the implementer [arxiv:2505.10468 §6.1]. The ReAct variant also runs this loop, but its call is a text action in a fixed trace format, and each thought is written into the trace without returning an observation 2210.03629 §2. The learned variants keep this loop at serving time, but the choice of tool comes from training: LLaVA-Plus learns it from tool-call dialogues 2311.05437 §2.1, LaMDA learns it from its tool-use data 2201.08239 §6.2, and DeepTravel learns it from the reward 2509.21842 §3.2.3. SAKE fixes the order of its two tools and learns which entities to name and which groups to keep 2505.15062 §3.2. The All Tools loop in ChatGLM follows the same shape, with each tool result added to the conversation before the next call. The card gives no step limit or error rule 2406.12793 §2.

The model-calls-inside-a-program variant does not use this loop. It generates several programs in one pass, runs each, and votes on the answers 2210.02875 §2.2 2210.02875 app. A.3. PAL does not loop either. It generates one program per question for its main results, runs it once, and returns the value, and it votes over sampled programs only for its 80.4% result 2211.10435 §3 2211.10435 §5.1.

Chameleon also plans once. Its planner writes a module sequence, the modules run in order, and the result is the last output. An invalid plan is replaced by the task's default sequence 2304.09842 §3 2304.09842 §4.2. Parse the plan against a whitelist of module names. The released code evaluates the plan text directly, which the Chameleon card flags.

# Describe the tools in the prompt, with no demonstrations [§5.1]
prompt = tool_instructions + "<|endofprompt|>" + question
for step in range(max_steps):        # our guard; no cap in the paper
    # Generate until a tool call or the end of the answer [fig. 5.2]
    out = generate(prompt)
    if not ends_with_tool_call(out):
        return out
    # Run the tool outside the model [§5.1]
    result = execute(parse_call(out))
    # Append the call and its result, then continue generating
    prompt = prompt + out + "\n> " + result
raise StepLimitExceeded

MemTX puts a gate in front of the irreversible calls inside this loop 2607.23929 §3.3. A call that the gate refuses returns its reason to the model, and the model may commit or abort before retrying.

# Gate before one irreversible tool call  [§3.3]
def gate_allows(tx, tool, store):
    # Reversible tools are never gated  [§3.3]
    if tool.reversible:
        return True
    # Any tentative record the transaction did not read blocks the call  [§3.3]
    for record in store.records:
        if record.state == "tentative" and record.id not in tx.snapshot:
            return False
    # The external-action tier also needs an action-safe record  [§3.3]
    if tx.risk_tier == "external_action":
        return any(store.is_action_safe(rid) for rid in tx.snapshot)
    return True

Contradictions

The cards disagree in several places, and none of the disagreements is resolved here.

The GPT-4 probe reports that the model often failed to decide when a tool was needed 2303.12712 §5.1.2. The navigation study finds a similar gap at the stop decision: 23 of 30 failures in one run claim success while 3.1 to 34.1 m from the goal 2607.26148 §4.6 2607.26148 tab. 7. The review presents tool-augmented reasoning as a mitigation for agent limitations 2505.10468 §6.1, and it gives no measurement that would test that claim. Neither result settles the other. Qwen's benchmark gives Qwen-Chat 14B 98 against 95 for GPT-4 on tool selection 2309.16609 tab. 6. That score covers which tool is picked from a set, not whether a call is needed, so it does not settle the probe's point. Binder avoids the run-time choice by writing its calls into the program before it runs 2210.02875 §2.1. Its error analysis still lists missing API use among the causes of failure 2210.02875 §4.2, so the paper does not show that writing calls up front removes the problem.

Worked examples are the first disagreement. The GPT-4 probe's tool prompt has none 2303.12712 §5.1. ReAct relies on one to six examples per task 2210.03629 §3.2. The Auto-GPT authors found that tool instructions without examples performed much worse 2306.02224 §2.2. The models and tasks differ, and no card compares examples with no examples for the same model.

The second is ReAct's ALFWorld score. AutoGen reports 54 for ReAct, averaged over three attempts on the 134 unseen tasks, and 66 for its best attempt 2308.08155 app. A3. The ReAct paper reports 57 averaged over six prompt sets and 71 for the best set 2210.03629 tab. 3. The averaging differs, so the two sets of numbers are not interchangeable.

Whether a tool helps depends on the model and the benchmark in the measured cards. In Auto-GPT, the tool loop beats the imitation baseline only with GPT-4 2306.02224 tab. 1. PAL shows the same dependence on the model. With text-davinci-001 it scores 8.6% on GSM8K against 26.5% for chain-of-thought, and with text-davinci-002 it scores 65.8% against 46.9% 2211.10435 §6. The navigation study shows the same pattern. Forcing waypoints raised success by 17 to 38 points for the Qwen and GPT rows, by 8.7 to 9.3 for sonnet-5 and opus-4.8, and by 0.7 to 1.3 for fable-5 and opus-5 2607.26148 tab. 4. In LLaVA-Plus, giving base LLaVA the tool outputs at test time lowers its scores on LLaVA-Bench, from 81.2 to 59.1 on COCO and from 57.1 to 47.5 on In-the-Wild, but raises its SEED-Bench average from 44.45 to 51.54 2311.05437 tab. 4. In ReAct, scores fall below chain-of-thought on HotpotQA, at 27.4 against 29.4, and rise above it on FEVER, at 60.9 against 56.3 2210.03629 tab. 1. Its chain-of-thought baseline uses no tools, so that comparison changes the tools and the trace together. The cards do not explain these directions. Only PAL's runner ablation and Chameleon's module removals change one tool alone. Each covers one model, and PAL's covers one benchmark.

TANGO's numbers do not test its LLM step. The paper says the LLM writes the program 2412.10402 §3. The released evaluator builds the generator in a debug mode that returns a fixed program per task, and the task templates insert the episode's target name code_generator.py utils.py. The paper does not say whether the agent receives the target label. Its results therefore measure the modules and a fixed program, not the planner.

Binder's numbers come from code-davinci-002, which the card records as retired, and the official evaluator changes its scores by 1 to 2.5 points 2210.02875 §3.1 2210.02875 app. C.2. Reproducing them needs a substitute model, so they do not transfer directly.

The measured cards do not test the same thing as each other. Binder tests table QA. Auto-GPT, ReAct, AutoGen, DeepTravel, MemTX, TANGO and the navigation study compare whole agents with baselines. Chameleon and SIRL compare whole methods with theirs. LaMDA compares its groundedness stage with the model before it, but that stage also changes the training data. LLaVA-Plus trains a model, and the protocol tests whole agent systems with several edits at once. PAL compares code with text reasoning, so its main comparison changes the prompt format as well as the executor. Only its runner ablation on GSM8K isolates the executor. Gorilla scores a call without running it, and ChatGLM's All Tools results come from a whole model with its tools 2305.15334 §3.3 2406.12793 tab. 9. None of the other cards isolates a tool-only effect.

Two cards compare a learned model with GPT-4 on choosing a tool or an API, and the margins differ. Gorilla's fine-tuned 7B model beats zero-shot GPT-4 by 20 to 66 points on its own API benchmark 2305.15334 tab. 1. Qwen-Chat 14B scores 98 on its own tool-selection benchmark, against 95 for GPT-4 2309.16609 tab. 6. The benchmarks differ in task and author, so this is not one test with two answers. Compare numbers only within each card.

Papers

• Sparks of Artificial General Intelligence: Early experiments with GPT-4: an early GPT-4 study that describes prompt-only tool use, shows chained tool sessions, and reports that tool use needs prompting and that the model's judgement about when to call tools is weak.
• PAL: Program-aided Language Models: few-shot prompts in which the model writes Python steps with comments and an interpreter produces the answer. It reports author-measured results on math and symbolic tasks with code-davinci-002, and an ablation that changes only the runner.
• AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges: a narrative review that describes the reason-act-observe tool loop, its multi-agent extension and the open challenges. It reports no experiments.
• A Survey on Large Language Model based Autonomous Agents: a 2023 survey whose action module separates external tools from the model's own abilities. It reports no experiments and covers work up to August 2023.
• Binding Language Models in Symbolic Languages: Binder, a training-free method in which a generated SQL or Python program calls the same model for steps the language cannot express. It reports author-measured table QA accuracy.
• Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents: a conceptual framework that defines plugins as tuples of callable functions, configuration and usage constraints. It reports no experiments.
• Autogenesis: A Self-Evolving Agent Protocol: a protocol that registers tools, prompts, agents, environments and memory as versioned resources, updated through propose, assess and commit. Its agent gains are author-reported and do not isolate tool changes.
• Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models: an LLM planner writes a module sequence over thirteen module types, run in order. It reports author-measured ScienceQA and TabMWP accuracy with GPT-4 and ChatGPT, and module-removal ablations.
• Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling: an RL method in which the reward comes from running generated solver code. It reports author-measured accuracy on five benchmark columns for 7B and 32B models.
• MemTX: Transactional Belief Commit for Stateful Agent Memory: a commit protocol for agent memory with an action gate before irreversible tool calls and typed repair. It reports author-measured results on the authors' own 90-case suite.
• Embodied Agents Take Control: Minimal-Interface Zero-Shot Agents Rival Industrial-Scale Policies in Vision-and-Language Navigation: a zero-shot navigation study in which a coding-agent harness calls observe and step tools on R2R-CE. It reports author-measured success rates on 100 episodes, with an optional trained waypoint tool.
• TANGO: Training-free Embodied AI Agents for Open-world Tasks: an LLM writes a short program over a pretrained navigation policy, a memory-augmented value map and detectors. It reports author-measured zero-shot results on three embodied benchmarks, and its released evaluator does not call the LLM.
• Qwen Technical Report: a family of base and chat models whose chat variants mix about 2,000 tool-use samples into fine-tuning. It reports author-measured tool-selection results on its own benchmark, among many other benchmarks.
• ReAct: Synergizing Reasoning and Acting in Language Models: few-shot prompts that interleave free-text thoughts with Wikipedia lookups and environment actions. It reports author-measured results on two question-answering tasks and two interactive tasks, using PaLM-540B.
• LaMDA: Language Models for Dialog Applications: a dialogue model fine-tuned to query a search tool, a calculator and a translator before replying. It reports author-measured groundedness and citation accuracy from crowd labels.
• AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation: a framework in which agents, some of which run code, exchange messages. It reports author-measured MATH and ALFWorld results for its agent designs.
• Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions: an Auto-GPT-style JSON tool loop on WebShop and ALFWorld, with an optional supervised suggestion in the prompt. It reports author-measured success rates on small test sets.
• LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents: a multimodal model fine-tuned on tool-call dialogues in a three-field format. It reports author-measured benchmark scores, and its strongest setting pre-computes tool outputs.
• DeepTravel: An End-to-End Agentic Reinforcement Learning Framework for Autonomous Travel Planning Agents: an RL-trained travel agent with a cached tool sandbox and a two-stage reward. It reports author-measured online and offline pass rates, graded by an LLM judge.
• Gorilla: Large Language Model Connected with Massive APIs: a LLaMA-7B model fine-tuned to write one API call per request, with an optional retrieved document in the prompt. It reports author-measured accuracy and hallucination rates on its own APIBench.
• ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools: a model family whose GLM-4 All Tools variant chooses among a browser, Python, an image model and user functions. It reports author-measured benchmark results and gives few training details.
• Agentic Large Language Models, a survey: a 2025 survey that places tool calling in its acting category. It reports no experiments of its own.
• From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review: a 2025 review that describes an agent loop and compares the MCP, ACP and A2A protocols. It reports no experiments of its own.
• TRiSM for Agentic AI: a 2025 review of trust and security for multi-agent LLM systems. Its threat list includes tool misuse, and it proposes a Tool Utilization Efficacy score. It reports no experiments.
• Empowering Biomedical Discovery with AI Agents: a 2024 perspective that defines four autonomy levels for biomedical agents and lists tool use among four modules. It reports no experiments.
• SAKE: Structured Agentic Knowledge Extrapolation for Complex LLM Reasoning via Reinforcement Learning: a GRPO-trained Qwen2.5 model that calls two deterministic knowledge-graph tools in a fixed three-turn rollout. It reports author-measured QA accuracy and token use on UMLS and ConceptNet.
• DREAMS: Density Functional Theory Based Research Engine for Agentic Materials Simulation: a multi-agent system for periodic DFT in which each tool call passes a provenance and judge guard. It reports author-measured lattice-constant and adsorption-energy results, and the token cost of the guard.