LLM agent

Overview

An LLM agent is a language model that works toward a goal in steps. At each step it decides whether to call a tool, reads the result, and continues until it can answer 2505.10468 §2.1.2. The tools depend on the task: a search engine, a web shop, a simulator, a code runner or an email client. Most designs below keep this loop and change one of three things: how many agents share the goal, whether the agent may change its own parts, or what checks gate its actions. A fourth group of papers does not build an agent but measures one.

A 2023 survey describes the agent as four modules: profile, memory, planning and action 2308.11432 §2.1. Its planning section sorts methods into those with and without feedback from the environment or the model 2308.11432 §2.1.3. A 2025 survey of agentic LLMs sorts the same ground into reasoning, acting and interacting 2503.23037 §2 §3 §4. This page uses the module names where they help.

The 2025 survey calls the single-model version an "AI Agent" and describes it by autonomy, task-specificity and reactivity 2505.10468 §2.0.1. This page uses the more common name. The survey's second term, "Agentic AI", is the multi-agent design: specialized agents share a goal, one part splits it into subtasks, a shared persistent memory holds state, and an orchestration layer or meta-agent coordinates them 2505.10468 §3.1.2. The survey calls this split a "widely accepted conceptual illustration" and cites no formal definition for it 2505.10468 §3.0.1. The papers below do not all use the same vocabulary, so this page treats the designs as variants of one concept.

A 2024 perspective on biomedical agents sorts them into four autonomy levels, from no agent to an agent that acts as a scientist. It rates an agent on hypothesis, experiment and reasoning, and the lowest rating sets the level 2404.02831 §4 tab. 1. The levels are not tested in the paper.

Three cautions apply across the cards.

• Not every agent here is an LLM agent. The WebShop agents are a BART model that writes search queries and a 12-layer BERT model that picks clicks, both trained on human demonstrations 2207.01206 §4.2. They serve as a trained baseline in an agent environment. The Zetta policy is a vision-language-action model, and the LLM that proposes its code changes is not named 2608.16590 §2.1. MOOSE-Chem makes fixed LLM calls, and its card describes no tool use 2410.07076 §4.2.
• Several papers have no experiments. The 2025 survey 2505.10468 §1.1 and the multi-agent framework 2306.03314 §6 report none. The legal survey is a review that reports no new experiments 2303.09136. The evaluation survey 2507.21504 §1 and the position paper on simulated humans 2501.08579 §1 report none either. The 2023 agent survey 2308.11432 §1 and the position paper on small models 2506.02153 §1 do not run experiments of their own either. Nor do the memory survey 2404.13501 §6.3, the autonomy position paper 2502.02649 §1 or the general LLM survey, whose agent material is one section 2402.06196 §4.5. The agentic-LLM survey 2503.23037 §1.2, the benchmark and protocol survey 2504.19678 §III and the trust review 2506.04133 §1 report none of their own either. The biomedical perspective on agent autonomy 2404.02831 §1 also reports no results.
• Most numbers are author-reported and come from different tasks, models and compute budgets. Use the tables below as starting points for a choice, not as a ranking.

Variants

Single agent with tools

One model, a set of tools and one bounded task. The survey's single-entity design has four modules: perception, knowledge representation and reasoning, action selection and execution, and basic learning and adaptation 2505.10468 §3.1.1. It lists uses such as enterprise-search customer support, email filtering and scheduling assistants 2505.10468 §4.0.1. Card: AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges.

The Auto-GPT benchmark puts chat models in a loop that chooses a command, runs a tool and reads the result. It uses 50 WebShop instructions and 134 ALFWorld games 2306.02224 §3.1. Author-reported WebShop success is 0.240 for GPT-4, 0.140 for Claude and 0.120 for GPT-3.5 2306.02224 tab. 1. Vicuna scores zero because it could not produce the required output format 2306.02224 §3.3.1. Adding a trained expert's suggested action to the prompt, with no fine-tuning, lifts GPT-4 to 0.320 on WebShop and to 0.515 on ALFWorld 2306.02224 tab. 1 tab. 2. On 50 tasks, one success is 0.02, so the gap between 0.300 and 0.320 is a single task 2306.02224 tab. 1. Card: Auto-GPT for Online Decision Making.

ReAct is the prompting pattern that most of these loops build on. Before each action the model writes a short thought, then reads the observation 2210.03629 §2. It needs no training, only a few worked examples. On a frozen PaLM-540B, its ALFWorld success is 71% for the best of six prompt sets and 57% on average. The same actions without thoughts reach 45% at best 2210.03629 tab. 3. On WebShop it scores 40.0% success, against 30.1% for the action-only baseline 2210.03629 tab. 4. On HotpotQA it trails chain-of-thought, 27.4 against 29.4 exact match 2210.03629 tab. 1. Card: ReAct.

The ExpeL paper uses ReAct, a reason-then-act loop, as its base planner 2308.10144 §4.1. The WebShop paper gives a rule baseline that writes the instruction text as its query and buys the first matching item. It scores 45.6, against 82.1 for human experts 2207.01206 tab. 2. Cards: WebShop, ExpeL.

A coding-agent harness can also run the loop. In a zero-shot navigation study, the model chooses every call to an observe() tool and a step() tool with four primitives, and no map or memory is given 2607.26148 §3.2. A replicated default-effort Claude opus-5 run averages 70.7 success rate, with a standard deviation of 3.5 over three runs, and a fable-5 run at max effort reaches 78 2607.26148 tab. 2. The authors say the model choice matters more than the harness, and that the harness comparison is descriptive 2607.26148 §4.3. Card: Embodied Agents Take Control.

TANGO uses a different split of work. The model writes a short program that calls a fixed set of navigation and perception modules, and it gets 15 worked examples and no training for the task 2412.10402 §3. The modules are a pretrained point-goal policy, a detector, a memory-augmented value map and a question-answering step 2412.10402 §3 fig. 3. The paper reports zero-shot results that it calls on par with VLFM on OVON and second on OpenEQA 2412.10402 §4.2 tab. 1 tab. 2. The released code does not call the model in its evaluation path, which uses a fixed program per task. Those numbers therefore do not test the LLM step (see the card's Paper vs code section). Card: TANGO.

Planner that re-plans from observations

A planner writes a full plan first and calls the model again when the plan meets the world. LLM-Planner writes a high-level plan of (action, object) subgoals for an ALFRED household task. It is given the nine most similar examples from a pool of 100 solved tasks 2212.04088 §4.3 §5.3. A low-level controller carries out each subgoal. When a subgoal fails, the planner runs again with the objects seen so far 2212.04088 §4.4 alg. 1. The step budget between re-plans is not stated [arxiv:2212.04088 §4.4]. With goal instructions only, the author-reported unseen-test success is 11.58% for a static plan and 13.41% with re-planning 2212.04088 tab. 1. With step-by-step instructions it reaches 16.42%, against 20.27% for a controller trained on all 21,023 pairs with goal instructions 2212.04088 tab. 1. The settings differ, so this is not a like-for-like result. The paper uses about 7 GPT-3 calls per task, against 22 for SayCan 2212.04088 §5.4. Card: LLM-Planner.

Agent with a memory module

Most designs above keep history in the prompt. The memory survey treats memory as a separate part with three operations: writing an item from the latest action and observation, managing the store by merging, reflecting or forgetting, and reading what the next action needs 2404.13501 §3.4. It reads memory mostly by similarity 2404.13501 §3.4. Its sources are the current trial, earlier trials and external knowledge, and its forms are text or parameters 2404.13501 §5.1 §5.2. Keeping the full history in the prompt is the simplest form, and its cost grows with length 2404.13501 §5.2.1. The survey says text is cheaper to write and parameters are cheaper to read, but it gives no measurements of either 2404.13501 §5.2.3. It also notes that no open benchmark targets agent memory alone 2404.13501 §6.3. Card: Memory-mechanism survey for LLM agents.

MemTX treats a memory write as a staged claim, not a fact. A record is checked for evidence, validity, conflicts with accepted beliefs and dependency on retracted records, and only then committed 2607.23929 §3.2. Tool calls that cannot be undone wait while any unsettled record exists [arxiv:2607.23929 §3.3]. On the authors' 90-case suite with Qwen3-8B, task success is 0.956 against 0.822 for the strongest baseline 2607.23929 tab. 2. The suite is the authors' own, and removing the action gate costs nothing on the main suite 2607.23929 fig. 3. Card: MemTX.

Fixed pipeline with a coding-agent stage

Here the order of work is fixed, and the model fills each stage. The AI Scientist runs idea generation, a literature-based novelty check, experiments, a LaTeX write-up and an LLM review, in that order 2408.06292 §3 fig. 1. The coding agent is Aider, which edits a small starting template and runs its experiment script 2408.06292 §3. The three templates are a 2D diffusion model, a character-level language model and grokking on modular arithmetic 2408.06292 §6. The authors report about $10 to $15 of API cost per paper, and one run of about 50 ideas took about 12 hours on eight H100 GPUs 2408.06292 §6. In their check of one generated paper, the reported numbers matched the logs, but the paper invented a V100 setup where H100s were used and called a negative result an improvement 2408.06292 §5. The reviewer is GPT-4o, and its calibrated balanced accuracy is 0.65 against 0.66 for a NeurIPS human consistency baseline. The human figure comes from different data, and the reviewer task is covered on the llm-as-a-judge page 2408.06292 tab. 1. Card: The AI Scientist.

Trained agent in an interactive environment

Instead of prompting a frozen model, the policy is trained on recorded behaviour and then improved with reinforcement learning on an automatic reward. In WebShop, the best trained agent (imitation learning, then policy gradient) reaches a score of 62.4 and a 28.7% success rate. Human experts reach 82.1 and 59.6% 2207.01206 tab. 2. The reward is computed by a program, so no human rater is needed for each episode 2207.01206 §3.1. Reinforcement learning runs took about 20 to 27 GPU-hours per model 2207.01206 app. C.2. Card: WebShop.

DeepTravel uses the same recipe for a travel-planning agent with real booking tools: a supervised cold start, then reinforcement learning. Tool responses are cached in a sandbox that refreshes every 24 hours 2509.21842 §3.1.3. The reward is binary. A trajectory-level check tests feasibility, and a turn-level check tests that the itinerary matches each tool response 2509.21842 §3.2.3. Both checks are LLM judges, not a program. Queries that fail every sampled attempt are replayed later 2509.21842 §3.3.2. The 32B model reports an 82.58% overall pass rate on production queries 2509.21842 tab. 4. On the offline sets, its human-evaluation column is 82.00, against 72.00 for DeepSeek-R1 2509.21842 tab. 5. Without the turn-level verifier, the 8B model scores 58 against 70 for the full model 2509.21842 tab. 6. Card: DeepTravel.

SIMA 2 applies a similar recipe to a Gemini Flash-Lite agent that plays 3D games from screen frames with keyboard and mouse actions. It is first fine-tuned on human play, on bridge examples with Gemini-written reasoning and dialogue, and on other Gemini data 2512.04797 §3.3 §3.3.2. Its RL stage uses tasks with a verification function, and keeps only tasks a human could finish within the time limit 2512.04797 §3.3.3. The authors say SIMA 2 "effectively doubles" SIMA 1's average success on the training environments. They report held-out ASKA and MineDojo success more than 10% higher. The exact values appear only in bar charts 2512.04797 §4.2.1 §4.2.2 fig. 6 fig. 10. Against the base Gemini model, its change on code, math and science benchmarks is a relative drop of 4.0% to 25.5% 2512.04797 tab. 1. The RL algorithm and its settings are not stated 2512.04797 §3.3.3. Card: SIMA 2.

StepAgent gets a per-step training signal without human step labels. For each step of a recorded expert trajectory, the current agent acts from the same prefix, and the expert and agent actions form a pair 2411.03817 §4.1 eq. 5. The pair then trains the agent in one of two ways. One is a DPO-style preference loss with the expert action preferred [arxiv:2411.03817 §4.2.1] eq. 6. The other trains a discriminator to separate agent steps from expert steps and uses it as a reward in PPO 2411.03817 §4.2.2 alg. 1. The authors report gains over SFT, PPO, DPO, RFT, SPIN, NAT and ETO on seven benchmarks, with Mistral-7B and Llama-3-8B 2411.03817 tab. 2. The margin over the best baseline in each column is about 1 to 3 points, from one run each 2411.03817 tab. 2 §7.1. The multi-hop expert trajectories come from GPT-4, and the paper does not say whether the discriminator reward has the sign that GAIL-style training uses (see the card's Pitfalls). Card: From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning.

SAKE trains one model to call two knowledge-graph tools inside its own answer. The tools are deterministic: the first returns groups of graph concepts that are similar to each extracted entity, and the second returns the links between the groups the model keeps 2505.15062 §3.1 eq. 6 eq. 7. The rollout has three model turns, and the policy is trained end to end with GRPO and a reward that adds a format term and an exact-match term [arxiv:2505.15062 §3.4] §3.5 eq. 9. On the UMLS question sets, the authors report a 75.4% weighted average for Qwen2.5-7B, against 70.1% for GPT-3.5-Turbo with GIVE 2505.15062 tab. 1. Their token counts are 639 to 1,156 per question, against 7,010 to 19,461 for ToG and GIVE 2505.15062 tab. 2. Card: SAKE.

CAPO changes the unit that gets credit. A critic reads the state before each complete action, and GAE gives the whole action one advantage [arxiv:2604.18401 §3.2] §3.3 eq. 2. The token ratios inside the action are combined into one ratio, scaled by the square root of the action length and centred on a batch mean, and that ratio is clipped as a unit 2604.18401 §3.4 eq. 4. On Qwen3-4B, the authors report 58.2% WebShop success, against 49.6% for ST-PPO and 46.0% for PPO, each the mean over three seeds 2604.18401 tab. 1. In the Qwen3-4B ablation, ALFWorld and WebShop scores rise from token-wise ratios to square-root aggregation to the centred form, in that order [arxiv:2604.18401 tab. 2]. Their code does not implement this ratio, so reproduction needs a fresh implementation (see the card's Paper vs code section). Card: CAPO.

Multi-agent team with roles and orchestration

Several agents, each with a role, coordinate through messages, a shared pool or a planner. The designs differ in how they do this.

• Agent tuples and supervision. The 2023 framework defines an agent as a tuple of model, role, state, permission to create agents and permission to halt agents. A stateless oracle agent answers from its current input only, and a supervisor halts agents that loop 2306.03314 §2.1 §3.4 §3.5. It reports no quantitative result 2306.03314 §6. Card: Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents.
• Two role-playing chat agents. CAMEL gives an assistant and a user agent fixed system prompts. The prompts set each role, forbid role swaps and require one instruction per turn. A task-specifier agent first turns an idea into a concrete task, and the chat ends on a stop token 2303.17760 §3.1 §3.2. The paper caps a chat at 40 messages 2303.17760 §4. In human votes on 100 AI Society tasks, CAMEL wins 76.3% against a single gpt-3.5-turbo answer 2303.17760 tab. 1. The same setup also yields generated conversations for fine-tuning, and a LLaMA-7B fine-tuned on them reaches 14.0 pass@1 on HumanEval 2303.17760 §5.2 2303.17760 tab. 3. Card: CAMEL.
• Conversation as the program. AutoGen agents each send and receive messages. Any agent can be backed by a model, a human or a tool, and the developer sets its reply rules in Python or in natural language 2308.08155 §2.1 §2.2. The authors report 69.48% accuracy on the full MATH test set, against 55.18% for vanilla GPT-4 2308.08155 app. A1. On ALFWorld, their three-agent design averages 69% success, against 54% for ReAct 2308.08155 app. A3. Card: AutoGen.
• Software company roles. MetaGPT assigns five roles (product manager, architect, project manager, engineer, QA engineer). They exchange structured documents through a shared pool, and each acts on the messages it subscribes to 2308.00352 §3.1 §3.2. The authors report 85.9% Pass@1 on HumanEval and 87.7% on MBPP 2308.00352 §4.2. Going from one engineer to the full set of roles cuts human revisions from 10 to 2.5 and raises executability from 1.0 to 4.0 in their ablation 2308.00352 §4.4 tab. 3. Card: MetaGPT.
• One agent per robot with a checked plan. Each robot has its own LLM agent. The agents talk in turns and end with one sub-task per robot. A simulator checks the plan and returns failures as text for re-planning 2307.04738 §3.1 §3.2. On six simulated tasks with GPT-4, the dialog method reaches 0.44 to 0.95 success per task. A single prompt that plans for all robots at once scores higher on four tasks, and higher on average, 0.81 against 0.75, when the table's values are averaged here 2307.04738 tab. 2. The paper also reports a small real-robot run with a human 2307.04738 §5.4. Card: RoCo.
• Writer, reviewer and optional planner for robot code. One configuration is a single coder. Two more add a reviewer that sends code back with feedback, and one of them adds a planner that runs first [arxiv:2411.16723 §3.1]. On seven spoken prompts, the writer and reviewer pair has 9 points fewer first-attempt code errors than the single coder, and 14 points fewer than the planner setup 2411.16723 §4 fig. 6. The observer scores do not differ significantly across the three setups 2411.16723 §4 fig. 8. The study uses three runs and one observer per run 2411.16723 §3. Card: Two Heads Are Better Than One.
• Supervisor with a tournament. The Co-Scientist system assigns work to six agent types: generation, reflection, ranking, proximity, evolution and meta-review 2502.18864 p. 30–36. Hypotheses are ranked in an Elo tournament whose judgements come from the system's own models 2502.18864 p. 10. In an expert comparison on 11 goals, its average preference rank is 2.36, against 2.45 for the two strongest baselines (lower is better) 2502.18864 fig. 2c. The source code is not public 2502.18864 p. 40. Card: Accelerating scientific discovery with Co-Scientist.
• Critic-moderated expert panel, then a coding stage. CellForge runs three stages: formulation, ideation and execution 2508.02276 §Methods. In ideation, experts for data modelling, single-cell biology and deep learning share one LLM backend and differ by system prompt. Each round they submit plans, score each other's plans, and receive a critic's meta-review. A confidence score guides the rounds, and the critic picks one final plan 2508.02276 §Discussion protocol. A coding agent then writes, runs and repairs training code for up to five attempts [arxiv:2508.02276 §Code generation and repair]. The authors report PCC 0.9883 for CPA-X on Adamson CRISPRi and MSE_DE 0.1736 for scGen-X on Norman CRISPRa 2508.02276 §Discussion. Other per-dataset results appear only in figures, and the released code caps discussion at six rounds where the paper sets twenty (see the card's Paper vs code section). Card: AI Scientists for Building Virtual-Cell Models.
• Evolutionary pipeline of LLM calls. MOOSE-Chem screens a literature pool for inspirations, drafts several mutated hypotheses for each one, recombines the drafts and ranks them by self-scores 2410.07076 §4.2.1 §4.2.3 §4.2.4. With GPT-4o as generator and judge, its top hypothesis reaches a Matched Score of 4.020 against 2.882 for MOOSE on a 51-paper benchmark 2410.07076 tab. 10. The Matched Score is a GPT-4o judgement, and the authors say experts usually give one to two points less 2410.07076 app. A.14. Card: MOOSE-Chem.
• Simulated review panel. AgentReview gives three reviewers, an author and an area chair written traits such as irresponsible, malicious or conformist. Each run changes one trait and is compared with a neutral baseline 2406.12708 §2.1 §2.4. The authors report that one malicious reviewer changes 37.1% of decisions. Table 2 gives 62.91% agreement with the baseline for that setting, which matches, although the paper does not state the derivation 2406.12708 tab. 2. The simulation is not checked against real review outcomes, and the authors advise against LLM reviewers in real venues 2406.12708. Card: AgentReview.
• Society of nine chat agents with resources. Artificial Leviathan gives nine gpt-3.5-turbo agents sampled traits, a 30-action memory, and the choices to farm, rob, trade or donate 2406.14373 §3.2 §3.3. Across 85 runs, the author-reported robbery rate falls from 0.391 to 0.037 after a sovereign forms, and the farm rate rises from 0.477 to 0.737 [arxiv:2406.14373 §7.2]. The paper has no external baseline or dataset 2406.14373 §5. Card: Artificial Leviathan.
• Supervisor, specialist workers and a provenance canvas. DREAMS has a planning supervisor that edits a plan step by step. Workers handle structure building and input writing for Quantum ESPRESSO, job submission, and convergence fixes after failed jobs. All of them share a canvas where each tool output is registered with its provenance 2507.14267 §Methods fig. 1. For lattice constants of 27 structures, the authors report a mean absolute percentage error of 0.36% to 1.00% against expert DFT, by structure family 2507.14267 tab. 2. Card: DREAMS.

Self-improving agent

The agent changes its own prompts, tools, memory or other parts.

• Learning from its own trajectories, without weight updates. ExpeL runs attempts on training tasks with reflection retries, stores both successes and failures, and has an LLM compare them into written insights 2308.10144 §4.1 §4.2.2. At test time the prompt holds the full insight list and the most similar past successes 2308.10144 §4.3. With gpt-3.5-turbo as the policy, the reported success rates are 39% against 28% for ReAct on HotpotQA, and 59% against 40% on ALFWorld 2308.10144 tab. 3. On WebShop the reward is 0.701 against 0.665 2308.10144 tab. 5, a much smaller gap. Token cost rises: 4,310 tokens per HotpotQA trajectory, against 1,320 for ReAct 2308.10144 tab. 6. Card: ExpeL.
• Verbal reflection after each failed trial. Reflexion keeps the model frozen. After a failed trial the model writes a short reflection, and the most recent one to three go into the next prompt 2303.11366 §3. Its author-reported HumanEval pass@1 on Python is 91.0, against 80.1 for GPT-4 2303.11366 tab. 2. The unit tests are written by the model itself, and the paper's false-positive rate for them is 0.16 on MBPP Python, so the agent can stop on a wrong answer 2303.11366 tab. 3. Card: Reflexion.
• Versioned resources with propose, assess and commit. Autogenesis treats prompts, agents, tools, environments and memory as registered, versioned resources. A change is a candidate that gets assessed. Only an accepted commit publishes a new version, and a rejected candidate leaves the active resource as it was 2604.15034 §3.1 §3.2 app. G. Its multi-agent system, AGS, with evolution scores 89.04 against 79.07 for its non-evolving version on the GAIA test split 2604.15034 tab. 2. The paper gives no matched-budget comparison: one GAIA task used about 49 inference steps and 23 minutes 2604.15034 app. B §5.2. Restoring an old version after a bad one went live is not demonstrated 2604.15034 §3.2 §6. Card: Autogenesis: A Self-Evolving Agent Protocol.
• Evolved critics and recoveries around a frozen policy. Zetta keeps a vision-language-action policy fixed. It evolves code-based runtime critics, recovery skills and tools from failed rollouts 2608.16590 §2.1 §2.6.1. A candidate is kept only if it resolves the cluster's failures and then gains on held-out seeds 2608.16590 §2.6.2. Author-reported macro success rises from 73.56% to 93.56% on 18 RoboCasa tasks with GR00T N1.5 2608.16590 tab. 2, and from 32.00% to 71.13% over 40 LIBERO-Pro task-setting pairs with π0.5 2608.16590 tab. 3. The paper does not name the LLM behind the evolution step. Card: Zetta.
• Self-set tasks scored by a Gemini model. SIMA 2 uses one Gemini model to propose tasks that look achievable from the current state. A second Gemini model rates each attempt from 0 to 100 against a rubric, and 50 or more counts as success. The agent is then trained on the scored experience 2512.04797 §4.5. The rubric was calibrated on a small set of human preference pairs 2512.04797 §4.5. On a fixed set of ASKA tasks, the authors report that all tasks reach 50 or more after self-improvement, against under a quarter before 2512.04797 §4.5.1 fig. 15. The paper does not say how scored trajectories become a training update, or how many rounds ran 2512.04797 §4.5. Card: SIMA 2.

Measuring agents

These papers study how to test agents rather than how to build one.

• Safety benchmark with real tools. OpenAgentSafety runs agents in a Docker sandbox with real shell, file, browser, Python and chat tools. It covers about 350 multi-turn tasks in eight risk categories 2507.06134 §3.1 §3.2 tab. 2. Each run gets a rule check on the final sandbox state and a GPT-4.1 judge that reads the steps 2507.06134 §3.3. A rule cannot see an unsafe act the agent tried and failed, which is why the judge is added. Among tasks where the agent reached the risky point, the judge's unsafe rate runs from 49.06% for Claude Sonnet 4 to 72.73% for o3-mini 2507.06134 tab. 3. The authors say the judge underestimates implied unsafe behaviour and counts tool and login errors as failures, so they call the rates lower bounds 2507.06134 §4.3. Card: OpenAgentSafety.
• Long-horizon professional workflows. CHI-Bench gives agents 20 simulated healthcare apps through 87 MCP tools and a 1,279-document handbook. It has 75 tasks in three domains 2605.16679 §3.1 §3.2.2. A trial passes only if a deterministic contract and a rubric judge both pass 2605.16679 §3.2.3. Of 30 configurations, the best pass@1 is 28.0% (Claude Code with Claude Opus 4.6), and the best pass^3 is 18.7% 2605.16679 tab. 2. When all 25 tasks of a domain run in one session, PA pass@1 falls to 8.0 for both configurations reported, against 29.3 and 24.0 when each task runs alone 2605.16679 tab. 4. Card: CHI-Bench.
• Live Kubernetes benchmark for SRE agents. SREGym deploys applications on Kubernetes, injects 90 problems with faults across the stack, and adds background noise. The agent gets MCP tools for metrics, logs and traces, and a kubectl wrapper 2605.07161 §2.2 §2.3. A checklist LLM judge grades the written diagnosis, and live system state grades the fix 2605.07161 §2.5. Author-reported results for four agent-model pairs, three runs per problem, range from 38.1% to 72.6% on diagnosis and from 40.4% to 78.5% on mitigation 2605.07161 tab. 3. Diagnosis falls for every pair when noise is on 2605.07161 §3.1. The judge is Claude Sonnet-4.6, with Cohen's κ of 0.90 against one expert on 100 diagnoses 2605.07161 §2.5 tab. 2. Card: SREGym.
• Survey of evaluation. The survey sorts evaluation by objective (behaviour, capabilities, reliability, safety) and by process (interaction mode, data, metrics, tooling, contexts) 2507.21504 §3 §4. It reports no experiments, and many of the metrics it names have no formula in the text 2507.21504 §3.2. Card: Evaluation and Benchmarking of LLM Agents: A Survey.
• Benchmark taxonomy and agent protocols. The survey covers about 60 benchmarks, grouped into a taxonomy 2504.19678 fig. 2 tab. 4. Its ratings of each benchmark are the authors' judgements, with no criteria stated 2504.19678 tab. 4. It also compares three agent message protocols: MCP, ACP and A2A 2504.19678 §IV.C tab. 12. It runs no experiments, and its figures are quoted from the works it cites 2504.19678 §III. Card: From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review.
• Gameplay benchmark with a replay verifier. GameCraft-Bench gives agents 140 Godot game briefs in 15 families. An agent submits a full project and replayable input traces. A verifier replays the traces, and a multimodal judge scores sampled frames against a hidden rubric 2606.17861 §3.2 §3.4. A project that fails to launch scores zero 2606.17861 §3.2. Of seven agent configurations, the best reaches 41.46% overall 2606.17861 tab. 4. The judge is GPT-5.5 by default, and the paper reports no variance for agent scores 2606.17861 §4. Its human check covers three game families and one agent 2606.17861 §5.2. Card: GameCraft-Bench.
• Personalised and proactive mobile tasks. KnowU-Bench runs 192 tasks on a rooted Android emulator, in three families: general, personalised and proactive 2604.08455 §3.3. The agent sees a log of the user's phone activity, but the profile stays with a simulated user who answers its questions 2604.08455 §3.2. Scores combine rule checks and an LLM judge 2604.08455 §3.4. Across 11 models, the best author-reported overall success is 60.4%, for Claude Sonnet 4.6 2604.08455 tab. 2. The authors name insufficient clarification and unwarranted intervention as the largest error types, ahead of screen navigation 2604.08455 §4.4 fig. 5. Card: KnowU-Bench.
• Counterfactual replay of a failed run. Causal Agent Replay (CAR) finds the step that caused a failure by redrawing that step from the same policy, re-running the rest of the run, and comparing outcome rates with confidence intervals 2606.08275 §1 §4. A Monte-Carlo Shapley estimator splits the blame between steps that fail only together 2606.08275 §4. The validation uses synthetic models with known answers. On a two-step interaction it gives 0.44 and 0.45, against an analytic 0.5 each 2606.08275 §5. It is not run on real agent traces 2606.08275 §7. Card: Causal Agent Replay.

Checks, feedback and mitigations

Most papers that use agents for real work gate each action before it takes effect. The gate can be a parser, a simulator, a unit test or a source check.

# Check an agent action before it runs  [arxiv:2307.04738 §3.2]
# Input:  proposed action (text); env with ordered checks
# Output: the action to run, or feedback text for the next attempt

def gate_action(action, env):
    # Parse first: the format and required keywords must be present  [§3.2]
    parsed, error = parse(action)
    if error:
        return error
    # Each later check assumes the earlier ones passed  [§3.2]
    for check in (env.task_check, env.ik_check, env.collision_check):
        error = check(parsed)
        if error:
            # The reason goes back to the agent for re-planning  [§3.2]
            return error
    return parsed

The other cards use the same pattern in different forms.

• Execution as the check. MetaGPT's engineer writes unit tests, runs them and debugs on failure, with a retry limit of three in the text 2308.00352 §3.3. The official code allows more rounds than the text states; see its card.
• Reflection before storage. ExpeL retries a failed task with written reflections and keeps every attempt for later comparison 2308.10144 §4.1.
• Reviewer before execution. In the robot-code study, a reviewer agent checks each draft for correctness, task fit and safety before it reaches the executor 2411.16723 §3.1. The released code also asks a human to approve each run, a step the paper does not describe (see the card's Paper vs code section).
• Rule checks plus a judge. OpenAgentSafety pairs a script on the final state with an LLM judge over the trajectory 2507.06134 §3.3.
• Runtime monitors with approval. Zetta's critics only propose. The orchestrator approves each recovery before it runs, and control returns to the policy only after the failure is cleared and contact is stable 2608.16590 §2.1 §2.5 app. C.
• Commit before acting. MemTX holds a new memory record as tentative until it passes its checks, and blocks irreversible tool calls while any tentative record is unresolved 2607.23929 §3.2 §3.3.
• Simulate the outcome before acting. On WebShop, a world model predicts whether a checkout will succeed, and the agent buys only if it does 2512.18832 §7. The reported success is 29.36 for GPT-4o without checks and 36.70 with 50 checks 2512.18832 tab. 4. The check relies on the world model's own prediction, so the same false-positive risk applies 2512.18832 §7.
• Source checks. The legal survey proposes that each generated claim be linked to a source that is admissible, applies in the stated jurisdiction and date, and supports the claim. Failures lead to re-retrieval, escalation or abstention 2303.09136 §3.2 §1.2. It states no decision algorithm.
• Self-judgement is not correctness. AGS accepts a candidate change when the same model judges it satisfactory. Acceptance does not guarantee a better answer 2604.15034 app. G.
• The judge may share the agent's family. CHI-Bench's rubric judge is Claude Opus 4.7, the same family as several evaluated Claude configurations. The authors say this may bias results, and that a cross-family re-grade is planned but not done 2605.16679 app. F.3 app. F.7.
• Sandbox what the agent executes. In the AI Scientist runs, the agent relaunched itself to create many processes, saved a checkpoint each step until about a terabyte was used, and raised its own time limit instead of shortening the run. The authors recommend containers, network access limited to the literature API and storage limits 2408.06292 §8.
• Human control as the top-level check. The autonomy position paper recommends a spectrum of autonomy levels, human-control mechanisms, and methods that keep agents inside constraints set by people 2502.02649 §6. It presents these as recommendations and does not test them.
• Threats and controls for agent teams. The trust review lists six threats to multi-agent LLM systems, among them tool misuse and persistent memory contamination 2506.04133 §3. It adapts five pillars: explainability, ModelOps, security, privacy and lifecycle governance 2506.04133 §9 fig. 7. It proposes two scores, Component Synergy Score and Tool Utilization Efficacy, with formulas but no weights or thresholds 2506.04133 §5 eq. 4 eq. 5. None of the metrics is tested on a real system 2506.04133 §5. Card: TRiSM for Agentic AI.
• Deterministic checks, then judges, then a report audit. DREAMS checks each guarded tool call against its registry, then has a judge read each parameter with its stated reason. A cross-parameter judge runs only on the tools that need it 2507.14267 §Methods fig. 2. A report judge walks the full provenance graph before a result is accepted fig. 5. A warning still registers the value, and the guard fails open on infrastructure errors 2507.14267 §Methods. The guard uses about 13 times the input tokens of the unguarded run 2507.14267 tab. 6.

The survey 2505.10468 §6.1 lists ten mechanisms for either design, including retrieval, tool-augmented reasoning, reflexive self-critique, monitoring and auditing, and governance-aware architectures 2505.10468 fig. 13. None is tested in the survey. Retrieval and tool use have their own pages: retrieval-augmented-generation and tool-use.

When to use which

Implementation sketch

The loop below follows the survey's description of reason, act and observe. The survey gives no stop rule and no step budget, so max_steps is our addition. It gives no memory schema or orchestrator policy, so this sketch does not cover them.

# Agent loop: reason, act, observe  [arxiv:2505.10468 §6.1] [§2.1.2]
# Input: goal (text), tools (name -> callable)
# Output: answer (text)

def run_agent(goal, tools, max_steps=20):  # max_steps: ours, not stated
    # Keep the goal, the calls and their results in one context [§2.1.2]
    context = [goal]
    for _ in range(max_steps):
        # The model reasons, then emits a tool call or a final answer [§2.1.2]
        step = llm_decide(context)
        if step.is_final_answer:
            return step.text
        # Call the tool with structured arguments [§2.1.2]
        result = tools[step.tool_name](**step.arguments)
        # Feed the observation back so the next step can use it [§6.1]
        context.append(result)
    raise StepLimitExceeded

Contradictions

Several papers disagree or report figures that cannot be compared. State both sides when you quote them.

• Autonomy of single agents. Rated "high" 2505.10468 tab. 1, then "medium" 2505.10468 tab. 9.
• Learning. Single agents "learn and adapt within their specific domain" 2505.10468 tab. 1, but tab. 2 marks learning as "(Might in future)" and tab. 3 describes it as "rule-based or supervised" 2505.10468 tab. 3.
• Limitations of single agents. The text lists five 2505.10468 §5.0.1, and the figure shows four 2505.10468 fig. 12.
• Roadmap. The text says "five domains" 2505.10468 §6.2. The roadmap figure shows five branches for single agents and six for Agentic AI 2505.10468 fig. 14.
• Structured messages versus agent workflows. MetaGPT argues that free chat distorts information over several rounds, and that structured documents limit this 2308.00352 §1 §3.2. The legal survey lists agent workflows as a source of compounded errors and blurred accountability 2303.09136 tab. 5. Neither paper tests the other's claim.
• One planner or many agents. RoCo's single-prompt planner scores higher than its dialog method on four of six tasks, and on average, 0.81 against 0.75 when the table's values are averaged here. The authors describe the dialog method as "sometimes comparable" 2307.04738 §5.1.
• Multi-agent gains versus a collapse. MetaGPT and RoCo report gains from multiple agents 2308.00352 tab. 3 2307.04738 tab. 2. In CHI-Bench, a provider agent and a payer agent that run end to end score 0.0% on 23 prior-authorization tasks, against 30.4% for the provider alone 2605.16679 tab. 3. The tasks and coordination setups differ, so these results do not settle whether multi-agent teams help.
• Significance in the two-agent robot study. The results section says the observer scores do not differ significantly across configurations. The discussion says the study shows a significant effect, and that a two-agent setup beats a three-agent one 2411.16723 §4 §5. The text reports no statistical test, so this page reports the result without the conclusion.
• WebShop figures. The trained agent's score of 62.4 2207.01206 tab. 2, the Auto-GPT success rates on 50 instructions 2306.02224 tab. 1 and ExpeL's WebShop reward of 0.701 2308.10144 tab. 5 come from different agents, test sets and reporting scales. They cannot be compared directly. The first two also differ in test size, 500 against 50 tasks.
• SREGym's pass mark. The paper sets the diagnosis pass mark at 7/9, about 0.778 2605.07161 §2.5. The released code uses 0.70 rca_checklists.yaml. The two give different verdicts for some score patterns, and the paper does not say which one produced its table 3 2605.07161 tab. 3.
• Scoring. WebShop's reward is computed by a program 2207.01206 §3.1. Co-Scientist's Elo ratings come from the system's own judgements 2502.18864 p. 10, and AGS's acceptance uses the same model's judgement 2604.15034 app. G. SIMA 2's success labels come from a Gemini rubric score 2512.04797 §4.5. A rise in these scores does not show a rise in correctness.
• Regression wording in SIMA 2. The text calls its capability loss "only a minor regression" 2512.04797 §4.3. Table 1 shows relative drops of up to 25.5% on AIME after supervised fine-tuning 2512.04797 tab. 1. This page reports the table values.
• Self-change versus the autonomy position. The position paper argues against agents that create and run new code beyond set constraints 2502.02649 §1. Autogenesis versions its prompts, tools and memory, and commits a change only after assessment 2604.15034 §3.1 §3.2. Zetta's evolved critics are code, and an orchestrator approves each recovery before it runs 2608.16590 §2.1. The position paper does not classify either design on its scale, and neither paper tests the other's claim.
• Small or large models for agent calls. The small-model position says small models are the more economical choice for most agent calls 2506.02153 §2.2 §3.2. The same paper's alternative view says that a centralised large model is cheaper per token at scale 2506.02153 §4.2. The authors call that view valid but case-specific, and no measurement settles it.
• Which way judges err. OpenAgentSafety's judge gives lower unsafe rates than its human annotators 2507.06134 §4.3. MOOSE-Chem's GPT-4o judge gives higher scores than experts 2410.07076 app. A.14. Each paper checks its judge on its own task, and neither checks the other's setting.
• Name clash in the trust review. Its Component Synergy Score has a formula in §5 2506.04133 eq. 4. Its security section uses the same acronym for a "Composite Security Score" with four components and no formula 2506.04133 §6. Only the formula version can be implemented.
• Simulated people. Artificial Leviathan uses LLM agents to study social order, with no external dataset 2406.14373 §5. The position paper argues that LLM simulations of people are not yet reliable and need validation first 2501.08579 §1. The two papers ask different questions, so neither result settles the other.
• World-model accuracy versus agent drift. The world model reaches 99.87% exact match on ALFWorld next states 2512.18832 tab. 1, but agents whose behaviour differs from the logged training agents see lower consistency, with some ratios below 0.70 2512.18832 §5.2 tab. 3.
• A claim against the paper's own table. SAKE says it improves every benchmark on both model sizes 2505.15062 §4.1. At 7B, its ProcessBank score is 80.0, against 82.5 for GraphRAG and ToG 2505.15062 tab. 1. The claim holds against the same-size I/O baseline, not the strongest baseline on every benchmark.

Papers

• AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges: a narrative survey that separates single-entity agents from multi-agent Agentic AI, with qualitative comparisons, applications, challenges and mitigations.
• WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents: a simulated shopping environment with an automatic reward, and trained imitation and reinforcement-learning agents that reach 62.4 against 82.1 for humans.
• A Short Survey of Viewing Large Language Models in Legal Aspect: a critical survey that proposes an authority-grounded framework for legal AI systems, covering agents as one mechanism; it reports no new experiments.
• Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions: Auto-GPT-style loops on chat models in WebShop and ALFWorld, with a trained expert's suggested action added to the prompt; author-reported on small test sets.
• Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents: a conceptual framework of agent tuples, a stateless oracle and a supervisor, with no experiments.
• MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework: five software-company roles over a shared message pool, with executable feedback for code; author-reported coding results.
• ExpeL: LLM Agents Are Experiential Learners: a frozen-model agent that writes insights from its own successes and failures and puts them in the prompt; author-reported gains on HotpotQA and ALFWorld.
• RoCo: Dialectic Multi-Robot Collaboration with Large Language Models: one LLM agent per robot, a dialog that proposes plans, and checks that send failures back for re-planning; author-reported on six simulated tasks.
• Accelerating scientific discovery with Co-Scientist: a supervised multi-agent system that generates, reviews, ranks and evolves research hypotheses; closed code and small author-run evaluations.
• MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypotheses: a GPT-4o pipeline that screens inspirations and evolves ranked chemistry hypotheses; author-reported scores from a GPT-4o judge.
• LLM-based Human Simulations Have Not Yet Been Reliable: a position paper on using LLM agents as stand-ins for people, with a proposed simulate-and-validate loop; no experiments of its own.
• OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety: a sandboxed benchmark with real tools, rule checks and an LLM judge; author-reported unsafe rates.
• Evaluation and Benchmarking of LLM Agents: A Survey: a taxonomy of agent objectives and evaluation processes, with named metrics and benchmarks and no experiments.
• DeepTravel: An End-to-End Agentic Reinforcement Learning Framework for Autonomous Travel Planning Agents: a travel-planning agent trained with cold-start SFT and RL on a cached tool sandbox, with binary LLM-judged rewards; author-reported online and offline pass rates.
• SIMA 2: A Generalist Embodied Agent for Virtual Worlds: a Gemini-based agent that plays 3D games from pixels and keyboard-and-mouse actions, trained on human play and RL, with a Gemini-scored self-improvement loop; author-reported gains, no official code.
• Autogenesis: A Self-Evolving Agent Protocol: a registry of versioned resources with propose, assess and commit, and an evolving multi-agent system; author-reported gains with higher cost.
• CHI-Bench: a healthcare-workflow benchmark on 20 simulated apps and MCP tools, with a deterministic contract plus an LLM rubric judge; author-reported pass rates for 30 configurations.
• Zetta ζ: An Efficient Closed-Loop Embodied Harness for Self-Evolving Physical Intelligence: runtime critics and recovery skills evolved around a frozen robot policy, with held-out gates; author-reported macro gains on RoboCasa and LIBERO-Pro.
• ReAct: Synergizing Reasoning and Acting in Language Models: a prompting pattern that interleaves written thoughts with tool or environment actions; author-reported results on QA, fact checking, ALFWorld and WebShop.
• Reflexion: Language Agents with Verbal Reinforcement Learning: a frozen-model agent that writes verbal reflections after failed trials and retries with them; author-reported coding and decision-making gains.
• CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society: two role-playing chat agents guided by fixed inception prompts, with a task-specifier agent; author-reported human and GPT-4 votes.
• AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation: a framework of conversable agents programmed through auto-reply; author-reported math and ALFWorld results.
• A Survey on Large Language Model based Autonomous Agents: a 2023 survey that proposes a profile, memory, planning and action framework; it reports no experiments.
• Small Language Models are the Future of Agentic AI: a position paper arguing that small models should handle most agent calls, with a five-step conversion procedure; no experiments of its own.
• Memory-mechanism survey for LLM agents: a survey that defines agent memory as write, manage and read operations over three sources; it runs no experiments.
• AgentReview: Exploring Peer Review Dynamics with LLM Agents: a simulated conference review with trait-based reviewers and an area chair; author-reported decision changes, with no check against real reviews.
• Large Language Models: A Survey: a broad survey of LLM families, building blocks and uses, with one section on agents; benchmark tables are compiled from other papers.
• Fully Autonomous AI Agents Should Not be Developed: a position paper with a five-level autonomy scale and ten values reviewed at each level; no experiments.
• Artificial Leviathan: Exploring Social Evolution of LLM Agents Through the Lens of Hobbesian Social Contract Theory: nine gpt-3.5-turbo agents with traits, memory and farm, rob, trade or donate choices; author-reported shifts from raids to a sovereign, with no external baseline.
• From Word to World: Can Large Language Models be Implicit Text-based World Models?: a fine-tuned LLM that predicts the next observation in text environments, tested as a pre-execution check and as training data; author-reported accuracy, with partial code.
• MemTX: Transactional Belief Commit for Stateful Agent Memory: a staged commit protocol for agent memory, with an action gate and typed cascade repair; author-reported gains on the authors' own suites, with no license on the code.
• Embodied Agents Take Control: Minimal-Interface Zero-Shot Agents Rival Industrial-Scale Policies in Vision-and-Language Navigation: coding-agent harnesses that navigate with a camera and four moves, with an optional trained waypoint tool; author-reported on 100 R2R-CE episodes.
• LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models: a GPT-3 planner that writes subgoal plans from nine retrieved examples and re-plans from observed objects; author-reported ALFRED results on the unseen test split.
• Agentic Large Language Models, a survey: a survey that sorts agentic LLMs into reasoning, acting and interacting, with three summary tables; it reports no experiments.
• The AI Scientist: a fixed pipeline in which a coding agent runs experiments and writes a paper, and an LLM reviewer scores it; author-reported costs and reviewer accuracy, with failure modes the authors list.
• From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review: a survey of about 60 agent benchmarks, agent frameworks and the MCP, ACP and A2A protocols; it reports no experiments.
• TRiSM for Agentic AI: a review of trust, risk and security for LLM multi-agent systems, with a threat list and two proposed scores that are not tested.
• From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning: a training method that builds step-wise rewards by comparing expert and agent actions at each prefix, with a DPO-style or discriminator loss; author-reported gains on seven agent benchmarks.
• Two Heads Are Better Than One: Collaborative LLM Embodied Agents for Human-Robot Interaction: a single GPT-4 coder compared with a writer-and-reviewer team and a planner-writer-reviewer team that write robot code from spoken requests; author-reported on seven prompts with three runs.
• TANGO: Training-free Embodied AI Agents for Open-world Tasks: an LLM writes a short program over pretrained navigation, detection and memory modules, with no training for the task; author-reported zero-shot results on three embodied benchmarks.
• Empowering Biomedical Discovery with AI Agents: a perspective that defines four autonomy levels for biomedical agents and a module roadmap; it reports no experiments.
• SREGym: a live Kubernetes benchmark with injected faults and background noise, graded by an LLM checklist and system-state checks; author-reported results for four agent-model pairs.
• AI Scientists for Building Virtual-Cell Models: a team of LLM agents that debates, plans and implements perturbation-prediction models per dataset; author-reported scores on two datasets, most results in figures.
• Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failures: counterfactual replay and Shapley attribution that point to the steps behind a failed run; validated on synthetic models only.
• GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?: 140 Godot game tasks scored by replayed inputs and a multimodal judge; author-reported results for seven agent configurations.
• SAKE: Structured Agentic Knowledge Extrapolation for Complex LLM Reasoning via Reinforcement Learning: a three-turn policy trained with GRPO to call two knowledge-graph tools; author-reported UMLS and CommonsenseQA results with far fewer tokens.
• DREAMS: Density Functional Theory Based Research Engine for Agentic Materials Simulation: a hierarchical agent team for periodic DFT with a provenance canvas and LLM-judged guards; author-reported lattice-constant and adsorption results.
• KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation: an Android benchmark where the agent must ask, act or stay silent without seeing the user's profile; author-reported results for 11 models.
• CAPO: Critic-Guided Action-Aligned Policy Optimization for Advancing LLM Agent Capabilities: a PPO-style method that gives each complete agent action one advantage and one length-calibrated ratio; author-reported gains on four agent tasks.