Evaluation and Benchmarking of LLM Agents: A Survey

In plain words

An LLM agent is a program that uses a language model to plan, call tools and take steps toward a goal. Testing one is harder than testing a plain language model, because the agent acts over many steps, may change things in the real world, and can give different results on the same input. This survey is a guide to the many ways people test such agents.

The authors sort existing work with two questions. The first is what to test: whether the agent finishes its task, what it can do, whether it is reliable, and whether it is safe. The second is how to test it: whether it runs on fixed test cases or live interaction, what data and benchmarks are used, how results are scored, which tools help, and where the test runs.

The authors also point out problems that matter in companies, such as limiting an agent to the data a user is allowed to see, keeping results repeatable enough for audits, and following internal rules and laws. They end by calling for evaluations that are more complete, closer to real use, cheaper to run, and automated.

TL;DR

The paper is a literature survey, not a new method. It organises agent evaluation into a two-dimensional taxonomy: evaluation objectives and evaluation process §2. It names the metrics and benchmarks that prior work uses for each part §3 §4 tab. 1. It adds four enterprise challenges §5 and four research directions §6. It runs no experiments of its own and reports no measured results.

Problem

Language models are usually judged on text generation or question answering. Agents act in dynamic environments, use tools, keep memory and work with other agents, so those tests miss important behaviour §1. Agent evaluation also differs from traditional software testing because agent behaviour is probabilistic §1. The authors say that prior surveys cover LLM evaluation narrowly or look at single agent capabilities, and that enterprise needs such as data access control, reliability, audit and compliance are seldom studied §1.

Method

The paper has no algorithm, model or training procedure. Its contribution is the taxonomy §2, which the paper presents with a tree figure fig. 1. The card follows the taxonomy's structure.

1. Evaluation objectives: what to evaluate §3.
   1. Agent behavior §3.1: the outside view of the agent, as the user sees it. Covers task completion, output quality, latency and cost.
   2. Agent capabilities §3.2: how an agent uses tools, plans and reasons, keeps memory, and works with other agents.
   3. Reliability §3.3: consistency across repeated runs of the same task, and robustness when inputs or the environment change.
   4. Safety and alignment §3.4: fairness and explainability, harm, toxicity and bias, and compliance and privacy.
2. Evaluation process: how to evaluate §4.
   1. Interaction mode §4.1: offline evaluation on static datasets, and online evaluation with simulated or real users and environments. The paper also presents evaluation-driven development as a process that runs both offline and online §4.1.
   2. Evaluation data §4.2: synthetic, human-annotated and interaction-generated data, plus leaderboards.
   3. Metric computation §4.3: code-based checks, LLM-as-a-judge, agent-as-a-judge, and human-in-the-loop review. The paper names the trade-offs of each §4.3.
   4. Evaluation tooling §4.4: frameworks and platforms that automate and repeat evaluations.
   5. Evaluation contexts §4.5: from controlled simulators and mocked APIs or sandboxes to live deployment. The paper calls this a trade-off between realism, cost and security §4.5.
3. Enterprise challenges §5. Role-based access control §5.1, reliability guarantees §5.2, dynamic and long-horizon interaction §5.3, and adherence to domain policies and compliance §5.4.
4. Future directions §6. Holistic evaluation, more realistic settings, automated and scalable evaluation, and time- and cost-bounded protocols §6.

The metrics and benchmarks the paper names for each part are listed in the Results section below.

# Two-dimensional taxonomy of agent evaluation  [§2] [fig. 1]
# Each leaf is a category the paper discusses in the cited section

taxonomy = {
    "objectives": {  # what to evaluate  [§3]
        "behavior": [...],  # task completion, output quality, latency and cost  [§3.1]
        "capabilities": [...],  # tool use, planning, memory, collaboration  [§3.2]
        "reliability": [...],  # consistency, robustness  [§3.3]
        "safety": [...],  # fairness, harm, compliance and privacy  [§3.4]
    },
    "process": {  # how to evaluate  [§4]
        "interaction_mode": [...],  # offline versus online  [§4.1]
        "data": [...],  # synthetic, real, benchmarks, leaderboards  [§4.2]
        "metrics": [...],  # code-based, LLM judge, human review  [§4.3]
        "tooling": [...],  # frameworks and platforms  [§4.4]
        "contexts": [...],  # simulators, sandboxes, live deployment  [§4.5]
    },
}

Hyperparameters

Not applicable. The paper is a survey and trains or runs no models. No learning rates, prompts, sampling settings, model versions or run counts are given anywhere in the paper §1–§6.

Results

Not applicable. The paper reports no experiments and no measured numbers of its own. The table lists the metrics and benchmarks the paper names for each part, with the section where the text discusses them. Each item is the survey's description of prior work, so no value is checked here. Figures for other papers are not restated, because the paper does not give them in the text.

The full table of objectives and metrics is tab. 1 in the paper.

Paper vs code

No official code. Searched: (1) the paper's LaTeX source, where neither the abstract nor the body links a code repository or project page; (2) no project page is linked, so none was opened; (3) GitHub repository search for the method-less title and the authors' organisation. Two repositories mention the paper's title: SAP-samples/llm-agents-eval-tutorial, described as tutorial materials for the paper, and mahmoodm2/KDD2025_Tutorial, a personal account. The paper does not link either, and neither is clearly the authors' official code, so neither is used.

Pitfalls

• No experiments, so no numbers to trust. The paper's claims about how models perform come from other papers, as summarised by the authors. Verify any figure against its source before you rely on it.
• Metrics are named, but few are defined. The text names many metrics, such as invocation accuracy, MRR, NDCG, progress rate, step success rate, consistency score and factual recall accuracy, without formulas or definitions §3.2. Formulas: not stated.
• Named metrics are not always tied to a benchmark. The "Latency & Cost" and "Multi-Agent Collaboration" rows in the table name metrics whose definitions are not in the text tab. 1 §3.2.
• Internal inconsistencies. The table lists AgentBoard twice in the task-completion row tab. 1. ToolEmu is cited for error handling and robustness in the text §3.3 but is missing from the robustness row of the table tab. 1; its error-handling row is commented out in the source. Explainability appears under the fairness heading §3.4.
• Source notes not used here. Commented-out text in the source (for example, a figure of "under 25%" for pass^k in the τ-benchmark) is not in the published paper, so this card does not use it §5.2.
• Tool-use scope is narrow. The paper defines tool use as one tool call, and treats multi-tool sequencing as planning §3.2.
• Enterprise topics are described, not measured. Role-based access control, reliability guarantees, long-horizon use and compliance are argued from examples, not tested §5. The paper cites a benchmark that embeds role-based access checks §5.1.
• Tooling claims rest on practice, not literature. The paper says OpenAI Evals is not formally described in academic literature and is included because of practical need §4.4.

Lineage

• Builds on: none stated. The introduction says prior surveys cover LLM evaluation narrowly or look at single agent capabilities §1.
• Compared against: the authors describe prior surveys as narrower in scope §1. No side-by-side comparison is given.
• Superseded by: not known from content in this wiki.