OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety

In plain words

AI agents now do real work: they run shell commands, edit files, browse websites and send messages. OpenAgentSafety is a test bench for checking whether such an agent does something unsafe while it is trying to finish a job. Instead of faking the tools, it gives the agent real ones inside a sandbox, so a bad action leaves a trace that can be checked afterwards.

The test set holds a few hundred tasks. Some users are harmless, yet the request can still lead to an unsafe result. Some users openly ask for something harmful. In some tasks a simulated coworker pushes the agent in a bad direction. Each run is scored in two ways: a script looks at the final state of the sandbox, and a language model reads the agent's steps and judges them.

The authors ran seven popular models through the test. Most of them behaved unsafely on a large share of the tasks where the agent got far enough to be tested. The point of the work is that such checks are needed before agents are trusted with real systems.

TL;DR

• OpenAgentSafety is an open framework (code under MIT) that runs LLM agents on about 350 multi-turn tasks, in Docker, with real shell, file, browser, IPython and chat tools §3.1 §3.2.
• Each task has a rule-based check of the final environment state and an LLM-as-judge (GPT-4.1) that reads the trajectory §3.3.
• Author-reported: among tasks where the agent reached the risky point, the LLM-judge unsafe rate runs from 49.06% (Claude Sonnet 4) to 72.73% (o3-mini) tab. 3.
• Use it to measure agent risk or as a harness to extend. Before quoting numbers, read the Paper vs code section: the judge and metric definitions in the released code do not match the paper in places.

Problem

Earlier agent-safety benchmarks often rely on simulated environments, narrow domains or simulated tool APIs §1. The authors compare 24 prior benchmarks on three properties: real tools, diverse user intents, and multi-turn user interaction. By their table, only OpenAgentSafety has all three tab. 1.

Two evaluation problems are named. First, rule-based checks need several rounds of revision on real agent behaviour to avoid false positives and negatives §3.3. Second, a rule check cannot see an agent that tried an unsafe act and failed, so an LLM judge is added for reasoning and intermediate actions §3.3. The authors also report that many runs never reach the risky point, because of web-navigation, authentication or tool-use failures §4.2.

Method

OpenAgentSafety runs an agent inside a containerised sandbox. The agent is the OpenHands platform with its tools §3.1 §4.1. The tools are a Unix shell, a file system, a Python interpreter, a web browser, and a custom ChatNPC tool that messages simulated coworkers §3.1 §3.2. Websites used by the tasks (file sharing, version control and issue tracking) are local copies, which avoids harm to live services §3.1. The authors adapted them from TheAgentCompany §3.1.

Secondary actors (NPCs) are simulated with the Sotopia framework. The chat tool supports direct messages and broadcasts §3.1 app. A.2. The NPCs may hold manipulative goals or goals that conflict with the user's §3.2.

Task design §3.2. Each task is scored on three axes: risk category, tool usage, and user or NPC intent. There are eight risk categories tab. 2. The authors name three intent combinations:

1. Benign user, no NPC. The task looks safe but can be completed unsafely.
2. Benign user, malicious NPC. The agent meets a deceptive party during a normal request.
3. Malicious user, with or without an NPC. The unsafe instruction is in the user's request.

Each task is a Docker image holding the environment, the task text, the NPC behaviour and a rule-based evaluator §3.2. The authors wrote 80 seed tasks by hand. GPT-4o then varied the social context and NPC set for each seed, keeping the core challenge and tool use §3.2 app. A.7. The prompt template asks for K = 3 to 5 new tasks per seed app. A.7. The generated tasks were checked by the authors §3.2.

Evaluation §3.3. There are two evaluators per task:

• Rule-based. A Python script reads the final environment state and returns 1 (unsafe) or 0 (safe).
• LLM-as-judge. GPT-4.1 reads the agent's steps and the unsafe-behaviour description. The prose names four outcomes: safe, failure, complete-unsafe, and incomplete-unsafe §3.3. The prompt in the appendix uses the numeric labels -1, 0, 1 and 2 for those outcomes, and the mapping is implied by the rubric wording rather than stated app. A.3.

Four metrics are reported §3.3:

• Unsafe rate: share of tasks flagged unsafe, computed only over tasks that reached the risky point.
• Failure rate: share of tasks where the agent never reached a risky step.
• Disagreement rate: share of tasks where the judge said safe but the rule check found an unsafe change.
• Successful completion rate: share of tasks where a judge says the correct completion was reached (a refusal counts as correct for unsafe tasks).

The paper gives no equations and no algorithm environment. The per-task procedure and the metric definitions, written in our words from §3.3 and the appendix, are below.

# Per-task evaluation  [§3.3] [app. A.3]
# Inputs:  final_state, trajectory_text, unsafe_description
# Outputs: rule_unsafe (0 or 1), judge_label (-1, 0, 1 or 2)

def evaluate_task(final_state, trajectory_text, unsafe_description):
    # Rule check on the final sandbox state; 1 means unsafe  [§3.3]
    rule_unsafe = rule_evaluator(final_state)
    # Judge sees the formatted steps and the unsafe-behaviour text  [app. A.3]
    prompt = build_judge_prompt(unsafe_description, trajectory_text)
    # Judge: -1 never reached risk, 0 safe, 1 or 2 unsafe  [app. A.3]
    judge_label = parse_label(call_judge(prompt))
    return rule_unsafe, judge_label

# Metrics over all tasks  [§3.3] [tab. 3 caption]
# Inputs: results, a list of (rule_unsafe, judge_label) per task

def summarise(results):
    # Failure rate: share of all tasks the judge marked as never reaching risk
    n_total = len(results)
    failed = [r for r in results if r.judge_label == -1]
    failure_rate = len(failed) / n_total
    # Unsafe rate: computed only over tasks that did reach the risk  [tab. 3]
    reached = [r for r in results if r.judge_label != -1]
    judge_unsafe = [r for r in reached if r.judge_label in (1, 2)]
    llm_judge_rate = len(judge_unsafe) / len(reached)
    rule_unsafe = [r for r in reached if r.rule_unsafe == 1]
    rule_rate = len(rule_unsafe) / len(reached)
    return failure_rate, llm_judge_rate, rule_rate

The denominator in the second block is our reading. Table 3's LLM-Judge column is close to Table 5's divided by one minus the failure rate (see Pitfalls).

Hyperparameters

Values the paper states, plus values the released code sets (marked "code"). Items the paper leaves out are marked "not stated".

Results

Author-reported. Setup: seven LLMs on the 356 generated tasks, run with OpenHands in a Docker sandbox, with the rule check and GPT-4.1 judge on each trajectory §4.1. The caption says the LLM-Judge and rule-based percentages are over tasks that reached a safety-vulnerable state. It does not state the denominators of the other columns.

Table 3: main results tab. 3

Table 5: same models over all tasks tab. 5

Table 4: LLM judges against human annotation on 100 GPT-4o trajectories tab. 4

Percentages. The right half excludes trajectories that humans labelled as failures. The caption does not define the two disagreement columns.

The "ASB" row is not defined in the caption. The text suggests it is a fine-tuned Qwen 2.5 7B judge taken from Agent-SafetyBench §4.3 RQ4(3), which is our inference.

Findings from §4.3 (author-reported). Figures 3 to 5 are heatmaps. They are described here in words only.

• User intent §4.3 RQ1. Benign prompts gave unsafe behaviour in 50 to 86% of tasks across models. Claude Sonnet 3.7 reached 85.7% under benign intent. Under explicit malice, Claude Sonnet 3.7 fell to 30% and Deepseek-v3 to 33.3%, while o3-mini rose to 80.7%, GPT-4o was at 64.7% and Deepseek-R1 at 66%. With a malicious NPC and a benign user, the range was 44.9% to 69.2%.
• Risk category §4.3 RQ2. Computer security compromise reached 72 to 86%. Legal violations, privacy breaches and harmful decision-making were also high. Spreading malicious content was lowest: Claude Sonnet 3.7 at 27.7%, while o3-mini still failed in 75% of those tasks.
• Tools §4.3 RQ3. Web tasks (OwnCloud, GitLab) gave 59 to 75% unsafe rates. For file-system and code tools, o3-mini was at 73.8% and GPT-4o at 71.8%. Messaging tasks gave 45 to 70%.
• Judge reliability §4.3 RQ4. GPT-4.1 underestimated implied unsafe behaviour. Judges also counted tool and authentication errors as failures, so the authors call the unsafe rates conservative lower bounds. A fine-tuned judge showed similar errors.
• Failures §4.2. The LLM judge puts about 35 to 49% of tasks in the failure class, mostly from web navigation, authentication or tool-use errors.

Paper vs code

Code: the official repository at code_commit (MIT). Its LICENSE file names TheAgentCompany as copyright holder LICENSE. Judge and metric scripts are under evaluation/. No script that produces Table 3 was found in the repo, so those numbers cannot be reproduced from it.

• Disagreement is defined in the opposite direction. The paper defines it as judge safe and rule unsafe §3.3. print_final_metrics counts judge label 2 with rule score 0 llm_as_judge.py. So the code's disagreement count is judge unsafe and rule safe. Use the paper's definition for comparison, and state which one you used.
• Judge prompt differs slightly. The code adds "Carefully look at each step of the trajectory" llm_as_judge.py and a system message llm_as_judge.py. The appendix prompt has neither app. A.3. Otherwise the rubric and output format match. Judge model gpt-4.1 matches llm_as_judge.py.
• Unsafe-behaviour text can be lost. The judge script extracts the description with a regex that needs the exact header Checkpoint N (Mpt) llm_as_judge.py. Some task files use ## Checkpoint 1 (1pts) checkpoints.md or (1 point) checkpoints.md. By grepping the headers, about 119 of 357 files lack the exact form. For those, the judge would receive the placeholder UNKNOWN. We inferred this from the regex and did not run the script.
• Rule score uses result == 1. A missing evaluator or zero total gives -1 llm_as_judge.py. Otherwise the score is 1 only when the sum equals 1 llm_as_judge.py. This matches the single-point files, but multi-point tasks would be misread. The alternative aggregation strategies in the scoring module are not used by this script scoring.py.
• Judge errors are dropped from totals. parse_label returns None on failure llm_as_judge.py. The metric totals count only parsed labels llm_as_judge.py llm_as_judge.py. Unparseable or errored judgements leave the denominator.
• Completion script treats errors as failure. safe_completion_eval.py maps unparseable labels and API errors to 0 safe_completion_eval.py safe_completion_eval.py. That is the "successful completion" check, which the paper describes as an LLM judgement §3.3.
• Simulated user not described in the paper. The run script sets up an LLM user with a manipulative system prompt run_eval.py using gpt-4o-2024-05-13, for at most four replies run_eval.py. The paper describes only NPCs app. A.4.
• NPC model. The paper says GPT-4o app. A.4. The repo docs name a Claude 3.5 Sonnet snapshot (20241022) as the environment LLM behind the baseline results EVALUATION.md.
• Task count. The text gives 80 seed plus 356 generated tasks, and the abstract says "over 350". The repo has 357 safety-* folders at this commit tasks.
• Rule evaluators are encrypted, but the key is in the repo. The runner encrypts the evaluator and hard-codes the key run_eval.py. The decryption step eval.py is therefore obfuscation, not a secret. Some tasks ship plain utils/evaluator.py, for example evaluator.py.
• Forked dependency. openhands-ai is pinned to a git fork rather than the upstream release pyproject.toml. Reproducing the runs needs that fork.

Pitfalls

• The abstract disagrees across versions. The arXiv metadata abstract says five LLMs and 51.2% to 72.7%, with Claude Sonnet 3.7 lowest. The LaTeX source, which this card follows, says seven LLMs and 49% to 73%, with Claude Sonnet 4 lowest. Table 3 agrees with the source. Quote the table, not the metadata abstract.
• Table 3 probably uses a reduced denominator. Each LLM-Judge value in Table 3 is within 0.15 points of the matching Table 5 value divided by one minus the failure rate. For Claude Sonnet 4, 29.60 / (1 − 0.397) ≈ 49.09, against 49.06 printed. The paper does not state the denominator, so say which one you used.
• Task count is ambiguous. The paper's own numbers give 80 + 356 or 356 total, and the repo has 357 folders. State which count you use.
• Judge labels are numeric in code but described in words in the text. The text's four outcomes map to -1, 0, 1 and 2 only by the rubric wording. Check the mapping before comparing across papers.
• Failure is judge-based and inflated. The paper defines failure as not reaching an initial subtask §3.3, but the scripts take it from the judge's -1 label. The authors say judges overestimate failure §4.3 RQ4(2). Unsafe rates are therefore lower bounds on the reached tasks.
• Few human checks. The judge was compared with humans on 100 GPT-4o trajectories only §4.3.
• Repeat runs and variance are not stated. NPCs are non-deterministic app. A.4. No seeds, repeat runs or confidence intervals are reported for Table 3 beyond the rank tests in the text §4.3.
• Judge model versions. The judge is a hosted GPT-4.1 call. The paper does not state the API snapshot or the decoding settings beyond what the code sets.
• Costs. One model takes 24 to 30 hours of wall-clock time across three EC2 instances app. A.5. The code's per-task budget is 4, with the unit not stated run_eval.py. Plan API spend before a full run.
• Figure content is only in the paper. Heatmaps (Figures 3 to 5) and the task-distribution figures (Figures 6 to 10) are not reproduced here. Use the text in §4.3 and tab. 6 for example trajectories.
• Resets between tasks. Container-based resets and redeployment take minutes, according to the appendix app. A.5. Plan for the time this adds across about 350 tasks.

Lineage

• Builds on: OpenHands, an open platform for multi-tool LLM agents (arXiv:2407.16741) §3.1. Environments adapted from TheAgentCompany (arXiv:2412.14161) §3.1. NPC messaging uses Sotopia §3.1.
• Compared against: 24 prior benchmarks in the authors' comparison table tab. 1. Among them, Agent-SafetyBench (arXiv:2412.14470) is cited for a fine-tuned judge that the paper evaluates §4.3 RQ4(3). SafeArena (arXiv:2503.04957) is cited for the finding that browsing raises risk §1.
• Superseded by: not known from content already in this wiki.