CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows?

In plain words

Most tests of AI agents use coding or web tasks. This paper builds a practice environment for a different job: the paperwork of US healthcare. A provider asks an insurer to approve a treatment, the insurer's reviewers decide whether it is covered, and a care manager follows up with a patient. The agent works through 20 imitation software apps, has to follow a handbook of 1,279 documents, and sometimes has to hold a back-and-forth conversation with a simulated person.

The authors built 75 such tasks, had clinicians check them, and scored each run from the final records plus a set of rubric questions. Across 30 agent setups, the best solved 28% of tasks on a first try, and no setup passed 20% when it had to succeed on all three runs. Doing every task inside one long session made results much worse.

The authors' takeaway is that agents that look strong on coding benchmarks may not carry over to rule-heavy, multi-role office work where a wrong step cannot be undone.

TL;DR

CHI-Bench (written χ-Bench in the abstract) tests whether agents can drive long-horizon healthcare operations end to end: provider prior authorization, payer utilization management and care management. Tasks run in a simulated environment of 20 apps exposed through 87 MCP tools and are scored by a deterministic contract plus an LLM rubric judge. The best of 30 configurations reaches 28.0% pass@1 and none reaches 20% pass^3, all author-reported. Read it if you evaluate agents on policy-dense, role-composed, irreversible workflows.

Problem

Healthcare operations combine three things that general agent benchmarks rarely test together: policy density (every decision must follow medical, insurance or operational rules), multi-role composition (one case passes through several roles with irreversible handoffs), and multilateral interaction (some steps are multi-turn dialogs with humans, such as peer-to-peer review or patient outreach) §1. The paper compares 29 healthcare and long-horizon agent benchmarks on nine evaluation axes, and says its own benchmark covers all nine §2; Table 1; app. B.

Method

CHI-Bench is a benchmark and a simulator, not a training method. Each task hands an agent one clinical case in one of three domains, a role-scoped tool surface, and instructions. The agent must drive the case to a terminal status through tool calls and by writing the role's artifacts such as submission packets, review notes, letters and care plans §1; §3; §3.2.1.

Environment. A local simulator runs 20 healthcare apps built with FastAPI, SQLite and SQLModel. They expose 151 REST routes as 87 MCP tools over 3 MCP servers, and the case state machines have 29 statuses §3.1; app. C.1; app. C.2. The shared world holds about 50 simulated patients, about 90 healthcare workers and about 5,000 chart activities §3.1. Agents act through MCP, the local database and the file system §3.1.

Domains. Provider prior authorization (PA), payer utilization management (UM) and care management (CM). Each domain has 25 tasks §3.1; app. E.

Handbook skill. Each agent also gets the Managed-Care Operations Handbook, a skill of 1,279 markdown documents. A top-level index routes to three role sub-skills (provider-pa, payer-um, care-manager), with shared medical-library and platform appendices §3.1.2; app. C.3. The paper says clinicians and operations leaders at Johns Hopkins Medicine helped develop it §3.1.2; app. C.3.

Task construction. A terminal world state is sampled. Claude Opus 4.7, with structured JSON output, then drafts the upstream artifacts (charts, submission packets or personas, and per-stage rubric prompts) conditioned on the state graph and the matching handbook section. Each rubric line must cite a policy section or a CM taxonomy node §3.2.2; app. D.1. Candidates pass three gates: every expected action resolves to a citation; chart and rubric agree without leaking the chosen path; and a dry run reaches one terminal state with lengths inside a difficulty band app. D.1. An annotator then walks each case through the live UI, and the recorded trajectory, database states and handoffs become the ground truth §3.2.2. The pipeline produced 523 candidate tasks and kept 75 for quality and diversity. The human walkthrough needed 21 steps on average and at most 40 §3.2.2. Clinicians review each trajectory, and a PHI scan and a clinical-realism check gate admission §3.2.2; app. D.1.

Dialogs. Peer-to-peer and patient-outreach steps are multi-turn. A persona LLM plays the patient and a second LLM plays the peer-to-peer counterpart, both via the Anthropic Python SDK. The model used for them is not stated app. F.5.

Reward. A trial is binary: pass only if the deterministic contract and the rubric judge both pass §3.2.3:

R = \mathrm{DeterministicPass} \land \mathrm{JudgePass} §3.2.3

The deterministic contract reads the world store, event log and transcripts the simulator persisted. The rubric judge is Claude Opus 4.7, run with three votes per rubric and a strict majority §3.2.3; app. F.3. A fractional scorecard is kept for diagnostics only §3.2.3.

# Binary reward for one trial  [§3.2.3]
# Input:  trial (world store, event log, transcripts, rubrics)
# Output: 1 if the trial passes, else 0

def trial_reward(trial):
    # Layer 1: deterministic contract on the persisted record  [§3.2.3]
    if not deterministic_contract_passes(trial):
        return 0
    # Layer 2: judge each rubric with V = 3 votes  [app. F.3]
    for rubric in trial.rubrics:
        # Strict majority; a tie fails  [app. F.3]
        # [code: src/chi_bench/verifier/judge/workspace_judge.py#L330]
        votes = judge_votes(rubric, trial, num_votes=3)
        if not majority_passes(votes):
            return 0
    # Every rubric passed  [§3.2.3]
    return 1

Trials and metrics. Each task is run three times, and the agent wall-clock cap is 1,800 s app. F.3. A trial that reaches the cap without a terminal state counts as a failure for both pass@k and pass^k app. F.3. Metrics are task-level averages, with n = 3 trials per task §4.1; app. F.2:

\mathrm{pass@}k = \mathbb{E}\left[1 - \binom{n-c}{k} \Big/ \binom{n}{k}\right], \qquad \mathrm{pass}^{k} = \mathbb{E}\left[\binom{c}{k} \Big/ \binom{n}{k}\right] app. F.2

Here c is the number of passing trials. With n = k = 3 the reported quantities are pass@1 (fraction of passing trials), pass@3 (any trial passes) and pass^3 (all three trials pass).

# Per-task metrics from n = 3 trial rewards  [app. F.2]
# Input:  rewards, three 0/1 values for one task
# Output: pass@1, pass@3 and pass^3 for that task

def task_metrics(rewards):
    # Count the passing trials c out of n = 3  [app. F.2]
    passes = sum(rewards)
    # pass@1 is the share of passing trials
    # [code: src/chi_bench/aggregator.py#L193-L206]
    pass_at_1 = passes / 3
    # pass@3 is 1 if any trial passes  [code: src/chi_bench/aggregator.py#L207]
    pass_at_3 = 1.0 if passes > 0 else 0.0
    # pass^3 is 1 only if all three pass
    # [code: src/chi_bench/aggregator.py#L208]
    pass_pow_3 = 1.0 if passes == 3 else 0.0
    return pass_at_1, pass_at_3, pass_pow_3

Configurations. Thirty agent harness and model pairs: nine on first-party CLIs (Claude Code, Codex, Gemini CLI) and 21 on open-source frameworks (OpenClaw, Hermes, the OpenAI Agents SDK and DeepAgents) with five open-weight models served via OpenRouter, plus one OpenClaw run with Claude Opus 4.7 §4.1. Each (configuration, task) pair gets three trials, so the full run has 30 × 75 × 3 = 6,750 trials app. F.2.

Variants. The Arena runs a provider agent and a payer agent on 23 PA tasks. They exchange information only through MCP tools and each side is scored; a trial passes only if every check on both sides passes §4.3. The Marathon loads all 25 tasks of a domain into one agent session. The agent may attempt them in any order, and context compaction follows the harness default §4.4.

Hyperparameters

Results

All numbers are author-reported, on this benchmark only, using the setup above. Table 2 gives the overall pass@1, pass@3 and pass^3 for all 30 configurations, with task-level bootstrap 95% CIs in brackets tab. 2. Per-domain pass@1 is also shown here. The per-domain pass@3 and pass^3 values are in tab. 2.

Table 2. Overall and per-domain results tab. 2. Values are percent. CI = task-level bootstrap 95%. Steps is the mean per trial over 225 trials. Cost is the mean per-trial USD.

Headline. Best overall pass@1 is Claude Code with Claude Opus 4.6 at 28.0%. Claude Sonnet 4.6 (26.2%), Claude Opus 4.7 (24.4%) and Codex with GPT-5.5 (20.9%) are close behind. Best per domain is Opus 4.6 on UM (41.3%), Opus 4.7 on CM (32.0%) and Codex with GPT-5.5 on PA (29.3%) §4.2. No configuration exceeds 18.7% pass^3 tab. 2. The paper's conclusion reports the same 28.0% and 20% claims §5.

Reliability. pass^3 falls well below pass@1 for the leading configurations: Opus 4.6 goes from 28.0 to 18.7, GPT-5.5 from 20.9 to 9.3 §4.2; tab. 2.

Arena (PA, two agents) tab. 3. Provider-only on 23 PA tasks: 30.4% pass@1. End-to-end two-agent setup with Codex and GPT-5.5 on both sides: 0.0%. Of the tasks, 2 were never submitted, 18 did not finish the MD decision and 5 failed the final judge. Peer-to-peer tasks failed on both sides §4.3.

Marathon (all 25 tasks in one session) tab. 4. pass@1 averaged over 3 sessions. Each row is Marathon versus per-task (isolated trials from tab. 2).

Neither agent submitted a PA authorization across the 25 queued cases, and on UM and CM each reached a finalized determination or care plan on only 3 to 8 of 25 cases per session §4.4. Both configurations drop sharply in this setting tab. 4.

Handbook ablation §4.5. Codex with GPT-5.5, UM pass@1 is 32.0 with the full handbook and 17.3 when the domain handbook is removed. Removing only the medical library barely changes it. For PA, removing both appendices does modestly better than the other trims. The paper attributes this to over-verification: with a handbook present, the agent refuses to submit when uncertain §4.5.

MCP versus CLI tab. 5. Codex with GPT-5.5 on MCP tools re-surfaced as CLI commands via MCPorter, 3 trials per task:

The paper calls this neutral to worse rather than uniformly beneficial §4.6; tab. 5.

Failure modes §4.7; app. G. Of 5,886 failed trials, the first-level split is Clinical-Reasoning 35.4%, Workflow-Completion 23.3%, Abstain-or-Stuck 15.6%, Policy-Compliance 13.2%, Tool-Use-Error 10.7%, Harness-Fault 1.0% and Hallucination 0.8%. Nearly half of the Abstain-or-Stuck cases hit the 1,800 s cap. CM also shows an illegitimate consent mode, where the agent keeps widening the care program until a reluctant patient agrees §4.7.

Paper vs code

Code paths are relative to the repository at code_commit (github.com/actava-ai/chi-bench, Apache-2.0). I cloned it at that commit and checked the judge, metrics and experiment config against the paper.

• Judge votes. Paper: V = 3 votes per rubric app. F.3. Code: the default is one vote unless CHI_BENCH_JUDGE_NUM_VOTES is set workspace_judge.py. The repo's docs/judge.md calls voting an optional way to "smooth further". The paper's three-vote runs need that variable set; the repo does not set it.
• Judge timeout. Paper: 1,200 s per rubric app. F.3. Code: the PA/UM judge defaults to 900 s task_runtime.py and the CM judge also to 900 s task_runtime.py. A separate session-mode default of 1,800 s exists claude_runner.py.
• Judge output cap. Paper: 128K tokens per rubric app. F.3. Code: the same 128,000 default claude_runner.py.
• Judge model. Paper: Claude Opus 4.7 app. F.3. Code: the default is claude-opus-4-7 task_runtime.py. No discrepancy.
• Tie rule. Paper: ties resolved as fail app. F.3. Code: passed = votes_for_pass * 2 > votes_total, so a tie fails workspace_judge.py. No discrepancy.
• Metrics. Paper: pass@k and pass^k formulas app. F.2. Code: pass@1 is passes/n, pass@3 is "any pass" and pass^3 is "all three pass" for n = 3 aggregator.py. Consistent with the paper.
• Agent wall-clock. Paper: 1,800 s cap app. F.3. Code: the value was not located; the config sets agent_timeout_multiplier: 2 on Harbor's defaults table1_main_matrix.yaml.
• CM outage. Code: when the judge cannot produce a verdict, unparsed rubrics count as not passing and the binary reward becomes 0 task_runtime.py. This matches the paper's rule that judge outages are failures. I did not verify the same rule for PA and UM.
• Marathon score. Paper: pass@1 per task averaged over 3 sessions tab. 4. Code: a session's score is passed/total session_verifier.py.
• Experiment matrix. The table-1 config lists 30 rows, matching the paper's 30 configurations table1_main_matrix.yaml.
• Bootstrap. Paper: percentile bootstrap, number of resamples not stated. Code: 1,000 resamples with seed 0 aggregator.py.

Pitfalls

• Headline 3.8%. The abstract and introduction say the Marathon falls to 3.8%. The paper does not state how that number is computed. Averaging the three domain values of Table 4 gives 3.6% for Codex and 4.0% for Claude Code, and averaging those two rows gives about 3.8%. Treat it as a derived figure §1; tab. 4.
• Handbook size. The arXiv abstract says "1,290+" documents; the body says 1,279 §3.1.2; §4.5. Use 1,279.
• Judge bias. The judge is Claude Opus 4.7, the same family as several evaluated Claude configurations. The paper says a cross-family re-grade is planned but not done. Its V = 3 agreement numbers measure within-judge stability only app. F.7.
• Small n, wide CIs. Each task has three trials and 75 tasks. The top configurations' CIs overlap heavily: Opus 4.6 at 28.0 [19.6, 36.9] against Sonnet 4.6 at 26.2 [18.2, 33.8] tab. 2. Ranking among the leaders is not established.
• Turn budgets differ by harness. The OpenAI Agents SDK is pinned to 50 turns; other harnesses use their defaults app. F.3. Harness comparisons therefore mix harness design with turn budget.
• Time caps decide failures. A timed-out trial counts as a failure in both pass@k and pass^k app. F.3. Speed and the cap therefore affect scores, and Abstain-or-Stuck is about 15.6% of failures §4.7.
• Task selection. 75 tasks were chosen from 523 "for quality and diversity". The paper does not give the exact selection rule §3.2.2.
• Marathon depends on context. Compaction follows each harness default, and Claude Code runs with a 1M-token context and never compacts §4.4. A Marathon score therefore reflects the harness setting as well as the model.
• Data licence. The paper says the handbook is released under a research Data Use Agreement that allows non-commercial research use and bars redistribution of the original documents app. C.3. This card contains no handbook text. Anyone reusing the handbook or the dataset must check that agreement first; I did not verify the dataset's own terms on its hosting page.
• Language-only. The benchmark has no imaging or speech, and the paper warns against using a 28% pass@1 as evidence of safety for live patient care §5.

Lineage

• Builds on: agent skills §3.1.2; pass@k from HumanEval and pass^k from τ-bench, as cited in §4.1; Harbor, the trial orchestrator app. F.1; and MCPorter for the MCP-to-CLI comparison §4.6. Cited works are named, not linked.
• Compared against: 29 healthcare and long-horizon agent benchmarks on nine axes §2; Table 1; app. B.
• Superseded by: not known.