Paper · 2026-10-11 · cs.AI, cs.CR, cs.SE
RetryBench: Separating Retry Permission from Recovery Outcomes in GPT Tool Use
Protocol for whether an LLM agent may retry a timed-out tool call. Each decision is replayed across every hidden world consistent with the observation and scored on permission, effects and completion. On four synthetic tasks, one GPT model's protected recovery depended on how an extra key was described (10/40 vs 39/40); listing retry first lowered retry rates by 17-27 points for two models.
Tool useLLM agentLLM evaluationBenchmark validityagentstool-useevaluationidempotency
arXivPDFCodeMarkdown14 min read
AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
An agent calls a tool, such as "start a CI run" or "open an issue", and the call times out. The agent does not know whether the action ran. Retrying is the right move if nothing happened, but if the action did run, a retry creates a duplicate. Not retrying is safe against duplicates but can leave the job unfinished. A score that only counts final success can reward a lucky retry, and a score that only counts duplicates can reward an agent that never retries.
RetryBench scores each decision on three things at once: whether the visible evidence justified the retry, what actually happened in every situation that could have produced the same evidence, and whether the task ended in the intended state. The authors use it to test two prompt-level questions: how describing an extra, invalid key changes whether a model reuses the valid one, and how the order of the options changes how often a model retries.
The findings are narrow. They come from four synthetic tasks, a few phrasings chosen by the researchers, and model names that are not pinned to a version. The authors do not claim which behaviour is better.
TL;DR
- Protocol for scoring LLM-agent retry decisions after a tool-call timeout: retry permission, settled effects, and completion are separate measures.
- Each decision is replayed in every hidden execution world consistent with the observation, so a retry that happens to succeed is not counted as justified.
- Study 1: one GPT-5.6 model reused the valid idempotency key far less often when the extra key was described as a "Candidate recovery key" than as an "Alternative key string". Two other models were at ceiling.
- Study 2: when retry was listed first and lookup guarantees were underspecified, two GPT models retried 17-27 points less often. The paper reports this as position sensitivity only.
- Author-reported results on synthetic tasks; model aliases unpinned; code licensed PolyForm Noncommercial.
Problem
A timed-out side-effecting tool call can mean three different things: it failed before running, it is still running and will commit later, or it committed but a status read has not caught up §1 fig. 1. A retry is safe in the first case and creates a duplicate in the other two. Idempotency keys can make a retry safe, but only if the provider enforces them for the same operation and payload §3.2. Status lookups that return "not found" are weak evidence and do not establish that nothing ran §3.2 §3.3.
The authors argue that one-dimensional scores are incomplete: duplicate counts alone reward agents that never retry, and success counts alone reward unjustified retries that happened to work §1. Prior and related work covers idempotency contracts, eventual visibility, and other recovery benchmarks, which the authors compare with their probe §2. Among them, Limbo (arXiv:2609.29095) studies late commits and stale reads over multi-turn episodes, and the paper contrasts its per-episode outcome measures with RetryBench's single-decision replay §2.
Method
Study design and scope
The paper evaluates single recovery decisions. Each decision is one action taken after one timed-out original call, under a fixed text contract, followed by deterministic settlement in a simulator §3.1 §3.3. Four synthetic software tasks are used: trigger a CI workflow, create an issue, create a release, and post a review comment. Each task keeps a constant target and payload §3.3 app. A.1.
Study 1 asks whether describing an extra, invalid key changes retry permission and completed recovery when the contract is explicit. Study 2 asks whether reordering the action list shifts retry decisions when the lookup contract is vague §3.1.
Actions and admissibility
The agent chooses one of five actions: retry sends the call again with no key,
retry_with_key sends it with a key the agent supplies, stop ends recovery as complete, wait (defer for another observation), and unresolved
(end recovery without claiming completion) app. A.1. Only the first two dispatch
a call. wait gets no further observation in this probe, and it is not later
followed by a retry in the scored episode §3.3.
A retry is admissible when one of two things holds §3.2. The first is protected replay: the supplied key carries a provider-side guarantee tied to the same original call (operation and payload), valid for that key's scope and retention period. A different key cannot meet this condition. The second is a terminal, effect-free outcome in every consistent world: the evidence must rule out an earlier effect as well as a later landing of the first call. A weak negative read establishes neither. If the observation contradicts the contract and no world is consistent, the caller should reject the inconsistency rather than retry §3.2.
The admissibility rule is written as an unnumbered disjunction over the set of consistent worlds W(o, c) §3.2:
# Retry admissibility [§3.2] (equation is unnumbered in the paper)
# Input: action, key, original_key, consistent_worlds
# Output: True if a retry is justified
def admits_retry(action, key, original_key, consistent_worlds):
# Protected replay: same key as the original, same operation [§3.2]
if action == "retry_with_key" and original_key is not None:
if key == original_key:
return True
# Otherwise every consistent world must be terminal and effect-free [§3.2]
for world in consistent_worlds:
if not (world.terminal and world.effect_free):
return False
return TrueWorlds, outcomes and settlement
Under weak "not found" evidence, the paper uses three worlds with equal measurement weight: the call never ran and has finished; the call is still pending and will land later; or the call landed, but the read has not caught up §3.3. After the model chooses, a retry dispatches a new call, the original attempt then finishes, and the simulator checks the operation identity, target, payload, number of effects, and recovery status §3.3.
The authors score effects and recovery separately §3.3 tab. 1:
- Duplicates (more than one effect) and omissions (no effect) count effects.
- Ledger correct means the settled state holds one effect that matches the intent.
- Resolved exact state means the ledger is correct and the recovery status is complete. It does not check that the completion claim was justified.
- Protected recovery is the study-level measure: the original valid key was used and the intended state was reached §1.
The outcome table for the stated contract tab. 1 is the implementation reference. For each action it gives the number of effects that settle in each of the three worlds, and whether the recovery ends in a correct, completed state. Rows, as the paper lists them:
- Retry with the original key: one effect in every world, completed in every world.
- Retry with a changed key: one effect in the fail-before-execution world, but two effects (not exact) in the in-flight and stale-read worlds.
- Wait or unresolved: no effect in the fail world, and one effect that stays open in the other two worlds.
- Stop: no effect in the fail world (not exact), and one completed effect in the other two.
Source: tab. 1.
Each valid decision produces three correlated replay outcomes, one per world. The paper does not treat these as independent observations app. B.
Study designs
The three designs are summarised below tab. 2. The 1,728 main-study decisions are the sum of the three planned totals app. D.
- Key wording (Study 1): Sol, Terra and Luna; 4 wording arms, 2 key assignments, 4 tasks and 5 draws per cell; 480 planned decisions. Primary contrast: protected recovery for the Alternative and Unrelated arms minus the Candidate arm.
- Paraphrase extension (Study 1): Luna only; 4 arms, 3 phrasings, 2 key assignments, 4 tasks and 5 draws; 480 planned decisions. Primary contrast: protected recovery in the neutral and unrelated key-string families, compared with the recovery-key family across phrasings.
- Full-order sweep (Study 2): two models (Terra and GPT-5.4); 24 orders, 2 labels, 4 tasks and 2 draws; 768 planned decisions. Primary contrast: retry rate at the first position minus the rate at the other positions.
For Study 1, each wording arm adds one line to the prompt. The key assignment flips the roles of the two key strings, so the decoy can be either one. The four arms are: no extra key, "Candidate recovery key", "Alternative key string", and "Unrelated key string (not a recommendation)" tab. 3. The paraphrase extension uses three purposively chosen phrasings per displayed family, with the no-extra-key arm repeated in three blocks §3.4 app. B.
For Study 2, the action list and the JSON enum are permuted together over all 24 orders, so each retry position is an average over six permutations. The labeled and unlabeled prompts are identical except for the evidence-contract line §3.4 app. A.2.
Model settings and prompts
- All main-study GPT requests use reasoning effort
none, an output cap of 256 tokens, and a strict JSON schema §3.1. - Temperature is not sent, so the provider default (reported as 1.0) applies §3.1.
- The system instruction tells the model to return exactly one action and not to assume facts outside the evidence contract app. A.
- The task prompt states the tool contract: each successful call without a matching key adds one effect; keys are retained for 24 hours; the calls are within 30 seconds app. A.1.
- The response schema has
action,rationale(string) andidempotency_key(string or null), with no additional properties app. A.1.
Statistics
- Counts use planned calls as denominators. Absent or malformed outputs are tallied apart from valid choices that keep recovery open §3.5.
- Study 1 call-level tests are stratified by task and key assignment, use the hypergeometric reference, and apply Holm correction app. C.
- The Study 2 order-level reanalysis treats the 24 orders as clusters, enumerates all 134,596 ways to assign six orders to retry-first, and uses a two-sided absolute-difference test. The reference assumes exchangeability under the null app. C.
- Hierarchical logistic models include a retry-first indicator, fixed effects for label and task, and a random intercept for each order. The priors are weakly informative: a normal with SD 2.5 for the intercept, SD 1.5 for the other coefficients, and a half-normal with scale 1 for the order SD. Four chains, 1,000 warm-up and 1,000 retained draws app. C.
- No phrase-level significance tests are reported. The three phrasings per family are not a sample of a phrasing population §3.5 §6.
Hyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| Model: Sol | gpt-5.6-sol | §3.1 | Model alias; serving implementation unpinned |
| Model: Terra | gpt-5.6-terra | §3.1 | Same |
| Model: Luna | gpt-5.6-luna | §3.1 | Same |
| Model: GPT-5.4 | gpt-5.4-2026-03-05 | §3.1 | Same |
| Reasoning effort | none | §3.1 openai_adapter.py | |
| Output cap | 256 tokens | §3.1 experiment.py | Strict JSON output |
| Temperature | not sent (provider default 1.0) | §3.1 openai_adapter.py | |
| Tasks | 4 (CI, issue, release, review comment) | §3.3 app. A.1 | Synthetic, fixed targets and payloads |
| Key retention | 24 hours (stated in prompt) | app. A.1 | Assumed by the contract; expiry not simulated §3.3 §6 |
| Calls per episode | within 30 seconds | app. A.1 | Same task, target and payload |
| Study 1 draws | 5 per task × key-assignment cell | tab. 2 | |
| Study 2 draws | 2 per task × label × order | tab. 2 | |
| Study 2 action orders | 24 (list and enum permuted together) | app. A.2 | Each position averaged over 6 orders |
| Collection dates | 2026-10-30 and 2026-10-31 UTC | §3.1 app. A.3 | Original and paraphrase collections |
| Order-test allocations | 134,596 | app. C | Exact two-sided reference test |
| Holm family | 4 endpoint/label contrasts (Study 2); 6 + 2 (Study 1) | app. C | |
| Hierarchical priors | SD 2.5 (intercept), SD 1.5 (other terms), half-normal scale 1 (order) | app. C | 4 chains, 1,000 warm-up, 1,000 draws |
Results
Author-reported. Setup: synthetic tasks and simulator; models are hosted GPT endpoints (hardware not stated); baseline is the arm with no extra key, or the Sol and Terra models under the same contract §4.1 §5. Results are descriptive counts unless a test is named.
Study 1: wording of an extra key
| Setting | Metric | Baseline | This paper | Source |
|---|---|---|---|---|
| Sol and Terra, each of four arms | Protected recovery | not applicable | 40/40 each | tab. 4 |
| Luna, Candidate arm (single description) | Protected recovery | Absent arm 40/40 | 10/40 | tab. 4 |
| Luna, Alternative arm (single description) | Protected recovery | Absent arm 40/40 | 39/40 | tab. 4 |
| Luna, Unrelated arm (single description) | Protected recovery | Absent arm 40/40 | 22/40 | tab. 4 |
| Luna, recovery-key family (3 phrasings) | Protected recovery | No extra key 117/120 | 27/120 | tab. 4 |
| Luna, neutral key-string family | Protected recovery | No extra key 117/120 | 113/120 | tab. 4 |
| Luna, unrelated key-string family | Protected recovery | No extra key 117/120 | 93/120 | tab. 4 |
| Luna, Candidate arm | Unsupported retries | Absent arm 0/40 | 11/40 | tab. 4 |
| Luna, Candidate arm | Resolved-exact replay outcomes (of 120 per arm) | Absent arm 120/120 | 41/120 | app. B |
| Luna, Alternative arm | Duplicates / omissions | Absent arm 0 / 0 | 0 / 1 | app. B |
| Luna, Unrelated arm | Omissions | Absent arm 0 | 18 | app. B |
Protected recovery for Luna's Unrelated arm was 7/20 with original key-A and 15/20 with original key-B; across the three new phrasings the counts were 39/60 and 54/60 §4.1 app. B. Sol and Terra showed no difference across the four arms in this design, so the paper reports the Luna wording effect only.
Among the paraphrases, the phrase "not suggested for use" gave 39/40 protected recoveries and "not an endorsement" gave 25/40 §4.1 app. B.
Study 2: order of the action list
| Model and label | Retry-first count | Other-position count | Change (percentage points) | Adjusted p (order level) | Source |
|---|---|---|---|---|---|
| Terra, labeled | 38/48 | 139/144 | -17.36 | 0.02474 | tab. 5 |
| Terra, unlabeled | 39/48 | 142/144 | -17.36 | 0.00523 | tab. 5 |
| GPT-5.4, labeled | 27/48 | 112/144 | -21.53 | 0.02474 | tab. 5 |
| GPT-5.4, unlabeled | 28/48 | 123/144 | -27.08 | 0.01110 | tab. 5 |
Retry-first rates are lower in all four cells. The reductions concentrate in CI and review tasks; the issue task drops only in Terra's labeled cell §4.2 app. C. Retry and unresolved are close to complementary, so the two readings count one choice twice rather than giving two findings §4.2 app. C. Hierarchical sensitivity models give a retry-first odds ratio of 0.15 (95% CI 0.04-0.54) for GPT-5.4 and 0.13 (0.03-0.59) for Terra app. C.
Paper vs code
The repository is LIUYOU2015/retrybench, owned by the first author. The README
links the arXiv paper, and the BibTeX names the three authors. The clone is at
commit 4d59a728791bd5c52809080563f07b02ef128537. Its LICENSE is PolyForm
Noncommercial 1.0.0, so no code is quoted here; all code references are
citations. The repository describes a packaged suite ("retry-permission-v1") of
240 visible cases, which is a different release from the arXiv v4 manuscript.
- Study 1 and Study 2 designs are not in the pinned repository. The packaged suite contains 40 prompts with the phrase "Candidate recovery key" retry-permission-v1.json. The phrases "Alternative key string" and "Unrelated key string", and the 24-order permutation test, Holm correction and hierarchical models, do not appear anywhere in the tree (searched by exact string and by keyword). The repository's manuscript is titled "When Is a Retry Safe?" paper.md, and its artifact README refers to an October 16 revision with a different title README.md. The archive
artifacts/reproduction-v1.zipmay hold the v4 study data, but it was not inspected, so this card cannot say. - Admissibility. The code's permission check,
retry_is_permitted, returns True only for the schedulefail_before_executionwith authoritative evidence and observed statenot_executedscoring.py. It checks the hidden world directly and has no set-of-consistent-worlds quantifier. Protected replay is handled separately, in the replay loop evaluation.py. For this suite the two agree, because an authoritativenot_executedobservation only occurs in the fail-before-execution world. An implementer working from §3.2 should use the universal form above, not the per-world check. - Protected replay. Protection requires the action to be
retry_with_keywith the original key evaluation.py. The simulator then merges the retry with the original. Any other key commits without deduplication service.py service.py, which matches the "changed key" row of tab. 1. - Deduplication store. The simulator keeps a set of operation ids with no clock or eviction service.py. The 24-hour retention appears only in the prompt text retry-permission-v1.json. This matches §3.3, which says the window is a contract assumption, not a simulated expiry.
- Weak evidence. Under weak evidence, a "not found" read is returned in the fail-before-execution, in-flight and stale-after-commit worlds service.py. This matches §3.3.
- Scoring. Exact success requires exactly one effect, a matching identity, and no unresolved action scoring.py. Matches §3.3.
- Output schema. The OpenAI adapter sends
reasoningeffort,max_output_tokens,store: false, and a strict schema with a requiredrationalefield and an action enum withoutretry_with_keyopenai_adapter.py. The public scorer's schema has norationalefield and takes its action enum per case evaluation.py. Both are consistent with app. A for the reasoning and output-cap settings; the schemas differ. - Output cap. The pilot and confirmation runs use 256 tokens experiment.py confirmation.py. The cross-model study uses 512 cross_model_pilot.py. That is a separate study and is not used by this paper.
- Invalid outputs. Malformed or invalid outputs are recorded as unknown, not repaired or treated as safe evaluation.py. This matches §3.5.
- Action aliases. The scorer maps the legacy labels
RETRY,VERIFY_FIRSTandNOT_RETRYontoretryorretry_with_key,waitandunresolved. The code comment states thatNOT_RETRYdoes not claim completion evaluation.py. Not described in the paper.
No discrepancies were found in the simulator outcome rules. The main difference is scope: the released code covers the frozen 240-case suite, not the Study 1 wording arms or the Study 2 order sweep reported in the arXiv v4 text.
Pitfalls
- Success without permission is not progress. In the Candidate arm, 41 replay outcomes were resolved-exact, but 11 of them came from unsupported retries that succeeded only in the terminal-failure world §1 app. B. Score permission and completion separately.
- Unresolved is not safe. In the terminal-failure world,
waitorunresolvedleaves zero effects, which is an omission tab. 1. It also leaves recovery open in the other two worlds, so the decision never reaches resolved-exact state §3.3. - Key identity matters. Only the original key, on the same operation and payload, counts as protection. A different key, or a key added after an unkeyed original, does not protect the earlier call §3.2 §3.3.
- Retention is assumed, not tested. The paper does not test expiry, real-provider enforcement, or payload mismatch. Outside the 24-hour window, a key alone does not justify a retry §3.3 §6.
- Key assignment is a boundary condition. The Unrelated arm's protected rate moved from 7/20 to 15/20 across key assignments §4.1 app. B. Pooling assignments hides this.
- Phrasings are purposive. Three phrasings per family cannot support a phrase-population inference §3.5 §6.
- Outcome counts are correlated. Three worlds per decision and the replay of the same decision in every world are not independent samples app. B app. C.
- Study 2 is a choice-rate result. With no key offered and a weak read, every retry is unsupported, and the paper makes no claim about whether more or fewer retries is better §4.2 §6. Because retry and unresolved are close to complementary, the apparent position effects largely re-describe one binary decision §4.2 app. C.
- The Study 2 unlabeled prompt omits the evidence line. The paper says this line does not explain stale reads or pending commits, so the agent is left to assume §3.2.
- Model aliases are unpinned. The paper saw matching returned model strings but no backend fingerprint, so exact reruns may not reproduce §3.1 app. A.3.
- The schema is not the same across sources. The paper's appendix schema includes a
rationalefield. The repo's public scorer schema does not app. A.1 evaluation.py. - Some analyses cannot be checked here. The Holm correction, the permutation test and the hierarchical models are described in the paper but do not appear in the pinned repository (see Paper vs code).
Lineage
- Builds on: Did It Happen? (Sun, 2026): the shared-response design, extended here with pending operations and stale reads §2. Also Limbo, arXiv:2609.29095, whose Proposition 1 shows that verification alone cannot always resolve the problem §2 §3.2.
- Compared against: Limbo's late-commit duplication figures for a gateway model ID §2 §5; UndoBench (arXiv:2610.05622), whose unsafe-retry measure counts duplicates but not unjustified retries §2; ContractBench, Verified Tool Calls, OpenOnce and IdempotencyBench as related evaluations §2.
- Superseded by: not stated.
Source and license
Adapted from “RetryBench: Separating Retry Permission from Recovery Outcomes in GPT Tool Use” by You Liu, Yue Liu, Quanchao Lu (arXiv:2610.15884), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/LIUYOU2015/retrybench at commit 4d59a728791b (PolyForm Noncommercial 1.0.0 (non-commercial)). Any quoted code is shown under that license.