Paper · 2026-10-16 · cs.AI
Agents' Last Doubt: Knowing When to Act and When to Step Back in Tool-Using Agents
SURE-Act trains a Qwen3-1.7B controller on counterfactual ToolSandbox rollouts, forcing each legal meta-action (ask, search, reversible act, irreversible commit) once per decision point. A pairwise ranking loss adds +0.0309 offline utility over regression. Handing control back after three repeated choices gives end-to-end gains (author-reported; unseen-family intervals include zero).
LLM agentTool useLearning-to-rank losses over candidate setsagentstool-usedecision-makingcounterfactual-datalearning-to-rank
AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
A tool-using AI assistant keeps facing the same small choice: act now, look something up first, ask the user, or do something that cannot be undone. Logs of past runs only show the option that was taken, so nobody learns what the other options would have done. This paper fixes that by going back to saved moments in the environment and trying every option once from the same point, then letting the rest of the episode play out with the same underlying assistant. The outcomes become labels. A small separate model learns to score the options, and at run time it picks one for the assistant.
The scoring model does better than a plain regression at ranking the options, but the bigger lesson is about how often it steps in. Overriding the assistant at almost every turn hurts the final score. A simple rule that hands control back once the model has picked the same option three times in a row turns a controller that hurts the score into one that helps it, according to the authors. The authors report that this helps on the benchmark they used, but it does not make the assistant safer, and the gains on new task families are uncertain.
TL;DR
SURE-Act is a controller that picks, at each decision point, one of four meta-actions for a frozen tool-using agent: ask the user, search, make a reversible change, or make an irreversible commit §3.1. It is trained on 3,115 counterfactual outcomes from 1,036 ToolSandbox decision points tab. 2. Offline, a pairwise ranking term improves selection over regression alone. End to end, the gains depend on a stopping rule that returns control after three identical proposals §3.4. Treat the end-to-end numbers as author-reported and as sensitive to the evaluation job.
Problem
Logged trajectories reveal only the action that was taken, so the value of unchosen actions is unobserved §1. A successful run does not show whether a cheaper action would have worked, and a failed run does not show which action would have prevented the failure §1. Acting before resolving uncertainty can cause irreversible errors, while asking or searching adds interaction cost §1. Without counterfactual labels, a selector has nothing to learn from and nothing to be scored against §1.
The paper also reports that intervening at every decision point lowers the ToolSandbox Overall Score below the unmodified backbone §4.7. Better offline selection alone therefore did not give better end-to-end performance §4.7.
Assumptions: labels come from one temperature-zero continuation per action §3.2; the backbone is frozen and the evaluation environment is the simulated ToolSandbox §4.1; the controller only sees the last twelve turns, the environment identifier and the legal-action set §3.3.
Method
Meta-actions and utility. Four meta-actions are defined: Ask (request missing information from the user), Search (read-only retrieval), Act-Reversible and Commit-Irreversible §3.1. The expected utility of an action is the success probability minus a weighted interaction cost minus a weighted probability of an unauthorised irreversible action §3.1. The paper uses a fixed cost profile for the cost term, but the values are not stated in the paper §3.1.
# Utility of one legal meta-action a at state x [§3.1]
# Q(x, a) = p_success - lam * cost - eta * p_critical
def expected_utility(x, a):
return p_success(x, a) - lam * cost(x, a) - eta * p_critical(x, a)Counterfactual data collection. Decision points are taken from agent trajectories and the environment state is saved at each one. For every legal meta-action, a copy of the state is made and the backbone's next turn is restricted to that category. Tool-based actions restrict the tool set to the chosen category and require a tool call; Ask makes the backbone write a question to the user. After that one turn the restriction is removed and the backbone completes the episode. The success, interaction cost and critical-error flag are recorded §3.2 alg. 1.
# Counterfactual labels for one decision point [§3.2] [alg. 1]
# Each legal action gets its own copy of the saved state.
for state in saved_states:
for action in legal_actions(state):
# Force the action for one turn, then let the backbone finish.
trajectory = roll_out(force(clone(state), action))
success, cost, critical = grade(trajectory)
# Observed utility of this single rollout [§3.2]
utility = success - lam * cost - eta * critical
dataset.append((context, action, success, cost, critical, utility))The utility label is y = s − λc − ηr, where r is 1 if a critical error occurred §3.2. Each action has one rollout at temperature zero, so each label is one observed outcome, not an expectation §3.2.
Controller and outputs. The controller is a 1.7B-parameter Qwen3 encoder fitted with low-rank adapters, which averages its token states and feeds one linear output layer §3.3. For a context h and a candidate action a it predicts three scores, for success, cost and critical error §3.3. Ranking uses the plug-in utility, which the paper does not treat as a calibrated probability §3.3.
f_\theta(h, a) = (\hat s, \hat c, \hat r), \qquad \hat Q_\theta(h, a) = \hat s - \lambda \hat c - \eta \hat r \qquad [\S 3.3]
Losses. The outcome loss is a masked squared error over the observed (context, action, component) triples. Missing outcomes contribute nothing §3.3. The ranking loss is a hinge over legal action pairs at the same decision point whose observed gap is positive. The margin is the observed gap; pairs with equal outcomes are dropped, and a decision point with no usable pair adds no ranking penalty §3.3. The objective adds the two with weight alpha. Alpha = 0 is the regression-only ablation §3.3.
\mathcal{L}_{\mathrm{out}} = \frac{1}{|\mathcal{M}|} \sum_{(h,a,j) \in \mathcal{M}} \bigl(\hat z_{\theta,j}(h,a) - z_j(h,a)\bigr)^2 \qquad [\S 3.3]
\mathcal{L}_{\mathrm{rank}}(h) = \frac{1}{|\mathcal{P}_h|} \sum_{(a,b) \in \mathcal{P}_h} \max\{0,\; g_{ab} - (\hat Q_a - \hat Q_b)\}, \quad g_{ab} = y_a - y_b > 0 \qquad [\S 3.3]
\mathcal{L} = \mathcal{L}_{\mathrm{out}} + \alpha \, \mathcal{L}_{\mathrm{rank}} \qquad [\S 3.3]
The appendix writes the same hinge over unordered pairs with an absolute gap and a sign term, and averages over all pairs in a batch rather than per decision point app. B.1. The two normalisations are not identical (see Pitfalls).
Inference with intervention control. At every turn the controller scores each legal option and suggests the one with the top predicted utility; the backbone then carries out that suggestion for one turn, under the same restriction used during label collection §3.4. The controller remembers its last suggestion and a streak length. Repeating that suggestion lengthens the streak; choosing something else restarts it at one §3.4. Once the streak reaches the threshold of three, the controller withdraws before carrying out that suggestion, and the untouched backbone completes the episode §3.4 alg. 1. Handing control back is a stop rule, not an additional action category §3.4. The rule is preregistered §3.4.
# Phase 3: inference with intervention control [§3.4] [alg. 1]
previous = None
repeats = 0
while not episode_done():
# Controller proposes the best-scoring legal meta-action [§3.4]
proposal = argmax_over_actions(q_hat(context, a))
# Count consecutive identical proposals; a change resets to 1 [§3.4]
if proposal == previous:
repeats = repeats + 1
else:
repeats = 1
previous = proposal
# Third identical proposal: hand the episode to the backbone [§3.4]
if repeats >= k:
backbone_finishes_episode()
break
# Otherwise run the proposal for one turn with the backbone [§3.4]
execute_one_turn(backbone, proposal)
advance_to_next_decision_point()Training procedure. Four epochs, batch size eight, learning rate 1e-4, inputs truncated to 1,024 tokens, validation every 50 steps, early-stopping patience of six evaluations app. A.2 tab. 8. The checkpoint is chosen by validation action-selection utility, not training loss app. A.2 §3.3.
Splits. Families, not decision points, are assigned to the three sets in a 60/20/20 ratio. Each split seed reshuffles the sorted family list, and paired methods share seeds app. A.1 §4.1.
Hyperparameters
Settings for the 120-partition runs that compare the controller with fixed-action baselines, read from the run artefacts tab. 8 app. A.2. The ranking comparison uses its own runs and does not centre its targets tab. 8 app. A.2.
| Name | Value | Where | Notes |
|---|---|---|---|
| encoder | Qwen/Qwen3-1.7B | tab. 8 app. A.2 | with LoRA adapters, mean pooling, linear head §3.3 |
| LoRA rank | 16 | tab. 8 app. A.2 | "set in the training script" per tab. 8 |
| LoRA alpha | 32 | tab. 8 | as above |
| LoRA dropout | 0.05 | tab. 8 | as above |
| LoRA target modules | not stated | ||
| epochs | 4 | tab. 8 app. A.2 | |
| batch size | 8 | tab. 8 app. A.2 | |
| learning rate | 1e-4 | tab. 8 app. A.2 | optimiser, schedule, warm-up, weight decay: not stated |
| sequence length | 1,024 tokens | tab. 8 app. A.2 | inputs truncated |
| utility weight lambda | 1 | tab. 8 app. A.2 | |
| utility weight eta | 1 | tab. 8 app. A.2 | |
| ranking-loss weight alpha | 1 | tab. 8 app. A.2 | ablation compares 1 with 0 app. A.2 |
| targets | centred (main); uncentred (ranking ablation) | tab. 8 app. A.2 | the centring rule is not stated |
| gradient checkpointing | on | tab. 8 | |
| validation fraction | 0.2 (of families) | tab. 8 app. A.1 | |
| test fraction | 0.2 (of families) | tab. 8 app. A.1 | |
| early-stopping patience | 6 evaluations | tab. 8 app. A.2 | evaluate every 50 steps |
| stopping threshold k | 3 | §3.4 app. A.5 | preregistered; sweep over 2 to 5 in tab. 6 |
| last turns in context | 12 | §3.3 | |
| rollout temperature | 0 | §3.2 | one rollout per action |
| backbone models | Qwen3-4B, 8B, 14B (end to end); Qwen3-8B and 14B as label sources | §3.2 §4.2 | |
| cost profile values | not stated | §3.1 | the more expensive profile is used in tab. 9 |
Results
Author-reported throughout. Setup: ToolSandbox; 82-family development pool for the offline and most end-to-end numbers; 89 unseen families for the transfer rows §4.1 app. A.1. Comparators: the unmodified backbone, a constant action picked in hindsight on each split, prompted and uncertainty-based selectors, and two decision rules from prior work, each re-implemented inside the authors' framework §4.1 app. A.3.
Table 2: counterfactual dataset tab. 2 (mean outcome per meta-action).
| Meta-action | Records | Success | Cost | Critical |
|---|---|---|---|---|
| ask the user | 1028 | 0.441 | 0.396 | 0.043 |
| search, read only | 1008 | 0.574 | 0.356 | 0.048 |
| act reversibly | 861 | 0.564 | 0.364 | 0.034 |
| commit irreversibly | 218 | 0.471 | 0.373 | 0.037 |
| all | 3115 |
Offline selection (ranking loss and heads). Ranking term versus regression-only: +0.0309 ± 0.0028 (SE) over 330 family-held-out partitions, positive on 250 §4.3 tab. 9. Median +0.0257 app. D.6. Three outcome heads versus one scalar: +0.0082 ± 0.0016 over 312 partitions tab. 9 §4.3.
Offline comparison with the hindsight-best constant action (120 development partitions) tab. 10 §4.3:
| Comparison | Gain | Better on | p |
|---|---|---|---|
| controller over best fixed action | +0.0092 ± 0.0016 | 93/120 | <0.0001 |
| SURE-Act vs cost-aware verbalised | +0.0068 ± 0.0023 | 80/120 | 0.0003 |
| SURE-Act vs best prompted | +0.0107 ± 0.0022 | 86/120 | <0.0001 |
A perfect-utility selector reaches +0.1107 over the same reference, so the controller captures 8.3% of the available headroom app. D.4 app. D.7.
All 18 selectors tab. 7, each scored against the same hindsight reference over 120 partitions (mean gain ± SE; partitions with a positive gain):
| Selector | Gain | Better on |
|---|---|---|
| SURE-Act | +0.0092 ± 0.0016 | 93/120 |
| cost-aware verbalised | +0.0024 ± 0.0019 | 71/120 |
| multi-signal gate | +0.0023 ± 0.0019 | 78/120 |
| prompted, plain | -0.0016 ± 0.0021 | 57/120 |
| prompted, cost-aware | -0.0028 ± 0.0025 | 58/120 |
| cost-aware entropy | -0.0057 ± 0.0015 | 35/120 |
| self-consistency | -0.0073 ± 0.0015 | 6/120 |
| always search | -0.0073 ± 0.0015 | 0/120 |
| cost-aware self-consist. | -0.0073 ± 0.0015 | 0/120 |
| token entropy | -0.0094 ± 0.0014 | 8/120 |
| verbalised confidence | -0.0111 ± 0.0020 | 31/120 |
| prompted, CoT | -0.0250 ± 0.0026 | 23/120 |
| always act | -0.0394 ± 0.0032 | 0/120 |
| ask-or-act heuristic | -0.0397 ± 0.0027 | 8/120 |
| SAGE-Agent (adapted) | -0.0550 ± 0.0030 | 6/120 |
| random legal action | -0.0629 ± 0.0022 | 0/120 |
| always ask | -0.1648 ± 0.0049 | 0/120 |
| always commit | -0.2104 ± 0.0039 | 0/120 |
Unseen families, offline. On 89 families never seen in development, the same controllers score -0.0000 (SE 0.0003), positive on 74 of 120 partitions. None of the four action rankings beats chance there app. D.3.
End to end, ToolSandbox Overall Score, gain over the unmodified backbone tab. 3 §4.2:
| Population | Gain | Better on | Serving floor |
|---|---|---|---|
| development pool, Qwen3-8B | +0.0344 ± 0.0082 | 20/22 | +0.0068 |
| unseen families, Qwen3-8B | +0.0216 ± 0.0093 | 18/21 | +0.0011 |
| unseen families, Qwen3-4B | +0.0575 ± 0.0095 | 20/22 | -0.0012 |
| unseen families, Qwen3-14B | +0.0252 ± 0.0101 | 17/20 | +0.0013 |
An independent repeat on the development pool gives +0.0386 ± 0.0093 §4.2. Family-clustered 95% intervals include zero for the 8B development, 8B unseen and 4B unseen rows §4.2 app. D.8.
Matched comparison against adapted rules tab. 4 §4.2:
| Rule | Score | vs agent | Better on | Override rate | SURE-Act over it |
|---|---|---|---|---|---|
| SURE-Act | 0.451 | +0.0299 ± 0.0104 | 17/22 | 0.1917 | |
| SAGE-Agent (adapted) | 0.300 | -0.1224 ± 0.0142 | 1/22 | 1.0000 | +0.1506 (22/22) |
| Calibrate-Then-Act (adapted) | 0.307 | -0.1152 ± 0.0143 | 0/22 | 1.0000 | +0.1438 (21/22) |
| agent run twice | 0.418 | -0.0051 ± 0.0033 | 11/22 | 0.0000 |
Family-clustered 95% intervals for the margins over the adapted rules are [0.0667, 0.2433] and [0.0583, 0.2367] §4.2.
Interaction cost tab. 5 §4.4 (paired per scenario; discounted cost):
| Rule | Override rate | Turns | Cost |
|---|---|---|---|
| SURE-Act | 0.192 | +0.133 | -0.068 |
| SAGE-Agent | 1.000 | +0.543 | +0.080 |
| Calibrate-Then-Act | 1.000 | +0.492 | +0.077 |
| agent run twice | 0.000 | -0.004 | -0.000 |
Threshold sweep tab. 6 §4.5. The registered k = 3 is marked.
| k | Rate | Turns | Gain | Interval | Better |
|---|---|---|---|---|---|
| dev 2 | 0.089 | -0.091 | +0.0376 | [+0.0148, +0.0605] | 17/21 |
| dev 3 (registered) | 0.192 | +0.281 | +0.0358 | [+0.0163, +0.0553] | 19/21 |
| dev 4 | 0.308 | +0.677 | +0.0353 | [+0.0165, +0.0542] | 19/21 |
| dev 5 | 0.439 | +1.064 | +0.0119 | [-0.0107, +0.0344] | 12/21 |
| dev floor | 0.000 | -0.008 | -0.0008 | [-0.0080, +0.0065] | 11/21 |
| unseen 2 | 0.096 | -0.158 | +0.0355 | [+0.0184, +0.0526] | 18/20 |
| unseen 3 (registered) | 0.201 | +0.204 | +0.0225 | [+0.0024, +0.0426] | 17/20 |
| unseen 4 | 0.316 | +0.616 | +0.0044 | [-0.0138, +0.0226] | 15/20 |
| unseen 5 | 0.418 | +1.066 | -0.0072 | [-0.0247, +0.0103] | 11/20 |
| unseen floor | 0.000 | -0.039 | +0.0008 | [-0.0022, +0.0038] | 12/20 |
No stopping rule. With the same controller applied at every turn, the intervention rate is 0.9273 and the score falls 0.0707 (SE 0.0154) below the unmodified backbone. With the stopping rule the rate is 0.2256 and the score rises 0.0344 §4.7. Across nine decision rules, intervention rate and score change correlate at -0.77 §4.7 fig. 2; restricted to five intervention-control rules, the rate-versus-score correlation is -0.8693 app. D.9.
Ceiling and worst action. The head picks the worst legal action on 13.5% of development checkpoints versus 31.3% for random and 10.3% for the best fixed action; on unseen families 8.7%, 30.9% and 4.8% app. D.5. The paper says this is not better than the hindsight best fixed action.
Ablation of the three heads and cost weights tab. 9 app. D.6. Paired differences, n:
| Change | Paired difference | n |
|---|---|---|
| within-checkpoint ranking term | +0.0309 ± 0.0028 | 330 |
| three outcome heads over a scalar | +0.0082 ± 0.0016 | 312 |
| a larger encoder | +0.0067 ± 0.0035 | 85 |
| batch size | -0.0028 ± 0.0034 | 90 |
| halving the cost weight | +0.0025 ± 0.0050 | 41 |
| doubling the cost weight | -0.0009 ± 0.0054 | 44 |
| halving the risk weight | +0.0039 ± 0.0051 | 44 |
| doubling the risk weight | +0.0038 ± 0.0062 | 44 |
| a cheaper cost profile | +0.0034 ± 0.0049 | 47 |
| a more expensive cost profile | -0.0205 ± 0.0060 | 47 |
| a different encoder family | -0.0106 ± 0.0048 | 36 |
Headline numbers for the 120-partition offline evaluation tab. 10: gain +0.0092 ± 0.0016 over the best fixed action, 93/120 (table above).
Other results. A second environment (tau-squared-bench, retail and airline) was run and the controller chose one action at every decision point. The paper traces this to its own branch truncation, which gives every non-handoff branch a score of zero by construction, and draws no transfer conclusion app. C.
Paper vs code
No official code.
The paper has no code link in its text, abstract or arXiv comments. The LaTeX
source has no URL except citations. The
source comments mention a Python package named sureact and a scripts/ folder,
but no repository was found under those names. Searched GitHub repositories for
sureact, sure-act ToolSandbox, ToolSandbox counterfactual, agents last doubt ToolSandbox and ToolSandbox Qwen3; the sureact hits are unrelated
projects, and the others returned nothing. Searched GitHub users for the first
authors Qisai Liu and Qingyue Jiao; nothing matched. No project page was linked.
Pitfalls
- Labels come from one rollout at temperature zero. Over three labelling runs that differed only in seed, the top-ranked action differed between runs at 345 of the 725 checkpoints they all covered (flip rate 0.476). Absolute action values do not transfer between label runs, though the best fixed action and the aggregate margin reproduced app. D.1. Expect label noise to dominate the ranking margin.
- Labels measure one forced turn followed by the original policy. Later behaviour can undo the intervention, and repeated interventions are not what the labels measure §3.4 app. D.1. The paper therefore adds the stopping rule as a separate heuristic rather than a learned value §3.4.
- Ranking loss normalisation differs between sections. §3.3 averages the hinge per decision point; app. B.1 averages over all pairs in a batch and uses an absolute-gap form §3.3 app. B.1. The per-batch version weights checkpoints with many legal actions more heavily. The paper does not say which was used for the 330-partition ablation, though app. B.1 describes the implemented objective.
- The stopping rule fires before the third proposal is executed. With k = 3, the controller can run at most two identical proposals in a row before handing over; the third identical proposal is replaced by the backbone's own action choice §3.4 alg. 1. Once triggered, control never returns in that episode §3.4.
- Intervention rates differ across jobs. The development 8B job reports a rate of 0.2256 §4.7; the matched rules job reports 0.1917 or 0.192 tab. 4 tab. 5; the sweep reports 0.192 at k = 3 on the development pool and 0.201 on the unseen pool tab. 6. The development 8B gain is +0.0344 while the matched-job gain is +0.0299 tab. 3 tab. 4. Quote each number with its job.
- The unseen-family results are weak. Offline, the 89 unseen families give -0.0000 with no action ranking above chance app. D.3. End to end, the 4B mean of +0.0575 (the abstract's "up to" figure) has a family-clustered interval of [-0.0063, 0.1303]; the 8B unseen interval includes zero too app. D.8. The paper states that partition-level standard errors describe sensitivity to the split, not uncertainty over new families app. A.6.
- No safety gain. Every repaired evaluation with component statistics has more critical-error occurrences than the unmodified backbone, while milestone scores rise §4.2 app. D.10. The result is about task competence, not safety.
- Overall Score and training utility differ in shape. The native score multiplies milestone by an indicator of no critical error, while the training utility subtracts a linear penalty app. A.5 app. D.7. Selecting with perfect knowledge of the training utility reaches 91.5% of the native ceiling app. D.7. Cost-profile values are not stated, so the utility cannot be reproduced from the card alone §3.1.
- The learned head's margin over a fixed schedule is not established. The head adds +0.0197 over a fixed-action schedule that intervenes about as often, with an interval including zero, and +0.0136 in a replication app. D.10. The paper claims the combination (head plus stopping rule), not either part alone app. D.10.
- Comparisons are against adapted rules. SAGE-Agent and Calibrate-Then-Act are re-implemented as one decision rule each inside the authors' framework, not the published systems §4.1 app. A.3. The paper says the numbers should be read under this framework only.
- Serving noise is measured per job. Each end-to-end gain is compared with a floor from the unmodified backbone run twice in the same job (+0.0068 for the development 8B job tab. 3; the appendix reports the same run pair differing by 0.0068, SE 0.0037 app. D.9). Gains near a job's floor are not distinguishable from serving variation.
- The family counts are inconsistent. The worst-action table uses 53 development and 80 unseen families app. D.5, while the rest of the paper uses 82 and 89. The paper does not explain the difference.
- Reproduction needs ToolSandbox and served policies. Offline numbers can be re-derived from saved labels, but the end-to-end numbers need the environment and a vLLM-served policy app. E.3. No labels or code are released with the paper.
Lineage
- Builds on: ToolSandbox, the multi-turn tool-use benchmark used for all labels and evaluation (Lu et al., 2025) §2 §4.1; Qwen3 (arXiv:2505.09388) as the controller encoder and as the policy family §3.3 tab. 8; LoRA (Hu et al., 2022) §3.3; ReAct-style tool use (Yao et al., 2023) §1.
- Compared against: SAGE-Agent, a structured-uncertainty clarification rule (Suri et al., 2026, ACL Findings) §2 §4.1; Calibrate-Then-Act, a cost-aware decision rule (Ding et al., 2026, arXiv:2602.16699) §2 §4.1. Rollout-based learners, REFUEL and CollabLLM, are discussed as related work but not compared §2.
- Superseded by: not stated.
Source and license
Adapted from “Agents' Last Doubt: Knowing When to Act and When to Step Back in Tool-Using Agents” by Qisai Liu, Qingyue Jiao, Bangwei Guo, Wenxi Dai, Changyi Yang, Jiahe Liu, Dimitris N. Metaxas, Yiyu Shi, Zhanhong Jiang, Minghao Guo (arXiv:2610.24194), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.