wikillm

Paper · 2026-10-16 · cs.AI

Agents' Last Doubt: Knowing When to Act and When to Step Back in Tool-Using Agents

SURE-Act trains a Qwen3-1.7B controller on counterfactual ToolSandbox rollouts, forcing each legal meta-action (ask, search, reversible act, irreversible commit) once per decision point. A pairwise ranking loss adds +0.0309 offline utility over regression. Handing control back after three repeated choices gives end-to-end gains (author-reported; unseen-family intervals include zero).

LLM agentTool useLearning-to-rank losses over candidate setsagentstool-usedecision-makingcounterfactual-datalearning-to-rank

arXivPDFMarkdown15 min read

AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

A tool-using AI assistant keeps facing the same small choice: act now, look something up first, ask the user, or do something that cannot be undone. Logs of past runs only show the option that was taken, so nobody learns what the other options would have done. This paper fixes that by going back to saved moments in the environment and trying every option once from the same point, then letting the rest of the episode play out with the same underlying assistant. The outcomes become labels. A small separate model learns to score the options, and at run time it picks one for the assistant.

The scoring model does better than a plain regression at ranking the options, but the bigger lesson is about how often it steps in. Overriding the assistant at almost every turn hurts the final score. A simple rule that hands control back once the model has picked the same option three times in a row turns a controller that hurts the score into one that helps it, according to the authors. The authors report that this helps on the benchmark they used, but it does not make the assistant safer, and the gains on new task families are uncertain.

TL;DR

SURE-Act is a controller that picks, at each decision point, one of four meta-actions for a frozen tool-using agent: ask the user, search, make a reversible change, or make an irreversible commit §3.1. It is trained on 3,115 counterfactual outcomes from 1,036 ToolSandbox decision points tab. 2. Offline, a pairwise ranking term improves selection over regression alone. End to end, the gains depend on a stopping rule that returns control after three identical proposals §3.4. Treat the end-to-end numbers as author-reported and as sensitive to the evaluation job.

Problem

Logged trajectories reveal only the action that was taken, so the value of unchosen actions is unobserved §1. A successful run does not show whether a cheaper action would have worked, and a failed run does not show which action would have prevented the failure §1. Acting before resolving uncertainty can cause irreversible errors, while asking or searching adds interaction cost §1. Without counterfactual labels, a selector has nothing to learn from and nothing to be scored against §1.

The paper also reports that intervening at every decision point lowers the ToolSandbox Overall Score below the unmodified backbone §4.7. Better offline selection alone therefore did not give better end-to-end performance §4.7.

Assumptions: labels come from one temperature-zero continuation per action §3.2; the backbone is frozen and the evaluation environment is the simulated ToolSandbox §4.1; the controller only sees the last twelve turns, the environment identifier and the legal-action set §3.3.

Method

Meta-actions and utility. Four meta-actions are defined: Ask (request missing information from the user), Search (read-only retrieval), Act-Reversible and Commit-Irreversible §3.1. The expected utility of an action is the success probability minus a weighted interaction cost minus a weighted probability of an unauthorised irreversible action §3.1. The paper uses a fixed cost profile for the cost term, but the values are not stated in the paper §3.1.

Pseudocode
# Utility of one legal meta-action a at state x  [§3.1]
# Q(x, a) = p_success - lam * cost - eta * p_critical
def expected_utility(x, a):
    return p_success(x, a) - lam * cost(x, a) - eta * p_critical(x, a)

Counterfactual data collection. Decision points are taken from agent trajectories and the environment state is saved at each one. For every legal meta-action, a copy of the state is made and the backbone's next turn is restricted to that category. Tool-based actions restrict the tool set to the chosen category and require a tool call; Ask makes the backbone write a question to the user. After that one turn the restriction is removed and the backbone completes the episode. The success, interaction cost and critical-error flag are recorded §3.2 alg. 1.

Pseudocode
# Counterfactual labels for one decision point  [§3.2] [alg. 1]
# Each legal action gets its own copy of the saved state.
for state in saved_states:
    for action in legal_actions(state):
        # Force the action for one turn, then let the backbone finish.
        trajectory = roll_out(force(clone(state), action))
        success, cost, critical = grade(trajectory)
        # Observed utility of this single rollout  [§3.2]
        utility = success - lam * cost - eta * critical
        dataset.append((context, action, success, cost, critical, utility))

The utility label is y = s − λc − ηr, where r is 1 if a critical error occurred §3.2. Each action has one rollout at temperature zero, so each label is one observed outcome, not an expectation §3.2.

Controller and outputs. The controller is a 1.7B-parameter Qwen3 encoder fitted with low-rank adapters, which averages its token states and feeds one linear output layer §3.3. For a context h and a candidate action a it predicts three scores, for success, cost and critical error §3.3. Ranking uses the plug-in utility, which the paper does not treat as a calibrated probability §3.3.

f_\theta(h, a) = (\hat s, \hat c, \hat r), \qquad \hat Q_\theta(h, a) = \hat s - \lambda \hat c - \eta \hat r \qquad [\S 3.3]

Losses. The outcome loss is a masked squared error over the observed (context, action, component) triples. Missing outcomes contribute nothing §3.3. The ranking loss is a hinge over legal action pairs at the same decision point whose observed gap is positive. The margin is the observed gap; pairs with equal outcomes are dropped, and a decision point with no usable pair adds no ranking penalty §3.3. The objective adds the two with weight alpha. Alpha = 0 is the regression-only ablation §3.3.

\mathcal{L}_{\mathrm{out}} = \frac{1}{|\mathcal{M}|} \sum_{(h,a,j) \in \mathcal{M}} \bigl(\hat z_{\theta,j}(h,a) - z_j(h,a)\bigr)^2 \qquad [\S 3.3]

\mathcal{L}_{\mathrm{rank}}(h) = \frac{1}{|\mathcal{P}_h|} \sum_{(a,b) \in \mathcal{P}_h} \max\{0,\; g_{ab} - (\hat Q_a - \hat Q_b)\}, \quad g_{ab} = y_a - y_b > 0 \qquad [\S 3.3]

\mathcal{L} = \mathcal{L}_{\mathrm{out}} + \alpha \, \mathcal{L}_{\mathrm{rank}} \qquad [\S 3.3]

The appendix writes the same hinge over unordered pairs with an absolute gap and a sign term, and averages over all pairs in a batch rather than per decision point app. B.1. The two normalisations are not identical (see Pitfalls).

Inference with intervention control. At every turn the controller scores each legal option and suggests the one with the top predicted utility; the backbone then carries out that suggestion for one turn, under the same restriction used during label collection §3.4. The controller remembers its last suggestion and a streak length. Repeating that suggestion lengthens the streak; choosing something else restarts it at one §3.4. Once the streak reaches the threshold of three, the controller withdraws before carrying out that suggestion, and the untouched backbone completes the episode §3.4 alg. 1. Handing control back is a stop rule, not an additional action category §3.4. The rule is preregistered §3.4.

Pseudocode
# Phase 3: inference with intervention control  [§3.4] [alg. 1]
previous = None
repeats = 0
while not episode_done():
    # Controller proposes the best-scoring legal meta-action  [§3.4]
    proposal = argmax_over_actions(q_hat(context, a))
    # Count consecutive identical proposals; a change resets to 1  [§3.4]
    if proposal == previous:
        repeats = repeats + 1
    else:
        repeats = 1
    previous = proposal
    # Third identical proposal: hand the episode to the backbone  [§3.4]
    if repeats >= k:
        backbone_finishes_episode()
        break
    # Otherwise run the proposal for one turn with the backbone  [§3.4]
    execute_one_turn(backbone, proposal)
    advance_to_next_decision_point()

Training procedure. Four epochs, batch size eight, learning rate 1e-4, inputs truncated to 1,024 tokens, validation every 50 steps, early-stopping patience of six evaluations app. A.2 tab. 8. The checkpoint is chosen by validation action-selection utility, not training loss app. A.2 §3.3.

Splits. Families, not decision points, are assigned to the three sets in a 60/20/20 ratio. Each split seed reshuffles the sorted family list, and paired methods share seeds app. A.1 §4.1.

Hyperparameters

Settings for the 120-partition runs that compare the controller with fixed-action baselines, read from the run artefacts tab. 8 app. A.2. The ranking comparison uses its own runs and does not centre its targets tab. 8 app. A.2.

NameValueWhereNotes
encoderQwen/Qwen3-1.7Btab. 8 app. A.2with LoRA adapters, mean pooling, linear head §3.3
LoRA rank16tab. 8 app. A.2"set in the training script" per tab. 8
LoRA alpha32tab. 8as above
LoRA dropout0.05tab. 8as above
LoRA target modulesnot stated
epochs4tab. 8 app. A.2
batch size8tab. 8 app. A.2
learning rate1e-4tab. 8 app. A.2optimiser, schedule, warm-up, weight decay: not stated
sequence length1,024 tokenstab. 8 app. A.2inputs truncated
utility weight lambda1tab. 8 app. A.2
utility weight eta1tab. 8 app. A.2
ranking-loss weight alpha1tab. 8 app. A.2ablation compares 1 with 0 app. A.2
targetscentred (main); uncentred (ranking ablation)tab. 8 app. A.2the centring rule is not stated
gradient checkpointingontab. 8
validation fraction0.2 (of families)tab. 8 app. A.1
test fraction0.2 (of families)tab. 8 app. A.1
early-stopping patience6 evaluationstab. 8 app. A.2evaluate every 50 steps
stopping threshold k3§3.4 app. A.5preregistered; sweep over 2 to 5 in tab. 6
last turns in context12§3.3
rollout temperature0§3.2one rollout per action
backbone modelsQwen3-4B, 8B, 14B (end to end); Qwen3-8B and 14B as label sources§3.2 §4.2
cost profile valuesnot stated§3.1the more expensive profile is used in tab. 9

Results

Author-reported throughout. Setup: ToolSandbox; 82-family development pool for the offline and most end-to-end numbers; 89 unseen families for the transfer rows §4.1 app. A.1. Comparators: the unmodified backbone, a constant action picked in hindsight on each split, prompted and uncertainty-based selectors, and two decision rules from prior work, each re-implemented inside the authors' framework §4.1 app. A.3.

Table 2: counterfactual dataset tab. 2 (mean outcome per meta-action).

Meta-actionRecordsSuccessCostCritical
ask the user10280.4410.3960.043
search, read only10080.5740.3560.048
act reversibly8610.5640.3640.034
commit irreversibly2180.4710.3730.037
all3115

Offline selection (ranking loss and heads). Ranking term versus regression-only: +0.0309 ± 0.0028 (SE) over 330 family-held-out partitions, positive on 250 §4.3 tab. 9. Median +0.0257 app. D.6. Three outcome heads versus one scalar: +0.0082 ± 0.0016 over 312 partitions tab. 9 §4.3.

Offline comparison with the hindsight-best constant action (120 development partitions) tab. 10 §4.3:

ComparisonGainBetter onp
controller over best fixed action+0.0092 ± 0.001693/120<0.0001
SURE-Act vs cost-aware verbalised+0.0068 ± 0.002380/1200.0003
SURE-Act vs best prompted+0.0107 ± 0.002286/120<0.0001

A perfect-utility selector reaches +0.1107 over the same reference, so the controller captures 8.3% of the available headroom app. D.4 app. D.7.

All 18 selectors tab. 7, each scored against the same hindsight reference over 120 partitions (mean gain ± SE; partitions with a positive gain):

SelectorGainBetter on
SURE-Act+0.0092 ± 0.001693/120
cost-aware verbalised+0.0024 ± 0.001971/120
multi-signal gate+0.0023 ± 0.001978/120
prompted, plain-0.0016 ± 0.002157/120
prompted, cost-aware-0.0028 ± 0.002558/120
cost-aware entropy-0.0057 ± 0.001535/120
self-consistency-0.0073 ± 0.00156/120
always search-0.0073 ± 0.00150/120
cost-aware self-consist.-0.0073 ± 0.00150/120
token entropy-0.0094 ± 0.00148/120
verbalised confidence-0.0111 ± 0.002031/120
prompted, CoT-0.0250 ± 0.002623/120
always act-0.0394 ± 0.00320/120
ask-or-act heuristic-0.0397 ± 0.00278/120
SAGE-Agent (adapted)-0.0550 ± 0.00306/120
random legal action-0.0629 ± 0.00220/120
always ask-0.1648 ± 0.00490/120
always commit-0.2104 ± 0.00390/120

Unseen families, offline. On 89 families never seen in development, the same controllers score -0.0000 (SE 0.0003), positive on 74 of 120 partitions. None of the four action rankings beats chance there app. D.3.

End to end, ToolSandbox Overall Score, gain over the unmodified backbone tab. 3 §4.2:

PopulationGainBetter onServing floor
development pool, Qwen3-8B+0.0344 ± 0.008220/22+0.0068
unseen families, Qwen3-8B+0.0216 ± 0.009318/21+0.0011
unseen families, Qwen3-4B+0.0575 ± 0.009520/22-0.0012
unseen families, Qwen3-14B+0.0252 ± 0.010117/20+0.0013

An independent repeat on the development pool gives +0.0386 ± 0.0093 §4.2. Family-clustered 95% intervals include zero for the 8B development, 8B unseen and 4B unseen rows §4.2 app. D.8.

Matched comparison against adapted rules tab. 4 §4.2:

RuleScorevs agentBetter onOverride rateSURE-Act over it
SURE-Act0.451+0.0299 ± 0.010417/220.1917
SAGE-Agent (adapted)0.300-0.1224 ± 0.01421/221.0000+0.1506 (22/22)
Calibrate-Then-Act (adapted)0.307-0.1152 ± 0.01430/221.0000+0.1438 (21/22)
agent run twice0.418-0.0051 ± 0.003311/220.0000

Family-clustered 95% intervals for the margins over the adapted rules are [0.0667, 0.2433] and [0.0583, 0.2367] §4.2.

Interaction cost tab. 5 §4.4 (paired per scenario; discounted cost):

RuleOverride rateTurnsCost
SURE-Act0.192+0.133-0.068
SAGE-Agent1.000+0.543+0.080
Calibrate-Then-Act1.000+0.492+0.077
agent run twice0.000-0.004-0.000

Threshold sweep tab. 6 §4.5. The registered k = 3 is marked.

kRateTurnsGainIntervalBetter
dev 20.089-0.091+0.0376[+0.0148, +0.0605]17/21
dev 3 (registered)0.192+0.281+0.0358[+0.0163, +0.0553]19/21
dev 40.308+0.677+0.0353[+0.0165, +0.0542]19/21
dev 50.439+1.064+0.0119[-0.0107, +0.0344]12/21
dev floor0.000-0.008-0.0008[-0.0080, +0.0065]11/21
unseen 20.096-0.158+0.0355[+0.0184, +0.0526]18/20
unseen 3 (registered)0.201+0.204+0.0225[+0.0024, +0.0426]17/20
unseen 40.316+0.616+0.0044[-0.0138, +0.0226]15/20
unseen 50.418+1.066-0.0072[-0.0247, +0.0103]11/20
unseen floor0.000-0.039+0.0008[-0.0022, +0.0038]12/20

No stopping rule. With the same controller applied at every turn, the intervention rate is 0.9273 and the score falls 0.0707 (SE 0.0154) below the unmodified backbone. With the stopping rule the rate is 0.2256 and the score rises 0.0344 §4.7. Across nine decision rules, intervention rate and score change correlate at -0.77 §4.7 fig. 2; restricted to five intervention-control rules, the rate-versus-score correlation is -0.8693 app. D.9.

Ceiling and worst action. The head picks the worst legal action on 13.5% of development checkpoints versus 31.3% for random and 10.3% for the best fixed action; on unseen families 8.7%, 30.9% and 4.8% app. D.5. The paper says this is not better than the hindsight best fixed action.

Ablation of the three heads and cost weights tab. 9 app. D.6. Paired differences, n:

ChangePaired differencen
within-checkpoint ranking term+0.0309 ± 0.0028330
three outcome heads over a scalar+0.0082 ± 0.0016312
a larger encoder+0.0067 ± 0.003585
batch size-0.0028 ± 0.003490
halving the cost weight+0.0025 ± 0.005041
doubling the cost weight-0.0009 ± 0.005444
halving the risk weight+0.0039 ± 0.005144
doubling the risk weight+0.0038 ± 0.006244
a cheaper cost profile+0.0034 ± 0.004947
a more expensive cost profile-0.0205 ± 0.006047
a different encoder family-0.0106 ± 0.004836

Headline numbers for the 120-partition offline evaluation tab. 10: gain +0.0092 ± 0.0016 over the best fixed action, 93/120 (table above).

Other results. A second environment (tau-squared-bench, retail and airline) was run and the controller chose one action at every decision point. The paper traces this to its own branch truncation, which gives every non-handoff branch a score of zero by construction, and draws no transfer conclusion app. C.

Paper vs code

No official code.

The paper has no code link in its text, abstract or arXiv comments. The LaTeX source has no URL except citations. The source comments mention a Python package named sureact and a scripts/ folder, but no repository was found under those names. Searched GitHub repositories for sureact, sure-act ToolSandbox, ToolSandbox counterfactual, agents last doubt ToolSandbox and ToolSandbox Qwen3; the sureact hits are unrelated projects, and the others returned nothing. Searched GitHub users for the first authors Qisai Liu and Qingyue Jiao; nothing matched. No project page was linked.

Pitfalls

  • Labels come from one rollout at temperature zero. Over three labelling runs that differed only in seed, the top-ranked action differed between runs at 345 of the 725 checkpoints they all covered (flip rate 0.476). Absolute action values do not transfer between label runs, though the best fixed action and the aggregate margin reproduced app. D.1. Expect label noise to dominate the ranking margin.
  • Labels measure one forced turn followed by the original policy. Later behaviour can undo the intervention, and repeated interventions are not what the labels measure §3.4 app. D.1. The paper therefore adds the stopping rule as a separate heuristic rather than a learned value §3.4.
  • Ranking loss normalisation differs between sections. §3.3 averages the hinge per decision point; app. B.1 averages over all pairs in a batch and uses an absolute-gap form §3.3 app. B.1. The per-batch version weights checkpoints with many legal actions more heavily. The paper does not say which was used for the 330-partition ablation, though app. B.1 describes the implemented objective.
  • The stopping rule fires before the third proposal is executed. With k = 3, the controller can run at most two identical proposals in a row before handing over; the third identical proposal is replaced by the backbone's own action choice §3.4 alg. 1. Once triggered, control never returns in that episode §3.4.
  • Intervention rates differ across jobs. The development 8B job reports a rate of 0.2256 §4.7; the matched rules job reports 0.1917 or 0.192 tab. 4 tab. 5; the sweep reports 0.192 at k = 3 on the development pool and 0.201 on the unseen pool tab. 6. The development 8B gain is +0.0344 while the matched-job gain is +0.0299 tab. 3 tab. 4. Quote each number with its job.
  • The unseen-family results are weak. Offline, the 89 unseen families give -0.0000 with no action ranking above chance app. D.3. End to end, the 4B mean of +0.0575 (the abstract's "up to" figure) has a family-clustered interval of [-0.0063, 0.1303]; the 8B unseen interval includes zero too app. D.8. The paper states that partition-level standard errors describe sensitivity to the split, not uncertainty over new families app. A.6.
  • No safety gain. Every repaired evaluation with component statistics has more critical-error occurrences than the unmodified backbone, while milestone scores rise §4.2 app. D.10. The result is about task competence, not safety.
  • Overall Score and training utility differ in shape. The native score multiplies milestone by an indicator of no critical error, while the training utility subtracts a linear penalty app. A.5 app. D.7. Selecting with perfect knowledge of the training utility reaches 91.5% of the native ceiling app. D.7. Cost-profile values are not stated, so the utility cannot be reproduced from the card alone §3.1.
  • The learned head's margin over a fixed schedule is not established. The head adds +0.0197 over a fixed-action schedule that intervenes about as often, with an interval including zero, and +0.0136 in a replication app. D.10. The paper claims the combination (head plus stopping rule), not either part alone app. D.10.
  • Comparisons are against adapted rules. SAGE-Agent and Calibrate-Then-Act are re-implemented as one decision rule each inside the authors' framework, not the published systems §4.1 app. A.3. The paper says the numbers should be read under this framework only.
  • Serving noise is measured per job. Each end-to-end gain is compared with a floor from the unmodified backbone run twice in the same job (+0.0068 for the development 8B job tab. 3; the appendix reports the same run pair differing by 0.0068, SE 0.0037 app. D.9). Gains near a job's floor are not distinguishable from serving variation.
  • The family counts are inconsistent. The worst-action table uses 53 development and 80 unseen families app. D.5, while the rest of the paper uses 82 and 89. The paper does not explain the difference.
  • Reproduction needs ToolSandbox and served policies. Offline numbers can be re-derived from saved labels, but the end-to-end numbers need the environment and a vLLM-served policy app. E.3. No labels or code are released with the paper.

Lineage

  • Builds on: ToolSandbox, the multi-turn tool-use benchmark used for all labels and evaluation (Lu et al., 2025) §2 §4.1; Qwen3 (arXiv:2505.09388) as the controller encoder and as the policy family §3.3 tab. 8; LoRA (Hu et al., 2022) §3.3; ReAct-style tool use (Yao et al., 2023) §1.
  • Compared against: SAGE-Agent, a structured-uncertainty clarification rule (Suri et al., 2026, ACL Findings) §2 §4.1; Calibrate-Then-Act, a cost-aware decision rule (Ding et al., 2026, arXiv:2602.16699) §2 §4.1. Rollout-based learners, REFUEL and CollabLLM, are discussed as related work but not compared §2.
  • Superseded by: not stated.

Source and license

Adapted from “Agents' Last Doubt: Knowing When to Act and When to Step Back in Tool-Using Agents” by Qisai Liu, Qingyue Jiao, Bangwei Guo, Wenxi Dai, Changyi Yang, Jiahe Liu, Dimitris N. Metaxas, Yiyu Shi, Zhanhong Jiang, Minghao Guo (arXiv:2610.24194), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.