Paper · 2026-10-09 · cs.AI
Rollout-Time Intervention: Unlocking Agentic Reinforcement Learning Beyond Policy Limits
When every rollout in an agent RL group fails, RTI reruns the group with a validator, corrector and verifier that repair the first task-critical error. A perplexity gate and a clipped likelihood ratio control the off-policy repairs. Author-reported Pass@1, Qwen3-8B, GRPO to RTI: Navigation 35.3 to 57.3, tau2-Telecom 36.8 to 61.4, Claw-Eval-Hard20 22.5 to 50.0.
Reinforcement learning with verifiable rewards (RLVR)Policy gradientImportance weightingLLM agentreinforcement-learningllm-agenttool-useoff-policygrpo
arXivPDFCodeMarkdown18 min read
AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
An AI agent that is trained with trial and error learns little when it almost never finishes a task. Every attempt fails, so there is nothing to reinforce. Rollout-time intervention (RTI) gets around this by watching a failed attempt, finding the first step that really derailed it, and rewriting only that step the way a careful reviewer would. The reviewer is steered by the grading rubric, but the rewrite must read as something the agent could have said without seeing the rubric. The rewritten attempt is then used as a training example, with two safeguards: examples that look too unlike the agent's own writing are thrown away, and the rewritten words that matter most get more weight in the update.
TL;DR
- Reinforcement learning for LLM agents stalls when the policy rarely succeeds, because a group of rollouts with no success gives no learning signal §1 §4.1.
- RTI reruns an all-failed group with an intervention pipeline: a validator flags the first task-critical error, a corrector rewrites that step, and a verifier rejects leaks and invented tool results §4.1.
- Rewritten trajectories are off-policy, so a perplexity gate (threshold delta) filters them and a clipped likelihood ratio (eps_cal) reweights their tokens §4.2 §4.3.
- Author-reported, mean Pass@1 over four evaluations, Qwen3-8B, GRPO versus RTI: Navigation 35.33 to 57.33, tau2-Telecom 36.84 to 61.40, Claw-Eval-Hard20 22.50 to 50.00 tab. 1.
- The released code departs from the paper in several places: the gate threshold, the Claw-Eval reward and trigger, and the model used as the tau2 intervener. See Paper vs code.
Problem
Standard on-policy RL mainly strengthens behaviour the current policy already produces. When a task lies beyond the policy's capability, successful rollouts are very rare, and training can become unstable or collapse §1. Teacher-guided alternatives, such as on-policy distillation and LUFFY-style mixing of expert rollouts, are limited by the teacher's ability §1 §2.
The authors observe that failed agent trajectories often contain one early, task-critical mistake (for example, calling a tool that belongs to the user side, or trusting a wrong search result), after which the rest of the run follows a failure loop §1. If that step were corrected, the rollout might succeed. The catch is that a corrected rollout is no longer sampled from the policy, which raises two risks: grader-side hints may show up in the generated text, and the shift can push importance ratios outside the PPO clipping range, where gradients become noisy or are suppressed §1.
Method
RTI has three parts §4. All three use the policy model itself as the intervener by default §4.1 app. C.1.
Trigger. Interventions run only when the whole group of G rollouts for a query gets zero reward §4.1. If any rollout succeeds, the group is an ordinary GRPO group and the reweighting is switched off §4.3 alg. 1.
Reward-aware intervention (§4.1). The intervention is a three-role pipeline that walks through a rollout one step at a time:
- Validator. Given the query, the observable history, the drafted step and the reward-side evidence (the evaluation rubric), it decides whether the step is still compatible with success. It is meant to flag only task-critical errors, not suboptimal but recoverable choices or different action orders. It returns a JSON object with a validity flag, error types, a diagnosis and a suggested next step §4.1.
- Corrector. It rewrites the flagged step with the history, the tool schema, the draft and the validator feedback as inputs. The rewrite should keep the draft's style and action format, fix the decisive error, and stay derivable from the visible history. Four principles govern it: style alignment, progressive correction, leakage suppression and minimal intervention §4.1.
- Verifier. It accepts a rewrite only if it is consistent with the gold requirement, does not leak privileged content, and is a plausible continuation. It rejects leaked keywords, fabricated tool results, tool arguments that contradict the history or the specified parameters, and unreasonable interventions §4.1.
Perplexity gate (§4.2). Accepted rewrites are rescored under the rollout policy, conditioned only on the original history, with the intervention removed. The rewrite is kept if its per-token perplexity is at most delta eq. 3 eq. 4. Otherwise the draft step is kept and the rollout continues.
Reasoning-shift reweighting (§4.3). For each token of a kept rewrite, the ratio between its likelihood under the correction prompt (which includes the validator feedback) and under the rollout prompt is computed eq. 5. The ratio is clipped to 1 plus or minus eps_cal and treated as a constant eq. 6. It multiplies the GRPO clipped surrogate only on rewritten tokens eq. 7.
Update. One failed group member is replaced by the successful rewritten trajectory, group advantages are computed as in GRPO from the rewards of the resulting group, and the policy ascends the RTI objective alg. 1. Environment-feedback tokens are masked out of the loss app. C.1.
Equations
Group-relative advantage, with mean and standard deviation over the G rollouts §3:
A_i = \frac{r(x,\tau_i)-\mu_r}{\sigma_r+\epsilon},\quad \mu_r=\tfrac{1}{G}\textstyle\sum_{i} r(x,\tau_i),\quad \sigma_r=\sqrt{\tfrac{1}{G}\textstyle\sum_{i}\big(r(x,\tau_i)-\mu_r\big)^2} eq. 1
GRPO objective, with KL weight beta and clip eps_clip §3:
\mathcal{J}_{\mathrm{GRPO}}(\theta)=\mathbb{E}\Big[\frac{1}{G}\sum_{i}\frac{1}{|\tau_i|}\sum_{t}\min\big(\rho^t_{i}A_i,\ \mathrm{clip}(\rho^t_{i},1-\epsilon_{\mathrm{clip}},1+\epsilon_{\mathrm{clip}})A_i\big)\Big]-\beta\,\mathbb{D}_{\mathrm{KL}}(\pi_\theta\|\pi_{\mathrm{ref}}) eq. 2
where rho is the ratio of the current policy to the behaviour policy that produced the trajectory, per token eq. 2.
Perplexity of a rewrite under the rollout policy, over its T_k tokens eq. 3:
\mathrm{PPL}_{\mathrm{roll}}(\tilde u_k)=\exp\Big(-\frac{1}{T_k}\sum_{t=1}^{T_k}\log\pi^{\mathrm{roll}}_\theta\big(\tilde u_k^{t}\mid h_{1:k-1},\tilde u_k^{<t}\big)\Big) eq. 3
Gate eq. 4:
\mathrm{PPL}_{\mathrm{roll}}(\tilde u_k)\le\delta eq. 4
Token ratio between correction prompt and rollout prompt eq. 5:
\rho^t=\exp\big(\log\pi^{\mathrm{cal}}(\tilde u_k^t\mid h_{1:k-1},\tilde u_k^{<t})-\log\pi^{\mathrm{roll}}_\theta(\tilde u_k^t\mid h_{1:k-1},\tilde u_k^{<t})\big) eq. 5
Clipped weight, a stop-gradient constant that is 1 on on-policy tokens eq. 6:
\tilde\rho^t_{\mathrm{cal}}=\mathrm{clip}\big(\rho^t,\ 1-\epsilon_{\mathrm{cal}},\ 1+\epsilon_{\mathrm{cal}}\big) eq. 6
RTI objective eq. 7:
\mathcal{J}_{\mathrm{RTI}}(\theta)=\mathbb{E}\Big[\frac{1}{G}\sum_{i}\frac{1}{|\tilde\tau_i|}\sum_{t}\tilde\rho^t_{\mathrm{cal}}\min\big(\rho^t_{i}A_i,\ \mathrm{clip}(\rho^t_{i},1-\epsilon_{\mathrm{clip}},1+\epsilon_{\mathrm{clip}})A_i\big)\Big] eq. 7
Algorithm
The paper's Algorithm 1 is reproduced here in simplified form alg. 1 app. A.2. It describes one intervened rollout per query; see Paper vs code for what the released code does.
# Group update with rollout-time intervention [alg. 1] [§4]
# Input: query, policy pi, group size G, gate threshold delta
# Output: group of G trajectories with rewards and token weights
def rti_group(query, pi, G, delta):
# Stage 1: plain on-policy rollouts, no intervention [alg. 1]
group = [rollout(pi, query) for _ in range(G)]
if any(traj.reward == 1 for traj in group):
# A success exists: ordinary GRPO group, weights stay 1 [§4.3]
return group, unit_weights(group)
# Stage 2: one intervened rollout for this query [§4.1] [alg. 1]
attempt = intervened_rollout(pi, query, delta)
if attempt.reward < 1:
# Intervention did not succeed: keep the stage 1 group [alg. 1]
return group, unit_weights(group)
# Replace one failed member with the repaired trajectory [alg. 1]
slot = random_failed_index(group)
group[slot] = attempt
return group, token_weights(group)# One intervened rollout [§4.1] [§4.2]
# A repair is kept only if the verifier accepts it and it passes the gate
def intervened_rollout(query, pi, delta):
# Observable steps so far: reasoning, action, observation [§3]
history = []
while not episode_done(history):
# Draft the next step with the policy itself [§4.1]
draft = pi.sample_step(query, history)
# Validator: flag only task-critical deviations [§4.1]
verdict = validator(query, history, draft)
if verdict.is_valid:
history.append(draft)
continue
# Corrector: minimal rewrite of the decisive error [§4.1]
fixed = corrector(query, history, draft, verdict)
# Verifier: reject leaks, invented tool results, bad arguments [§4.1]
if not verifier(query, history, draft, fixed).ok:
history.append(draft)
continue
# Gate: perplexity under the rollout policy [eq. 3] [eq. 4]
ppl = perplexity(pi, query, history, fixed)
if ppl <= delta:
history.append(fixed)
else:
history.append(draft)
return history# Per-token weights on repaired tokens [eq. 5] [eq. 6]
# logp_cal: log-probs under the correction prompt (has validator feedback)
# logp_roll: log-probs under the rollout prompt (original history only)
def token_weights(logp_cal, logp_roll, eps_cal):
# Likelihood ratio of each repaired token [eq. 5]
ratio = exp(logp_cal - logp_roll) # (T,)
# Two-sided clip around 1, a constant with no gradient [eq. 6]
return clip(ratio, 1 - eps_cal, 1 + eps_cal)# Per-token term of the RTI objective [eq. 7] [eq. 2]
# Weight applies to repaired tokens only; other tokens use 1 [§4.3]
def rti_token_term(ratio_grpo, advantage, weight, eps_clip):
# PPO-style clipped surrogate, as in GRPO [eq. 2]
unclipped = ratio_grpo * advantage
clipped = clip(ratio_grpo, 1 - eps_clip, 1 + eps_clip) * advantage
# Multiply the clipped surrogate by the reasoning-shift weight [eq. 7]
return weight * minimum(unclipped, clipped)Theory
The appendix gives an O(1/sqrt(K)) stationarity bound for RTI. It is conditional on an alignment assumption on the update direction and on bounded second moments. The expected fraction of intervened updates is treated as a constant lambda app. B.1. It is a convergence argument, not evidence about benchmark performance.
Hyperparameters
Values stated in the paper are given first. Code values come from the released scripts and defaults, with citations. "not stated" means the paper and the code do not give the value.
| Name | Paper | Where | Code (released) |
|---|---|---|---|
| Backbones | Qwen3-8B; Qwen3.5-9B | app. C.1 | same, via hf_checkpoint |
| Intervener (validator, corrector, verifier) | the policy model itself (default) | §4.1 app. C.1 | Navigation Qwen3-8B: policy endpoint run-qwen3-8B.sh; tau2: default is USER_SIM_MODEL = Qwen3.5-397B-A17B run-qwen3-8B.sh run-qwen3-8B.sh |
| Group size G | 8 | app. C.1 | 8 run-qwen3-8B.sh |
| Rollout temperature | 0.8 | app. C.1 | 0.8 run-qwen3-8B.sh |
| Optimizer | Adam, learning rate 1e-6, constant, weight decay 0.1 | app. C.1 | Adam, beta1 0.9, beta2 0.98 run-qwen3-8B.sh |
| GRPO clip eps_clip | 0.2 (symmetric) | app. C.1 | 0.2 low, 0.28 high run-qwen3-8B.sh |
| Reweighting clip eps_cal | 0.2, so weights lie in [0.8, 1.2] | app. C.1 eq. 6 | lower bound 0.8, upper bound 1.2 run-qwen3-8B.sh |
| KL weight beta | not stated | eq. 2 | 0.0 Navigation run-qwen3-8B.sh; 0.0 Claw run-qwen3.5-9B.sh; 0.001 tau2 run-qwen3-8B.sh |
| Gate threshold delta | 5, used in all experiments; swept over {0, 3, 5, 20, inf} on Navigation with Qwen3-8B | §5.4 fig. 3 | 5.0 Navigation run-qwen3-8B.sh; 20.0 tau2 run-qwen3-8B.sh; 20.0 Claw run-qwen3.5-9B.sh |
| Replaced members per group | one failed member | alg. 1 | 1 Navigation run-qwen3-8B.sh; 1 tau2 run-qwen3-8B.sh; Claw: one champion slot (see Paper vs code) |
| Claw success threshold | task score above 0.75 | app. C.2.3 | 0.75 run-qwen3.5-9B.sh |
| Max interaction turns | 30 for tau2-Bench | app. C.2.2 | 40 tau2 run-qwen3-8B.sh; 6 Navigation run-qwen3-8B.sh |
| Max response length | not stated | 8192 Navigation run-qwen3-8B.sh; 9096 tau2 run-qwen3-8B.sh; 20000 Claw run-qwen3.5-9B.sh | |
| Context length | not stated | 32000 Navigation run-qwen3-8B.sh | |
| Global batch size | not stated | 128 Navigation run-qwen3-8B.sh; 128 tau2 run-qwen3-8B.sh; 64 Claw run-qwen3.5-9B.sh | |
| Training length | not stated | 500 rollout steps in all three scripts run-qwen3-8B.sh | |
| Intervener temperature | not stated | 0.0 default args.py config.py; 0.3 tau2 run-qwen3-8B.sh and Claw run-qwen3.5-9B.sh | |
| Intervener max new tokens | not stated | 800 default config.py; 9096 tau2 run-qwen3-8B.sh and Claw run-qwen3.5-9B.sh | |
| Evaluation samples | 4 rollouts per task; metrics reported at each k from 1 to 4 | §5.1 | n-samples-per-eval-prompt 4 run-qwen3-8B.sh |
| Hardware | 8 NVIDIA H20 GPUs, tensor parallel 4 for RL; teachers on 16 PPU-ZW810E | app. C.1 | tensor parallel 4 run-qwen3-8B.sh |
| tau2 user simulator | Qwen3.5-397B-A17B for all methods | app. C.1 | USER_SIM_MODEL run-qwen3-8B.sh |
| Claw judge | Qwen3.7-Max | app. C.1 | not inspected |
| Claw score | task score = safety x (0.8 completion + 0.2 robustness) | app. C.2.3 | not inspected |
Results
Author-reported. Setup: Qwen3-8B and Qwen3.5-9B backbones; five benchmarks (Navigation on Amap APIs, tau2-Bench Telecom, Retail and Airline, Claw-Eval-Hard20) §5.1 app. C.2. Baselines: vanilla backbone, a larger same-family model, GPT and Claude models, GRPO, GSPO, AEPO, OPD and LUFFY §5.1 app. C.3.
Abbreviations: p@1 is Pass@1 averaged over the four evaluations. The averaged columns take the mean over k = 1 to 4 §5.1. Values are percentages.
Table 1: Qwen3-8B backbone tab. 1
| Method | Nav p@1 | Nav avg p@k | Nav avg p^k | Tel p@1 | Tel avg p@k | Tel avg p^k | Retail p@1 | Retail avg p@k | Retail avg p^k | Airline p@1 | Airline avg p@k | Airline avg p^k | Claw p@1 | Claw avg p@k | Claw avg p^k |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-8B | 22.67 | 28.50 | 14.17 | 36.84 | 57.02 | 21.49 | 45.61 | 71.93 | 35.53 | 31.00 | 50.50 | 16.00 | 18.75 | 24.38 | 15.94 |
| Qwen3-235B-A22B | 28.00 | 35.50 | 16.67 | 44.74 | 64.04 | 28.07 | 64.04 | 79.61 | 47.15 | 50.50 | 64.50 | 32.00 | 20.00 | 20.00 | 12.50 |
| GPT-4.1 | 23.33 | 31.33 | 18.67 | 39.47 | 58.99 | 23.68 | 71.93 | 83.55 | 52.19 | 53.00 | 66.50 | 34.00 | 30.00 | 30.00 | 18.75 |
| Claude-4-Sonnet | 26.00 | 32.83 | 20.67 | 49.12 | 67.76 | 32.02 | 74.56 | 85.09 | 53.95 | 54.00 | 68.50 | 35.50 | 40.00 | 46.25 | 30.00 |
| GRPO | 35.33 | 40.00 | 29.00 | 36.84 | 55.48 | 21.71 | 67.54 | 80.92 | 50.22 | 46.50 | 57.50 | 26.50 | 22.50 | 24.17 | 20.83 |
| GSPO | 28.00 | 34.50 | 18.00 | 44.96 | 64.48 | 30.66 | 63.16 | 74.78 | 46.27 | 39.00 | 55.50 | 22.50 | 26.25 | 31.67 | 19.90 |
| AEPO | 34.00 | 37.67 | 26.17 | 45.68 | 62.41 | 29.23 | 63.16 | 81.14 | 49.78 | 51.00 | 65.50 | 33.50 | 20.00 | 27.50 | 16.25 |
| OPD | 37.33 | 41.50 | 32.00 | 55.26 | 68.64 | 36.62 | 68.42 | 82.24 | 46.71 | 47.50 | 56.50 | 30.50 | 26.25 | 31.46 | 21.98 |
| LUFFY | 36.00 | 51.50 | 26.67 | 49.12 | 72.59 | 27.19 | 71.93 | 84.87 | 49.34 | 46.00 | 59.50 | 26.50 | 30.00 | 32.81 | 27.19 |
| RTI | 57.33 | 62.33 | 46.67 | 61.40 | 81.14 | 44.30 | 73.68 | 85.09 | 55.92 | 56.00 | 70.00 | 35.50 | 50.00 | 62.29 | 37.71 |
Table 2: Qwen3.5-9B backbone tab. 2
| Method | Nav p@1 | Nav avg p@k | Nav avg p^k | Tel p@1 | Tel avg p@k | Tel avg p^k | Retail p@1 | Retail avg p@k | Retail avg p^k | Airline p@1 | Airline avg p@k | Airline avg p^k | Claw p@1 | Claw avg p@k | Claw avg p^k |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.5-9B | 39.33 | 53.00 | 26.17 | 92.11 | 96.49 | 89.69 | 82.46 | 93.64 | 71.05 | 60.50 | 73.00 | 47.00 | 26.25 | 31.67 | 19.90 |
| Qwen3.5-397B-A17B | 44.00 | 53.67 | 38.00 | 95.61 | 98.25 | 93.64 | 84.21 | 94.74 | 73.90 | 63.00 | 77.00 | 51.00 | 30.00 | 40.00 | 26.25 |
| GPT-5.1 | 42.00 | 54.00 | 30.17 | 95.61 | 98.03 | 93.64 | 77.19 | 92.11 | 62.28 | 66.50 | 79.00 | 53.00 | 40.00 | 47.50 | 40.00 |
| Claude-4.6-Sonnet | 58.00 | 64.17 | 53.83 | 97.37 | 98.90 | 96.05 | 91.23 | 96.93 | 85.53 | 80.00 | 88.50 | 71.50 | 51.25 | 59.27 | 44.17 |
| GRPO | 71.33 | 74.83 | 66.33 | 95.61 | 97.59 | 93.64 | 84.21 | 94.74 | 73.90 | 75.00 | 82.50 | 63.00 | 52.50 | 57.71 | 45.42 |
| GSPO | 68.67 | 72.33 | 63.17 | 95.61 | 98.03 | 94.30 | 85.09 | 94.96 | 75.44 | 79.00 | 85.50 | 68.00 | 51.25 | 58.75 | 44.69 |
| AEPO | 70.67 | 77.17 | 64.83 | 96.49 | 98.47 | 94.52 | 85.96 | 95.18 | 76.75 | 78.50 | 86.50 | 70.00 | 51.25 | 56.66 | 44.90 |
| OPD | 70.00 | 75.00 | 58.67 | 96.49 | 98.68 | 95.18 | 86.84 | 95.61 | 78.29 | 79.50 | 84.50 | 65.00 | 51.25 | 55.94 | 47.50 |
| LUFFY | 71.33 | 76.17 | 65.17 | 96.49 | 98.68 | 95.18 | 85.96 | 95.18 | 76.75 | 78.00 | 86.50 | 67.00 | 61.25 | 66.67 | 54.90 |
| RTI | 74.67 | 78.50 | 68.67 | 96.49 | 98.68 | 95.18 | 88.60 | 96.05 | 81.14 | 82.00 | 87.50 | 70.50 | 85.00 | 94.17 | 73.96 |
The text states that with Qwen3-8B, RTI gains 22.00 points over GRPO on Navigation, 24.56 on tau2-Telecom and 27.50 on Claw-Eval-Hard20 (Pass@1). With Qwen3.5-9B, RTI reaches 85.00 on Claw-Eval-Hard20, 23.75 points above the best competing method §5.2.
Table 3: Ablation, Qwen3-8B, Navigation and tau2-Telecom tab. 3
| Method | Nav pass@1 | Nav pass@2 | Nav pass@3 | Nav pass@4 | Nav pass^1 | Nav pass^2 | Nav pass^3 | Nav pass^4 | Tel pass@1 | Tel pass@2 | Tel pass@3 | Tel pass@4 | Tel pass^1 | Tel pass^2 | Tel pass^3 | Tel pass^4 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GRPO | 35.33 | 40.00 | 41.33 | 43.33 | 35.33 | 29.33 | 26.67 | 24.67 | 36.84 | 52.63 | 63.16 | 69.30 | 36.84 | 21.93 | 17.54 | 10.53 |
| RTI w/o Post-Check | 26.00 | 33.33 | 36.00 | 38.00 | 26.00 | 14.00 | 9.33 | 7.33 | 38.60 | 53.51 | 59.65 | 67.54 | 38.60 | 24.56 | 16.67 | 14.91 |
| RTI w/o PPL Gating | 50.00 | 60.00 | 64.00 | 65.33 | 50.00 | 37.33 | 29.33 | 24.00 | 54.39 | 68.42 | 75.44 | 79.82 | 54.39 | 27.19 | 21.05 | 18.42 |
| RTI w/o Reweight | 55.33 | 58.67 | 60.67 | 62.00 | 55.33 | 45.33 | 42.00 | 33.33 | 60.53 | 79.82 | 85.96 | 88.60 | 60.53 | 38.60 | 30.70 | 20.18 |
| RTI | 57.33 | 61.33 | 64.67 | 66.00 | 57.33 | 48.00 | 42.00 | 39.33 | 61.40 | 83.33 | 87.72 | 92.11 | 61.40 | 46.49 | 37.72 | 31.58 |
The authors report that removing the verifier drops the Telecom gains almost entirely and pushes Navigation Pass@1 below GRPO. Removing the gate is the second-largest drop on Telecom, most visible in the all-of-k metrics (pass^2 falls from 46.49 to 27.19). Removing the reweighting is the mildest change §5.3.
Table 4: Benchmark sizes tab. 4
| Statistic | tau2-Telecom | tau2-Retail | tau2-Airline | Navigation | Claw-Eval-Hard20 |
|---|---|---|---|---|---|
| Train | 1148 | 563 | 271 | 600 | 160 |
| Test | 114 | 114 | 50 | 150 | 20 |
Table 5: Claw-Eval-Hard20 composition tab. 5 app. C.2.3
| Group | Count | No success in 3 rollouts | One success in 3 rollouts |
|---|---|---|---|
| Office automation family | 12 | 12 | 0 |
| Research and VQA family | 3 | 1 | 2 |
| Document family | 5 | 4 | 1 |
| All selected | 20 | 17 | 3 |
Table 6: Intervention statistics, tau2-Telecom, Qwen3-8B tab. 6 app. D.3
| Measure | Observed | Share |
|---|---|---|
| Groups that entered the intervention stage | 150 of 1,000 | 15.0% |
| Steps the validator marks invalid | 8,739 of 15,440 | 56.60% |
| Corrections the verifier accepts | 5,166 of 8,739 | 59.11% |
| Corrections the verifier turns down | 3,573 of 8,739 | 40.89% |
| Repaired trajectories with at least one accepted fix | 1,169 of 1,200 | 97.42% |
Average tool calls per trajectory, corrected versus original: 0.31 versus 0.55. Average characters: 1,097 versus 2,045.
Table 7: Out-of-distribution check, full text-only Claw-Eval (199 tasks), Qwen3.5-9B tab. 7 app. D.6
| Method | Overall p@1 | Overall avg p@k | Overall avg p^k | Hard20-ID p@1 | Hard20-ID avg p@k | Hard20-ID avg p^k | Rest-OOD p@1 | Rest-OOD avg p@k | Rest-OOD avg p^k |
|---|---|---|---|---|---|---|---|---|---|
| Claude-4.6-Sonnet | 60.10 | 67.99 | 45.78 | 51.25 | 59.27 | 44.17 | 61.07 | 68.71 | 45.89 |
| Base | 51.51 | 61.79 | 41.23 | 26.25 | 31.67 | 19.90 | 54.34 | 65.13 | 43.62 |
| GRPO | 56.06 | 65.63 | 45.83 | 52.50 | 57.71 | 45.42 | 56.56 | 66.53 | 45.87 |
| RTI | 65.20 | 72.95 | 56.47 | 85.00 | 94.17 | 73.96 | 62.92 | 70.73 | 54.48 |
Rest-OOD tasks come from families that never contributed training data. The gain there is smaller, and the authors attribute part of that to environment limits on those tasks app. D.6.
Figures. Fig. 1 contrasts on-policy RL, teacher-guided RL and RTI on the two failure modes described in the introduction fig. 1 §1. Fig. 2 is the framework overview fig. 2 §4. Fig. 3 plots Pass@4 on Navigation against delta: the extremes (0 and infinity) underperform delta = 5 fig. 3 §5.4. Fig. 4 compares training-time success of GRPO and RTI on Navigation, where RTI's intervened rollouts supply successes early on fig. 4 §5.5. Fig. 5 shows the average perplexity of intervened steps falling over training (a) and a broader, multi-modal turn distribution under RTI (b) fig. 5 §5.5. Fig. 6 shows Claw-Eval training curves; the text compares self-intervention with a 397B intervener and reports that self-intervention is stronger, but gives no table fig. 6 §5.5. Fig. 7 visualises which tokens receive the largest reweighting fig. 7 app. D.5.
Paper vs code
Official code: https://github.com/Alibaba-NLP/qqr, Apache-2.0, at commit
156f7355977e34cb44133ee3f59092f9e03ed92c (cloned with --depth 1). The paper
links it in the abstract, so it is the official repository. The RTI code is in
qqr/off_policy/rti/, and the stage-2 group logic is in
qqr/off_policy/group_rollout.py. The paper and the code disagree in these places:
- Gate threshold for tau2 and Claw. The paper says delta = 5 "in all experiments" §5.4. The tau2 and Claw scripts use 20 run-qwen3-8B.sh run-qwen3.5-9B.sh. The README says the same, with 5 only for Navigation.
- Gate comparison. The paper keeps a rewrite when its perplexity is at most delta eq. 4. The code keeps it only when the perplexity is strictly below the threshold metrics.py.
- Scope of stage 2. Algorithm 1 runs one intervened rollout per query alg. 1. The Navigation and tau2 code reruns all G members with the intervener on, then replaces one failed member with a randomly chosen success group_rollout.py.
- Claw-Eval reward. The paper says training uses a binary reward app. C.1. The Claw code writes the official grader score, a value between 0 and 1, into the sample reward group_rollout.py.
- Claw-Eval trigger and replacement. The paper triggers only when all rollouts fail §4.1. The Claw code triggers when the best-scoring member is below 0.75 group_rollout.py. It replaces one slot only if the new champion beats the old by more than 0.1 or scores 1.0 group_rollout.py group_rollout.py. The Claw script sets
RTI_REPLACE_NUM=2run-qwen3.5-9B.sh but also--inspector-replace-num 0run-qwen3.5-9B.sh. The Claw group code only checks whether the count is zero. It then replaces at most one slot, and only when the improvement test passes group_rollout.py. - Intervener model for tau2. The paper says all three roles are the policy model §4.1 app. C.1. The tau2 script defaults the intervener model name to
USER_SIM_MODEL, which is Qwen3.5-397B-A17B run-qwen3-8B.sh run-qwen3-8B.sh run-qwen3-8B.sh. The Navigation and Claw scripts use the policy model. - Leakage de-leak step. The paper names leakage suppression as a corrector principle §4.1 and ablates it app. D.2. The released code also has a separate de-leak agent, but its call is commented out in the active pipeline pipeline.py.
- Extra checks absent from the paper. A rule-based pre-check can trigger a correction even when the judge says the step is valid pipeline.py pipeline.py. A heuristic post-check runs before the LLM verifier pipeline.py. Duplicate POI-search detection is in heuristic.py.
- Failure defaults. If the judge's output cannot be parsed, the step is treated as valid pipeline.py. If the LLM verifier fails or cannot be parsed, the repair is rejected pipeline.py pipeline.py. The paper says nothing about either.
- GRPO clip. The paper uses a symmetric clip of 0.2 app. C.1. The scripts use 0.2 below and 0.28 above run-qwen3-8B.sh.
- KL weight. The paper gives no value for beta eq. 2. The code uses 0.0 for Navigation and Claw run-qwen3-8B.sh run-qwen3.5-9B.sh, and 0.001 for tau2 run-qwen3-8B.sh.
- tau2 turn limit. The paper says 30 turns app. C.2.2. The tau2 script uses 40 run-qwen3-8B.sh.
- Reweighting mask. The code finds repaired tokens by comparing teacher and rollout log-probabilities, flagging any difference above 1e-6, not by an explicit flag utils.py. Its docstring says the ratio uses the current policy, but the code uses rollout log-probs utils.py, which matches the paper's ratio. The clamp is hard, not gradient-preserving, but the ratio is a constant anyway.
- Tool-call format. For Qwen3.5 the repaired step is written in XML tool-call form, and for Qwen3 in JSON form pipeline.py pipeline.py. The paper does not mention this.
The README says the default hyperparameters follow the paper. The points above show that this holds for Navigation with Qwen3-8B, but not for every benchmark.
Pitfalls
- The gate threshold is task-specific. The paper's sweep was on Navigation with Qwen3-8B only §5.4. The released code uses a value four times larger for tau2 and Claw. Expect to tune delta per task, and check the response format first, because Qwen3 and Qwen3.5 serialise tool calls differently pipeline.py.
- Intervention is rare. Only 15.0% of tau2-Telecom groups trigger it. The validator flags 56.6% of steps, and the verifier rejects 40.9% of the resulting corrections tab. 6. Gains depend on the quality of the validator and verifier, which are the same policy model by default.
- Each intervened step costs several extra LLM calls. The code runs a validator, a corrector and a verifier, plus two scoring passes for the reweighting pipeline.py, plus perplexity scoring. The paper reports no wall-clock or token cost.
- Privileged information enters three ways. The validator and corrector see the evaluation rubric, and the reweighting ratio is computed under a prompt that contains validator feedback §4.3. The paper's leakage check is an LLM judgment with no measured false-negative rate. Without the verifier the Telecom Pass@1 falls from 61.40 to 38.60 tab. 3.
- The reweighting is small. Its weights stay within [0.8, 1.2] eq. 6. Removing it changes Telecom Pass@1 from 61.40 to 60.53 and Navigation Pass@1 from 57.33 to 55.33 tab. 3.
- Small test sets and coarse steps. Claw-Eval-Hard20 has 20 test tasks. Reported Pass@1 values move in steps of 1.25 points. Navigation has 150 tasks tab. 4. No confidence intervals are reported, and the number of training seeds is not stated.
- Hard20 is selected for failure. The 20 tasks were chosen from a pool where the base policy has a Pass@3 of at most 1/3. 17 of the 20 never succeed in three rollouts tab. 5 app. C.2.3. Training data are 160 synthetic variants of the same 20 task families app. C.2.3, so the in-distribution gains are not evidence of general capability. The out-of-distribution gains are smaller tab. 7.
- Evaluation depends on large models. The tau2 user simulator is Qwen3.5-397B-A17B for every method, and the Claw judge is Qwen3.7-Max app. C.1. The teacher baselines (OPD and LUFFY) use 235B and 397B teachers app. C.1, so the RTI-versus-teacher comparison is neither compute-matched nor teacher-matched.
- The larger-intervener result is qualitative. The text compares self-intervention with a 397B intervener and calls self-intervention stronger §5.5, but there is no table, and the figure is the only evidence fig. 6.
- The theorem is conditional. It assumes an alignment condition on the update direction and bounded second moments, and treats the mixing ratio lambda as constant app. B.1. It does not show that RTI improves benchmark scores.
- Evaluation temperature is not stated. Only the training rollout temperature (0.8) is given app. C.1.
Lineage
- Builds on: GRPO, the group-relative advantage and clipped surrogate §3 eq. 1 eq. 2. The paper cites it as Shao et al., 2024 (DeepSeekMath).
- Compared against: GRPO, GSPO, AEPO (on-policy RL); OPD and LUFFY (teacher-guided) §5.1 app. C.3.
- Superseded by: not known.
Source and license
Adapted from “Rollout-Time Intervention: Unlocking Agentic Reinforcement Learning Beyond Policy Limits” by Xi Chen, Ruixue Ding, Zhigaoyuan Wang, Fanrui Zhang, Qiang Zhang, Boli Chen, Shihang Wang, Tianfu Wang, Ailin Ren, Pengyong Wang, Pengjun Xie, Hui Xiong (arXiv:2610.13438), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/Alibaba-NLP/qqr at commit 156f7355977e (Apache-2.0). Any quoted code is shown under that license.