Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

In plain words

Training a language model with reinforcement learning usually means generating a batch of answers, scoring them, and only then updating the model. Newer setups let the model learn from each answer as soon as it finishes, which keeps the hardware busy. That creates a problem: the model that wrote an old answer is no longer the model being trained, so the feedback can be out of date.

The usual fix, GRPO, scores several answers to the same question together and compares them to each other. That makes it wait for the slowest answer in the group, and it does not work when the feedback is one answer per question, as in long agent tasks. This paper keeps one answer per question and instead learns a critic, a second network that estimates how good a situation is, to say which parts of an answer were better than expected. To keep learning stable, it trusts only tokens whose probability has not moved too far, drops the rest, and trains the critic more often than the main model. The authors report that this beats GRPO on math and coding tasks, but the reported gains come from their own runs.

TL;DR

• Replaces GRPO's group of rollouts per prompt with one rollout per prompt, so a finished answer can train the model at once §1 §3.2.
• Uses the probabilities logged during generation as the reference for the importance ratio, with no separate old-policy copy, and masks tokens whose ratio leaves a fixed interval §3.1 eq. 2 eq. 3.
• Updates the value model twice per policy update, freezes its attention layers, and computes advantages that skip environment-feedback tokens §3.2 §4.1.
• Author-reported: 97.3 on AIME2025 and 29.8 on SWE-Bench Verified with Qwen3-30B-A3B, against 84.2 and 27.0 for GRPO variants tab. 1 tab. 2.

Problem

Group-based policy-gradient methods such as GRPO make a batch wait for its slowest rollout, which raises off-policy drift in asynchronous training. They also need several answers per prompt, which an online or agentic setting that gives one trajectory per prompt cannot supply §1 §2. Asynchronous training also leaves each token with an unknown mix of old model versions, so the exact behaviour probability is hard to track; storing every checkpoint is called infeasible §3.1.

The paper assumes a PPO-style clipped objective and a learned value function §2. Its claims are tested on agentic reasoning with a Python tool, SWE-Bench Verified with the OpenHands scaffold, and a simulated preference-shift writing task §4.1 §4.5.

Method

The method has three parts: a token-level importance mask, a single-rollout design with a stronger critic, and a skip-observation advantage.

Token-level importance mask §3.1. The paper uses the rollout engine's log-probabilities as the behaviour proxy and drops the old-policy model. The ratio is the current policy over the rollout policy, computed per token from logged log-probabilities eq. 2. A calibration function passes the ratio through when it lies inside an open interval and returns zero otherwise eq. 3. The objective multiplies this calibrated ratio by the advantage and the policy log-probability eq. 1. Tokens outside the interval [1 − ε_ℓ, 1 + ε_h] are masked from the gradient on both sides, unlike standard PPO clipping, which acts on one side per advantage sign. The paper notes this is similar to IcePop but without the old policy §3.1.

Single rollout and value training §3.2.

• Faster value update. For each policy update, the value network gets K updates, with K > 1. The experiments use K = 2 §3.2 §4.1.
• Frozen attention in the value model. The paper traces critic gradient instability mainly to the attention layers and not to the MoE layers. During RL it freezes the value model's attention parameters and updates only the MoE projections §3.2 fig. 4.
• Value pretraining. The authors say a large value-pretraining corpus is needed to avoid a cold start. The main text gives no size §3.2.

Skip-observation token-level GAE §3.2. Environment feedback tokens are not generated by the model, so the paper removes them from the advantage recursion. For the last token of action i, written a_{i,N}, the advantage is the TD residual plus a discounted advantage of the first token of the next action eq. 4. The residual bridges the observation gap, using the value of the next action's first token eq. 5. Observation tokens are excluded from value training, and the paper states this removes noise from the advantage §3.2 §4.3.

Policy and advantage details §4.1. The policy uses the length-adaptive GAE of VAPO with λ_policy = 1 − 1/(α·l) and α = 1.5 §4.1. The appendix describes l as the raw token length when this is applied per step app. A.1. The critic uses λ_critic = 1 §4.1.

# One training batch, assembled from §3.1, §3.2 and §4.1 (not in the paper)
# Input: 128 single-rollout trajectories with per-token rollout log-probs
# Output: updated policy and value parameters

def train_on_batch(trajectories, policy, value, eps_low, eps_high):
    # Ratio from logged log-probs only, no old-policy model  [§3.1] [eq. 2]
    ratios = exp(policy_logp(tokens) - rollout_logp(tokens))  # (B, T)
    # Zero every token whose ratio falls outside the interval  [§3.1] [eq. 3]
    gate = inside_open_interval(ratios, 1 - eps_low, 1 + eps_high)
    # Advantage per action token, skipping observation tokens
    #   [§3.2] [eq. 4] [eq. 5]
    advantages = skip_observation_gae(trajectories, value, gamma, lam)
    # Two critic updates per batch, attention frozen  [§3.2] [§4.1]
    for _ in range(2):
        critic_step(value, advantages, frozen_attention=True)
    # One policy update, objective as written in eq. 1  [§3.1] [eq. 1]
    policy_loss = -mean(gate * ratios * advantages * policy_logp(tokens))
    update(policy, policy_loss)

Key equations, as numbered in the source §3.1 §3.2:

L(\theta) = \hat{\mathbb{E}}_t \left[ f(r_t(\theta), \epsilon_l, \epsilon_h)\, \hat{A}_t \log \pi_{\theta}(a_t|s_t) \right] eq. 1

r_t(\theta) = \exp\left( \log \pi_\theta(a_t|s_t) - \log \pi_{\text{rollout}}(a_t|s_t) \right) eq. 2

f(x; \epsilon_\ell, \epsilon_h) = \begin{cases} x & 1-\epsilon_\ell < x < 1+\epsilon_h \\ 0 & \text{otherwise} \end{cases} eq. 3

\hat{A}(a_{i,N}) = \delta + \gamma\lambda\, \hat{A}(a_{i+1,0}), \quad \delta = r_t + \gamma V(a_{i+1,0}) - V(a_{i,N}) eq. 4 eq. 5

Tables for the appendix alternative (step-level advantage) are in tab. 5.

Hyperparameters

Reported in §4.1 unless the table says otherwise. "Not stated" means the paper and its source give no value.

Results

Author-reported. Setup: Qwen3-30B-A3B-Thinking-2507 SFT on TIR data for math with a Python tool, and the same backbone for SWE-Bench Verified §4.1.

Math reasoning (accuracy %, Pass@1, mean of repeated runs) tab. 1:

Coding (accuracy %) tab. 2:

Ablations (AIME2025 / BeyondAIME) tab. 3 tab. 4:

Step-level advantage (appendix, at 400 steps) tab. 5 app. A.1:

Qualitative, from figures only: SAO and GRPO (w/ DIS) are similar early and diverge after about 400 steps fig. 3 §4.2; standard GRPO collapses near 160 steps and VAPO near 90 steps §4.2 §4.4; SAO's critic has higher explained variance after about 400 steps and the frozen-attention critic has a smoother gradient norm fig. 4 §4.4. Online writing simulation: SAO re-adapts its style after each reward switch, while the running-mean baseline lags fig. 5 §4.5. The paper gives no numbers for these two results.

Paper vs code

No official code. The paper and its LaTeX source contain no repository, project page or arXiv comment link. A GitHub repository search for "single-rollout", "SAO agentic rl" and organisation-scoped queries found three candidates. One, fooSynaptic/Single-rollout-async-Optimization, describes itself as an unofficial re-implementation; the other two are not owned by the authors. None was used.

Pitfalls

• Clip interval mapping. The paper names the clip ε_low and ε_high in §4.1, but the method defines ε_ℓ and ε_h and the interval [1 − ε_ℓ, 1 + ε_h] §3.1. Reading ε_low as ε_ℓ gives (0.7, 6.0) for math and (0.2, 4.0) for coding. The paper never states this mapping, so check it before copying. The upper bounds of 5.0 and 3.0 are far wider than the one-sided clip the paper contrasts with §3.1, so most upward ratio moves are kept.
• Loss form is ambiguous. Equation 1 puts f(r_t) and log π_θ in the objective, so with f(x) = x inside the interval the gradient is scaled by the ratio itself, not the usual ratio gradient eq. 1 eq. 3. The paper does not say whether the ratio is detached from the graph. The preliminaries write a PPO clipped objective with a min term §2, which the method drops; the method's objective has no min term.
• Open-interval edge. Equation 3 uses strict inequalities, so a ratio exactly equal to a bound is masked to zero eq. 3. This is unlikely to matter, but it is stated.
• Rollout log-probs as the reference. The method uses the rollout engine's log-probs in place of the old policy §3.1. Any difference between the inference and training numerics then enters the ratio directly. The paper accepts this bias but gives no measurement of its size app. B.
• Running-mean window conflicts. The ablation text describes a window of the 8 most recent rewards per prompt §4.3 (bullet), while the online simulation uses 128 §4.5. The two are different baselines.
• Stopping points differ. GRPO and VAPO numbers are the last valid checkpoint before collapse §4.2. The step-level table compares at 400 steps tab. 5, which is not the main-result step count, so its numbers (89.8 / 66.8) should not be compared with the main table's 97.3 / 74.8.
• Training length. The introduction claims stable training for "around one thousand steps" §1. The body's curves and text reach about 400 steps explicitly, and the abstract repeats the thousand-step claim. Treat the longer run as author-stated only.
• Single run per setting. Tables give means over repeated evaluation runs (16 or 4) §4.1, but no training-seed variance is reported. The gaps between SAO and the baselines (for example 97.3 vs 94.2 on AIME2025) are small relative to that unknown.
• Advantage details left open. The discount γ, the reward placed on action-boundary tokens inside eq. 5, the critic's regression target, and the exact definition of l in λ_policy = 1 − 1/(α·l) are not stated. The appendix uses "raw token length" app. A.1, which suggests l is the response length in tokens, but that is inferred.
• Frozen attention claim. The paper ties instability to "Full Attention" layers §3.2, which implies a hybrid architecture, but the experiment backbone is Qwen3-30B-A3B. The paper does not say which layers are "attention" in the value model, so an implementer should check their architecture before freezing.
• Hidden costs. The paper argues that dropping the old-policy model saves memory and that asynchrony improves utilisation §1 §3.1. It reports no throughput or memory numbers. The critic is also trained twice per batch and needs a large pretraining corpus §3.2; the cost of both is not measured.
• Deployment claim. The abstract says SAO was deployed to train GLM-5.2 (750B-A40B). The body reports no results for that model.
• Online simulation is controlled. The writing task uses a synthetic reward shift and a GLM-4.7 judge §4.5. Phase lengths, number of steps per phase, and the judge's error rate are not stated. Limitations are noted in app. B.
• Figures are described only. Several claims rest on figures without numbers (explained variance, gradient norms, clip ratio) fig. 4 fig. 3. The clip-ratio figure is described qualitatively; no clip-ratio values are given.

Lineage

• Builds on: PPO, with its clipped objective and value function §2; GRPO, which SAO replaces with single-rollout sampling §1 §3.2; VAPO, whose length-adaptive GAE and clip-higher setting are used for the baseline and the policy advantage §4.1.
• Related token masking: IcePop, which the paper says is similar but keeps an old-policy model §3.1.
• Compared against: GRPO with clip-higher and GRPO with DIS §4.2; VAPO without DIS §4.3; SPO and running-mean baselines app. A.2; step-level advantages app. A.1.
• Superseded by: not known from content in this wiki.