CAPO: Critic-Guided Action-Aligned Policy Optimization for Advancing LLM Agent Capabilities

In plain words

An AI agent often works in whole steps: it writes one message that contains a short reasoning note and a tool call, the environment answers, and then the next step begins. Standard reinforcement learning for language models scores and adjusts every single word of that message separately, even though the world only reacts once the whole message is finished.

CAPO keeps the step as the unit. A learned critic estimates how good the situation was just before each step, so the step can be credited or blamed for what it led to. The whole step then receives one update weight. Because steps differ in length, CAPO also rescales how the word-level changes inside a step are combined, so a long step does not dominate the update just by containing more words.

The aim is a training method that matches how agents actually act, while still using the usual word-level machinery that language models are built on.

TL;DR

CAPO is a PPO-style method for multi-turn LLM agents. A critic reads the state at the start of each complete action, and GAE gives that action one advantage. The token probability ratios inside the action are summed, divided by the square root of the action length, and centred on a batch mean before clipping. The paper reports gains over PPO, GRPO, GiGPO and ST-PPO on four agent settings with Qwen3-1.7B and Qwen3-4B. The results are author-reported. The released repository does not contain the action-aware ratio.

Problem

Critic-free methods such as GRPO give every token in a trajectory one shared advantage §1. Token-level critic methods such as PPO estimate values, assign credit and clip ratios per token, although the environment changes only after a complete action §1 fig. 1. The paper asks how critic-based policy optimisation can be aligned with the action while staying computable with token-level RL training §1.

The paper also argues that directly multiplying or averaging token ratios within an action makes weighting and clipping depend on action length §1 §3.4.

Method

Overview. CAPO uses complete actions as the decision unit. The advantage says which action to reinforce; the action-aware ratio says how strongly that action is weighted and clipped §3.1 fig. 2.

1. Decision granularity. At step t the state s_t is the interaction history (M_t tokens). The action a_t is the complete generated response (L_t tokens, reasoning and tool-call tokens). The environment changes only after the full action, and the trajectory is the sequence of (state, action, reward, next state) tuples §3.2 eq. 1.

2. State value at the action boundary. CAPO reads the critic's value at the last state token, which is conditioned on the whole state prefix but not on the action. No extra state encoder is needed §3.3. Appendix app. A gives the same definition.

3. Action-level advantage. Steps are ordered within each trajectory. The TD residual uses the next step's boundary value, which is set to zero after the terminal step. GAE runs backwards along the step timeline. The resulting advantage is assigned to every token of the action §3.3 eq. 2. The reward of an action may be intermediate feedback or the sum of token penalties §3.3.

4. Token ratios. For token i of action t, w_i = π_θ / π_old given the same prefix. Its log is z_i §3.4 eq. 3.

5. Aggregation family. The exact action likelihood ratio is the product of token ratios. A length-scaled family exp((1/L^α) Σ z_i) interpolates between the product (α = 0) and the geometric mean (α = 1). Clipping fractions grow with length for α = 0 and shrink with length for α = 1, and α = 0.5 gives more comparable fractions in a toy experiment §3.4 fig. 3.

6. Action-aware ratio (CAPO). Let B be the set of actions in the optimisation batch, and sg the stop-gradient. The batch mean μ̂ is the token-weighted mean of all z in the batch. The action ratio is exp(μ̂ + Σ_i (z_i − μ̂) / √L_t). The paper calls this a length-calibrated ratio, not an exact likelihood ratio §3.4 eq. 4. The centring removes the mean shift that direct square-root aggregation would add §3.4.

7. Objective. The clipped PPO form is applied to the action ratio and the action advantage: the minimum of the unclipped and clipped products, with clip interval [1 − ε, 1 + ε] §3.5 eq. 5. The paper writes this as J_CAPO, an expectation to be maximised eq. 5; the sign convention below is ours.

8. Gradient. Before clipping activates, the gradient is the action's advantage times its ratio, divided by √L_t, times the sum of the token score gradients inside the action §3.5 eq. 6. When L_t = 1 the ratio reduces to the ordinary token ratio §3.5.

# Action-level advantage  [§3.3] [eq. 2]
# Input:  values (T + 1,) critic values at the boundary before each step,
#         with the value after the terminal step set to 0  [§3.3]
#         rewards (T,) reward of each complete action  [§3.3]
# Output: advantages (T,), one per action, shared by its tokens

def action_advantages(values, rewards, gamma, lam):
    advantages = [0.0] * T
    last = 0.0
    for t in reversed(range(T)):
        # TD residual between this step and the next boundary  [eq. 2]
        delta = rewards[t] + gamma * values[t + 1] - values[t]
        # GAE recursion along the interaction timeline  [eq. 2]
        last = delta + gamma * lam * last
        advantages[t] = last
    return advantages

# Action-aware policy ratio  [§3.4] [eq. 4]
# Input:  z[b][i] token log-ratios for every action b in the batch
#         L[b] token count of action b
# Output: ratio[b] one ratio per action, used for weighting and clipping

def action_aware_ratios(z, L):
    # Token-weighted batch mean; stop-gradient, so it adds no derivative
    # terms  [§3.4] [eq. 4]
    total = sum(sum(z[b]) for b in range(len(z)))
    mu = stop_gradient(total / sum(L))
    ratios = []
    for b in range(len(z)):
        # Centre each token log-ratio, sum, scale by 1/sqrt(L_b)  [eq. 4]
        centred = sum(z_i - mu for z_i in z[b])
        ratios.append(exp(mu + centred / sqrt(L[b])))
    return ratios

# Clipped objective per action  [§3.5] [eq. 5]
# The paper maximises J; this loss is its negative  (sign convention ours)
# The paper does not state how the per-action term is averaged over tokens
# or actions.  [not stated]

def capo_loss(ratio, advantage, epsilon):
    unclipped = ratio * advantage
    clipped = clip(ratio, 1 - epsilon, 1 + epsilon) * advantage
    return -min(unclipped, clipped)

Hyperparameters

Values not stated in the paper are marked "not stated". Code values come from the released repository at code_commit.

Results

Author-reported. Setup: 3 seeds per RL method; ReAct evaluated once app. E. Metric columns: HotpotQA (in-domain) and 2Wiki/MuSiQue (out-of-domain) answer accuracy; RealResearchQuery F1@all and Recall@all; ALFWorld win rate on seen and unseen splits; WebShop task score and purchase success §4.1 tab. 1. The paper marks CAPO best in every column for both backbones tab. 1.

Qwen3-1.7B tab. 1

Qwen3-4B tab. 1

Policy-ratio ablation (Qwen3-4B, action-level credit assignment) tab. 2

Credit-assignment ablation (WebShop, λ = 1.00, Qwen3-4B) tab. 3

Clipping-threshold sensitivity (Qwen3-4B) tab. 4

OOR is the share of action ratios outside [1 − ε, 1 + ε]; Active is the share where clipping is applied tab. 4.

Paper-search tool use (Qwen3-4B) tab. 5: CAPO F1@all 0.318 with 2646.35 action tokens per trajectory, 3.21 search calls and 19.84 expansion calls. The baselines are PPO (0.303; 2719.78; 1.98; 16.76), GRPO (0.294; 2951.57; 2.74; 15.48), GiGPO (0.306; 2814.79; 4.57; 16.54) and ST-PPO (0.302; 2798.10; 3.85; 17.17). Format: F1@all; action tokens; search; expand tab. 5.

Training time (WebShop, 8 × H100, seconds per iteration) tab. 6: PPO 14.12 rollout, 76.65 actor, 104.60 critic, 195.37 total. CAPO 14.01, 74.35, 101.18, 189.54. The paper reports a single measurement per method §4.4.

Figures (described, not reproduced)

• fig. 1: the granularity mismatch between agent actions and token-level optimisation.
• fig. 2: the three parts of CAPO: (a) action as decision unit, (b) action-level GAE along the step timeline, (c) the action-aware ratio.
• fig. 3: toy experiment on α across action lengths, with ε = 0.001; it motivates α = 0.5 §3.4.
• fig. 4: ablation on four benchmarks removing credit assignment (CA), policy updates (PU), or both. Values are only in the plot; the text says CA matters more on RealResearchQuery, ALFWorld and WebShop, and PU more on HotpotQA §4.3.
• fig. 5: training curves on HotpotQA and ALFWorld against PPO and others; the paper says CAPO's critic loss is more stable on ALFWorld §4.4.
• fig. 6: on WebShop, the variance of the centred log-ratio sums grows about as L^0.90, and batch-mean restoration removes the negative shift of plain square-root aggregation §4.4.
• fig. 7: ALFWorld success and step gap by golden-step group; CAPO has the lowest step gap in every group from three to nine golden steps §4.4.
• fig. 8: a paper-search trajectory mixing search and citation/reference expansion §4.4.

Paper vs code

Repository: https://github.com/AgentR1/CAPO, linked by the paper's code field, with the same author list in its citation block. Cloned at code_commit. The repository has no LICENSE at its root; the bundled verl/ directory has an Apache-2.0 LICENSE. Code is cited by path and line only.

• The action-aware ratio is not in the release. No code computes the square-root-scaled, centred ratio of eq. 4 or clips it. A search for sqrt, length-calibration and action-aware names in the project code finds nothing relevant. The only ratio implementation the step-level scripts use is GSPO, which takes the mean of token log-ratios over the action (α = 1, the geometric mean) and clamps the log-ratio at 10 core_algos.py. The registered gspo_ratio_diag loss only wraps GSPO and adds diagnostics policy_losses.py. The released step-level scripts pair action-level GAE with that GSPO loss run_steppo.sh. Whether the paper's tables used these scripts is not stated.

• Clip ε differs. The paper gives ε = 0.001 for CAPO §4.1 app. B. The step-level scripts set clip_ratio_low and clip_ratio_high to 0.0003 run_steppo.sh run_steppo.sh run_steppo.sh. The paper gives 0.0003 for its GSPO baseline app. B.

• Step advantages are whitened over the batch. The released step-level GAE normalises raw step advantages by batch mean and standard deviation (+1e-8) core_algos.py. Equation 2 has no such step §3.3.

• Action value and reward follow the paper. The critic value is read at the last prompt token core_algos.py, the step reward is the sum of token rewards in the step core_algos.py, and the value after the terminal step is zero core_algos.py. These match §3.3.

• γ, λ, KL and learning rates match. γ = 0.99 run_ppo.sh; λ = 1.0 ppo_trainer.yaml; KL coefficient 0.001 run_ppo.sh; actor LR 1e-6 run_ppo.sh; critic LR 1e-5 run_ppo.sh.

• Length limits differ. The paper gives WebShop max prompt 16384 and max response 4096 tab. 7. The WebShop script uses 8192 and 1024 run_ppo.sh. The ALFWorld script defaults to 1024 response tokens run_ppo.sh.

• GPU count differs. The paper uses 8 H100s §4.1; the WebShop script sets 4 GPUs run_ppo.sh.

• Not checked. Reward functions in recipe/ and the environment servers were not compared with Appendix B.

Pitfalls

• Implement eq. 4 yourself. The released repository does not contain it, so any reproduction needs a new implementation. The paper does not say how μ̂ is computed: over the mini-batch or the whole rollout batch [not stated]. Its own text says μ̂ is an empirical statistic that depends on the optimisation set app. A.
• Per-action loss aggregation is not stated. The paper does not say whether the per-action term is averaged over tokens, actions or the batch [not stated]. The GSPO code uses sequence-mean token-mean aggregation core_algos.py.
• ε = 0.001 clips a lot. At ε = 1e-3 the paper's measured out-of-range fraction is 48.7% on ALFWorld and 52.3% on WebShop, and active clipping is 24.5% and 26.4% tab. 4. Larger ε weakens performance, and ε = 0.1 reaches 83.57 on ALFWorld tab. 4.
• The length-calibration argument is a toy plus an approximation. fig. 3 is a toy experiment. The variance argument assumes i.i.d. centred token log-ratios §3.4. Figure 6 gives only the empirical exponent 0.90 on WebShop §4.4.
• Not an importance-sampling correction. The paper says the action ratio is not an unbiased correction and that a TRPO-style guarantee does not apply app. A. Its gradient analysis holds only near the rollout policy §3.5.
• Critic cost remains. CAPO trains a critic, unlike GRPO-style methods app. F. On WebShop it is about 3% faster per iteration than PPO in one measurement tab. 6.
• Prompt construction matters. On ALFWorld and WebShop the prompt is rebuilt from the current observation and recent actions at each step rather than accumulated as a dialogue app. B.
• Paper-search scores depend on a selector. Reward and final scoring both use pasa-7b-selector, a Qwen2.5-7B-based relevance model that the agent cannot call app. B. Scores therefore inherit that model's judgments. The test set is 50 queries app. B.
• Small test sets. ALFWorld uses 140 seen and 134 unseen validation tasks app. B. The paper search test set has 50 queries.
• Gaps versus variance. Several CAPO margins are a few standard deviations of a 3-seed estimate, for example RealResearchQuery F1@all on Qwen3-1.7B, 0.319 ± 0.007 against GiGPO's 0.298 ± 0.010 tab. 1.
• Setup-dependent claims. Results are for two backbones on four tasks. The limitations section names single ReAct-style agents, multi-agent settings, very large backbones and non-agent post-training as untested app. F.
• Released scripts are not the paper's configuration. The scripts above differ in ε, length limits and GPU count. Use tab. 7 and §4.1 for the paper's settings, and treat the scripts as starting points.

Lineage

• Builds on: PPO, arXiv:1707.06347; GAE, arXiv:1506.02438 (cited in the code core_algos.py); GRPO, arXiv:2402.03300; GSPO, arXiv:2507.18071 (cited in the code core_algos.py).
• Compared against: ReAct; PPO; REINFORCE++ (arXiv:2501.03262); GRPO; RLOO; GSPO; GiGPO [feng2026gigpo, no arXiv id in the paper's bibliography]; ST-PPO (arXiv:2511.20718) §4.1.
• Training framework: Agent-R1 (arXiv:2511.14460) on veRL §4.1.
• Superseded by: not known.