ExGRPO: Learning to Reason from Experience

In plain words

To teach a language model to reason with reinforcement learning, the model tries several answers to each question, a checker marks each one right or wrong, and the model is nudged toward the right ones. The usual recipe uses each batch of attempts once and then throws it away. ExGRPO keeps the correct attempts, sorts the questions by how often the model got them right, and mixes some of those saved attempts back into later batches.

Questions the model always gets right are dropped from the saved set, so the replay effort goes to questions it solves some of the time. For each replayed question it keeps the saved answer the model is most confident about, which the authors found tends to be the cleaner reasoning. The replayed answer is also weighted by how far the current model has moved away from the one that wrote it, with a formula that amplifies the low-probability parts of that answer and damps the high-probability parts.

The aim is more learning from each attempt, and steadier training on models where the usual method breaks down. The paper's experiments are on maths and reasoning benchmarks, and the numbers are the authors' own.

TL;DR

• Method for RL with verifiable rewards (RLVR): mixes replayed correct rollouts into about half of each batch, using a bucketed buffer, a retired set, a lowest-entropy selection rule and a mixed-policy loss §4.
• Motivation from one 7B study: medium-difficulty questions and low-entropy correct trajectories were the most useful experience §3.2, fig. 1.
• Author-reported: +3.0 average over on-policy RLVR on Qwen2.5-Math-7B, and stable training on Llama-3.1-8B where on-policy RLVR collapses §5.2.
• Several details differ between the paper and the released code (bucket weighting, clipping in the mixed phase, loss normalisation, entropy term); read "Paper vs code" before implementing.

Problem

On-policy RLVR discards each batch of rollouts after one gradient update. The paper argues this wastes compute and blocks the model from learning again from its earlier successes §1. Experience replay addresses this in classic RL, but the paper says efficient replay for RLVR on large reasoning models is largely unexplored §1, §2. Prior LLM replay methods (ReMix, RePO, RLEP, RRL, ARPO) are discussed in §2. The open question is which stored experiences are worth replaying §1, §3.2.

Method

Preliminary findings §3.2, fig. 1. With on-policy RLVR on OpenR1-Math using Qwen2.5-Math-7B, the authors trained three models on questions of one difficulty band each. Medium-difficulty questions gave the largest gain on test accuracy. Correct trajectories had lower token entropy than incorrect ones, and the gap was measured with a Qwen3-32B judge of reasoning validity. The authors conclude: prioritise medium-difficulty questions, and pick low-entropy trajectories §3.2.

Setup of the objective §3.1. GRPO scores each rollout against its group. The paper follows Dr.GRPO: no standard-deviation scaling and no length normalisation of the loss. The reward is 1 for a correct final answer and 0 otherwise eq. 1.

Experience management §4.1, alg. 1:

1. Collection. For each question in a batch, the rollout policy generates K answers. Every correct answer is stored under its question in buffer E, together with the question's correct count k.
2. Partition. The buffer is bucketed by each question's latest correct count.
3. Retired set. A question whose K answers are all correct is removed from E and never replayed again §4.1, alg. 1.
4. Question sampling. A bucket is drawn with probability proportional to a Gaussian of its accuracy, centred at 0.5 with σ = 1. Then a question is drawn uniformly within the bucket (sampling without replacement) §4.1, alg. 2.
5. Answer selection. For each sampled question, the stored answer with the lowest mean token entropy under the current policy is chosen. No external judge is used at training time §4.1.

The bucket weight in the paper, with accuracy k/K alg. 1:

p_k \propto \exp\!\left(-\frac{(k/K - 0.5)^2}{2}\right) \qquad [\text{alg. 1}],\ [\text{§4.1}]

Mixed batch §4.2, alg. 1. With batch size B and ratio ρ = 0.5, the number of replayed questions is min(⌊ρB⌋, |E|). The rest of the batch is fresh questions. A replayed question gets K−1 new rollouts from the current policy, and its stored answer joins them as the K-th member of the group. A fresh question gets K new rollouts. Advantages are normalised within the group §4.2, eq. 2; the code drops the std term (see Paper vs code). When the buffer is empty, the batch is all fresh alg. 1.

Mixed-policy objective §4.2, eq. 4. Let w = π_θ / π_old for fresh tokens and w* = π_θ / π_past for the replayed answer, where π_past is the policy that generated it:

\mathcal{J}(\theta) = (1-\rho)\,\mathbb{E}_{q}\!\left[\tfrac{1}{K}\sum_{i=1}^{K} \mathrm{CLIP}(w_i, \hat{A}_i)\right] + \rho\,\mathbb{E}_{q^*}\!\left[\tfrac{1}{K}\!\left( \mathrm{CLIP}(w^*, \hat{A}^*) + \sum_{i=1}^{K-1}\mathrm{CLIP}(w_i, \hat{A}_i) \right)\right] \qquad [\text{eq. 4}]

Two changes are made for the replayed answer §4.2:

• Policy shaping. The CLIP term for the replayed answer is replaced by f(w*) · Â, with f(x) = x / (x + β) and β = 0.1. The stated purpose is to amplify low-probability signal and damp high-probability signal in the replay §4.2. The paper reports that shaping alone in on-policy RLVR gives no improvement §6.2.
• Delayed start. ExGRPO is switched on only after the batch pass rate passes 35%. Before that, the run is plain on-policy RLVR §4.2, §5.1.

The paper gives the per-step complexity as O(BK) reward evaluations and O(BK) likelihood evaluations app. C. The extra cost is an entropy pass over the stored answers of the sampled questions only app. E.5.

Pseudocode (one training step, after alg. 1; the paper's pseudocode numbers its steps, which is not repeated here):

# One ExGRPO training step  [alg. 1]
# Inputs: dataset, buffer E, retired set S, batch size B, K rollouts, rho
def training_step(policy, dataset, buffer, retired, B, K, rho):
    # Phase 1: choose replayed questions from the buffer  [§4.1]
    n_replay = min(floor(rho * B), len(buffer))
    # Bucket weight is a Gaussian over the bucket's accuracy  [alg. 1]
    replay_questions = bucket_sample(buffer, n_replay)
    replay_answers = {}
    for question in replay_questions:
        # Keep the stored answer with the lowest entropy under the
        # current policy; no judge is used  [§4.1]
        replay_answers[question] = pick_lowest_entropy(
            policy, buffer[question]
        )
    # Remaining slots are fresh questions from the dataset  [§4.2]
    fresh_questions = sample(dataset, B - n_replay)

    # Phase 2: rollouts  [§4.2] [alg. 1]
    groups = {}
    for question in replay_questions:
        # K-1 new samples; the stored answer is the K-th member  [§4.2]
        fresh = sample_rollouts(policy, question, K - 1)
        groups[question] = [replay_answers[question]] + fresh
    for question in fresh_questions:
        groups[question] = sample_rollouts(policy, question, K)

    for question, answers in groups.items():
        num_correct = count_correct(answers)
        # All K correct: retire the question  [§4.1]
        if num_correct == K:
            retired.add(question)
        # Some correct: store the correct answers in their bucket  [§4.1]
        elif num_correct > 0:
            buffer.update(question, correct_only(answers), num_correct)

    # Advantage: group mean subtracted, no std scaling  [§3.1] [eq. 2]
    # Update with the mixed objective  [eq. 4]
    update_policy(policy, groups)

Per-token term for one mixed batch. Eq. 4 writes CLIP; the code's mixed phase does not clip, and the sketch follows the code (see Paper vs code):

# Per-token policy-gradient term for one answer  [eq. 4] [§4.2]
# Shapes: logp_new, logp_old, advantage are (tokens,) for one answer
def token_term(logp_new, logp_old, advantage, is_replayed):
    # Ratio of current to generating policy: pi_past for replayed
    # answers (recorded log-probs), pi_old otherwise  [§4.2]
    ratio = exp(logp_new - logp_old)
    # Replayed tokens use f(w) = w / (w + beta), beta = 0.1  [§4.2]
    shaped = ratio / (ratio + 0.1)
    weight = shaped if is_replayed else ratio
    # Unclipped in the mixed phase, as in the code
    # [code: ExGRPO/exgrpo/verl/verl/mix_src/mix_core_alg.py#L121-L127]
    return -advantage * weight

Hyperparameters

Values for the Qwen2.5-Math-7B main run. "Code" column cites the released script ExGRPO/exp_scripts/run_exgrpo.sh and config files, with paths relative to the clone root raw/code/2510.02245/.

Results

Author-reported. Setup: Qwen2.5-Math-7B trained on OpenR1-Math 45k and tested on six in-distribution math benchmarks and three out-of-distribution tasks; the average over math is in the first "Avg." column §5.1, tab. 1. Baselines are described in app. E.2. The on-policy row is the comparison for the claims in §5.2.

Main table tab. 1

Headline reading §5.2: ExGRPO is +3.0 on the math average over on-policy RLVR on Qwen2.5-Math-7B, with a larger gain on AIME24/25. In the continual setting on LUFFY, ExGRPO is below Continual LUFFY on the out-of-distribution average (60.2 vs 61.8) §5.2, tab. 1.

Other backbones app. E.3. Each row is the average over the benchmarks in its group. Gains are the authors' claim.

Within-backbone gains in math average: +5.3 (Qwen2.5-Math 1.5B), +4.1 (Qwen2.5-7B Instruct), +2.7 (Llama3.1-8B base), +3.1 (Llama3.1-8B Instruct), and +3.0 for Qwen2.5-Math-7B (computed from the tables above). The text reports that the Llama3.1-8B on-policy run collapses §5.2 §6.1, fig. 4.

Ablations app. E.4. All rows are on the six math benchmarks; the out-of-distribution columns are not in the table source.

Retired-question replay (10% of retired examples every 25 steps) lowers both in- and out-of-distribution averages (ID 45.3 vs 48.4) app. E.4.

Bucket-weight sensitivity §6.2. Averages on the same six math benchmarks (ID) and three OOD tasks, with the paper's accuracy-unit parameters:

Continuous-reward extension §5.3. A small preliminary study on 7k WildChat-IF prompts with Qwen2.5-7B-Base, one 8B reward model, and two epochs of on-policy training. Averages over ArenaHardV2, IFEval and IFBench: GRPO 20.0, ExGRPO 21.7. The authors call this preliminary.

Cost app. E.5. Qwen2.5-Math-7B on 8 GPUs: average step time 155.4 s on-policy vs 184.9 s with ExGRPO (+19.0%). Replay memory was about 1.77 GB for 202,011 stored trajectories. Qwen2.5-Math-1.5B: +2.1% average step time.

Paper vs code

Code citations are relative to the clone root raw/code/2510.02245/, which is the LUFFY repository. ExGRPO lives in its ExGRPO/ folder. The paper's abstract links this repository, which is the basis for using it. The repository owner is not among the paper's authors, so treat its provenance with care. Commit 01f4ae066249e6965ce0d5512ccdf99a12895c74. The ExGRPO-specific code sits in a verl fork (Apache-2.0 license file at ExGRPO/exgrpo/verl/LICENSE), with no separate license file for mix_src/.

1. Bucket weight scale differs. The paper weights buckets with N(k/K; 0.5, σ = 1), with accuracy k/K in [0,1] alg. 1 §4.1. The code uses the success count k directly, with mean K/2 = 4 and σ = 1 weighted_bucket_sampler.py selected by exp_bucket_manager=normal run_exgrpo.sh. Relative bucket weights for counts 1 to 7 are about 0.01, 0.14, 0.61, 1, 0.61, 0.14, 0.01. Under the paper's text, accuracy 0.5 gets weight 1 and accuracy 0 or 1 gets about 0.88, so the paper's sampler is almost uniform. The code's sampler is strongly peaked. In accuracy units, the code's σ = 1 (in counts) is σ = 1/8, so the code matches a much narrower sampler than the paper's text describes. The σ = 0.5 and σ = 1.5 runs in §6.2 are stated in accuracy units. Which scale produced the reported numbers is not stated.

2. Clipping is removed in the mixed phase. Eq. 4 and §4.2 keep CLIP for both fresh and replayed tokens. In the code, the mixed-phase flag passes loss_remove_clip from the script (True) mix_trainer_experience.py, and the loss skips the clip branch mix_core_alg.py. Before the switch, the flag is False and clipping stays on mix_trainer_experience.py. The appendix theory section assumes clipping is removed app. D, "Policy shaping via f(w)=w/(w+β) and removing clip". The main text does not say so.

3. Loss normalisation is not the one the text describes. §3.1 says length normalisation is removed, but the GRPO clip definition still has a 1/|o| factor. The code, with loss_remove_token_mean=True run_exgrpo.sh, sums token losses and divides by the padded response width of the micro-batch mix_core_alg.py, then divides by the gradient-accumulation count mix_actor.py. The effective normalisation therefore depends on micro-batch padding and on dynamic batching run_exgrpo.sh. The paper does not mention this.

4. Entropy term is missing from eq. 4. The paper's objective has no entropy term, but its training table lists entropy coefficient 0.001 app. E.1. The actor subtracts entropy times that coefficient from the policy loss mix_actor.py.

5. Gradient handling is not in the paper. The code clips the gradient norm to 1.0 mix_ppo_trainer.yaml and skips the optimizer step when the norm exceeds 500 mix_actor.py run_exgrpo.sh. The skip threshold is set in the script; the 1.0 clip is the config default.

6. KL and reference model are disabled. The paper does not mention a KL term. The script sets its coefficient to 0, turns off the KL loss, and disables the reference model run_exgrpo.sh run_exgrpo.sh run_exgrpo.sh.

7. Training length. app. E.1 says 700 steps for Qwen2.5-Math-7B. The script sets trainer.total_training_steps=701 run_exgrpo.sh. Small, but it changes the schedule by one step.

8. Replayed log-probs come from rollout time, as the paper says. The replayed answer's old log-probs are replaced with the recorded ones when exp_is_correct is set, which is the default experience_trainer_ops.py mix_ppo_trainer.yaml. See Pitfalls for the fallback when no recorded value exists.

9. Trajectory selection matches. Mean per-token entropy under the current policy, minimised experience_helpers.py experience_helpers.py. No discrepancy found.

10. Batch composition matches. Replayed count is int(B·ρ), capped by the buffer size experience_collate_fn.py. No discrepancy found.

11. Delayed-start metric. batch/solved is the share of correct responses in the batch mix_trainer_experience.py. It is checked after each step against 0.35, and once it passes, the mode is never switched back mix_trainer_experience.py. This matches §4.2. The Llama exception is not in the repo.

12. Advantage and reward match. Std scaling is off run_exgrpo.sh. Math-Verify is used for reward_impl_version=3 main_mix_ppo.py. No discrepancy found.

13. Only one run script is in the repo. The Llama, 1.5B and instruct-model settings (500 steps, simplified prompt, no delayed start for Llama) are not in the repository. They are not checked.

14. Analysis buckets differ from the method. The preliminary study uses accuracy bands (Easy [75%, 100%), Medium (25%, 75%], Hard (0, 25%]) §3.2. The method uses count buckets 1 to 7 experience_trainer_ops.py. Consistent with the paper, but a reader should not expect the analysis bands to appear in the code.

Pitfalls

• Bucket scale: see Paper vs code item 1. Implementing the paper's text gives an almost uniform question sampler. Implementing the code gives a sharp one. The paper's ablations do not say which.
• No trust region in the mixed phase. Without CLIP, a large ratio on either fresh or replayed tokens is not bounded. The f(w) term damps large ratios only on replayed tokens §4.2. The paper does not report a ratio-spread diagnostic for this.
• Silent fallback for replayed log-probs. The replacement only happens when the recorded log-prob sum is nonzero experience_trainer_ops.py. Otherwise the current policy's log-prob is kept, so the importance ratio for that answer becomes about 1 and the replay is treated like on-policy data with no correction. An implementation must record log-probs at rollout time and check for missing values.
• Shaping function is bounded. f(w) = w/(w+0.1) stays below 1 for any w, so large ratios on replayed tokens cannot raise their weight above 1. Near w = 0, f(w) ≈ 10w, so small ratios are scaled up about tenfold. The ablation without shaping drops to 41.2 app. E.4. The shape matters.
• Entropy scoring cost. The paper scores only the stored answers of the sampled questions, usually a few dozen app. E.5. A naive implementation that rescores the whole buffer each step will be much slower.
• Buffer memory. About 1.77 GB for 202k trajectories in the 7B run app. E.5. Stored answers include full token sequences, so memory grows with run length.
• Retired questions are never revisited. Reintroducing them every 25 steps lowers both averages app. E.4, so keep the retired set permanent.
• One-way switch. Once the batch pass rate passes 35%, ExGRPO stays on mix_trainer_experience.py. A run that later collapses will not fall back to on-policy.
• Continual setting trades OOD for ID. The paper reports that ExGRPO is below Continual LUFFY on OOD in the continual setting, which it attributes to less on-policy exposure §5.2.
• Llama base is not like-for-like. It uses a simplified prompt and no delayed start app. E.1 §5.1. The large OOD gain there (1.2 to 30.8) mixes the method with a different setup.
• Scope. Only verifiable tasks were studied. Open-ended tasks, valuable failures and other RL algorithms are not covered app. A.
• Averaging is not defined. Using the extension table and the main table, a plain mean over the five backbones gives +3.6 ID and +9.2 OOD, not the +3.5 and +7.6 in §1. The paper does not say how the averages are taken. Quote the per-backbone numbers.
• GPG-Zero OOD average does not follow its columns. Its three OOD columns (70.3, 40.4, 50.5) average 53.7, but the table reports 41.6 tab. 1. Verify before citing.
• Text-only RePO claim. §5.2 and app. E.4 quote an OOD average of 52.3 for ExGRPO^RE vs 46.8 for RePO. The ablation table has no OOD columns, and the OOD values are in commented-out source. Only the 46.8 matches the main table.
• Algorithm 1 typo. The argmin is written over e ∈ E, but it should range over the stored answers o_i of the sampled question alg. 1. The code does the per-question argmin experience_helpers.py.
• Seeds and variance. The paper does not state how many seeds were run. The Avg@32 protocol is for AIME and AMC only. Differences of 1 to 2 points, such as the 1.5B and bucket-width ablations, are not shown to be outside run-to-run noise.
• Continuous-reward section is a sketch. It uses one dataset, one reward model, two epochs, and a binned reward-variance difficulty rule §5.3. Do not rely on it.

Lineage

• Builds on: GRPO and Dr.GRPO for the advantage and length-normalisation choices §3.1. Prioritised experience replay (Schaul et al.) as cited in §2. The shaping function is "inspired by" the LUFFY off-policy method §4.2.
• Compared against: PRIME-Zero, Oat-Zero, GPG-Zero, RePO-Zero, SFT, SFT+RL and Continual LUFFY app. E.2 tab. 1. The closest replay baseline is RePO, which the paper says lacks experience management app. E.2.
• Superseded by: not stated; no later version is recorded in this wiki.