Experience replay

Overview

On-policy training uses each batch of rollouts for one update and then discards it. Experience replay keeps some of those rollouts and mixes them into later batches. The paper frames this as wasted compute and a missed chance to learn again from earlier successes 2510.02245 §1. Replay is well established in classic RL; the paper says efficient replay for RL with verifiable rewards (RLVR) on large reasoning models is largely unexplored 2510.02245 §1, §2.

Every variant has to answer three questions: what to store, which stored item to replay, and how to correct for the fact that the replayed item was produced by an older policy. The shared correction is an importance ratio between the current policy and the policy that wrote the item (see importance weighting). Not every variant applies a correction: the continual-learning and uniform DQN variants replay stored examples as they are. The prioritised DQN variant also reweights its loss with per-sample weights from the buffer 2412.02574 §4.1. Those weights handle non-uniform sampling, not the age of the policy.

Replay also has a second meaning in continual learning. There it protects earlier tasks from being overwritten, rather than reusing successful answers on the same task 1802.07569 §1, §3.4. The tiny-memory study below uses this meaning and reports its results against fine-tuning and three other continual-learning methods 1902.10486 §4, §5.4.

Seven cards in this wiki use replay. The first variant below is the only one that studies replay for RLVR. The second is the uniform replay buffer of a deep Q-learning paper, where replay is one of the standard parts of the method. The third adds prioritised sampling to a Double DQN agent that tests self-driving cars; the card does not ablate its replay. The next two are continual-learning papers: a tiny-memory study with experiments, and a review. The sixth is a 2020 survey of curriculum learning for RL, which files replay under sample ordering. The seventh is a continued-pretraining paper that replays general text while training on a medical corpus. That is a different use of the word, covered in its own variant. Other replay methods the RLVR paper names in its related work (the LLM methods it lists in 2510.02245 §2) have no card here, so they are not covered.

Variants

Correct-rollout buffer with retirement (ExGRPO)

Stores only correct answers, grouped by how many of the question's K answers were correct. A question whose K answers are all correct is retired and not replayed again. Each step replays about half the batch: the stored answer is the K-th member of a group of K answers, and its tokens are weighted by a damped importance ratio 2510.02245 §4.1, §4.2, alg. 1. Card: ExGRPO: Learning to Reason from Experience.

• Storage. Correct answers only, kept under their question with the question's correct count 2510.02245 §4.1.
• Question choice. The buckets are weighted by a Gaussian over accuracy, centred at 0.5 with σ = 1 in the paper's text. A question is then drawn uniformly from its bucket 2510.02245 §4.1, alg. 1.
• Answer choice. The stored answer with the lowest mean token entropy under the current policy. No external judge is used at training time 2510.02245 §4.1.
• Correction. The replayed answer's term is multiplied by f(w) = w / (w + β) with β = 0.1, where w is the ratio of the current policy to the policy that wrote the answer. The stated aim is to amplify low-probability signal and damp high-probability signal 2510.02245 §4.2.
• Schedule. Replay switches on only after the batch pass rate passes 35%. Before that the run is plain on-policy RLVR 2510.02245 §4.2, §5.1.

Results are author-reported on Qwen2.5-Math-7B, trained on OpenR1-Math 45k. Average over six math benchmarks: 48.3 for ExGRPO and 45.3 for on-policy RLVR with the same Dr.GRPO-style setup 2510.02245 tab. 1. The paper also reports Llama3.1-8B on-policy RLVR collapsing where the replay run stays stable 2510.02245 §5.2.

Uniform replay buffer in deep Q-learning

Deep Q-learning stores each transition and trains on random minibatches from the stored set. The quantum-circuit paper keeps this recipe and changes only the Q-function, which it makes a variational circuit. It adopts replay and a separate target network from the earlier DQN work 1907.00397 §2.3. The settings are a memory of 80 transitions for frozen lake and 1000 for cognitive radio, minibatches of 5, and a target copy refreshed every 20 steps 1907.00397 §4. The replay rule is the same as in DQN; the paper contributes the circuit, not a new replay method. No row in its results isolates the effect of replay: the simulator baselines are "not reported" 1907.00397 §6.4. Card: Variational Quantum Circuits for Deep Reinforcement Learning.

Prioritised replay in Double DQN (scenario search)

This variant stores each transition with a priority and draws batches in proportion to those priorities. The Q-network is a Double DQN: the evaluation network picks the next action and the target network scores it 2412.02574 §4.1. The paper says the priority reflects how well the selection network evaluated an action, but it gives no formula and no values for the priority exponents or the small offset 2412.02574 §4.1. Each action is followed by one 6 s observation period in the simulator 2412.02574 §4, §5.2.2.

Results are author-reported for the whole system, not for replay alone. On four roads with Apollo 7.0 in LGSVL, the agent finds 11.6 to 18.9 collisions per road, against 5.4 to 12.6 for DeepCollision and 3.2 to 7.0 for random search 2412.02574 tab. 5. The card lists no run that removes prioritised replay or the Double DQN target, so the share of that gain due to replay is not measured. Card: Generating Critical Scenarios for Testing Automated Driving Systems.

Tiny episodic memory with class-balanced writers (continual learning)

Experience replay (ER) trains each step on the current minibatch plus a minibatch of the same size drawn from an episodic memory. The memory is written at every step, and it is used as training data rather than as a gradient constraint, which is the approach of GEM and A-GEM 1902.10486 §4, alg. 1. In the single-pass runs the memory holds about one example per class, and each step stacks 10 current and 10 memory examples 1902.10486 §5.2.

The paper compares four writers for the memory: reservoir sampling, a ring buffer per class, online k-means and a mean-of-features rule. The class-balanced writers are best when the memory is tiny, and reservoir is better for larger memories 1902.10486 §4, §5.4. It also proposes a hybrid that begins with reservoir sampling and switches to the ring buffer [arxiv:1902.10486 §5.4, fig. 2]. The authors read the new task's data as a regulariser that keeps repeated replay on a tiny memory from overfitting 1902.10486 §5.5.

• Storage. One stack per class and task for the ring buffer, sized by the total class count 1902.10486 §4. The ring buffer therefore needs that count in advance.
• Cost. One extra memory batch per step. On Split CIFAR the learning time is 116 s, against 87 s for fine-tuning and 230 s for A-GEM [arxiv:1902.10486 tab. 2]. The paper gives no hardware.
• Results. Author-reported, with one example per class. On Split CIFAR, ER with a ring buffer reaches 56.2% average accuracy, against 54.9% for A-GEM and 40.6% for fine-tuning 1902.10486 tab. 3–tab. 6. The same table shows ER beating EWC and MER on all four benchmarks.

Card: On Tiny Episodic Memories in Continual Learning.

Rehearsal and dual memories in continual learning (review)

The review puts replay among three families of fixes for catastrophic forgetting, next to regularisation and dynamic architectures 1802.07569 §3.2–3.4. Its basic form, rehearsal, stores raw old examples and mixes them into training on new data [arxiv:1802.07569 §3.1, §3.4]. The review states no rule for choosing which old examples to keep.

• Storage. The review names the cost directly: replay needs explicit storage of old data, which raises memory use 1802.07569 §3.1.
• Pseudo-rehearsal. Pseudo-samples are drawn from a generative model instead of stored data. The review cites Shin et al., who pair a deep generative model with a task solver 1802.07569 §3.4. It recommends this direction for future work, which is a recommendation, not a measured result 1802.07569 §3.5.
• Dual-memory models. A fast, sparse episodic store feeds a slower general store, and replay from the first consolidates into the second 1802.07569 §2.3. The review names GeppNet+STM, FearNet and a dual-memory self-organising model for video 1802.07569 §3.4. On CORe50 it reports, without numbers, that its self-organising dual-memory model significantly outperforms prior lifelong learning methods 1802.07569 §3.4.

The review reports no numbers of its own and takes its comparisons from cited work. Card: Continual Lifelong Learning with Neural Networks: A Review.

Prioritised and hindsight replay as sample sequencing (survey)

The curriculum survey puts replay under "sample sequencing": reordering the experience an agent already has from the target task, without changing the task 2003.04960 §4.2.1. It names prioritised replay and hindsight replay as examples 2003.04960 §4.2.1. Its table lists these methods with one target task, no transfer, an automatic sequencer and adaptive ordering, and most rows report asymptotic performance 2003.04960 tab. 2. The survey runs no experiments, so it gives no score or speed-up for either method. Card: Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey.

General-text replay in continued pretraining (medical LLM)

Here the replayed data is not the model's own past experience. It is an external general corpus, sampled at random and mixed into a domain corpus while a pretrained model keeps training. The aim is to limit forgetting of general knowledge 2311.16079 §2.3. For MEDITRON, the stream is 420M tokens drawn from RedPajama, roughly 1% of the continued-pretraining mixture (GAP-Replay), on top of the medical text 2311.16079 §2.3, tab. 1. The mixture is then used for continued pretraining of Llama-2 at 7B and 70B 2311.16079 §4.1.

The card reports no run that trains without the replay stream. Its data-mixture comparison at 7B pairs two mixtures that both include replay (PMC + Replay against GAP + Replay) 2311.16079 §7.2, tab. 8. So the benefit of this 1% is not measured on the page. Card: MEDITRON-70B: Scaling Medical Pretraining for Large Language Models.

When to use which

The constraints below are the ones the papers' results speak to. The first row is the only one with reported numbers for RLVR; the continual-learning rows report the author's own benchmarks.

Implementation sketch

Language-agnostic pseudocode of one training step. It follows the paper's algorithm, with notes where the choices matter. The released code differs in several places; see Contradictions.

The sketch covers only the RLVR buffer. The tiny-memory paper's training loop is in its card, the prioritised DQN learning step is in its card, the continued-pretraining loop is in its card, and the review gives no pseudocode for rehearsal. Those variants have no sketch here.

# One training step with a replay buffer  [arxiv:2510.02245 alg. 1]
def training_step(policy, dataset, buffer, retired, B, K, rho):
    # Replay slots: rho * B questions, fewer if the buffer is small  [§4.2]
    n_replay = min(floor(rho * B), len(buffer))
    # Draw questions by bucket weight, then one stored answer each;
    # keep the answer with the lowest mean token entropy  [§4.1]
    replayed = sample_buckets(buffer, n_replay)
    groups = {}
    for q in replayed:
        stored = lowest_entropy(policy, buffer[q])
        # K-1 fresh rollouts; the stored answer is the K-th member  [§4.2]
        groups[q] = [stored] + sample_rollouts(policy, q, K - 1)
    # Remaining slots are fresh questions with K rollouts each  [§4.2]
    for q in sample(dataset, B - n_replay):
        groups[q] = sample_rollouts(policy, q, K)
    # Retire questions that are all correct; store the correct answers
    # of the rest in their bucket  [§4.1]
    for q, answers in groups.items():
        n_ok = count_correct(answers)
        if n_ok == K:
            retired.add(q)
        elif n_ok > 0:
            buffer.put(q, correct_only(answers), n_ok)
    # Advantage is the group mean subtracted, with no std scaling  [§3.1]
    # Replayed tokens use the damped ratio f(w) = w / (w + 0.1)  [§4.2]
    update_policy(policy, groups)

The replayed answer's old log-probabilities must be recorded when it is generated. If they are missing, the ratio for that answer becomes about 1 and the replay is treated as on-policy data 2510.02245 §4.2.

Contradictions

• Bucket weight scale. The paper's text puts the Gaussian over accuracy (k/K) with σ = 1. Its released code uses the success count with σ = 1. In accuracy units those differ by a factor of about eight, so the two samplers are very different 2510.02245 alg. 1, §4.1. The paper does not say which scale produced the reported numbers.
• Writers in the tiny-memory paper. The paper reports ER results for the ring buffer, k-means and mean-of-features writers. The released ER code writes with reservoir sampling only, so those three writer results cannot be reproduced from it 1902.10486 tab. 3–tab. 6. The memory budget is also counted differently: per class in total in the CIFAR script, and per class per task in the MNIST script. The card's Paper vs code section lists the differences.
• DQN training start. The paper's algorithm samples a minibatch at every step. The released code trains only once the buffer holds more than five transitions 1907.00397 alg. 1, §4. The card's Paper vs code section notes this and the other differences.
• Priority rule and constants. The paper states the priority rule in words only, and gives no values for the priority exponents or the offset 2412.02574 §4.1. The released code uses the absolute TD error as the priority and a weighted squared TD error as the loss, with fixed values for the exponents and offset. Its buffer holds 3000 transitions and its batch is 64. The card's Paper vs code section and Hyperparameters table give the values. Use the paper's description as the spec and treat those values as the authors' choice.
• Survey and RLVR framing. The survey (2020) predates the RLVR paper (2025) and reports no results of its own, so no number on this page conflicts with another. Its "sample sequencing" category is a framing of replay, not a measured comparison with the buffer method 2003.04960 §4.2.1.
• Clipping in the mixed phase. The method text keeps the clipped objective for both fresh and replayed tokens. The released code removes the clip for the mixed phase, and the appendix theory assumes it is removed 2510.02245 eq. 4, §4.2, app. D. Read the paper card's Paper vs code section before implementing.

Papers

• ExGRPO: Learning to Reason from Experience: the correct-rollout buffer with question retirement, lowest-entropy answer selection and the damped importance ratio for replayed answers.
• On Tiny Episodic Memories in Continual Learning: experience replay with one example per class, and four memory writers compared in single-pass continual learning.
• Variational Quantum Circuits for Deep Reinforcement Learning: a deep Q-learning recipe with uniform minibatch replay, where the Q-function is a variational quantum circuit; replay is not ablated.
• Generating Critical Scenarios for Testing Automated Driving Systems: prioritised replay with a Double DQN agent that sets simulator scene parameters; replay is not ablated.
• Continual Lifelong Learning with Neural Networks: A Review: rehearsal, pseudo-rehearsal and dual-memory models as replay for continual learning; no new experiments.
• Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey: a classification that places prioritised and hindsight replay under sample sequencing; no experiments.
• MEDITRON-70B: Scaling Medical Pretraining for Large Language Models: continued pretraining of Llama-2 with about 1% general-text replay mixed into a medical corpus; the replay share is not ablated.