--- kind: concept slug: experience-replay title: Experience replay summary: >- Storing past examples and mixing them into later training. RLVR: a correct-answer buffer with retirement (author-reported +3.0 over on-policy, one 7B model). Continual learning: a tiny per-class memory beats four baselines (author-reported). DQN and LLM pretraining: replay not isolated. tags: [reinforcement-learning, rlvr, llm-reasoning, sampling, continual-learning, episodic-memory, q-learning, prioritized-replay, continued-pretraining] papers: ["1802.07569", "1902.10486", "1907.00397", "2003.04960", "2311.16079", "2412.02574", "2510.02245"] related: [reinforcement-learning-verifiable-rewards, policy-gradient, importance-weighting, continual-learning, catastrophic-forgetting, curriculum-learning, deep-q-learning, double-q-learning, language-model-pretraining, domain-adaptation] status: draft updated_at: 2026-10-31 --- ## Overview On-policy training uses each batch of rollouts for one update and then discards it. Experience replay keeps some of those rollouts and mixes them into later batches. The paper frames this as wasted compute and a missed chance to learn again from earlier successes [arxiv:2510.02245 §1]. Replay is well established in classic RL; the paper says efficient replay for RL with verifiable rewards (RLVR) on large reasoning models is largely unexplored [arxiv:2510.02245 §1, §2]. Every variant has to answer three questions: what to store, which stored item to replay, and how to correct for the fact that the replayed item was produced by an older policy. The shared correction is an importance ratio between the current policy and the policy that wrote the item (see [importance weighting](/concepts/importance-weighting)). Not every variant applies a correction: the continual-learning and uniform DQN variants replay stored examples as they are. The prioritised DQN variant also reweights its loss with per-sample weights from the buffer [arxiv:2412.02574 §4.1]. Those weights handle non-uniform sampling, not the age of the policy. Replay also has a second meaning in continual learning. There it protects earlier tasks from being overwritten, rather than reusing successful answers on the same task [arxiv:1802.07569 §1, §3.4]. The tiny-memory study below uses this meaning and reports its results against fine-tuning and three other continual-learning methods [arxiv:1902.10486 §4, §5.4]. Seven cards in this wiki use replay. The first variant below is the only one that studies replay for RLVR. The second is the uniform replay buffer of a deep Q-learning paper, where replay is one of the standard parts of the method. The third adds prioritised sampling to a Double DQN agent that tests self-driving cars; the card does not ablate its replay. The next two are continual-learning papers: a tiny-memory study with experiments, and a review. The sixth is a 2020 survey of curriculum learning for RL, which files replay under sample ordering. The seventh is a continued-pretraining paper that replays general text while training on a medical corpus. That is a different use of the word, covered in its own variant. Other replay methods the RLVR paper names in its related work (the LLM methods it lists in [arxiv:2510.02245 §2]) have no card here, so they are not covered. ## Variants ### Correct-rollout buffer with retirement (ExGRPO) Stores only correct answers, grouped by how many of the question's K answers were correct. A question whose K answers are all correct is retired and not replayed again. Each step replays about half the batch: the stored answer is the K-th member of a group of K answers, and its tokens are weighted by a damped importance ratio [arxiv:2510.02245 §4.1, §4.2, alg. 1]. Card: [ExGRPO: Learning to Reason from Experience](/papers/2510.02245). - **Storage.** Correct answers only, kept under their question with the question's correct count [arxiv:2510.02245 §4.1]. - **Question choice.** The buckets are weighted by a Gaussian over accuracy, centred at 0.5 with σ = 1 in the paper's text. A question is then drawn uniformly from its bucket [arxiv:2510.02245 §4.1, alg. 1]. - **Answer choice.** The stored answer with the lowest mean token entropy under the current policy. No external judge is used at training time [arxiv:2510.02245 §4.1]. - **Correction.** The replayed answer's term is multiplied by f(w) = w / (w + β) with β = 0.1, where w is the ratio of the current policy to the policy that wrote the answer. The stated aim is to amplify low-probability signal and damp high-probability signal [arxiv:2510.02245 §4.2]. - **Schedule.** Replay switches on only after the batch pass rate passes 35%. Before that the run is plain on-policy RLVR [arxiv:2510.02245 §4.2, §5.1]. Results are author-reported on Qwen2.5-Math-7B, trained on OpenR1-Math 45k. Average over six math benchmarks: 48.3 for ExGRPO and 45.3 for on-policy RLVR with the same Dr.GRPO-style setup [arxiv:2510.02245 tab. 1]. The paper also reports Llama3.1-8B on-policy RLVR collapsing where the replay run stays stable [arxiv:2510.02245 §5.2]. ### Uniform replay buffer in deep Q-learning Deep Q-learning stores each transition and trains on random minibatches from the stored set. The quantum-circuit paper keeps this recipe and changes only the Q-function, which it makes a variational circuit. It adopts replay and a separate target network from the earlier DQN work [arxiv:1907.00397 §2.3]. The settings are a memory of 80 transitions for frozen lake and 1000 for cognitive radio, minibatches of 5, and a target copy refreshed every 20 steps [arxiv:1907.00397 §4]. The replay rule is the same as in DQN; the paper contributes the circuit, not a new replay method. No row in its results isolates the effect of replay: the simulator baselines are "not reported" [arxiv:1907.00397 §6.4]. Card: [Variational Quantum Circuits for Deep Reinforcement Learning](/papers/1907.00397). ### Prioritised replay in Double DQN (scenario search) This variant stores each transition with a priority and draws batches in proportion to those priorities. The Q-network is a Double DQN: the evaluation network picks the next action and the target network scores it [arxiv:2412.02574 §4.1]. The paper says the priority reflects how well the selection network evaluated an action, but it gives no formula and no values for the priority exponents or the small offset [arxiv:2412.02574 §4.1]. Each action is followed by one 6 s observation period in the simulator [arxiv:2412.02574 §4, §5.2.2]. Results are author-reported for the whole system, not for replay alone. On four roads with Apollo 7.0 in LGSVL, the agent finds 11.6 to 18.9 collisions per road, against 5.4 to 12.6 for DeepCollision and 3.2 to 7.0 for random search [arxiv:2412.02574 tab. 5]. The card lists no run that removes prioritised replay or the Double DQN target, so the share of that gain due to replay is not measured. Card: [Generating Critical Scenarios for Testing Automated Driving Systems](/papers/2412.02574). ### Tiny episodic memory with class-balanced writers (continual learning) Experience replay (ER) trains each step on the current minibatch plus a minibatch of the same size drawn from an episodic memory. The memory is written at every step, and it is used as training data rather than as a gradient constraint, which is the approach of GEM and A-GEM [arxiv:1902.10486 §4, alg. 1]. In the single-pass runs the memory holds about one example per class, and each step stacks 10 current and 10 memory examples [arxiv:1902.10486 §5.2]. The paper compares four writers for the memory: reservoir sampling, a ring buffer per class, online k-means and a mean-of-features rule. The class-balanced writers are best when the memory is tiny, and reservoir is better for larger memories [arxiv:1902.10486 §4, §5.4]. It also proposes a hybrid that begins with reservoir sampling and switches to the ring buffer [arxiv:1902.10486 §5.4, fig. 2]. The authors read the new task's data as a regulariser that keeps repeated replay on a tiny memory from overfitting [arxiv:1902.10486 §5.5]. - **Storage.** One stack per class and task for the ring buffer, sized by the total class count [arxiv:1902.10486 §4]. The ring buffer therefore needs that count in advance. - **Cost.** One extra memory batch per step. On Split CIFAR the learning time is 116 s, against 87 s for fine-tuning and 230 s for A-GEM [arxiv:1902.10486 tab. 2]. The paper gives no hardware. - **Results.** Author-reported, with one example per class. On Split CIFAR, ER with a ring buffer reaches 56.2% average accuracy, against 54.9% for A-GEM and 40.6% for fine-tuning [arxiv:1902.10486 tab. 3]–[tab. 6]. The same table shows ER beating EWC and MER on all four benchmarks. Card: [On Tiny Episodic Memories in Continual Learning](/papers/1902.10486). ### Rehearsal and dual memories in continual learning (review) The review puts replay among three families of fixes for catastrophic forgetting, next to regularisation and dynamic architectures [arxiv:1802.07569 §3.2–3.4]. Its basic form, rehearsal, stores raw old examples and mixes them into training on new data [arxiv:1802.07569 §3.1, §3.4]. The review states no rule for choosing which old examples to keep. - **Storage.** The review names the cost directly: replay needs explicit storage of old data, which raises memory use [arxiv:1802.07569 §3.1]. - **Pseudo-rehearsal.** Pseudo-samples are drawn from a generative model instead of stored data. The review cites Shin et al., who pair a deep generative model with a task solver [arxiv:1802.07569 §3.4]. It recommends this direction for future work, which is a recommendation, not a measured result [arxiv:1802.07569 §3.5]. - **Dual-memory models.** A fast, sparse episodic store feeds a slower general store, and replay from the first consolidates into the second [arxiv:1802.07569 §2.3]. The review names GeppNet+STM, FearNet and a dual-memory self-organising model for video [arxiv:1802.07569 §3.4]. On CORe50 it reports, without numbers, that its self-organising dual-memory model significantly outperforms prior lifelong learning methods [arxiv:1802.07569 §3.4]. The review reports no numbers of its own and takes its comparisons from cited work. Card: [Continual Lifelong Learning with Neural Networks: A Review](/papers/1802.07569). ### Prioritised and hindsight replay as sample sequencing (survey) The curriculum survey puts replay under "sample sequencing": reordering the experience an agent already has from the target task, without changing the task [arxiv:2003.04960 §4.2.1]. It names prioritised replay and hindsight replay as examples [arxiv:2003.04960 §4.2.1]. Its table lists these methods with one target task, no transfer, an automatic sequencer and adaptive ordering, and most rows report asymptotic performance [arxiv:2003.04960 tab. 2]. The survey runs no experiments, so it gives no score or speed-up for either method. Card: [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](/papers/2003.04960). ### General-text replay in continued pretraining (medical LLM) Here the replayed data is not the model's own past experience. It is an external general corpus, sampled at random and mixed into a domain corpus while a pretrained model keeps training. The aim is to limit forgetting of general knowledge [arxiv:2311.16079 §2.3]. For MEDITRON, the stream is 420M tokens drawn from RedPajama, roughly 1% of the continued-pretraining mixture (GAP-Replay), on top of the medical text [arxiv:2311.16079 §2.3, tab. 1]. The mixture is then used for continued pretraining of Llama-2 at 7B and 70B [arxiv:2311.16079 §4.1]. The card reports no run that trains without the replay stream. Its data-mixture comparison at 7B pairs two mixtures that both include replay (PMC + Replay against GAP + Replay) [arxiv:2311.16079 §7.2, tab. 8]. So the benefit of this 1% is not measured on the page. Card: [MEDITRON-70B: Scaling Medical Pretraining for Large Language Models](/papers/2311.16079). ## When to use which The constraints below are the ones the papers' results speak to. The first row is the only one with reported numbers for RLVR; the continual-learning rows report the author's own benchmarks. | Situation | Use | Why | |---|---|---| | RLVR with a checkable answer and on-policy training plateaus | Correct-rollout buffer with retirement | Gain over on-policy in the paper's math average; gain largest on AIME24 and AIME25 [arxiv:2510.02245 tab. 1] | | Base model with near-zero accuracy on the task | Test first; do not copy the paper's gain | Llama base gains come with a different prompt and no delayed start [arxiv:2510.02245 app. E.1] | | Tight compute per step | Expect extra cost | Step time rose from 155.4 s to 184.9 s on 8 GPUs (+19%) [arxiv:2510.02245 app. E.5] | | Long runs with many stored answers | Budget memory for the buffer | About 1.77 GB for 202k stored trajectories in the 7B run [arxiv:2510.02245 app. E.5] | | Reward is not a checkable rule | Do not use from this page | The only non-verifiable test is a small preliminary study [arxiv:2510.02245 §5.3] | | Continual training on a prior method's traces | Compare with Continual LUFFY first | Out-of-distribution average was 60.2 against 61.8 for Continual LUFFY in the paper [arxiv:2510.02245 tab. 1] | | Need a clean ablation of the replay ratio | Keep ρ = 0.5 unless tested | ρ = 25% and 75% both scored lower in the paper's ablation [arxiv:2510.02245 app. E.4] | | Replay in a non-RLVR RL task, and you want a map of options | Prioritised or hindsight replay, as the survey classifies them | The survey gives no results for them; their original papers are not in this wiki [arxiv:2003.04960 §4.2.1, tab. 2] | | DQN-style agent with a parametric value function and correlated samples | Uniform random minibatches from a buffer, with a target network | The standard fix the paper adopts; its results do not isolate replay's effect [arxiv:1907.00397 §2.3, §4, §6.4] | | Value-based agent with discrete actions, where each decision runs a simulator for a fixed period | Prioritised replay with a Double DQN target | Author-reported system result of 11.6 to 18.9 collisions per road against 5.4 to 12.6 for DeepCollision; no run isolates replay [arxiv:2412.02574 tab. 5, §6] | | Continual learning, single pass, and a memory of about one example per class | Tiny episodic memory with a class-balanced writer (ring buffer) | Beats fine-tuning, EWC, A-GEM and MER on the paper's four benchmarks, with modest extra time [arxiv:1902.10486 tab. 2]–[tab. 6] | | Continual learning, tiny memory, but the number of classes is not known in advance | Reservoir writer | The ring buffer needs the total class count [arxiv:1902.10486 §4]; the paper finds reservoir best for larger memories [§5.4] | | Continual learning, and storing old raw examples is allowed and affordable | Rehearsal | The review's basic form; it names storage as the cost [arxiv:1802.07569 §3.1, §3.4] | | Continual learning, and old data cannot be stored (privacy, memory limit) | Pseudo-rehearsal from a generative model | The review's option for this case, recommended for future work and not measured [arxiv:1802.07569 §3.4, §3.5] | | Continual learning with a fixed memory budget and a need for stability on old tasks | Dual-memory model, if its implementation cost is acceptable | Qualitative gain reported on CORe50 without numbers; the review gives no cost figures for it [arxiv:1802.07569 §3.4] | | Continued pretraining of an LLM on a narrow domain, where general ability should not be lost | Mix a small share of general text into the domain corpus | MEDITRON used about 1% general replay (420M tokens) for this; the card reports no run without it, so the benefit is not measured [arxiv:2311.16079 §2.3, tab. 1, §7.2] | ## Implementation sketch Language-agnostic pseudocode of one training step. It follows the paper's algorithm, with notes where the choices matter. The released code differs in several places; see [Contradictions](#contradictions). The sketch covers only the RLVR buffer. The tiny-memory paper's training loop is in [its card](/papers/1902.10486), the prioritised DQN learning step is in [its card](/papers/2412.02574), the continued-pretraining loop is in [its card](/papers/2311.16079), and the review gives no pseudocode for rehearsal. Those variants have no sketch here. ```pseudocode # One training step with a replay buffer [arxiv:2510.02245 alg. 1] def training_step(policy, dataset, buffer, retired, B, K, rho): # Replay slots: rho * B questions, fewer if the buffer is small [§4.2] n_replay = min(floor(rho * B), len(buffer)) # Draw questions by bucket weight, then one stored answer each; # keep the answer with the lowest mean token entropy [§4.1] replayed = sample_buckets(buffer, n_replay) groups = {} for q in replayed: stored = lowest_entropy(policy, buffer[q]) # K-1 fresh rollouts; the stored answer is the K-th member [§4.2] groups[q] = [stored] + sample_rollouts(policy, q, K - 1) # Remaining slots are fresh questions with K rollouts each [§4.2] for q in sample(dataset, B - n_replay): groups[q] = sample_rollouts(policy, q, K) # Retire questions that are all correct; store the correct answers # of the rest in their bucket [§4.1] for q, answers in groups.items(): n_ok = count_correct(answers) if n_ok == K: retired.add(q) elif n_ok > 0: buffer.put(q, correct_only(answers), n_ok) # Advantage is the group mean subtracted, with no std scaling [§3.1] # Replayed tokens use the damped ratio f(w) = w / (w + 0.1) [§4.2] update_policy(policy, groups) ``` The replayed answer's old log-probabilities must be recorded when it is generated. If they are missing, the ratio for that answer becomes about 1 and the replay is treated as on-policy data [arxiv:2510.02245 §4.2]. ## Contradictions - **Bucket weight scale.** The paper's text puts the Gaussian over accuracy (k/K) with σ = 1. Its released code uses the success count with σ = 1. In accuracy units those differ by a factor of about eight, so the two samplers are very different [arxiv:2510.02245 alg. 1, §4.1]. The paper does not say which scale produced the reported numbers. - **Writers in the tiny-memory paper.** The paper reports ER results for the ring buffer, k-means and mean-of-features writers. The released ER code writes with reservoir sampling only, so those three writer results cannot be reproduced from it [arxiv:1902.10486 tab. 3]–[tab. 6]. The memory budget is also counted differently: per class in total in the CIFAR script, and per class per task in the MNIST script. The card's [Paper vs code](/papers/1902.10486) section lists the differences. - **DQN training start.** The paper's algorithm samples a minibatch at every step. The released code trains only once the buffer holds more than five transitions [arxiv:1907.00397 alg. 1, §4]. The card's [Paper vs code](/papers/1907.00397) section notes this and the other differences. - **Priority rule and constants.** The paper states the priority rule in words only, and gives no values for the priority exponents or the offset [arxiv:2412.02574 §4.1]. The released code uses the absolute TD error as the priority and a weighted squared TD error as the loss, with fixed values for the exponents and offset. Its buffer holds 3000 transitions and its batch is 64. The card's [Paper vs code](/papers/2412.02574) section and [Hyperparameters](/papers/2412.02574) table give the values. Use the paper's description as the spec and treat those values as the authors' choice. - **Survey and RLVR framing.** The survey (2020) predates the RLVR paper (2025) and reports no results of its own, so no number on this page conflicts with another. Its "sample sequencing" category is a framing of replay, not a measured comparison with the buffer method [arxiv:2003.04960 §4.2.1]. - **Clipping in the mixed phase.** The method text keeps the clipped objective for both fresh and replayed tokens. The released code removes the clip for the mixed phase, and the appendix theory assumes it is removed [arxiv:2510.02245 eq. 4, §4.2, app. D]. Read the [paper card's Paper vs code section](/papers/2510.02245) before implementing. ## Papers - [ExGRPO: Learning to Reason from Experience](/papers/2510.02245): the correct-rollout buffer with question retirement, lowest-entropy answer selection and the damped importance ratio for replayed answers. - [On Tiny Episodic Memories in Continual Learning](/papers/1902.10486): experience replay with one example per class, and four memory writers compared in single-pass continual learning. - [Variational Quantum Circuits for Deep Reinforcement Learning](/papers/1907.00397): a deep Q-learning recipe with uniform minibatch replay, where the Q-function is a variational quantum circuit; replay is not ablated. - [Generating Critical Scenarios for Testing Automated Driving Systems](/papers/2412.02574): prioritised replay with a Double DQN agent that sets simulator scene parameters; replay is not ablated. - [Continual Lifelong Learning with Neural Networks: A Review](/papers/1802.07569): rehearsal, pseudo-rehearsal and dual-memory models as replay for continual learning; no new experiments. - [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](/papers/2003.04960): a classification that places prioritised and hindsight replay under sample sequencing; no experiments. - [MEDITRON-70B: Scaling Medical Pretraining for Large Language Models](/papers/2311.16079): continued pretraining of Llama-2 with about 1% general-text replay mixed into a medical corpus; the replay share is not ablated.