Paper · 2026-02-15 · cs.LG, cs.AI
Experiential Reinforcement Learning
Experiential Reinforcement Learning adds an attempt, reflect, retry and distil cycle to RL for language models. After a failed first attempt the model writes a reflection, retries with it, trains on the retry, and distils the successful retry into the base policy so no reflection is needed at inference. Author-reported: Sokoban reward 0.06 to 0.87 on Qwen3-4B, up to +0.11 on HotpotQA.
Policy gradientReinforcement learning with verifiable rewards (RLVR)Context distillationSelf-refinement (draft, critique, rewrite loop)reinforcement-learningagentsself-reflectionlanguage-models
arXivPDFCodeMarkdown13 min read
AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Language models often learn from a score that arrives only at the end of a task: the attempt worked or it did not. That score says something went wrong, but not what to change, so the model has to work out the fix from many failures.
This paper adds a pause into the training loop. The model makes a first try, looks at how it went, and writes itself a short note on what to fix. It then makes a second try guided by that note. When the second try succeeds, training rewards it, and the model is also trained to produce that better answer directly from the original question, without the note. It is a bit like a student who marks a wrong test, writes down the mistake, retries, and then keeps the corrected method.
The note is a training aid only. At use time the model answers directly, so the extra step adds no cost when the model is deployed. The authors test this on grid puzzles and on a question-answering task that uses a search tool. The gains they report are their own measurements.
TL;DR
Experiential Reinforcement Learning (ERL) adds an attempt, reflection, retry and consolidation cycle to reinforcement learning fine-tuning of language models §2. When a first attempt fails, the model writes a reflection, makes a second attempt with it, and trains on the retry; the successful retry is then distilled into the base policy, so no reflection is needed at inference §2 app. A. Author-reported gains over RLVR with GRPO on FrozenLake, Sokoban and HotpotQA, with ablations that remove memory or structured reflection tab. 1. Seeds and variance are not reported, the evaluation sets are small, and the released code differs from the paper in several places (see Paper vs code). Re-measure before relying on the numbers.
Problem
Sparse, delayed outcome rewards tell the policy little about which behaviour to change. The policy must then infer the correction through undirected exploration, which the authors describe as often unstable and sample-inefficient. The problem is worse in multi-step agentic tasks, where small errors compound and credit assignment is hard §1. Inference-time self-reflection methods help, but they usually need reflection or memory at deployment §5. ERL assumes that each attempt yields a textual feedback message and a scalar reward from the environment §3.1 app. B.
Method
ERL keeps the standard RL objective of the base method and wraps it in an inner loop per task. The policy is a language model π_θ, trained with GRPO as the policy-gradient optimiser for both the baseline and ERL §3.2. ERL adds reflection and retry trajectories, plus an internalization step §2.
For each task x, the procedure is:
- First attempt. The policy samples y1 from x alone. The environment returns feedback f1 and reward r1 §2 alg. 2.
- First-attempt update. The first attempt is trained with the policy objective alg. 2.
- Gate. A reflection is triggered only when r1 is below the threshold τ, which is 1 in the paper app. A. Successful first attempts are not reflected on.
- Reflection. The model writes a reflection Δ conditioned on x, y1, f1, r1 and the memory m §2.
- Second attempt. The model writes y2 conditioned on x and Δ. The environment returns f2 and r2 §2.
- Memory write. If r2 > τ, Δ is stored in m. The memory is plain text placed in the system prompt, and it persists across episodes §2 app. A.
- Joint update. Δ and y2 are both trained with the policy objective, and the reflection receives the reward r2 §2.
- Internalization. The policy is trained to produce y2 from x alone, with the reflection removed, weighted by the indicator I(r2 > 0) §2.
The simplified Algorithm 1 in the main text omits the gate in step 3; the full Algorithm 2 in Appendix A includes it alg. 1 alg. 2. The card below follows Algorithm 2.
The policy objective is written as a policy-gradient term over the first attempt, the reflection and the second attempt, where A is an advantage computed from the associated rewards §2. The paper gives it unnumbered:
\mathcal{L}_{\text{policy}}(\theta) = - \mathbb{E}\left[ A \, \log \pi_\theta(y \mid x, \cdot) \right] §2
The internalization objective is a reward-indicator-weighted negative log-likelihood of the second attempt given the task alone:
\mathcal{L}_{\text{distill}}(\theta) = - \mathbb{E}\left[ \mathbb{I}(r^{(2)} > 0) \, \log \pi_\theta(y^{(2)} \mid x) \right] §2
Appendix A also describes an on-policy alternative that matches the policy without reflection to the policy with reflection under a reverse KL. The paper presents it as an option, and the released training scripts do not use it app. A train_erl_frozenlake.sh.
# One ERL update for a task x [alg. 2]
# Inputs: policy pi_theta, memory m, reward threshold tau (= 1)
def erl_update(policy, task, memory, tau):
# First attempt, conditioned on the task only [alg. 2] [§2]
first = policy.sample(task)
feedback_1, reward_1 = environment.run(task, first)
# Policy update on the first attempt [alg. 2]
policy_update([first], [reward_1])
# Gate: reflect and retry only after a failed attempt [app. A]
if reward_1 >= tau:
return memory
# Reflection conditioned on attempt, feedback, reward and memory [§2]
reflection = policy.sample(task, first, feedback_1, reward_1, memory)
# Second attempt, conditioned on the task and the reflection [§2]
second = policy.sample(task, reflection)
feedback_2, reward_2 = environment.run(task, second)
# Keep the reflection only if the retry clears the threshold [alg. 2]
if reward_2 > tau:
memory = reflection
# Reflection and retry both receive the retry's reward [§2]
policy_update([reflection, second], [reward_2, reward_2])
# Internalize: learn the retry from the task alone [§2]
if reward_2 > 0:
distill(policy, task, second) # weight I(reward_2 > 0) [§2]
return memoryEnvironments app. B:
- FrozenLake: an n×n grid with n sampled from 2 to 9 and a frozen-tile probability p sampled from [0.6, 0.85). Symbols A (agent), B (goal), C (hole), D (frozen). Actions are Up, Down, Left and Right. Reward is 1 on reaching the goal and 0 otherwise. An episode ends at the goal, at a hole, or after 8 actions. Transitions are deterministic.
- Sokoban: an n×n grid with n from 6 to 8, one box and one goal, with layouts accepted only if the shortest solution is at most 8 moves. Reward is 1 only when all boxes are on goals. Moves are deterministic and cannot pull boxes.
- HotpotQA: a multi-hop question-answering task. The agent may call local_search(query, top_k) for up to 5 turns, and must put its final answer in a boxed command. Retrieval is dense (e5-base-v2 embeddings, FAISS) over an indexed Wikipedia corpus. Reward is 1.0 for an exact match, token F1 when F1 is at least 0.3, and 0 otherwise §3.1.
Ablations §4.4: the no-memory variant keeps reflection but does not store reflections across episodes. The no-reflection variant replaces the reflection with the full first-attempt history and a generic instruction to improve.
Hyperparameters
Unless stated, values are from the paper's training appendix. Code values are from the official repository at the commit in the frontmatter.
| Name | Value | Where | Notes |
|---|---|---|---|
| Base models | Qwen3-4B-Instruct-2507; Olmo-3-7B-Instruct | §3.2 | Paper trains both with RLVR and ERL. See Paper vs code for which scripts exist |
| Policy-gradient optimiser | GRPO | §3.2 | Used for RLVR and ERL |
| Learning rate | 1e-6 | app. C train_erl_frozenlake.sh | Optimiser learning rate |
| Training batch size | 64 tasks | app. C train_erl_frozenlake.sh | |
| Actor mini-batch size | 64 | app. C train_erl_frozenlake.sh | Dynamic batch sizing on |
| Max prompt length | 8,196 tokens | app. C | Code: 6114 for FrozenLake and Sokoban train_erl_frozenlake.sh; 8192 for HotpotQA train_erl_hotpot.sh |
| Max response length | 8,196 tokens | app. C | Code: 10240 for FrozenLake and Sokoban train_erl_frozenlake.sh; 8192 for HotpotQA train_erl_hotpot.sh |
| Sampling temperature | 0.7 | app. C train_erl_frozenlake.sh | |
| Validation sampling | 4 samples per prompt, temperature 0.7, top-p 0.8, top-k 20 | app. C train_erl_frozenlake.sh | |
| RLVR rollouts per task | 10 | §3.2 app. C train_grpo_frozenlake.sh | |
| ERL rollouts per attempt | 4 | app. C train_erl_frozenlake.sh | §3.2 says "half as many per attempt"; see Paper vs code |
| Clip ratio (upper bound) | 0.28 | app. C train_erl_frozenlake.sh | |
| KL loss | low-variance KL, coefficient 0.001 | app. C train_erl_frozenlake.sh | Fixed KL control coefficient 0.001 |
| Entropy coefficient | 0 | app. C train_erl_frozenlake.sh | |
| Rejection sampling, stepwise advantage | disabled | app. C train_erl_frozenlake.sh | |
| Gate threshold τ | 1 | app. A train_erl_frozenlake_flow.py | Code default success threshold is 1.0 |
| FrozenLake budget | 8 actions per episode | app. B train_erl_frozenlake_flow.py | |
| FrozenLake grid size | n in 2 to 9, uniform | app. B prepare_frozenlake_data.py | |
| FrozenLake frozen probability | p in [0.6, 0.85), uniform | app. B prepare_frozenlake_data.py | |
| FrozenLake instances | 10,000 train; 100 disjoint evaluation | app. B prepare_frozenlake_data.py | |
| Sokoban grid size | n in 6 to 8 | app. B prepare_sokoban_data.py | |
| Sokoban solution limit | shortest solution at most 8 moves | app. B prepare_sokoban_data.py | |
| HotpotQA turns | up to 5 | app. B train_erl_hotpot_flow.py | |
| HotpotQA retrieval | top_k default 5, max 50; e5-base-v2 embeddings, FAISS | app. B (tool schema) | |
| Solver temperature (HotpotQA) | 0.7 | train_erl_hotpot_flow.py | Not stated in the paper for this task |
| Updater sampling | temperature 0.5, top-p 0.9 | not stated in paper; train_erl_frozenlake_flow.py | Code default |
| Attempts shown to updater | last 8 | not stated in paper; erl_frozenlake_flow.py | |
| Evaluation frequency | every 5 iterations | app. C train_erl_frozenlake.sh | |
| Training length | stopped manually at convergence | app. C; code sets 10 epochs for FrozenLake and Sokoban train_erl_frozenlake.sh and 100 for HotpotQA train_erl_hotpot.sh | |
| Hardware | one node, 8 H100 GPUs; vLLM; FlashAttention | app. C |
Results
Author-reported. Metric is final evaluation reward. Setup: FrozenLake and Sokoban are evaluated on 100 held-out instances each app. B; the HotpotQA evaluation set size is not stated; seeds and variance are not reported §4; training uses the same models and hardware as in Hyperparameters app. C.
| Model · task | RLVR | ERL | ERL w/o memory | ERL w/o reflection | Source |
|---|---|---|---|---|---|
| Qwen3-4B-Instruct-2507 · FrozenLake | 0.86 | 0.94 | 0.86 (−0.08) | 0.60 (−0.34) | tab. 1 |
| Qwen3-4B-Instruct-2507 · HotpotQA | 0.45 | 0.56 | 0.56 (−0.00) | 0.48 (−0.08) | tab. 1 |
| Qwen3-4B-Instruct-2507 · Sokoban | 0.06 | 0.87 | 0.87 (−0.00) | 0.59 (−0.28) | tab. 1 |
| Olmo3-7B-Instruct · FrozenLake | 0.39 | 0.66 | 0.64 (−0.02) | 0.54 (−0.12) | tab. 1 |
| Olmo3-7B-Instruct · HotpotQA | 0.47 | 0.50 | 0.47 (−0.03) | 0.46 (−0.04) | tab. 1 |
| Olmo3-7B-Instruct · Sokoban | 0.04 | 0.20 | 0.24 (+0.04) | 0.06 (−0.14) | tab. 1 |
Parenthesised values are differences from ERL as printed in the paper tab. 1.
Headline claims: gains of up to +0.81 on Sokoban, +0.27 on FrozenLake and +0.11 on HotpotQA, which the abstract and introduction call "+81%", "+27%" and "+11%" §1. These are absolute differences in reward, not relative gains tab. 1. The largest effect is Sokoban for Qwen3-4B, from 0.06 to 0.87, and Olmo3-7B from 0.04 to 0.20 §4.1. The paper reports ERL ahead of RLVR in all six settings §1 §4.1.
Learning dynamics: validation reward against wall-clock training time is higher for ERL and the gap persists, most visibly on FrozenLake and Sokoban fig. 4 §4.2. Training reward after the reflection step is higher than before it and than RLVR across environments and models fig. 6 §4.3. On FrozenLake with Qwen3-4B, removing memory slows convergence and removing reflection reduces both speed and final reward fig. 7 §4.4.
Caveat from the paper: on Olmo3-7B Sokoban, the no-memory variant slightly beats full ERL (0.24 vs 0.20). The authors attribute this to early reflections propagating through memory when the model's self-reflection is weak or the environment is stochastic §4.4.
Figure 5 plots the same final rewards as bars fig. 5. The paper's figures are described here, not reproduced, as the licensing rules require.
Paper vs code
Official repository: microsoft/experiential_rl at commit 20cd6460753973b9a5834769e2feed02736f44c6, MIT licence. Code citations give the file name only, to keep them short; each name is unique in the tree at this commit except core_algos.py, which is cited as ppo/core_algos.py. Its README cites arXiv 2602.13949 and lists the six authors, which is how it was matched to this paper. The owner is the Microsoft organisation rather than a personal account; two authors are listed with Microsoft affiliations.
- Memory content. The paper stores the reflection Δ §2 app. A. The code stores the full system prompt that the updater rewrites from the reflection, a consolidated rule-and-strategy prompt erl_frozenlake_flow.py erl_frozenlake_flow.py.
- Memory write condition. The paper writes memory per reflection when r2 > τ. The code writes it when any retried attempt in the episode succeeds, and the paper's per-example rule collapses into a per-episode rule erl_frozenlake_flow.py erl_frozenlake_flow.py. With the default batch size of 1 these coincide train_erl_frozenlake_flow.py.
- Memory scope. The paper's memory is cross-episode §2. In the code the memory is an attribute of each pooled workflow object, and the pool creates one object per slot agent_workflow_engine.py. Whether that gives one shared memory or many depends on how the trainer creates the pool, which was not traced.
- Updater reward. The paper gives each reflection its own reward r2 §2. The code gives the updater trajectory the mean second-attempt reward over the minibatch erl_frozenlake_flow.py. These match when the minibatch holds one task.
- Distillation weight. The paper uses the indicator I(r2 > 0) §2. The code trains distillation with RAFT, whose advantage is the reward clipped at zero core_algos.py. On binary rewards this is the indicator; on HotpotQA's partial F1 credit it weights by the F1 value.
- Gate. Algorithm 1 in the main text has no gate, Algorithm 2 has one, and the code follows Algorithm 2 erl_frozenlake_flow.py erl_frozenlake_flow.py.
- Rollouts per attempt. §3.2 says "half as many per task per attempt", which is 5 for RLVR's 10. Appendix C says 4. The code uses 4 train_erl_frozenlake.sh.
- Sequence lengths. Appendix C says 8,196 for both prompt and response. The FrozenLake and Sokoban scripts use 6114 and 10240 train_erl_frozenlake.sh. The HotpotQA scripts use 8192 train_erl_hotpot.sh.
- Models per task. The paper uses both models for all tasks. The repository has Qwen3-4B for FrozenLake and Sokoban ERL, and Qwen3-4B and Olmo-3-7B for HotpotQA train_erl_hotpot_olmo.sh. The FrozenLake RLVR baseline script points to a local Llama-3.1-8B-Instruct path, which the paper does not mention train_grpo_frozenlake.sh.
- Training length. The paper stops training manually on convergence app. C. The code sets 10 epochs for FrozenLake and Sokoban and 100 for HotpotQA train_erl_frozenlake.sh train_erl_hotpot.sh.
- Sokoban step budget. The paper says 8 actions per instance app. B. The dataset preparation writes max_steps 8 per instance prepare_sokoban_data.py, but the Sokoban training script's default agent budget is 64 train_erl_sokoban_flow.py. Which value applies during training was not traced.
- Environment feedback wording. The paper quotes environment messages in Appendix B. The ERL FrozenLake environment file contains no such strings. The updater instead sees a summary written by the flow: "Fell into a hole.", "Hit the max step limit", and counts of ineffective moves erl_frozenlake_flow.py.
- Updater sampling and window. Temperature 0.5, top-p 0.9, and a window of the last 8 attempts are code defaults that the paper does not state train_erl_frozenlake_flow.py erl_frozenlake_flow.py.
- Prompt base. The paper's reflection starts from the same prompt for every attempt. The code's updater starts from the last improved prompt, so each rewrite builds on the previous one erl_frozenlake_flow.py erl_frozenlake_flow.py.
- Matching settings. Learning rate, batch and mini-batch size, clip ratio 0.28, KL coefficient 0.001, entropy 0, temperature 0.7 and validation sampling match Appendix C train_erl_frozenlake.sh.
Pitfalls
- The memory is a rewritten full prompt in the code, not a list of reflections. A prompt that is rewritten repeatedly can drift, and the paper reports that early wrong reflections can persist through memory §4.4. Store and inspect prompts across rounds if you reimplement this.
- Gating. When every first attempt in a batch succeeds, the code skips the reflection step entirely erl_frozenlake_flow.py. Reflection runs only on failures, so easy tasks contribute no reflection data.
- Distillation weights on partial-credit tasks. Under RAFT the weight equals the reward. For HotpotQA's F1 scores this differs from the paper's indicator, so a 0.3 F1 retry gets a weight of 0.3, not 1 core_algos.py.
- Cost. Each failed first attempt adds one reflection generation and one extra rollout on the same task §2. The authors match the training compute per task by using 4 rollouts per attempt against RLVR's 10 §3.2 app. C. The "no extra inference cost" claim §1 applies to deployment, not training.
- Small evaluation sets. FrozenLake and Sokoban use 100 instances each app. B, so one instance is 0.01 of reward and a 0.02 difference is two instances. The paper reports no seeds or variance, so differences of this size may be noise.
- Units in the headline. "+81%" and "+11%" are absolute reward differences, not relative improvements tab. 1.
- Internal inconsistency in the source. The LaTeX source contains a commented-out copy of the main table with ERL at 0.96 for Qwen3-4B on FrozenLake; the rendered ablation table gives 0.94, which this card uses tab. 1.
- Memory across workers. Because memory sits on each pooled workflow object, parallel training may keep several independent memories rather than one. Check this when you port the method agent_workflow_engine.py.
- HotpotQA reward function. The code scores answers with the repository's RewardSearchFn, which was not inspected. The paper's reward is the exact-match and F1 rule described above §3.1.
Lineage
- Builds on: GRPO, the policy-gradient optimiser used throughout §3.2, and Search-R1 for the agentic HotpotQA setup §3.1. Neither has a card in this wiki yet.
- Training stack: the rLLM agent training stack, vLLM and FlashAttention app. C. Neither has a card in this wiki.
- Related inference-time reflection: STaR, Self-Refine and Reflexion, which the paper says need reflection or memory at deployment §5.
- Related self-distillation from feedback: concurrent work that distils a feedback-conditioned teacher into a student, which the paper relates to its internalization step §5.
- Compared against: RLVR trained with GRPO and 10 rollouts per task §3.2.
- Superseded by: not known.
Source and license
Adapted from “Experiential Reinforcement Learning” by Taiwei Shi, Sihao Chen, Bowen Jiang, Linxin Song, Longqi Yang, Jieyu Zhao (arXiv:2602.13949), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/microsoft/experiential_rl at commit 20cd64607539 (MIT). Any quoted code is shown under that license.