Paper · 2026-06-03 · cs.LG, cs.AI, cs.CL
Reinforcement Learning from Rich Feedback with Distributional DAgger
DistIL trains a student on its own rollouts by minimising forward cross-entropy to a feedback-conditioned teacher, crediting earlier tokens with later disagreement. Author-reported: beats SDPO and GRPO on science and coding, and SDPO, OPSD and GRPO on hard maths. Monotonic improvement and regret bounds are proved for an idealised natural-gradient variant.
On-policy distillationReinforcement learning with verifiable rewards (RLVR)Credit assignmentImitation learningreinforcement-learningself-distillationimitation-learningrich-feedbackreasoningrlvr
arXivPDFCodeMarkdown13 min read
AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Teaching a language model to reason usually rewards each attempt with a single bit: right or wrong. Many real settings give far more than that, such as error messages from running code, a teacher's critique, or a correct worked solution. A model that sees that extra material can act as a better "teacher" for its own, unaided self.
The method has the model write its own attempts, then nudges it toward what the feedback-informed version of itself would have chosen at every step. It does not only match the single final answer. It also tells each earlier choice how much trouble the later steps ran into, so a wrong turn early on gets some of the blame.
The authors argue that some earlier self-teaching methods can move the model in the wrong direction even when the teacher is better on average, and that their version does not have this flaw. Their experiments cover science questions, coding and hard maths.
TL;DR
- DistIL is an on-policy, DAgger-style objective: the student's own rollouts are scored against a feedback-conditioned teacher, with forward cross-entropy at each position §4.
- The gradient splits into a per-position part and a later-credit part. The later-credit part uses later teacher-student cross-entropy as a return §4.
- Author claims: monotonic improvement under an idealised natural-gradient step, a regret bound, and a link to pass@N §5.
- Author-reported gains over SDPO, OPSD and GRPO on three benchmark families §6. The released science and coding configs appear not to run as shipped (see Pitfalls).
Problem
Reinforcement learning with verifiable rewards gives one scalar per response, so credit is spread evenly over all tokens and gives no signal when every sample gets the same outcome §2. Self-distillation methods use a teacher that sees feedback and distil it into the student. Those methods use reverse KL (SDPO) or Jensen-Shannon (OPSD) §2§3. The paper's central assumption is that the feedback-informed teacher's expected reward is at least the student's §2.
The paper shows two failures of these methods. First, an f-divergence step toward a better teacher need not improve reward, even with an idealised natural-gradient step (Proposition 1 §3). For reverse KL, a three-action bandit makes the reward fall (Proposition 2 §3). Second, the common token-wise gradient drops the future-credit term. It can then assign zero credit to an early decision whose consequences appear later, and converge to a strictly worse policy (Proposition 3 §3).
Method
Objective. For each prompt x, sample a response y from the student. At each prefix s_t = (x, y_<t), sum a cross-entropy that takes the teacher's distribution as target, where pi_T is the same network conditioned on feedback f and held fixed (stop-gradient) §4. The paper says this objective needs only teacher samples, so it also works with a black-box teacher §4.
Gradient. The paper's gradient has two parts §4app. B:
- A local term: the gradient of the per-position cross-entropy, which weights each log-probability by the teacher's probability.
- A future-credit term: the score function of each sampled token, multiplied by the sum of later cross-entropies.
Terms with an index at or before the current position drop out of the second term. Each such loss does not depend on the sampled action, and the score function has zero mean under the policy app. B.
Practical variant alg. 1 §4. The experiments use a PPO-style update with a group of G rollouts per prompt. Two changes from the idealised objective: the future sum is divided by the number of remaining tokens, to avoid length bias §4app. C; and the teacher is an exponential moving average (EMA) of the student, or a fixed copy of the initial model §4.
# DistIL with rich feedback, practical variant [alg. 1] [§4]
# Inputs: student theta, teacher phi (starts equal to theta), prompt x,
# group size G
# Output: loss whose gradient is the DistIL update [§4]
def distil_loss(student, teacher, prompt, group_size):
# Sample G student rollouts and collect feedback for each [alg. 1]
rollouts = [student.sample(prompt) for _ in range(group_size)]
feedbacks = [get_feedback(prompt, y) for y in rollouts]
terms = []
for y, feedback in zip(rollouts, feedbacks):
# Teacher sees the feedback; treated as a fixed target [§4]
teacher_probs = stop_grad(teacher.probs(prompt, feedback, y))
student_logps = student.log_probs(prompt, y) # (T, V)
# Local term: forward cross-entropy per position [§4]
local_ce = -(teacher_probs * student_logps).sum(axis=-1) # (T,)
# Future credit: later cross-entropies [alg. 1]
future = later_sum(stop_grad(local_ce)) # (T,)
# Scale by tokens remaining, at least one [app. C]
future = future / max(remaining_length(y), 1) # (T,)
# Score term: credit each sampled token with its future [§4]
sampled_logp = student_logps[positions(y), y] # (T,)
terms.append(local_ce + future * sampled_logp)
# Average over all response tokens in the group [alg. 1]
return mean_over_tokens(terms)
# Teacher is an EMA of the student [alg. 1]
def update_teacher(teacher, student, beta):
for phi_p, theta_p in zip(teacher.params, student.params):
phi_p = beta * phi_p + (1 - beta) * theta_pTheory §5.
- Proposition 4: under a fixed state distribution, a natural-gradient step on the forward cross-entropy moves the student toward the teacher. Reward changes by eta times the teacher-student gap, plus O(eta^2). It assumes the map from parameters to policies is twice continuously differentiable, and tangent-space realizability. The paper says that assumption holds exactly for tabular softmax policies §5app. B.
- Theorem 1: regret for NPG-DistIL, the per-state mirror-descent variant in Algorithm 2 app. B. It needs finite ratio-based concentrability C and finite initial KL concentrability C0. The bound is written with a worst-case teacher variance and a signed recoverability parameter mu_T. The rate is n^(-1/2) for a nearly deterministic teacher and n^(-1/4) for a stochastic one §5.
- Proposition 5: for binary success rewards, minus the forward cross-entropy lower-bounds the teacher-weighted log-probability of success on each prompt §5. This is the paper's explanation for the pass@N gains.
Hyperparameters
Values for the DistIL runs. "not stated" means the paper does not give the value in the parts read, including the appendix. Code values are marked.
| Name | Science | Coding | Maths | Where |
|---|---|---|---|---|
| Model | Qwen3-8B; Olmo3-7B-Instruct | Qwen3-8B | Qwen3-8B; Qwen3-4B-Instruct-2507 | tab. 4 tab. 6 §6 |
| Thinking mode (training) | False | False | not stated | tab. 4 |
| Max prompt length | 2048 | 2048 | not stated | tab. 4 |
| Max response length | 8192 | 8192 | 16,384 completion | tab. 4 tab. 6 |
| Question batch size | 32 | 32 | effective batch 32 | tab. 4 tab. 6 |
| Mini-batch size | 32 | 1 | not stated | tab. 4 |
| Rollouts per question (G) | 8 | 8 | 1 | tab. 4 tab. 6 |
| Rollout temperature | 1.0 | 1.0 | 0.7 | tab. 4 tab. 6 |
| Top-K teacher tokens | 100 | 20 | 100 (code only) | tab. 4 run_distil_8b.sh |
| Teacher averaging rate | 0.01 | 0.01 | fixed teacher (code) | tab. 4 run_distil_8b.sh |
| Importance-ratio clip | 2 | 2 | not stated | tab. 4 |
| Learning rate | 5e-5 (constant) | 1e-6 (constant) | 5e-6 | tab. 4 tab. 6 |
| Warm-up steps | 10 | 0 | not stated | tab. 4 |
| Weight decay | 0.01 | 0.01 | not stated | tab. 4 |
| Gradient clip norm | 1.0 | 1.0 | 0.1 (code only) | tab. 4 run_distil_8b.sh |
| Optimiser | AdamW | AdamW | not stated | tab. 4 |
| LoRA rank / alpha / targets | not stated | not stated | 64 / 128 / q, k, v, o, gate, up, down | tab. 6 |
| Training steps | not stated | not stated | 100 | tab. 6 |
| Validation | 16 rollouts, T 0.6, top-p 0.95 | 4 rollouts, T 0.6, top-p 0.95 | see Table 8 and 9 | tab. 4 tab. 8 tab. 9 |
| Hardware | 4 x H200 | 4 x H200 | 4 x H200 | app. C |
Maths evaluation settings tab. 8 tab. 9:
| Parameter | Qwen3-4B-Instruct-2507 | Qwen3-8B |
|---|---|---|
| Output length cap (tokens) | 16,384 | 38,912 |
| Reasoning switch | On | On |
| Nucleus / top-k / min-p cutoffs | 0.95 / 20 / 0.0 | 1.0 / -1 / 0.0 |
| Presence penalty | 0.0 | 0.0 |
| Samples per prompt | 64 | 64 |
| Temperature | 0.7 | 1.0 |
Baseline settings: science and coding use SDPO's settings, and maths uses OPSD's app. C. The learning rate and teacher EMA rate for science differ from SDPO (1e-5, and 0.05 for SDPO's EMA rate) tab. 3 tab. 4.
Results
Author-reported throughout. Science rows are best Avg@16 within 1 h and 5 h of wall-clock training on the SciKnowEval L3 benchmark, averaged over four domains tab. 1. Base models have one value per domain (shown in both columns).
| Model | Method | Chem 1h | Chem 5h | Phys 1h | Phys 5h | Bio 1h | Bio 5h | Mat 1h | Mat 5h | Avg 1h | Avg 5h |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-8B | base | 41.2 | 41.2 | 59.2 | 59.2 | 30.8 | 30.8 | 58.9 | 58.9 | 47.5 | 47.5 |
| Qwen3-8B | GRPO (off-policy) | 65.9 | 74.5 | 63.8 | 72.7 | 35.1 | 59.9 | 74.3 | 77.1 | 59.8 | 71.1 |
| Qwen3-8B | GRPO (on-policy) | 63.3 | 63.4 | 63.6 | 63.6 | 49.8 | 49.8 | 73.9 | 74.1 | 62.7 | 62.7 |
| Qwen3-8B | SDPO | 73.0 | 80.2 | 68.0 | 72.4 | 52.9 | 63.6 | 72.2 | 75.9 | 66.5 | 73.0 |
| Qwen3-8B | DistIL | 75.8 | 80.8 | 72.7 | 80.8 | 53.3 | 66.6 | 74.9 | 76.2 | 69.2 | 76.1 |
| Olmo3-7B-Instruct | base | 22.8 | 22.8 | 37.7 | 37.7 | 16.2 | 16.2 | 36.7 | 36.7 | 28.4 | 28.4 |
| Olmo3-7B-Instruct | GRPO (off-policy) | 39.7 | 56.7 | 55.3 | 63.3 | 35.6 | 55.8 | 70.9 | 75.0 | 50.4 | 62.7 |
| Olmo3-7B-Instruct | GRPO (on-policy) | 51.4 | 57.5 | 62.7 | 62.7 | 49.8 | 49.8 | 73.3 | 73.5 | 59.3 | 60.9 |
| Olmo3-7B-Instruct | SDPO | 70.2 | 79.2 | 59.8 | 64.9 | 49.5 | 52.9 | 71.8 | 78.1 | 62.8 | 68.8 |
| Olmo3-7B-Instruct | DistIL | 72.1 | 81.0 | 67.4 | 74.5 | 47.8 | 55.3 | 73.5 | 76.9 | 65.2 | 71.9 |
Figure 1 fig. 1 plots validation Best@16 and Maj@16 over training for SDPO and DistIL on Qwen3-8B, in four science domains. The paper describes DistIL as steadier, and SDPO as showing a drop in biology Best@16 after about 100 steps §6.
Maths, Avg@16 and Pass for Qwen3 models, with the best checkpoint up to step 100 tab. 2:
| Model | Method | AIME24 Avg | AIME24 Pass | AIME25 Avg | AIME25 Pass | HMMT25 Avg | HMMT25 Pass | AMC23 Avg | AMC23 Pass | Minerva Avg | Minerva Pass |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-4B | base | 61.4 | 82.0 | 50.3 | 69.5 | 30.3 | 48.8 | 93.8 | 98.8 | 43.2 | 48.6 |
| Qwen3-4B | SFT | 55.8 | 80.9 | 43.1 | 67.4 | 29.7 | 46.0 | 91.7 | 97.0 | 41.9 | 50.4 |
| Qwen3-4B | GRPO | 61.4 | 82.0 | 50.3 | 69.5 | 30.3 | 48.8 | 93.8 | 98.8 | 43.2 | 48.6 |
| Qwen3-4B | SDPO | 60.9 | 85.4 | 49.6 | 74.1 | 32.8 | 50.9 | 93.9 | 99.8 | 43.4 | 52.0 |
| Qwen3-4B | OPSD | 63.2 | 85.4 | 51.5 | 76.1 | 33.0 | 54.2 | 94.5 | 99.6 | 43.4 | 51.8 |
| Qwen3-4B | DistIL | 65.3 | 87.5 | 55.3 | 77.6 | 33.0 | 56.6 | 94.8 | 100.0 | 44.2 | 52.9 |
| Qwen3-8B | base | 75.9 | 86.8 | 66.9 | 79.6 | 44.7 | 65.6 | 95.6 | 100.0 | 48.9 | 56.3 |
| Qwen3-8B | SFT | 74.2 | 85.3 | 65.3 | 80.7 | 42.2 | 64.4 | 96.3 | 100.0 | 48.2 | 56.2 |
| Qwen3-8B | GRPO | 75.9 | 86.8 | 66.9 | 79.6 | 44.7 | 65.6 | 95.6 | 100.0 | 48.9 | 56.3 |
| Qwen3-8B | SDPO | 76.9 | 89.5 | 68.3 | 84.0 | 45.6 | 71.2 | 96.5 | 100.0 | 49.2 | 57.7 |
| Qwen3-8B | OPSD | 76.5 | 91.3 | 69.7 | 83.3 | 45.5 | 68.2 | 96.2 | 100.0 | 49.2 | 57.8 |
| Qwen3-8B | DistIL | 76.4 | 90.7 | 71.1 | 85.0 | 46.4 | 71.4 | 96.6 | 100.0 | 49.5 | 58.4 |
The paper does not say which k the Pass column uses. The eval tables list 64 samples per prompt tab. 8 tab. 9.
Coding, LiveCodeBench v6 with public tests for feedback and private tests for evaluation, Qwen3-8B §6:
- Temperature 0.2, Avg@16: DistIL Accuracy 0.656 and Score 0.482; SDPO 0.643 and 0.467; GRPO 0.600 and 0.405 §6.
- Temperature 0.6, Avg@16 (appendix): DistIL 0.660 and 0.482; SDPO 0.643 and 0.467; GRPO 0.639 and 0.411 app. D. The appendix calls the metric "Mean@16".
- Best@k and Maj@k curves for k in {2, 4, 8, 16} are in fig. 2 and app. D.
Ablations §6:
- Against a cross-entropy-only baseline, which uses the local term alone, DistIL is reported to be higher throughout training on the Material domain fig. 3.
- Teacher top-K: K = 1 is weakest, K = 100 is best, and the full distribution is slightly worse than K = 100 on Physics fig. 4 §6.
Maths headline claims §6: on AIME25, DistIL leads the next-best method on Avg@16 by 3.8 points on Qwen3-4B and 1.4 points on Qwen3-8B tab. 2.
Paper vs code
Code: https://github.com/rishabh-1086/distIL at commit
e00096415c5fc53ddc1216dcd4f5dbad1b7a32ea (shallow clone, checked on
2026-11-12). The repository contains an SDPO tree (verl-based, used for
science and coding) and an OPSD tree (used for maths). The repository has no top-level LICENSE.
The paper links this repository. Algorithm 2 (NPG-DistIL) has no matching
implementation: a name search for "npg" finds nothing relevant.
- Teacher batch never built for the DistIL loss. The trainer builds the
teacher inputs only when the loss mode is
sdpo, while the DistIL config setsdistilray_trainer.py distil.yaml. The actor then asserts that the teacher keys exist dp_actor.py. On reading, the science and coding DistIL runs would fail that assertion. This was not run (no GPUs and no approval to run the job). - Two different future-term estimators. The science and coding code takes the score-function term, weighting log pi(y_t | s_t) by the future return core_algos.py. The maths code (OPSD) weights the cross-entropy itself by the future return, with the return detached, so the gradient flows through the local term [code: OPSD/opsd_trainer.py#L541] opsd_trainer.py. The paper's gradient §4 matches the first, not the second.
- Length normalisation. Algorithm 1 divides by (|y| - t - 1), which is undefined at the last token alg. 1. The appendix uses T - t - 1 app. C. The science and coding code clamps the denominator to at least one core_algos.py. The appendix formula takes the teacher from the student (pi_theta), while Algorithm 1 uses the EMA teacher (pi_phi) app. C alg. 1.
- Top-K. The paper's ablation says K = 100 "in all experiments" §6. The coding script uses 20 run_distil_coding.sh, as does the DistIL config default distil.yaml. The paper's coding table also lists 20 tab. 4.
- Top-K construction differs by domain. Science and coding evaluate the teacher at the student's top-K indices and add a tail bucket for the mass outside them dp_actor.py core_algos.py distil.yaml. Maths takes the teacher's own top-K and renormalises, with no tail opsd_trainer.py. The paper does not describe either.
- Extra weights not in the paper. The loss is multiplied by a detached importance ratio capped at 2 core_algos.py distil.yaml. The science and coding scripts also enable token-level rollout IS weights [code: SDPO/verl/trainer/ppo/core_algos.py#L1303-L1304] run_distil_science.sh. The maths loss has neither opsd_trainer.py.
- EMA rate. The paper's Algorithm 1 writes phi <- beta phi + (1 - beta) theta alg. 1. The code updates the teacher as 0.99 times the teacher plus 0.01 times the student dp_actor.py. So the table's "0.01" is the weight on the student, i.e. beta = 0.99 tab. 4.
- Teacher for maths. The paper does not say which teacher the maths runs use. The script sets a fixed teacher, with LoRA disabled for the teacher pass run_distil_8b.sh opsd_trainer.py.
- Learning rates and a carried-over row. The DistIL table lists "Jensen-Shannon" and "Reverse-KL" as the distillation divergence, which DistIL does not use; the row appears to be carried over from the SDPO table tab. 4 tab. 3. The DistIL science learning rate (5e-5) is five times SDPO's (1e-5) tab. 3 tab. 4. The comparison is not at matched learning rate.
- Clip value in maths. Gradient clip norm is 0.1 in the maths script but not in the maths table run_distil_8b.sh tab. 6.
- Feedback per domain. Science sets
include_environment_feedback=Falseand coding sets it to True run_distil_science.sh run_distil_coding.sh. Successful peer solutions from the same prompt group are used as demonstrations whenever one exists ray_trainer.py.
Pitfalls
- The released science and coding DistIL runs appear not to start. See
Paper vs code, item 1. Setting the loss mode to
sdpowould avoid the assertion, but then the loss is SDPO's, not DistIL's. Nothing here was run. - Which estimator to implement. The paper's math gradient §4 uses a score-function term. The maths code uses the cross-entropy term. The paper says the maths runs use the OPSD codebase app. C, so the second form is the one that produced Table 2. Pick one, and say which in your implementation.
- Normalisation is needed. The paper says the un-normalised future sum grows with length and produces large gradients app. C. The clamp in the code is the only guard at the last token.
- Teacher detail. The stop-gradient matters: if the teacher's probabilities carry gradient, the local term changes. The science code computes the teacher under no-grad dp_actor.py.
- Top-K and tail. K = 100 versus K = 20 changes results (the paper's own ablation). A tail bucket changes the normalisation. Neither is described in the paper, so pin both.
- The figure caption does not match the text. Figure 4's caption names the Material domain, but the file and the text say Physics fig. 4 §6.
- Proofs need strong assumptions. Proposition 4 needs a fixed state distribution and tangent-space realisability. The practical algorithm resamples states every step and uses PPO-style updates, so the guarantee does not carry over directly §5. Theorem 1 covers NPG-DistIL app. B, not Algorithm 1. Neither C nor C0 nor the variance or recoverability constants are estimated for any experiment in the parts read.
- Checkpoint selection. Table 1 reports the best checkpoint within a time budget, and the maths table reports the best checkpoint up to step 100 tab. 1 tab. 2. Those numbers are optimistic compared with a fixed final checkpoint.
- Train and eval thinking modes differ for maths. Training disables thinking tab. 4 for the science and coding runs, while maths evaluation enables it tab. 8 tab. 9. The maths training mode is not stated.
- Coding temperatures. The main-text coding table uses temperature 0.2 and the appendix uses 0.6. Both appear in the paper §6 app. D.
- Wall-clock budget. Science reports 1 h and 5 h, which depends on 4 x H200 GPUs and vLLM app. C. The parts read report no throughput or memory figures.
- Reprompt details. The science and coding DistIL configs remove thinking from the demonstration, truncate reprompts on the right at 10,240 tokens, and leave out samples with no demonstration from the loss [code: SDPO/verl/trainer/config/distil.yaml#L22-L25] [code: SDPO/verl/trainer/ppo/ray_trainer.py#L773-L777]. The paper does not describe these choices.
Lineage
- Builds on: DAgger, with a distributional variant that gives a teacher distribution rather than one action §4. Also on-policy self-distillation methods SDPO and OPSD, which the paper analyses and compares against §3. The length normalisation is attributed to MiniLLM app. C. None of these papers is carded here, so nothing from them is used.
- Compared against: GRPO (on- and off-policy), SFT, SDPO, OPSD §6.
- Related: the maximum-likelihood RL result the paper cites for pass@N §5. Not carded.
- Superseded by: not known from content on this wiki.
Source and license
Adapted from “Reinforcement Learning from Rich Feedback with Distributional DAgger” by Rishabh Agrawal, Jacob Fein-Ashley, Paria Rashidinejad (arXiv:2606.05152), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/rishabh-1086/distIL at commit e00096415c5f (Apache-2.0 (SDPO/ subtree only; no LICENSE for OPSD/ or scripts/)). Any quoted code is shown under that license.