wikillm

Paper · 2026-06-03 · cs.LG, cs.AI, cs.CL

Reinforcement Learning from Rich Feedback with Distributional DAgger

DistIL trains a student on its own rollouts by minimising forward cross-entropy to a feedback-conditioned teacher, crediting earlier tokens with later disagreement. Author-reported: beats SDPO and GRPO on science and coding, and SDPO, OPSD and GRPO on hard maths. Monotonic improvement and regret bounds are proved for an idealised natural-gradient variant.

On-policy distillationReinforcement learning with verifiable rewards (RLVR)Credit assignmentImitation learningreinforcement-learningself-distillationimitation-learningrich-feedbackreasoningrlvr

arXivPDFCodeMarkdown13 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Teaching a language model to reason usually rewards each attempt with a single bit: right or wrong. Many real settings give far more than that, such as error messages from running code, a teacher's critique, or a correct worked solution. A model that sees that extra material can act as a better "teacher" for its own, unaided self.

The method has the model write its own attempts, then nudges it toward what the feedback-informed version of itself would have chosen at every step. It does not only match the single final answer. It also tells each earlier choice how much trouble the later steps ran into, so a wrong turn early on gets some of the blame.

The authors argue that some earlier self-teaching methods can move the model in the wrong direction even when the teacher is better on average, and that their version does not have this flaw. Their experiments cover science questions, coding and hard maths.

TL;DR

  • DistIL is an on-policy, DAgger-style objective: the student's own rollouts are scored against a feedback-conditioned teacher, with forward cross-entropy at each position §4.
  • The gradient splits into a per-position part and a later-credit part. The later-credit part uses later teacher-student cross-entropy as a return §4.
  • Author claims: monotonic improvement under an idealised natural-gradient step, a regret bound, and a link to pass@N §5.
  • Author-reported gains over SDPO, OPSD and GRPO on three benchmark families §6. The released science and coding configs appear not to run as shipped (see Pitfalls).

Problem

Reinforcement learning with verifiable rewards gives one scalar per response, so credit is spread evenly over all tokens and gives no signal when every sample gets the same outcome §2. Self-distillation methods use a teacher that sees feedback and distil it into the student. Those methods use reverse KL (SDPO) or Jensen-Shannon (OPSD) §2§3. The paper's central assumption is that the feedback-informed teacher's expected reward is at least the student's §2.

The paper shows two failures of these methods. First, an f-divergence step toward a better teacher need not improve reward, even with an idealised natural-gradient step (Proposition 1 §3). For reverse KL, a three-action bandit makes the reward fall (Proposition 2 §3). Second, the common token-wise gradient drops the future-credit term. It can then assign zero credit to an early decision whose consequences appear later, and converge to a strictly worse policy (Proposition 3 §3).

Method

Objective. For each prompt x, sample a response y from the student. At each prefix s_t = (x, y_<t), sum a cross-entropy that takes the teacher's distribution as target, where pi_T is the same network conditioned on feedback f and held fixed (stop-gradient) §4. The paper says this objective needs only teacher samples, so it also works with a black-box teacher §4.

Gradient. The paper's gradient has two parts §4app. B:

  1. A local term: the gradient of the per-position cross-entropy, which weights each log-probability by the teacher's probability.
  2. A future-credit term: the score function of each sampled token, multiplied by the sum of later cross-entropies.

Terms with an index at or before the current position drop out of the second term. Each such loss does not depend on the sampled action, and the score function has zero mean under the policy app. B.

Practical variant alg. 1 §4. The experiments use a PPO-style update with a group of G rollouts per prompt. Two changes from the idealised objective: the future sum is divided by the number of remaining tokens, to avoid length bias §4app. C; and the teacher is an exponential moving average (EMA) of the student, or a fixed copy of the initial model §4.

Pseudocode
# DistIL with rich feedback, practical variant  [alg. 1] [§4]
# Inputs: student theta, teacher phi (starts equal to theta), prompt x,
#         group size G
# Output: loss whose gradient is the DistIL update  [§4]

def distil_loss(student, teacher, prompt, group_size):
    # Sample G student rollouts and collect feedback for each  [alg. 1]
    rollouts = [student.sample(prompt) for _ in range(group_size)]
    feedbacks = [get_feedback(prompt, y) for y in rollouts]
    terms = []
    for y, feedback in zip(rollouts, feedbacks):
        # Teacher sees the feedback; treated as a fixed target  [§4]
        teacher_probs = stop_grad(teacher.probs(prompt, feedback, y))
        student_logps = student.log_probs(prompt, y)  # (T, V)
        # Local term: forward cross-entropy per position  [§4]
        local_ce = -(teacher_probs * student_logps).sum(axis=-1)  # (T,)
        # Future credit: later cross-entropies  [alg. 1]
        future = later_sum(stop_grad(local_ce))  # (T,)
        # Scale by tokens remaining, at least one  [app. C]
        future = future / max(remaining_length(y), 1)  # (T,)
        # Score term: credit each sampled token with its future  [§4]
        sampled_logp = student_logps[positions(y), y]  # (T,)
        terms.append(local_ce + future * sampled_logp)
    # Average over all response tokens in the group  [alg. 1]
    return mean_over_tokens(terms)


# Teacher is an EMA of the student  [alg. 1]
def update_teacher(teacher, student, beta):
    for phi_p, theta_p in zip(teacher.params, student.params):
        phi_p = beta * phi_p + (1 - beta) * theta_p

Theory §5.

  • Proposition 4: under a fixed state distribution, a natural-gradient step on the forward cross-entropy moves the student toward the teacher. Reward changes by eta times the teacher-student gap, plus O(eta^2). It assumes the map from parameters to policies is twice continuously differentiable, and tangent-space realizability. The paper says that assumption holds exactly for tabular softmax policies §5app. B.
  • Theorem 1: regret for NPG-DistIL, the per-state mirror-descent variant in Algorithm 2 app. B. It needs finite ratio-based concentrability C and finite initial KL concentrability C0. The bound is written with a worst-case teacher variance and a signed recoverability parameter mu_T. The rate is n^(-1/2) for a nearly deterministic teacher and n^(-1/4) for a stochastic one §5.
  • Proposition 5: for binary success rewards, minus the forward cross-entropy lower-bounds the teacher-weighted log-probability of success on each prompt §5. This is the paper's explanation for the pass@N gains.

Hyperparameters

Values for the DistIL runs. "not stated" means the paper does not give the value in the parts read, including the appendix. Code values are marked.

NameScienceCodingMathsWhere
ModelQwen3-8B; Olmo3-7B-InstructQwen3-8BQwen3-8B; Qwen3-4B-Instruct-2507tab. 4 tab. 6 §6
Thinking mode (training)FalseFalsenot statedtab. 4
Max prompt length20482048not statedtab. 4
Max response length8192819216,384 completiontab. 4 tab. 6
Question batch size3232effective batch 32tab. 4 tab. 6
Mini-batch size321not statedtab. 4
Rollouts per question (G)881tab. 4 tab. 6
Rollout temperature1.01.00.7tab. 4 tab. 6
Top-K teacher tokens10020100 (code only)tab. 4 run_distil_8b.sh
Teacher averaging rate0.010.01fixed teacher (code)tab. 4 run_distil_8b.sh
Importance-ratio clip22not statedtab. 4
Learning rate5e-5 (constant)1e-6 (constant)5e-6tab. 4 tab. 6
Warm-up steps100not statedtab. 4
Weight decay0.010.01not statedtab. 4
Gradient clip norm1.01.00.1 (code only)tab. 4 run_distil_8b.sh
OptimiserAdamWAdamWnot statedtab. 4
LoRA rank / alpha / targetsnot statednot stated64 / 128 / q, k, v, o, gate, up, downtab. 6
Training stepsnot statednot stated100tab. 6
Validation16 rollouts, T 0.6, top-p 0.954 rollouts, T 0.6, top-p 0.95see Table 8 and 9tab. 4 tab. 8 tab. 9
Hardware4 x H2004 x H2004 x H200app. C

Maths evaluation settings tab. 8 tab. 9:

ParameterQwen3-4B-Instruct-2507Qwen3-8B
Output length cap (tokens)16,38438,912
Reasoning switchOnOn
Nucleus / top-k / min-p cutoffs0.95 / 20 / 0.01.0 / -1 / 0.0
Presence penalty0.00.0
Samples per prompt6464
Temperature0.71.0

Baseline settings: science and coding use SDPO's settings, and maths uses OPSD's app. C. The learning rate and teacher EMA rate for science differ from SDPO (1e-5, and 0.05 for SDPO's EMA rate) tab. 3 tab. 4.

Results

Author-reported throughout. Science rows are best Avg@16 within 1 h and 5 h of wall-clock training on the SciKnowEval L3 benchmark, averaged over four domains tab. 1. Base models have one value per domain (shown in both columns).

ModelMethodChem 1hChem 5hPhys 1hPhys 5hBio 1hBio 5hMat 1hMat 5hAvg 1hAvg 5h
Qwen3-8Bbase41.241.259.259.230.830.858.958.947.547.5
Qwen3-8BGRPO (off-policy)65.974.563.872.735.159.974.377.159.871.1
Qwen3-8BGRPO (on-policy)63.363.463.663.649.849.873.974.162.762.7
Qwen3-8BSDPO73.080.268.072.452.963.672.275.966.573.0
Qwen3-8BDistIL75.880.872.780.853.366.674.976.269.276.1
Olmo3-7B-Instructbase22.822.837.737.716.216.236.736.728.428.4
Olmo3-7B-InstructGRPO (off-policy)39.756.755.363.335.655.870.975.050.462.7
Olmo3-7B-InstructGRPO (on-policy)51.457.562.762.749.849.873.373.559.360.9
Olmo3-7B-InstructSDPO70.279.259.864.949.552.971.878.162.868.8
Olmo3-7B-InstructDistIL72.181.067.474.547.855.373.576.965.271.9

Figure 1 fig. 1 plots validation Best@16 and Maj@16 over training for SDPO and DistIL on Qwen3-8B, in four science domains. The paper describes DistIL as steadier, and SDPO as showing a drop in biology Best@16 after about 100 steps §6.

Maths, Avg@16 and Pass for Qwen3 models, with the best checkpoint up to step 100 tab. 2:

ModelMethodAIME24 AvgAIME24 PassAIME25 AvgAIME25 PassHMMT25 AvgHMMT25 PassAMC23 AvgAMC23 PassMinerva AvgMinerva Pass
Qwen3-4Bbase61.482.050.369.530.348.893.898.843.248.6
Qwen3-4BSFT55.880.943.167.429.746.091.797.041.950.4
Qwen3-4BGRPO61.482.050.369.530.348.893.898.843.248.6
Qwen3-4BSDPO60.985.449.674.132.850.993.999.843.452.0
Qwen3-4BOPSD63.285.451.576.133.054.294.599.643.451.8
Qwen3-4BDistIL65.387.555.377.633.056.694.8100.044.252.9
Qwen3-8Bbase75.986.866.979.644.765.695.6100.048.956.3
Qwen3-8BSFT74.285.365.380.742.264.496.3100.048.256.2
Qwen3-8BGRPO75.986.866.979.644.765.695.6100.048.956.3
Qwen3-8BSDPO76.989.568.384.045.671.296.5100.049.257.7
Qwen3-8BOPSD76.591.369.783.345.568.296.2100.049.257.8
Qwen3-8BDistIL76.490.771.185.046.471.496.6100.049.558.4

The paper does not say which k the Pass column uses. The eval tables list 64 samples per prompt tab. 8 tab. 9.

Coding, LiveCodeBench v6 with public tests for feedback and private tests for evaluation, Qwen3-8B §6:

  • Temperature 0.2, Avg@16: DistIL Accuracy 0.656 and Score 0.482; SDPO 0.643 and 0.467; GRPO 0.600 and 0.405 §6.
  • Temperature 0.6, Avg@16 (appendix): DistIL 0.660 and 0.482; SDPO 0.643 and 0.467; GRPO 0.639 and 0.411 app. D. The appendix calls the metric "Mean@16".
  • Best@k and Maj@k curves for k in {2, 4, 8, 16} are in fig. 2 and app. D.

Ablations §6:

  • Against a cross-entropy-only baseline, which uses the local term alone, DistIL is reported to be higher throughout training on the Material domain fig. 3.
  • Teacher top-K: K = 1 is weakest, K = 100 is best, and the full distribution is slightly worse than K = 100 on Physics fig. 4 §6.

Maths headline claims §6: on AIME25, DistIL leads the next-best method on Avg@16 by 3.8 points on Qwen3-4B and 1.4 points on Qwen3-8B tab. 2.

Paper vs code

Code: https://github.com/rishabh-1086/distIL at commit e00096415c5fc53ddc1216dcd4f5dbad1b7a32ea (shallow clone, checked on 2026-11-12). The repository contains an SDPO tree (verl-based, used for science and coding) and an OPSD tree (used for maths). The repository has no top-level LICENSE. The paper links this repository. Algorithm 2 (NPG-DistIL) has no matching implementation: a name search for "npg" finds nothing relevant.

  1. Teacher batch never built for the DistIL loss. The trainer builds the teacher inputs only when the loss mode is sdpo, while the DistIL config sets distil ray_trainer.py distil.yaml. The actor then asserts that the teacher keys exist dp_actor.py. On reading, the science and coding DistIL runs would fail that assertion. This was not run (no GPUs and no approval to run the job).
  2. Two different future-term estimators. The science and coding code takes the score-function term, weighting log pi(y_t | s_t) by the future return core_algos.py. The maths code (OPSD) weights the cross-entropy itself by the future return, with the return detached, so the gradient flows through the local term [code: OPSD/opsd_trainer.py#L541] opsd_trainer.py. The paper's gradient §4 matches the first, not the second.
  3. Length normalisation. Algorithm 1 divides by (|y| - t - 1), which is undefined at the last token alg. 1. The appendix uses T - t - 1 app. C. The science and coding code clamps the denominator to at least one core_algos.py. The appendix formula takes the teacher from the student (pi_theta), while Algorithm 1 uses the EMA teacher (pi_phi) app. C alg. 1.
  4. Top-K. The paper's ablation says K = 100 "in all experiments" §6. The coding script uses 20 run_distil_coding.sh, as does the DistIL config default distil.yaml. The paper's coding table also lists 20 tab. 4.
  5. Top-K construction differs by domain. Science and coding evaluate the teacher at the student's top-K indices and add a tail bucket for the mass outside them dp_actor.py core_algos.py distil.yaml. Maths takes the teacher's own top-K and renormalises, with no tail opsd_trainer.py. The paper does not describe either.
  6. Extra weights not in the paper. The loss is multiplied by a detached importance ratio capped at 2 core_algos.py distil.yaml. The science and coding scripts also enable token-level rollout IS weights [code: SDPO/verl/trainer/ppo/core_algos.py#L1303-L1304] run_distil_science.sh. The maths loss has neither opsd_trainer.py.
  7. EMA rate. The paper's Algorithm 1 writes phi <- beta phi + (1 - beta) theta alg. 1. The code updates the teacher as 0.99 times the teacher plus 0.01 times the student dp_actor.py. So the table's "0.01" is the weight on the student, i.e. beta = 0.99 tab. 4.
  8. Teacher for maths. The paper does not say which teacher the maths runs use. The script sets a fixed teacher, with LoRA disabled for the teacher pass run_distil_8b.sh opsd_trainer.py.
  9. Learning rates and a carried-over row. The DistIL table lists "Jensen-Shannon" and "Reverse-KL" as the distillation divergence, which DistIL does not use; the row appears to be carried over from the SDPO table tab. 4 tab. 3. The DistIL science learning rate (5e-5) is five times SDPO's (1e-5) tab. 3 tab. 4. The comparison is not at matched learning rate.
  10. Clip value in maths. Gradient clip norm is 0.1 in the maths script but not in the maths table run_distil_8b.sh tab. 6.
  11. Feedback per domain. Science sets include_environment_feedback=False and coding sets it to True run_distil_science.sh run_distil_coding.sh. Successful peer solutions from the same prompt group are used as demonstrations whenever one exists ray_trainer.py.

Pitfalls

  • The released science and coding DistIL runs appear not to start. See Paper vs code, item 1. Setting the loss mode to sdpo would avoid the assertion, but then the loss is SDPO's, not DistIL's. Nothing here was run.
  • Which estimator to implement. The paper's math gradient §4 uses a score-function term. The maths code uses the cross-entropy term. The paper says the maths runs use the OPSD codebase app. C, so the second form is the one that produced Table 2. Pick one, and say which in your implementation.
  • Normalisation is needed. The paper says the un-normalised future sum grows with length and produces large gradients app. C. The clamp in the code is the only guard at the last token.
  • Teacher detail. The stop-gradient matters: if the teacher's probabilities carry gradient, the local term changes. The science code computes the teacher under no-grad dp_actor.py.
  • Top-K and tail. K = 100 versus K = 20 changes results (the paper's own ablation). A tail bucket changes the normalisation. Neither is described in the paper, so pin both.
  • The figure caption does not match the text. Figure 4's caption names the Material domain, but the file and the text say Physics fig. 4 §6.
  • Proofs need strong assumptions. Proposition 4 needs a fixed state distribution and tangent-space realisability. The practical algorithm resamples states every step and uses PPO-style updates, so the guarantee does not carry over directly §5. Theorem 1 covers NPG-DistIL app. B, not Algorithm 1. Neither C nor C0 nor the variance or recoverability constants are estimated for any experiment in the parts read.
  • Checkpoint selection. Table 1 reports the best checkpoint within a time budget, and the maths table reports the best checkpoint up to step 100 tab. 1 tab. 2. Those numbers are optimistic compared with a fixed final checkpoint.
  • Train and eval thinking modes differ for maths. Training disables thinking tab. 4 for the science and coding runs, while maths evaluation enables it tab. 8 tab. 9. The maths training mode is not stated.
  • Coding temperatures. The main-text coding table uses temperature 0.2 and the appendix uses 0.6. Both appear in the paper §6 app. D.
  • Wall-clock budget. Science reports 1 h and 5 h, which depends on 4 x H200 GPUs and vLLM app. C. The parts read report no throughput or memory figures.
  • Reprompt details. The science and coding DistIL configs remove thinking from the demonstration, truncate reprompts on the right at 10,240 tokens, and leave out samples with no demonstration from the loss [code: SDPO/verl/trainer/config/distil.yaml#L22-L25] [code: SDPO/verl/trainer/ppo/ray_trainer.py#L773-L777]. The paper does not describe these choices.

Lineage

  • Builds on: DAgger, with a distributional variant that gives a teacher distribution rather than one action §4. Also on-policy self-distillation methods SDPO and OPSD, which the paper analyses and compares against §3. The length normalisation is attributed to MiniLLM app. C. None of these papers is carded here, so nothing from them is used.
  • Compared against: GRPO (on- and off-policy), SFT, SDPO, OPSD §6.
  • Related: the maximum-likelihood RL result the paper cites for pass@N §5. Not carded.
  • Superseded by: not known from content on this wiki.

Source and license

Adapted from “Reinforcement Learning from Rich Feedback with Distributional DAgger” by Rishabh Agrawal, Jacob Fein-Ashley, Paria Rashidinejad (arXiv:2606.05152), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/rishabh-1086/distIL at commit e00096415c5f (Apache-2.0 (SDPO/ subtree only; no LICENSE for OPSD/ or scripts/)). Any quoted code is shown under that license.