wikillm

Paper · 2026-11-02 · cs.LG, cs.CL

Train the Sampler You Run: Few-Step Masked Diffusion Language Models via Mixture-Likelihood Fine-Tuning

Mix-FT fine-tunes a masked diffusion language model on an importance-weighted bound of the likelihood of its own few-step, random-order sampler, with no teacher and no architecture change. Author-reported: on a 110M block diffusion model it beats continued CE, Alpha diffusion and SDTT at 4, 8 and 16 steps per block; on LLaDA-8B (LoRA) 16 steps beat the original 32 under three judges.

Discrete diffusionImportance weightingDiffusion modelsLow-rank adaptation (LoRA and variants)masked-diffusiondiffusion-language-modelsfew-step-samplingimportance-weightingfine-tuning

arXivPDFMarkdown14 min read

AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

A masked diffusion language model writes text by filling in blanks. To go faster it fills several blanks at once, and each batch of blanks is guessed without seeing the others. The text it produces is then the result of many possible orders in which the blanks could have been filled. Ordinary training teaches the model to guess each blank from what is already known, and does not teach it how well the whole sequence of guesses explains the final text.

This paper's fix changes only the training signal. The model is fine-tuned on a score that asks how likely the text is under the exact fast sampler it will be run with, averaged over several random fill orders. The fill orders that explain a given text well count for more. No teacher model, no new parts and no change to the sampler are needed, and the extra training is short. The author reports that the resulting text is judged better than the original model's text when both are given the same number of steps, and that the fine-tuned model can use fewer steps and still come out ahead.

TL;DR

  • Mix-FT replaces the per-step cross-entropy loss with an importance-weighted bound on the log-likelihood of the random-order few-step sampler §3.
  • It needs a CE-trained starting model. It is not a from-scratch recipe §8.
  • Author-reported gains are judged by language-model NLL and two LLM raters. They are not human quality scores §4, §5.
  • No official code was found.

Problem

A masked diffusion language model samples a block of B positions in K parallel steps, revealing B/K positions per step. Within a step the positions are drawn independently, and the plan that sets which positions are revealed when is drawn at random §2. This factorisation error is the cost of few-step sampling. Existing fixes act at sampling time (temperature, schedules), add latents or heads, or distil from a many-step teacher §1.

The paper's claim is that the training loss itself is mismatched. When positions are revealed in random order, the sampler in use draws from a blend over reveal plans d, with the sampler law q_θ(x) = Σ_d μ(d) q_{θ,d}(x) §1, §2, eq. 1. Masked cross-entropy takes the mean of per-plan log-likelihoods, and its optimum is the true per-step conditionals. Those conditionals do not minimise the negative log of the mixture §1, §3, Proposition 1. The gap between the two objectives is a plan-information term §3, Proposition 2.

Method

Setting. The setup is block diffusion: a sequence is split into chunks of B positions that are produced from left to right, and each chunk is conditioned on the clean chunks before it. All statements concern one block given its prefix §2. A plan splits the B positions into K ordered groups of equal size g = B/K, and a plan is picked uniformly at random §2. The denoiser r_θ(x_i | x_S) gives a distribution over the token at a masked position given the revealed set S. A plan's law is a product over its steps:

q_{\theta,d}(x) = \prod_{k=1}^{K} \prod_{i \in G_k} r_\theta\big(x_i \mid x_{G_{<k}}\big), \qquad q_\theta(x) = \sum_d \mu(d)\, q_{\theta,d}(x) §2, eq. 1

Objective (Mix-FT). Mix-FT minimises an IWAE-style bound that averages the likelihood over M plans sampled from μ §3, eq. 2:

\mathcal{L}_M(\theta) = \mathbb{E}_{x,\, d_{1:M}}\Big[ -\log \frac{1}{M} \sum_{m=1}^{M} q_{\theta,d_m}(x) \Big] §3, eq. 2

The bound upper-bounds −E_p log q_θ(x) and tightens as M grows §3. Its gradient amounts to a cross-entropy over the states the sampler visits, where every plan's term is scaled by its self-normalised posterior weight w̃_m ∝ q_{θ,d_m}(x) §3:

\nabla_\theta \mathcal{L}_M = -\mathbb{E}\Big[\sum_m \tilde w_m \sum_k \sum_{i \in G^{(m)}_k} \nabla_\theta \log r_\theta\big(x_i \mid x_{G^{(m)}_{<k}}\big)\Big] §3, unnumbered display

At M = 1 the weight is 1 and the loss reduces to the per-path loss, whose optimum is the true conditionals. So M > 1 is needed §3. On exact toy problems the paper shows that the one-plan loss leaves a free table where it started, while M = 16 removes about 94% of the error of the true conditionals at K = 3 [§3, app. F]. Evaluating the bound on one sequence takes 1 + M(K − 1) forward passes: a single pass with the whole block masked is reused by every plan, and each plan adds K − 1 more. Passes can also be shared across blocks, since the blocks are independent once the clean prefix is fixed §3.

Training plans. The training plans may use a different step count from sampling. The 110M main model uses K = 2 plans, with M = 8 plans for each sequence, and is evaluated at 2, 4, 8 and 16 steps §4, app. A. LLaDA-8B runs use four-step plans and M = 8, since a two-step plan at block 32 would expose 16 tokens in a single step app. C.

Algorithm (pseudocode). Plan likelihoods are computed with the model. The weights are treated as constants, so no gradient flows through them. The paper's implementation does this with a no-gradient pass that gives the weights and then a gradient pass, which costs more FLOPs but less memory app. J.

Pseudocode
# log q_{θ,d}(x) for one plan d = (G_1, ..., G_K)  [§2] [eq. 1]
# x: the block's B tokens; the clean prefix is given
# The first call (nothing revealed) is the same for all plans: cache it  [§3]
def plan_log_likelihood(model, x, plan):
    revealed = set()
    total = 0.0
    for group in plan:
        # Predict each position of this group from the revealed ones  [§2]
        probs = model(x, revealed)                  # (B, V) per position
        for i in group:
            total = total + log(probs[i, x[i]])
        revealed = revealed | set(group)
    return total
Pseudocode
# Mix-FT loss for one block, with M plans  [§3] [eq. 2] [app. J]
# plans: M ordered partitions of the B positions, drawn uniformly  [§2]
def mixft_loss(model, x, plans):
    # Log-likelihood of x under each sampled plan  [§3]
    plan_logs = [plan_log_likelihood(model, x, d) for d in plans]
    # Self-normalised plan weights (plan posterior), held constant  [§3]
    weights = softmax(stop_gradient(plan_logs))     # (M,)
    # Weighted negative log-likelihood; its gradient is the formula above  [§3]
    return -sum(w * log_q for w, log_q in zip(weights, plan_logs))

Sampling. The fine-tuned model is sampled with the same sampler as the original, block by block, with K steps per block. Random order is the sampler Mix-FT trains on. LLaDA's default low-confidence remasking is evaluated as a transfer, and Mix-FT was never trained on it §5, §4. The paper also tests temperature schedules that vary by step; they are tuned on separate development samples and help every model app. A, app. J.

Exact likelihood. For small blocks, summing over plans can be done exactly with a dynamic program whose states are reveal sets. A block of 8 with K = 4 needs 127 forward passes per sequence §3, app. G. A block of 16 with K = 4 is not affordable, so the paper reports IWAE-64 and IWAE-512 there app. G.

Hyperparameters

Values are as the paper states them. Where the table gives a recipe for a variant, it says which.

NameValueWhereNotes
Start model (110M)BD3-LM OpenWebText checkpoint, block 16, 12 layers, width 768, 12 headsapp. Ipublic checkpoint kuleshov-group/bd3lm-owt-block_size16; NELBO perplexity 23.0 on the authors' held-out split
Training data (110M)OpenWebText; sequences of 1,024 GPT-2 tokensapp. Iheld-out split: the final 3,000 documents of the final shard
Batch (110M)8 × 1,024 tokensapp. I
Mix-FT main (110M, 1×)K = 2 plans, M = 8, 1,152 optimiser steps§4 app. Ibudget 276,480 forward-equivalents per arm
Mix-FT 8× (110M)K = 2, M = 256, 357 updates, 2,856 sequencesapp. Jmicro-batch 1, batch 8 sequences; 2,210,544 FE (7.9953× the 1× budget)
Mix-FT 16× (110M)K = 2, M = 512, 358 updates of 8 sequencesapp. J4,416,288 FE; two T4s
Optimiser (110M)AdamW (0.9, 0.999), no weight decayapp. I
Peak learning rate (110M)5e-5app. I
Warm-up (110M)min(100, floor(steps / 8)) stepsapp. I8× recipe: 44 steps (min(100, floor(357 / 8))) app. J
Schedule (110M)cosine to 10% of peakapp. I
Clip / precision (110M)gradient clip 1; fp16 autocastapp. I
Chunked backward (110M, M ≥ 128)chunks of 32 plans; block-chunked variant also describedapp. I app. Jthe chunked backward executes about 1.4× the counted FE at M = 256
Continued CE (110M)5,760 steps, trained on K = 4 sampler states, same batchesapp. I
Alpha diffusion (110M)κ = 0.5, block weights centred by a constant measured on the startapp. Ibest of κ ∈ {0.1, 0.25, 0.5, 2, 8}
SDTT-4 / SDTT-8 (110M)SDTT-4: 2 rounds of 1,920 steps (16 → 8 → 4); SDTT-8: 1 round of 3,840 (16 → 8)app. Ireverse KL; lr 6e-5 constant after warm-up; EMA 0.999; teacher passes counted
Generation (110M)512 samples per temperature, each 1,024 tokens long; β ∈ {0.95, 1, 1.05, 1.1, 1.15, 1.25}; K ∈ {2, 4, 8, 16}app. Iexact categorical sampling in fp32; one T4
Judges (110M)GPT-2-large (fp32) and Pythia-1.4B (fp16)app. Icontent tokens only; matched per-sample entropy at real text minus 0.3, 0.2, 0.1, 0 nats
Likelihood bound (110M)IWAE-32 at K = 4 and 2; 256 held-out sequencesapp. Iβ ∈ {1, 1.1, 1.25}, best reported
LLaDA-8B settingLLaDA-8B-Base; LoRA adapters of rank 32, alpha 32, on every attention and MLP linear layer (83,886,080 trainable parameters)app. Cblock 32; prompt 64 tokens; 192 generated
Mix-FT (LLaDA-8B)K = 4, M = 8; 8 examples per update with 25 rows each; 600 updates; 480,000 FEapp. CM = 32 run (seed 0) is 309 updates
Optimiser (LLaDA-8B)AdamW (0.9, 0.999), no weight decay, lr 2e-4, 5% warm-up, cosine to 0, clip 1app. Cfp16 loss scaling starting at 4096 (2^12); 8 rows per micro-batch
Continued CE, Alpha (LLaDA-8B)267 rows per update, taken from K = 4 random-order states; 600 updatesapp. CAlpha κ = 0.5
text8 model6 layers, d = 256, 4 heads, 4.82M parameters; sequences of 256 charactersapp. Eblock-causal attention over [clean
text8 Mix-FTone unit = 864 steps of Mix-FT (M = 8, 25 passes per sequence)app. EAdamW (0.9, 0.95), weight decay 0.01, clip 1, batch 32; lr for Mix-FT not stated in the table
text8 Basemasked-CE pre-training (20,000 steps, lr 1e-3, warm-up 200, cosine to 1e-4), then four units of CE on the states the K = 4 sampler visitsapp. E

Results

Author-reported. Every number below is the paper's own measurement under the stated protocol.

110M block diffusion model (BD3-LM, block 16) §4, Table 1; app. A. Judge NLL (GPT-2-large, content tokens, equal per-sample entropy), in nats per token, lower is better. Real held-out text scores 2.78.

Modelopt. stepsIWAE-32 sampler bound K=4K=2NELBO PPLjudge K=2K=4K=8K=16
Start (BD3-LM)--3.2233.77523.007.316.104.603.71
Continued CE5,7603.2093.76422.41--6.204.823.89
SDTT-42 × 1,9203.2453.79624.416.685.834.754.02
SDTT-83,8403.2283.77523.42----4.583.80
Alpha (κ = 0.5)5,7603.2023.75823.167.165.794.543.78
Mix-FT (K = 2, M = 8)1,1523.1813.72324.517.075.544.103.37
  • Time per 512 samples on one T4: 55 s (K = 2), 93 s (K = 4), 162 s (K = 8), 305 s (K = 16) §4, Table 1.
  • At K = 2, SDTT-4 has the best judge NLL in every seed. Mix-FT roughly ties Alpha diffusion there §4, app. A.
  • Mix-FT is more than 0.3 nats per token ahead of the start at K = 16. With the 8× budget, its K = 8 result recovers about 95% of the loss the start suffers when going from 16 to 8 steps under GPT-2-large (bootstrap range 92 to 99%), and about 97% under Pythia §4, app. J.
  • Blinded LLM raters (ChatGPT and Claude Sonnet 5.5, first token fixed) pick the 8× model's K = 8 text over the start's K = 16 text in 45% of pairs pooled. This closes 88% of the gap that halving the steps opens in win rate (95% interval 79 to 97%). At equal steps the raters favour the fine-tuned text in 76% and 79% of pairs §4, app. J.
  • Under LLaDA's low-confidence planner, the 110M Mix-FT model also outscores all baselines and the start at each of 4, 8 and 16 steps §4.

LLaDA-8B, LoRA, OpenWebText prefix continuation §5, Table 2; app. C. Judge NLL (GPT-2-large, content tokens, equal entropy), 3-seed means.

Modelrandom K=8random 16random 32low-conf. K=8low-conf. 16low-conf. 32
Start5.013.983.545.725.024.48
Continued CE4.883.89--5.694.83--
Alpha (κ = 0.5)4.673.74--5.554.76--
Mix-FT (K = 4, M = 8)4.293.48--5.174.38--
  • At K = 16, two tokens per step, Mix-FT's samples beat the start's K = 32 samples in each seed, under all three judges and both planners. Start at K = 32 minus Mix-FT at K = 16, seed mean with 95% interval under GPT-2-large: +0.07 [−0.02, +0.15] (random) and +0.10 [+0.01, +0.21] (low-confidence); under Qwen2.5-7B: +0.17 [+0.09, +0.25] and +0.13 [+0.04, +0.22] §5, Table 2.
  • In blinded pairwise tests with low-confidence decoding, the raters choose Mix-FT at K = 16 over the start at K = 32 in 69% (ChatGPT) and 71% (Claude Sonnet) of 288 pairs §5, app. J.
  • At eight steps per block (four tokens revealed at once), Mix-FT recovers 66 to 91% of the start's loss from halving the steps, yet still trails by 0.10 to 0.31 nats per token §5.

text8 (exact likelihood, 4.82M model, block 8, K = 4) §6, Table 3; app. E. Exact NLL in bits per character, best of ten temperatures, 3-seed mean.

ArmNLL (bpc)Δ vs continued CEjudge gap G
Base 4× (start)1.804----
Continued CE1.79200
SDTT1.802+0.009−0.038
Per-path FT (equal plan weights)1.782−0.010+0.014
Hybrid (Mix-FT + CE loss)1.764−0.028+0.076
Mix-FT1.751−0.042+0.156
  • Equal plan weights recover only about a quarter of the likelihood gain; the paper attributes about three quarters of it to the plan weighting §6, app. E.
  • Trained from the start in place of CE, the mixture objective ends about 0.06 bpc worse on the sampler's likelihood after four units, and it makes 25 times fewer optimiser steps §6.

Figures. Figure 1 is a schematic. A sampler with K steps first draws a plan, then samples each group of positions on its own, conditioned on the earlier groups. Plain CE averages the per-plan log-likelihoods, whereas Mix-FT trains on an importance-weighted bound applied to the log of that average fig. 1. Figure 2 relates judged quality to measured generation time at 110M fig. 2. Figure 3 covers the half-steps comparison at 110M, 8B and 1.1B together with the blinded-rater results fig. 3.

Paper vs code

No official code. The paper names no code repository, project page or artifact link in its source (main.tex, app_8b.tex, app_repro.tex). A GitHub search of the method name (mixft), its title words (mixture-likelihood diffusion, train sampler diffusion, iwae masked diffusion, block diffusion mixture likelihood, sampler masked diffusion finetune mixture) and Mix-FT diffusion / mix-ft llada returned nothing that belongs to the authors. The author-named accounts (najafgholizadeh, which has zero public repositories, and AminWork, which has a résumé repository and unrelated projects) hold no matching code.

Pitfalls

  • The start must already be CE-trained. Mix-FT refines a CE-trained model. Used from scratch on text8 it is worse than continued CE on the sampler's likelihood, and weights that are under-trained leave the mixture variants behind continued CE on a 34M model as well §8, app. A, app. E.
  • M = 1 is not Mix-FT. With one plan the weight is 1, and the loss is the per-path loss, which has the true conditionals as its optimum and gives no gain §3.
  • Weights collapse. The self-normalised plan weights concentrate on few plans. Measured on eight plans, the median effective sample size is 1.32 for the start at K = 2 at 110M, 1.54 at K = 4 and 1.95 at K = 8; at block 1,024 it is 1.00 app. A, §8. The bound is then loose, and the paper reports that Mix-FT's exact likelihood is not known at 8× §4.
  • The NELBO worsens. The one-token-per-step bound (NELBO perplexity) rises from 23.0 to 24.5 at 1× and to 28.8 at 8×, while the tighter IWAE-64 bound improves. A reader who scores with NELBO alone will see a regression §4, §8, app. J.
  • Cost. A sequence costs 1 + M(K − 1) forward passes for the bound §3. Wall-clock training for the 8× Mix-FT run was 1.45 times that of 8× continued CE, mainly because the chunked backward executes about 1.4× the counted FE app. J. The paper says the block-chunked variant executes exactly the counted FE, but this is an implementation the paper describes, not a released one.
  • Plan granularity depends on block size. At 110M, K = 2 plans beat K = 4 plans at equal compute, and the paper gives a hypothesis (coarse plans give the objective most to correct), which it marks as untested §4. At block 32 the paper trained on K = 4 plans instead app. C. The paper's reasoning is about reveal size per step, so use that, not the sampler's step count, as the guide (an inference from §4 and app. C).
  • The first token. BD3-LM skips training on the first position of each window and forces that token to end-of-text when it samples. The paper's fine-tunes train on it and draw it. Fixing the first token changes the start's judge NLL by 0.05 to 0.11 nats per token at K = 8 and 16, and it changes the ranking of claims. The paper states each claim under the convention that favours the start §4, app. B.
  • Judges are language models. The main numbers are GPT-2-large and Pythia-1.4B NLL on content tokens at matched entropy, plus two LLM raters. They are not human quality scores, and the paper says so (judges and raters are models) §8. Counting every token, including newline and end-of-text targets, rewards whichever model emits more newlines, which is why the paper reports content tokens §4, app. A.
  • Confidence planners are not covered by the theory. Proposition 1 covers fixed or state-determined plans only. LLaDA's confidence planner is outside it, and Mix-FT was trained only on random order §3, §8.
  • Per-step temperature schedules help every model. Schedules are tuned on separate development samples. They add about 0.3 nats per token at K = 8 for Mix-FT, Alpha and SDTT-8, and leave their order unchanged §4, app. A. They are an extra tuning step, not part of the core method.
  • Abstract rounding. The abstract gives the LLaDA compute as 0.002%; the setting table gives 0.0018% §5, app. C.
  • Scale. Full fine-tuning at scale is open. The 8B experiments use LoRA only, and the 1.1B and 8B comparisons use continued CE and Alpha only, with 256-token sequences §8. On a 1.1B GSM8K model trained further on solutions it had generated and checked, Mix-FT showed no gain over continued CE §8, app. D.
  • Hardware. The 110M runs used one T4 per arm (two for the 16× recipe). Times in the paper are T4-hours and are not comparable across hardware app. J.

Lineage

  • Builds on: the masked diffusion LM and block-diffusion line (MDLM, MD4, SEDD, LLaDA, BD3-LM); importance-weighted bounds (IWAE) §7.
  • Extends: the sampling-time analysis of few-step samplers as mixtures over paths §7, Theorem 1 in the cited work.
  • Compared against: continued cross-entropy at the sampler's states; Alpha diffusion (teacher-free α-divergence fine-tune); SDTT (distillation from a many-step teacher); per-step temperature schedules §4, §7, app. A.
  • Superseded by: not known.

Source and license

Adapted from “Train the Sampler You Run: Few-Step Masked Diffusion Language Models via Mixture-Likelihood Fine-Tuning” by Amin Najafgholizadeh (arXiv:2611.02666), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.