---
kind: concept
slug: self-conditioning
title: Self-conditioning
summary: >-
  Feeding a denoiser its own previous prediction as an extra input, so each step sees the last guess. Covers latent video diffusion (W.A.L.T, prediction channels), continuous text diffusion (LangFlow, expected-embedding input) and pure state carry without sampling tokens between steps (LAW). Costs: an extra forward pass on part of the training steps; results depend on the metric.
tags: [diffusion, self-conditioning, text-generation, video-generation, sampling]
papers: ["2201.07284", "2312.06662", "2604.11748", "2610.30068"]
related: [diffusion-models, discrete-diffusion, continuous-text-diffusion, latent-diffusion, flow-matching]
status: draft
updated_at: 2026-11-22
---

## Overview

An iterative denoiser usually throws away its prediction after each step and
only keeps the noisy input. Self-conditioning keeps the prediction and passes
it to the next call as an additional input. In training, the model first runs
with an empty self-condition, then runs again with that first estimate,
detached from gradients, as input [arxiv:2312.06662 §4.3]
[arxiv:2604.11748 alg. 1]. At sampling, the estimate from the previous step is
always available [arxiv:2604.11748 alg. 2]. The name is also used for an
unrelated mechanism in anomaly detection, where a focus score from a first
decoder pass feeds back into the encoder [arxiv:2201.07284 §3.4]. That is not
the technique described here.

The invariant all variants share is that the carried quantity is computed from
the model's own prediction, so the next step starts from a better-informed
guess. The variants differ in what is carried, where it enters the network,
and whether tokens are sampled between steps.

## Variants

### Prediction channels in latent video diffusion

W.A.L.T concatenates the first-pass estimate to the noisy latent along the
channel axis. It applies this with probability 0.9 during training and runs
the single-pass branch otherwise [arxiv:2312.06662 §4.3]. The paper's own
ablation on UCF-101 reports FVD 109.9 without self-conditioning and 61.4 with
it at p_sc = 0.9 [arxiv:2312.06662 tab. 3]. Card: [W.A.L.T](/papers/2312.06662). The paper does not state
which quantity is fed back or which pass carries the loss, so implementers
must choose and document it [arxiv:2312.06662 §4.3].

### Expected-embedding feedback in continuous text diffusion

LangFlow feeds back the probability-weighted average of the embedding table,
computed from the predicted token distribution. It enters through a projection
that starts at zero, and it is used on about one training step in four
[arxiv:2604.11748 §5] [arxiv:2604.11748 alg. 1]. At sampling it is always on,
starting from zero at the first step [arxiv:2604.11748 §4.2] [arxiv:2604.11748
alg. 2]. On LM1B, the paper reports PPL 49.0 without it and 30.0 with it, and
notes that the effect is much larger for the continuous model than for MDLM
[arxiv:2604.11748 tab. 1] [arxiv:2604.11748 §4.2]. Card: [LangFlow](/papers/2604.11748).

### Pure state carry without token sampling

LAW removes the discrete sample between steps altogether. Each pass sends the
expected embedding to the next pass, which is the only input to the denoiser,
and tokens are drawn once after the last pass [arxiv:2610.30068 §3.1]. It
trains with one or two unrolled passes, p = 0.9 for the two-pass case by
default, and a stop-gradient between passes [arxiv:2610.30068 §3.2]. The paper
reports that carrying the hidden representation instead drops GSM8K accuracy
below 1.5% without temperature regularization, and that removing the
stop-gradient drops accuracy from 42.2% to 20.2% at matched FLOPs
[arxiv:2610.30068 §4.3]. Card: [LAW](/papers/2610.30068). The earlier self-conditioning
baseline it builds on, Loopholing (arXiv:2510.19304), has no card here, so its
content is not used.

## When to use which

| Situation | Use | Why |
|---|---|---|
| Latent video or image diffusion, transformer backbone, enough compute for an extra pass on most steps | Prediction channels [arxiv:2312.06662 §4.3] | Reported FVD gain in the paper's own ablation [arxiv:2312.06662 tab. 3]; the paper uses p_sc = 0.9 |
| Continuous embedding diffusion for text, PPL is the target | Expected-embedding feedback [arxiv:2604.11748 §4.2] | The paper reports the gap to discrete models closes with it [arxiv:2604.11748 tab. 1] |
| Discrete masked or uniform diffusion, PPL is the target | Evaluate both ways and report which | Gen. PPL improves but PPL worsens for MDLM with it [arxiv:2604.11748 tab. 1] |
| Few-step generation, where information lost between steps matters | Pure state carry [arxiv:2610.30068 §4.3] | Reported gains on GSM8K and Sudoku at matched training FLOPs, small scale only [arxiv:2610.30068 §6] |
| Training budget is tight | Compare variants at equal FLOPs, not equal steps | The LAW card estimates about 3.9 F per update against 3 F for a baseline, so it trains for fewer steps [arxiv:2610.30068 app. B.1] [arxiv:2610.30068 app. B] |

The choice depends on the metric. LangFlow's gains are measured on PPL and
Gen. PPL, and the same self-conditioning raises MDLM's PPL [arxiv:2604.11748
tab. 1]. Any comparison should use the same protocol for every row.

## Implementation sketch

The common core is one extra input and, in training, one extra forward pass
without gradient. Shapes are (batch, positions, dim). Variants differ in what
`estimate` is and where it is concatenated or added.

```pseudocode
# Training step with self-conditioning  [arxiv:2604.11748 alg. 1]
# p_sc: probability of running the first pass
def train_step(model, x_t, t, target, p_sc):
    # Empty self-condition by default  [arxiv:2312.06662 §4.3]
    prev = zeros_like(x_t)
    if random() < p_sc:
        # First pass without gradient, its output is the estimate
        # [arxiv:2604.11748 alg. 1]
        prev = stop_gradient(model(x_t, t, zeros_like(x_t)))
    # Second pass receives the estimate as an extra input
    pred = model(x_t, t, prev)
    return loss(pred, target)
```

```pseudocode
# Sampling loop with the previous estimate carried forward  [alg. 2]
def sample(model, x, timesteps, to_estimate, step):
    # Starts with an empty self-condition  [arxiv:2604.11748 alg. 2]
    prev = zeros_like(x)
    for t_now, t_next in pairs(timesteps):
        pred = model(x, t_now, prev)
        # Carried quantity: a prediction, or its expected embedding
        # [arxiv:2610.30068 eq. 8] [arxiv:2604.11748 alg. 2]
        prev = to_estimate(pred)
        x = step(x, pred, t_now, t_next)
    return x
```

Reported settings, for reference:

| Variant | Feedback | Probability | Gradient through first pass | Source |
|---|---|---|---|---|
| Prediction channels | Estimate, concatenated along channels | 0.9 | Detached | [arxiv:2312.06662 §5.2] |
| LangFlow | Expected embedding through a zero-initialised projection | 0.25 in training, always on at sampling | Detached | [arxiv:2604.11748 §5] |
| LAW | Expected embedding, sole denoiser input | Two passes with p = 0.9 by default | Stop-gradient between passes | [arxiv:2610.30068 §3.2] |

## Contradictions

- **LangFlow's LM1B Gen. PPL.** The same model is reported at 81.5 in
  [arxiv:2604.11748 tab. 1] and at 92.2 in [arxiv:2604.11748 tab. 2]. The
  appendix NFE table agrees with 92.2 [arxiv:2604.11748 tab. 2]. Both values
  are cited here, and the card does not settle which one the authors intend.
- **W.A.L.T's default ablation row.** The default configuration scores 60.7
  [arxiv:2312.06662 tab. 3], while the p_sc = 0.9 row scores 61.4 with the
  same setting [arxiv:2312.06662 tab. 3]. The paper does not say which run is
  the baseline.
- **LAW's stop-gradient.** The paper's equation puts the stop-gradient on the
  whole next state [arxiv:2610.30068 eq. 10]. The released code detaches only
  the probabilities and keeps gradient through the embedding matrix via the
  carry [code: models/dit_loop.py#L156-L160]. This comes from reading the code,
  not from running it.

## Papers

- [Photorealistic Video Generation with Diffusion Models](/papers/2312.06662):
  prediction-channel self-conditioning in a latent video transformer, with an
  ablation at p_sc = 0.9.
- [LangFlow](/papers/2604.11748): expected-embedding self-conditioning in
  continuous text diffusion, with an LM1B ablation against MDLM.
- [LAW](/papers/2610.30068): self-conditioning as the only state carry,
  trained with a stop-gradient.
- [TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Data](/papers/2201.07284):
  uses the term for a focus-score feedback in anomaly detection; included only
  to mark the name collision.
