Concept · 2 papers · updated 2026-11-21
Score matching
Trains a network to match the score, the gradient of the log density, by regressing onto per-sample targets built from noised data. Two theory papers analyse it in diffusion: a linear model with a closed-form symmetry error, and an online fresh-sample objective with a total-variation bound. Neither has code or benchmarks. Choose by whether you need an exact analysis or a training bound.
generative-modelsdiffusiontheorysample-complexitysymmetrystochastic-gradient
Markdown5 min read
AI-compiled overview, updated 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
Overview
Score matching fits a network to the score of a distribution, which is the gradient of its log density. In diffusion models the score is taken at each noise level, and the reverse process uses it to denoise a sample. The true score is rarely available, so training regresses the network onto a per-sample target that is built from a clean data point and fresh noise. The expectation of that target equals the true score, which is why the regression is valid even though each individual target is noisy 2611.01562 §2 2610.35720 §2.1.
Both papers in this concept use the same variance-preserving corruption and the
same target form, -(x_t - alpha x_0) / sigma^2, written with the noise
coefficients of each paper 2611.01562 §2 2610.35720 §2.1. They
differ in what they analyse. Linear diffusion with symmetry
studies one linear score model trained by gradient flow, which gives closed
forms. Online conditional diffusion studies a general
conditional network trained on fresh minibatches, which gives a bound.
Variants
Linear score model with a proxy target
The model is linear, s(x, t) = W x, and is trained by gradient flow on the
score-matching loss with an L2 penalty 2610.35720 §2.1. The per-sample
target is a proxy: its mean given x is the true score, but its covariance is
non-zero 2610.35720 §2.1 app. B. The learned weights have a closed
form after training time tau 2610.35720 §3. The paper's error measure
is how far the weights break a rotational symmetry of Gaussian data
2610.35720 §2.2. It splits the expected error into three terms: the
average weight, the random initialisation, and the target noise
2610.35720 §3 app. C.
The initialisation term decays with training time. It vanishes once the
symmetric sector is learned 2610.35720 §4 §5. The target-noise term
does not decay in the same way. For fixed ratio gamma = D/N below one and
vanishing regularisation, it stays non-zero, and it grows as gamma approaches
one 2610.35720 §5 §6.3. The results are asymptotic and the paper runs
no experiments 2610.35720 §7.
Online objective with a randomized time index
The network is a time-gated sum of blockwise approximants, and its parameters are updated by SGD on fresh minibatches, with no stored dataset 2611.01562 §2. Each step draws a clean point, a condition, a grid index and Gaussian noise. The grid index is sampled with probability proportional to its step length, so the time average matches the stability argument 2611.01562 §2. The population loss equals a constant plus the squared distance to the true score, so its minimum over the class is the best score the class can reach 2611.01562 §2. The paper then bounds the total variation of the sampler, with terms for initialisation, time discretisation, approximation and finite-time optimisation 2611.01562 §3.3.
The approximation term decays with the complexity index M at a rate set by
the smoothness and the intrinsic dimensions of the response and the condition
2611.01562 §3.1. The optimisation term is O(1/N) under uniform
constants, which gives a sample requirement of order eps^-2 for a target
error eps 2611.01562 §3.5. These rates depend on assumptions the
authors state but do not prove for neural networks 2611.01562 §3.2
§4.
When to use which
The concept is used in theory here, not in a tested recipe. Neither card reports a benchmark, and neither has released code. Pick by the question you need answered.
| Situation | Use | Why |
|---|---|---|
| You want an exact error breakdown for a linear or Gaussian setting | Linear model with proxy target, Linear diffusion with symmetry | Closed forms for each error term 2610.35720 §3 §5 |
| You are checking whether a symmetry in the data will be learned | Same card | Its split shows which term persists and which decays 2610.35720 §4 §5 |
| You train from a stream of fresh samples and cannot store the data | Online objective, Online conditional diffusion | Each step uses new samples and a randomized time index 2611.01562 §2 |
| You need a budget for samples, network size and steps | Same card | It gives orders for M, N, K, T under assumptions 2611.01562 §3.5 |
| You need a bound on the stopped law rather than the data law | Same card | The bound compares to the law at a positive stopping time; the data-law result needs an extra density assumption 2611.01562 §3.6 |
Practical limits. The linear analysis uses Gaussian data and one noise level 2610.35720 §7. The online bound assumes a localized Polyak-Lojasiewicz inequality and that every iterate stays in a local region 2611.01562 §3.2. Its step size needs constants that the paper does not estimate, so the bound does not give a training recipe with numbers 2611.01562 §3.2.
Implementation sketch
The core of the denoising variant, using the target both papers share. The step size, time distribution and sampler follow the cards; this is a sketch from the formulas, not the authors' code, which does not exist.
# Denoising score matching with a per-sample target [arxiv:2611.01562 §2]
# Input: clean batch x0 (B, d); condition y; grid times t (B,)
# Output: scalar loss for one minibatch
def score_matching_loss(score_net, x0, y, t):
# Fresh Gaussian noise of the same shape as x0 [arxiv:2611.01562 §2]
eps = sample_normal(x0.shape) # (B, d)
# Noise coefficients of the variance-preserving process [§2]
alpha = exp(-t) # (B,)
sigma2 = 1 - alpha ** 2 # (B,)
# Noised state at time t [arxiv:2611.01562 §2]
x_t = alpha * x0 + sqrt(sigma2) * eps # (B, d)
# Per-sample target: the score of p(x_t | x0) [arxiv:2610.35720 §2.1]
target = -(x_t - alpha * x0) / sigma2 # (B, d)
# Regress the network output onto the target [arxiv:2611.01562 §2]
pred = score_net(x_t, y, t) # (B, d)
return mean(sum((pred - target) ** 2, dim=-1))The two variants ask different questions of the same noisy target. The linear analysis asks how much symmetry error that noise leaves in a linear model 2610.35720 §5. The online bound asks how far the sampler's distribution can drift, given the approximation and optimisation errors of the class 2611.01562 §3.3. The online paper does not split out a target-noise term.
Contradictions
None directly. The two cards measure different things: a symmetry error for a linear model 2610.35720 §2.2, and a total-variation bound for a general conditional class 2611.01562 §3.3. Do not read one as confirming or refuting the other. The linear paper's target-noise term stays non-zero at fixed ratio 2610.35720 §5. The online paper states that its population loss is minimised by the true score, up to a constant 2611.01562 §2. Whether the noisy target biases a trained network in practice is not settled by either card.
Papers
- Linear diffusion with symmetry: analytic split of the symmetry error of a linear score model trained by gradient flow.
- Online conditional diffusion: online SGD on fresh randomized-time samples, with a total-variation bound split into four terms.
Source and license
Compiled from the papers listed on this page, all published under CC BY, CC BY-SA or CC0, with citations. See each paper's page for its authors and license.