wikillm

Concept · 2 papers · updated 2026-11-21

Score matching

Trains a network to match the score, the gradient of the log density, by regressing onto per-sample targets built from noised data. Two theory papers analyse it in diffusion: a linear model with a closed-form symmetry error, and an online fresh-sample objective with a total-variation bound. Neither has code or benchmarks. Choose by whether you need an exact analysis or a training bound.

generative-modelsdiffusiontheorysample-complexitysymmetrystochastic-gradient

Markdown5 min read

AI-compiled overview, updated 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

Overview

Score matching fits a network to the score of a distribution, which is the gradient of its log density. In diffusion models the score is taken at each noise level, and the reverse process uses it to denoise a sample. The true score is rarely available, so training regresses the network onto a per-sample target that is built from a clean data point and fresh noise. The expectation of that target equals the true score, which is why the regression is valid even though each individual target is noisy 2611.01562 §2 2610.35720 §2.1.

Both papers in this concept use the same variance-preserving corruption and the same target form, -(x_t - alpha x_0) / sigma^2, written with the noise coefficients of each paper 2611.01562 §2 2610.35720 §2.1. They differ in what they analyse. Linear diffusion with symmetry studies one linear score model trained by gradient flow, which gives closed forms. Online conditional diffusion studies a general conditional network trained on fresh minibatches, which gives a bound.

Variants

Linear score model with a proxy target

The model is linear, s(x, t) = W x, and is trained by gradient flow on the score-matching loss with an L2 penalty 2610.35720 §2.1. The per-sample target is a proxy: its mean given x is the true score, but its covariance is non-zero 2610.35720 §2.1 app. B. The learned weights have a closed form after training time tau 2610.35720 §3. The paper's error measure is how far the weights break a rotational symmetry of Gaussian data 2610.35720 §2.2. It splits the expected error into three terms: the average weight, the random initialisation, and the target noise 2610.35720 §3 app. C.

The initialisation term decays with training time. It vanishes once the symmetric sector is learned 2610.35720 §4 §5. The target-noise term does not decay in the same way. For fixed ratio gamma = D/N below one and vanishing regularisation, it stays non-zero, and it grows as gamma approaches one 2610.35720 §5 §6.3. The results are asymptotic and the paper runs no experiments 2610.35720 §7.

Online objective with a randomized time index

The network is a time-gated sum of blockwise approximants, and its parameters are updated by SGD on fresh minibatches, with no stored dataset 2611.01562 §2. Each step draws a clean point, a condition, a grid index and Gaussian noise. The grid index is sampled with probability proportional to its step length, so the time average matches the stability argument 2611.01562 §2. The population loss equals a constant plus the squared distance to the true score, so its minimum over the class is the best score the class can reach 2611.01562 §2. The paper then bounds the total variation of the sampler, with terms for initialisation, time discretisation, approximation and finite-time optimisation 2611.01562 §3.3.

The approximation term decays with the complexity index M at a rate set by the smoothness and the intrinsic dimensions of the response and the condition 2611.01562 §3.1. The optimisation term is O(1/N) under uniform constants, which gives a sample requirement of order eps^-2 for a target error eps 2611.01562 §3.5. These rates depend on assumptions the authors state but do not prove for neural networks 2611.01562 §3.2 §4.

When to use which

The concept is used in theory here, not in a tested recipe. Neither card reports a benchmark, and neither has released code. Pick by the question you need answered.

SituationUseWhy
You want an exact error breakdown for a linear or Gaussian settingLinear model with proxy target, Linear diffusion with symmetryClosed forms for each error term 2610.35720 §3 §5
You are checking whether a symmetry in the data will be learnedSame cardIts split shows which term persists and which decays 2610.35720 §4 §5
You train from a stream of fresh samples and cannot store the dataOnline objective, Online conditional diffusionEach step uses new samples and a randomized time index 2611.01562 §2
You need a budget for samples, network size and stepsSame cardIt gives orders for M, N, K, T under assumptions 2611.01562 §3.5
You need a bound on the stopped law rather than the data lawSame cardThe bound compares to the law at a positive stopping time; the data-law result needs an extra density assumption 2611.01562 §3.6

Practical limits. The linear analysis uses Gaussian data and one noise level 2610.35720 §7. The online bound assumes a localized Polyak-Lojasiewicz inequality and that every iterate stays in a local region 2611.01562 §3.2. Its step size needs constants that the paper does not estimate, so the bound does not give a training recipe with numbers 2611.01562 §3.2.

Implementation sketch

The core of the denoising variant, using the target both papers share. The step size, time distribution and sampler follow the cards; this is a sketch from the formulas, not the authors' code, which does not exist.

Pseudocode
# Denoising score matching with a per-sample target  [arxiv:2611.01562 §2]
# Input:  clean batch x0 (B, d); condition y; grid times t (B,)
# Output: scalar loss for one minibatch

def score_matching_loss(score_net, x0, y, t):
    # Fresh Gaussian noise of the same shape as x0  [arxiv:2611.01562 §2]
    eps = sample_normal(x0.shape)                  # (B, d)
    # Noise coefficients of the variance-preserving process  [§2]
    alpha = exp(-t)                                # (B,)
    sigma2 = 1 - alpha ** 2                        # (B,)
    # Noised state at time t  [arxiv:2611.01562 §2]
    x_t = alpha * x0 + sqrt(sigma2) * eps          # (B, d)
    # Per-sample target: the score of p(x_t | x0)  [arxiv:2610.35720 §2.1]
    target = -(x_t - alpha * x0) / sigma2          # (B, d)
    # Regress the network output onto the target  [arxiv:2611.01562 §2]
    pred = score_net(x_t, y, t)                    # (B, d)
    return mean(sum((pred - target) ** 2, dim=-1))

The two variants ask different questions of the same noisy target. The linear analysis asks how much symmetry error that noise leaves in a linear model 2610.35720 §5. The online bound asks how far the sampler's distribution can drift, given the approximation and optimisation errors of the class 2611.01562 §3.3. The online paper does not split out a target-noise term.

Contradictions

None directly. The two cards measure different things: a symmetry error for a linear model 2610.35720 §2.2, and a total-variation bound for a general conditional class 2611.01562 §3.3. Do not read one as confirming or refuting the other. The linear paper's target-noise term stays non-zero at fixed ratio 2610.35720 §5. The online paper states that its population loss is minimised by the true score, up to a constant 2611.01562 §2. Whether the noisy target biases a trained network in practice is not settled by either card.

Papers

Source and license

Compiled from the papers listed on this page, all published under CC BY, CC BY-SA or CC0, with citations. See each paper's page for its authors and license.