wikillm

Paper · 2026-10-26 · cs.LG

Quantifying the failure of symmetry learning in linear diffusion models due to initialization and finite-sample effects

A linear diffusion model trained by gradient descent on a score-matching loss, asking how well it learns rotational symmetry. The average symmetry error splits into initialization, noisy-target and finite-sample terms, with closed forms from random matrix theory. Target noise breaks symmetry most persistently, even after infinite training and when oversampled. Analytic only; no experiments.

Diffusion modelsEquivariance constraintScore matchingWeight initializationdiffusionsymmetryequivariancerandom-matrix-theorylinear-modelstheory

arXivPDFMarkdown11 min read

AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Many kinds of data have symmetries. Rotate a photo of a dog and it is still a dog, so a good generative model should treat all orientations alike. Diffusion models, which learn to undo added noise, do not have this built in, and they may learn such symmetries only partly. This paper asks how much a diffusion model misses, and why. To keep the question solvable, it uses the simplest model that still has a clean answer: a straight linear map, trained on Gaussian data.

The authors name three possible sources of the error. The first is the random starting point of training, which fades as training goes on. The second is the noisy training target that replaces the true score. The third is having only a finite number of data samples, so that the measured data statistics are not perfectly symmetric. The starting point causes only a mild, temporary error. The noisy target keeps the model asymmetric even when training has run for a long time and the data are plentiful. Finite samples also leave an error, which is largest when data are scarce. Smaller symmetries are a little harder to learn than larger ones.

This is a theory paper. It has no experiments on real images or benchmarks. Its results are formulas, derived with tools from random matrix theory, that say how large each error is.

TL;DR

The paper analyses a linear diffusion model whose weights are fitted by plain gradient steps on the score-matching objective, and measures how far they break a rotational symmetry of the data §2 eq. 2 eq. 5. The average error is split into an average-weight term, an initialisation term and a target-noise term eq. 9. The initialisation term decays with training time, while the target-noise term persists after infinite training and in the oversampled regime §4 §6.3. The results are asymptotic closed forms from random matrix theory, not measured numbers, and the paper has no experiments §7.

Problem

Symmetries in the data distribution are not guaranteed to be learned by diffusion models, and missing them can lower sample efficiency and generalisation §1. The paper asks to what extent diffusion models fail to learn such symmetries, and which source of symmetry breaking is responsible §1.

Assumptions, all stated by the paper: data are Gaussian with mean zero app. B; the score model is linear, \hat{s}(x,t) = W x §2.1; only one noise time t is learned §2.1; and closed forms are taken in the limit N, D \to \infty with \gamma = D/N fixed §5 §6. The paper lists these as limitations, together with the omission of the sampling dynamics §7.

Method

Setting. The clean data covariance is \Sigma^0, and the noisy covariance at time t is \Sigma(t) = \alpha_t^2 \Sigma^0 + \sigma_t^2 I eq. 1 app. B. The true score is -\Sigma(t)^{-1} x, so the optimal weight is W^* = -\Sigma^{-1} §2.1 app. B. The paper writes \Sigma for this noise-corrupted covariance throughout the main text; see Pitfalls.

Learning target and loss. The model is trained on the proxy score \tilde{s}(x, t; x_0) = (\alpha_t x_0 - x)/\sigma_t^2. Its mean given x is the true score, and its covariance is C = I/\sigma_t^2 - \Sigma^{-1} §2.1 app. B. The loss is the score-matching objective with an L2 penalty of strength \kappa \ge 0 eq. 2.

Symmetry. The symmetry group is G = SO(d) acting on a d-dimensional subspace, with d \ge 3 §2.2. Equivariance means W = U W U^T for all U \in G eq. 3. The covariance is \Sigma = vP + Q\Sigma Q, where P projects onto the symmetric sector and Q = I - P eq. 4 §2.2. The symmetry error is

\mathcal{E}_G(W) = \frac{1}{2d}\,\mathbb{E}_{U \in G}\,\lVert W - U W U^T \rVert_F^2 eq. 5

It equals \frac{1}{d}\lVert W - W^G \rVert_F^2, where W^G is the group average of W. For SO(d) with d \ge 3 this has the closed form W^G = \frac{1}{d}\,\mathrm{tr}(PWP)\,P + QWQ app. A. The trace form is eq. 6. Its first two terms measure failure to learn the symmetric sector, and the last measures leakage between the two sectors §2.3.

Learned weights. With X \in \mathbb{R}^{D \times N} the data matrix, \tilde{S} the proxy-score matrix and K = XX^T/N the empirical covariance, gradient flow gives eq. 7 app. B:

W(\tau) = W(0)\, e^{-(K + \kappa I)\tau} + \frac{1}{N}\tilde{S} X^T (K + \kappa I)^{-1} \big(I - e^{-(K + \kappa I)\tau}\big) eq. 7

  1. Split the weights into three parts. W = W^{\text{avg}} + W^{\text{init}} + W^{\text{proxy}}, with W^{\text{avg}} = -\Sigma^{-1} K (K+\kappa I)^{-1} (I - e^{-(K+\kappa I)\tau}), W^{\text{init}} = W(0)e^{-(K+\kappa I)\tau}, and W^{\text{proxy}} the proxy-noise part app. C.
  2. Average the error over the randomness. W(0) and the target noise have mean zero and are independent, so cross terms vanish and the expected error is \mathcal{E}^{\text{avg}} + \mathcal{E}^{\text{init}} + \mathcal{E}^{\text{proxy}} eq. 8 app. C. The three terms are defined in eq. 9 and derived in app. C.
  3. Initialisation term. Entries of W(0) are drawn from \mathcal{N}(0, \xi^2/D). The term is \mathcal{E}^{\text{init}} = \xi^2\{\frac{1}{D} \mathrm{tr}\,e^{-2(K+\kappa I)\tau} + (1 - \frac{d}{D} - \frac{1}{Dd}) \frac{1}{d}\mathrm{tr}[P e^{-2(K+\kappa I)\tau}]\} eq. 9 app. C.
  4. Target-noise term. With F = (K+\kappa I)^{-1}(I - e^{-(K+\kappa I)\tau}), the exact boxed form is eq. 9 app. C: \mathcal{E}^{\text{proxy}} = (\frac{1}{\sigma_t^2} - \frac{1}{v}) \frac{1}{N}\mathrm{tr}[F^T K F] + [\frac{\mathrm{tr}(QC)}{N} - \frac{1}{Nd} (\frac{1}{\sigma_t^2} - \frac{1}{v})] \frac{1}{d}\mathrm{tr}[PF^TKF]. Later sections drop the 1/(Nd) term as subleading app. C.
  5. Infinite samples. With N \to \infty the objective becomes a trace in \Sigma eq. 10, and the weights follow a linear ODE with solution eq. 11. The initialisation term then depends on the eigenvalues of \Sigma outside the symmetric sector, plus one term from the symmetric sector eq. 12. For full symmetry it reduces to eq. 13 §4.
  6. Isotropic finite samples (\Sigma = vI, d = D). The three terms reduce to traces over functions of K eq. 14. Their averages over the Marchenko–Pastur law app. D with \gamma = D/N are eq. 15 §5. Infinite training time gives eq. 16 with eq. 17. Without regularisation the oversampled and undersampled limits are eq. 18 and eq. 19.
  7. General covariance (partial symmetry). Traces now involve the deterministic matrices P, Q and \Sigma as well as K eq. 20 §6. The paper uses single- and multi-resolvent local laws §6.1 app. F. The deterministic equivalent of the resolvent is eq. 21, and the companion Stieltjes transform m(z) solves eq. 22.
  8. General case, infinite time. With m_\kappa = m(-\kappa) and m'_\kappa = m'(-\kappa) app. H, the result is eq. 23, with constants c_\perp and J_\perp over the non-symmetric density \rho_\perp eq. 24. The \kappa \to 0 limits are eq. 25 (oversampled) and eq. 26 (undersampled). The leakage terms carry the factor (1 - r), with r = d/D §6.2 §6.4.

Notation used in the equations: \gamma = D/N; r = d/D; \lambda_\pm = v(1 \pm \sqrt{\gamma})^2 are the Marchenko–Pastur bulk edges §5; \Delta(\kappa) = \sqrt{(\kappa + \lambda_-)(\kappa + \lambda_+)} §5; \mu_{-1} = \mathbb{E}_{\lambda \sim \rho}[1/\lambda] §6.3.

Key results, for isotropic infinite-time, with q_1, q_2 defined in eq. 17:

\mathcal{E}^{\text{avg}} = \frac{\kappa^2}{v^2}\,[q_2(\kappa) - q_1(\kappa)^2], \qquad \mathcal{E}^{\text{proxy}} = \Big(\frac{1}{\sigma_t^2} - \frac{1}{v}\Big) \frac{\kappa + v(1+\gamma) - \Delta(\kappa)}{2 v\, \Delta(\kappa)} eq. 16

Two steps are easiest to implement as code, the learned weights eq. 7 and the isotropic error terms eq. 14 with eq. 15. Both are sketches from the formulas in the paper, not the authors' code, which does not exist.

Pseudocode
# Learned weights after training time tau  [eq. 7] [app. B]
# Input:  W0 (D, D); S_tilde (D, D); X (D, N); kappa, tau
# Output: W_tau (D, D)
def learned_weights(W0, S_tilde, X, kappa, tau):
    N = X.shape[1]
    # Empirical covariance K = X X^T / N  [app. B]
    K = X @ X.T / N
    # Eigen-decompose K; the rest is per eigen-direction  [app. B]
    lam, U = eigh(K)                           # lam (D,)
    rate = lam + kappa                         # (D,)
    # Decay of the initial weights  [eq. 7]
    decay = exp(-rate * tau)                   # (D,)
    # Gain of the proxy term; take the limit tau where rate is 0  [eq. 7]
    fit = where(rate > 0, (1 - decay) / rate, tau)   # (D,)
    # Rebuild the matrices in the original basis
    init_part = W0 @ (U * decay) @ U.T
    proxy_part = (S_tilde @ X.T / N) @ (U * fit) @ U.T
    return init_part + proxy_part
Pseudocode
# Isotropic case, Sigma = v I and d = D  [eq. 14] [eq. 15]
# Input:  lam (D,) eigenvalues of K; v, sigma2, xi2, gamma, kappa, tau
# Output: the three terms of the average symmetry error
def isotropic_error_terms(lam, v, sigma2, xi2, gamma, kappa, tau):
    rate = lam + kappa                         # (D,)
    decay = exp(-rate * tau)                   # (D,)
    fit = where(rate > 0, (1 - decay) / rate, tau)   # (D,)
    # Average-associated: population variance of the mean weight  [eq. 14]
    avg = variance(lam * fit) / v**2
    # Initialisation-associated: mean of the squared decay  [eq. 14]
    init = xi2 * mean(decay**2)
    # Target-noise-associated  [eq. 14]
    proxy = (1 / sigma2 - 1 / v) * gamma * mean(lam * fit**2)
    return avg, init, proxy

Hyperparameters

The paper has no training runs, so most entries are symbols, not values.

NameValueWhereNotes
\kappa (weight decay)\ge 0; 0 in Fig. 1, 0.001 in Fig. 2eq. 2 fig. 1 fig. 2\kappa = 0 is a limit in the text; Fig. 2 is evaluated at small \kappa
\tau (training time)[0, \infty), gradient-flow time§3 app. Blearning rate is infinitesimal in the analysis; a numeric learning rate is not stated
\sigma_t, \alpha_t\sigma_t > 0, 0 \le \alpha_t \le 1app. Bno values given
veigenvalue of \Sigma on the symmetric sector§2.2symbolic
\xi^2variance scale of W(0); entries \sim \mathcal{N}(0, \xi^2/D)§3 app. Cno value given
ddimension of symmetric sector§2.23 \le d \le D app. A
\gamma = D/N, r = d/Dratios§5 §6limits taken with \gamma, r fixed
N (samples)N \ge 1§3only the limit matters for the closed forms
m_\kappa"numerically determined"fig. 2the paper does not give the solver
Figure simulation settingsnot statedfig. 1 fig. 2values of D, N, d, seeds and number of draws are not given

Results

Author-reported. The paper has no benchmark or experiment. All results are analytic formulas, most of them asymptotic in N, D \to \infty §5 §6. The figures plot these formulas, and the text does not give the numbers behind them fig. 1 fig. 2.

SettingTerm(s)ResultSource
Infinite samples, full symmetry (d = D)\mathcal{E}^{\text{init}}\xi^2 (1 - 1/D^2)\, e^{-2(v+\kappa)\tau}; goes to 0 as \tau growseq. 13 §4
Infinite samples, partial symmetry\mathcal{E}^{\text{init}}initialisation error with a complement-sector average plus (1 - 1/(Dd))e^{-2(v+\kappa)\tau}eq. 12
Isotropic, \tau \to \infty, \kappa > 0\mathcal{E}^{\text{avg}}, \mathcal{E}^{\text{proxy}}\frac{\kappa^2}{v^2}[q_2 - q_1^2] and the \Delta(\kappa) expression aboveeq. 16
Isotropic, \tau \to \infty, \kappa \to 0, \gamma < 1\mathcal{E}^{\text{proxy}} only(\frac{1}{\sigma_t^2} - \frac{1}{v})\frac{\gamma}{v(1-\gamma)}; \mathcal{E}^{\text{avg}} = 0eq. 18
Isotropic, \tau \to \infty, \kappa \to 0, \gamma > 1all three\mathcal{E}^{\text{avg}} = \frac{\gamma-1}{v^2\gamma^2}; \mathcal{E}^{\text{init}} = \xi^2(1 - \frac{1}{\gamma}); \mathcal{E}^{\text{proxy}} = (\frac{1}{\sigma_t^2} - \frac{1}{v})\frac{1}{v(\gamma-1)}eq. 19
General, \tau \to \infty, \kappa > 0\mathcal{E}^{\text{avg}}, \mathcal{E}^{\text{proxy}}eq. 23 in terms of m_\kappa; \mathcal{E}^{\text{init}} = 0eq. 23 app. H
General, \tau \to \infty, \kappa \to 0, \gamma < 1\mathcal{E}^{\text{proxy}} only(\frac{1}{\sigma_t^2} - \frac{1}{v})\frac{\gamma \mu_{-1}}{1-\gamma} + \frac{\gamma(1-r)c_\perp}{v(1-\gamma)}eq. 25
General, \tau \to \infty, \kappa \to 0, \gamma > 1\mathcal{E}^{\text{init}} and the others\mathcal{E}^{\text{init}} = \xi^2(1 - \frac{1}{\gamma}) + \xi^2\frac{1-r}{1 + v m_0}eq. 26

Qualitative claims the paper makes from these formulas:

  • Initialisation error is mild and transient. It vanishes once the symmetric sector is learned, around \tau \sim 1/(v+\kappa) §4 §5.
  • The target-noise term persists in the oversampled regime, where the other two vanish at \kappa \to 0, and it blows up as \gamma \to 1^- §5 §6.3.
  • The target-noise term's size does not depend on \kappa, while the average-associated term carries a \kappa^2 prefactor §4.
  • In the general case, Figure 2 shows the proxy term dominating the average error, as in the isotropic case fig. 2. For the isotropic undersampled case the paper says this holds "anecdotally" §5.
  • Smaller symmetries (small r = d/D) are slightly harder to learn because of the (1 - r) leakage terms §6.2 fig. 2.

Paper vs code

No official code.

The search covered: the arXiv abstract page, whose comments name only the NeurReps 2026 workshop and give no code link; the TeX source, whose only code line is an unfilled template comment, Link to code: http://??; GitHub repository searches for "symmetry diffusion linear", "vastola", "kanaka rajan", "symmetry learning diffusion" and "proxy score diffusion noise", which returned nothing relevant; and the Rajan lab organisation, which has four repositories, none for this paper. The checks were done without cloning anything.

Pitfalls

  • No experiments, so no numbers to match. Figures 1 and 2 are plots without printed values, and the sizes D, N, d, seeds and draws behind them are not stated. Figure 1's y-axis is in arbitrary units fig. 1. Treat the figures as shape checks only.
  • Asymptotic formulas, with dropped corrections. The closed forms hold in the limit N, D \to \infty with \gamma and r fixed. The exact finite-size expressions carry terms of order 1/(Nd) and 1/D^2, which the paper drops app. C. Compare eq. 9 (exact) with eq. 20 (dropped term). Eq. 13 keeps (1 - 1/D^2), but eq. 14 does not. If you need exact finite-size values, use the boxed exact forms in app. C.
  • Empirical eigenvalues are one realisation. Eq. 14 averages over the eigenvalues of K. Averaging over the eigenvalues of a single finite draw of X gives one realisation, not the expectation over the data. Average over several draws, or use the Marchenko–Pastur forms eq. 15 app. D.
  • Zero eigenvalues when \kappa = 0. For \gamma > 1, K is singular. The factor (1 - e^{-(\lambda+\kappa)\tau})/(\lambda+\kappa) must be taken as \tau where \lambda + \kappa = 0. Each zero eigenvalue then contributes e^0 = 1 to \mathcal{E}^{\text{init}}. This is the mass 1 - 1/\gamma in eq. 19 (derived here from the formula, not stated in the paper).
  • Which covariance is \Sigma. The main text writes \Sigma for the true covariance, but app. B makes clear that this is the noise-corrupted \Sigma(t) = \alpha_t^2\Sigma^0 + \sigma_t^2 I. Using the clean \Sigma^0 changes v, W^* and every formula above.
  • Sign of the target-noise prefactor. (1/\sigma_t^2 - 1/v) is non-negative only if v \ge \sigma_t^2. Because v is an eigenvalue of \Sigma(t) \succeq \sigma_t^2 I, this holds for the noisy covariance. It is not stated in the paper and is derived here. If you plug in a value of v below \sigma_t^2, the "error" becomes negative and the formulas are invalid.
  • Literal reading of \mathrm{spec}(Q\Sigma Q). Eq. 12 writes a sum over \mathrm{spec}(Q\Sigma Q). Read on the full D-dimensional space, that matrix has d extra zero eigenvalues, which would add d terms equal to 1 at \kappa = 0. The text says the sum covers only the eigenvalues outside the symmetric sector, and app. C writes it as an average over D - d values. Use those D - d eigenvalues only.
  • Density normalisation for the partial case. c_\perp and J_\perp integrate over \rho_\perp. The paper does not state that \rho_\perp is normalised to
    1. The identity \gamma(1-r)\,\mathrm{tr}(QC)/(D-d) = \mathrm{tr}(QC)/N eq. 20 suggests it is normalised over the D - d complement eigenvalues. Check this before implementing.
  • Branch of m(z). The self-consistent equation eq. 22 needs the branch with m(z) \sim -1/z for large |z| app. H. The paper says m_\kappa is "numerically determined" fig. 2 but gives no solver or starting point.
  • Equation numbers. Numbers in this card were counted from the TeX source, in order, since equations are numbered consecutively in the class used. The compiled PDF was not checked. If a citation seems off, cite the section.
  • Limits checked by hand. The isotropic \kappa \to 0 results eq. 18 and eq. 19 follow from eq. 16 when \Delta(0) = v\,|1-\gamma| and q_1(0) = 1/(v(1-\gamma)) for \gamma < 1. The partial-symmetry formulas reduce to the isotropic ones at r = 1. This was checked by reading the formulas, not by simulation.
  • Scope. The analysis covers score learning at one noise level, with Gaussian data and a linear model. Sampling dynamics are not analysed, so the paper does not show that the sampler produces symmetric samples §7.

Lineage

  • Builds on: the proxy-score result in Vastola, "Generalization through variance: how noise shapes inductive biases in diffusion models" (ICLR 2025), cited for the target's mean and covariance §2.1. No card exists for it, so only the name is given.
  • Linear and Gaussian diffusion dynamics: Wang and Vastola, "The Unreasonable Effectiveness of Gaussian Score Approximation for Diffusion Models" (TMLR) and Wang and Pehlevan, "An Analytical Theory of Spectral Bias in the Learning Dynamics of Diffusion Models" (NeurIPS 2025) §1 §7. Named only.
  • Random matrix tools: "Anisotropic local laws for random matrices" and Lin and Pan, "Eigenvector overlaps in large sample covariance matrices and nonlinear shrinkage estimators" (Annals of Statistics) §6.1. Named only.
  • Compared against: none. The paper does not run baselines.
  • Superseded by: not known.

Source and license

Adapted from “Quantifying the failure of symmetry learning in linear diffusion models due to initialization and finite-sample effects” by John J. Vastola, Kanaka Rajan (arXiv:2610.35720), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.