wikillm

Paper · 2026-10-30 · stat.ML, cs.LG, math.ST

Model Collapse Is Avoidable Even as Real Data Vanishes: A Sharp Criterion for Iterative Gaussian Estimation

Iterated mean and covariance estimation of a Gaussian, where each batch mixes real samples with the previous fit's samples, unlabelled. Consistency holds iff the sum of (1-alpha_t)(1-rho_t) diverges: any fixed real share works, a shrinking one if it shrinks slowly. Mean error rate: N^-1 below synthetic share 1/2, N^-2(1-alpha) above. Theory; CIFAR-10 DDPM agrees qualitatively.

Model collapseSynthetic training dataDiffusion modelsmodel-collapsesynthetic-dataself-consuming-traininggaussian-estimationtheory

arXivPDFMarkdown12 min read

AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Models trained on the open web increasingly see text and images that earlier models produced. Repeat that loop and the model can slowly drift away from the real world. This is called model collapse. Yet current generative models keep improving, and the real and the generated data cannot be cleanly separated.

The paper studies a deliberately simple version of the problem. It tries to learn the average and the spread of a bell-shaped data distribution. Each round it draws new samples from a mix of the real distribution and the current learned version, pools everything, and refits. The model never gets to tell the two kinds of sample apart.

The answer is a single test on how the real share of new data shrinks over the rounds. If the total amount of fresh real information keeps growing without bound, the learned version converges to the truth. A fixed real share is enough, and a vanishing share is enough if it vanishes slowly. Less real data still makes the estimate converge more slowly, which is why data curation still matters. A small diffusion experiment on CIFAR-10 shows the same qualitative pattern.

TL;DR

  • Iterated Gaussian mean and covariance estimation with indistinguishable real and synthetic samples. The mean is unbiased. The covariance estimate sits low by exactly C_t, the mean estimator's own covariance §3.2.1.
  • Both estimators converge to the truth if and only if the series of (1 - alpha_t)(1 - rho_t) diverges §3.2.2 eq. 12. So any fixed real share above zero suffices, and a vanishing share suffices if it vanishes slowly.
  • If the real share vanishes too fast, or is zero after round 1, the error approaches a positive limit. The mean's rate is N^-1 for synthetic share below 1/2 and N^-2(1 - alpha) above it §3.2.3 eq. 17.
  • Theory only, for Gaussians. The diffusion experiment is a single seed and is not a proof.

Problem

Iterative retraining on model output is commonly studied in two forms: Replace, which discards earlier data and can degrade without bound, and Accumulate, which keeps all data and degrades in a bounded way §1 §2. Prior theory that explains continued progress assumes new real data can be perfectly separated from generated data §1 §2. Perfect separation is unrealistic, and unfiltered data remains common §1. The paper addresses the Accumulate setting in which real and generated samples are pooled and treated identically §1.

The model is a multivariate Gaussian with mean mu and positive-definite covariance Sigma. The real data are assumed Gaussian §3.1. The paper does not claim the result for neural networks; it uses the Gaussian model to keep the analysis tractable §1.

Method

Setup (Accumulate, mixed batches) §3.1.

  1. Iteration 1: draw n_r i.i.d. real samples and form the empirical mean and the biased (divide by n_r) covariance eq. 1.
  2. Iteration t >= 2: draw n_b(t-1) samples. Each is drawn independently from the mixture with weight alpha_{t-1} on the current fit N(mu^(t-1), Sigma^(t-1)) and weight 1 - alpha_{t-1} on the true N(mu, Sigma) eq. 2. Conditional on the past, these draws are independent of earlier batches §3.1, conditional-independence assumption.
  3. Pool counts: N_t = n_r + sum of n_b(s) for s < t, and rho_t = N_t / N_{t+1} eq. 3. Note rho_t lies in (0, 1).
  4. Refit on the whole pool, with no labels: mu^(t) is the pool mean and Sigma^(t) is the pool's covariance divided by N_t, not N_t - 1 eq. 4.
  5. The fitted model D_t = N(mu^(t), Sigma^(t)) is compared to D_real = N(mu, Sigma) using the 2-Wasserstein distance W_2 eq. 5 §3.1.

Notation for the criterion. delta^(t) = mu^(t) - mu eq. 7. C_t = Cov(mu^(t)) eq. 6. Because mu^(t) is unbiased, C_t = E[delta delta^T] and E||mu^(t) - mu||_2^2 = Tr(C_t) eq. 10 eq. 11.

Results.

  • Bias §3.2.1. E[mu^(t)] = mu eq. 8. E[Sigma^(t)] = Sigma - C_t eq. 9. The covariance estimator is biased downwards by exactly C_t, which is positive semi-definite. At t = 1, C_1 = Sigma / n_r, which gives the usual (n_r - 1)/n_r factor for the maximum-likelihood covariance.
  • Bound on C_t §3.2.1. 0 preceq C_t preceq (2 / n_r) Sigma for all t. The downward bias stays under double its round-1 size, whatever the synthetic share. (Unnumbered lemma in the paper.)
  • Sharp criterion §3.2.2 eq. 12. The following are equivalent: (a) the series sum_t (1 - alpha_t)(1 - rho_t) diverges; (b) E||mu^(t) - mu||_2^2 tends to 0 eq. 13; (c) E||Sigma^(t) - Sigma||F tends to 0 eq. 14; (d) E[W_2^2(D_t, D_real)] tends to 0 eq. 15. The paper notes that 1 - rho_t = n_b(t) / N{t+1} is the weight of batch t in the pool, so the summand is the share of fresh real information that batch t adds §3.2.2.
  • Constant synthetic share below one §3.2.2 eq. 16. If sup_t alpha_t < 1, the series diverges, so consistency holds. This covers any fixed alpha in [0, 1).
  • Vanishing real share §3.2.2. With constant n_b, 1 - rho_t is of order 1/t, so the criterion becomes the sum of (1 - alpha_t)/t. Schedules alpha_t = 1 - 1/t and alpha_t = 1 - 1/sqrt(t) give a convergent sum and therefore no consistency. Schedules alpha_t = 1 - 1/ln(t+2) and alpha_t = 1 - 1/ln ln(t+16) give a divergent sum and consistency. All four have sup_t alpha_t = 1, so the sufficient lemma does not apply to them.
  • Pure synthetic §3.2.2 (Proposition, unnumbered). If alpha_t = 1 for all t, the criterion sum is zero. Yet the error stays bounded: Tr(Sigma)/n_r <= E||mu^(t) - mu||_2^2 <= 2 Tr(Sigma)/n_r, and the covariance error is bounded above and below by expressions in Tr(Sigma), ||Sigma||_F and n_r stated in the paper §3.2.2. The mean stays unbiased but inconsistent, and the covariance stays biased.
  • Rate, constant alpha §3.2.3 eq. 17. With alpha_t = alpha and n_b(t) = n_b, and N_t = n_r + (t-1) n_b (order t): E||mu^(t) - mu||_2^2 is of order N_t^-1 if alpha < 1/2; N_t^-1 ln N_t if alpha = 1/2; and N_t^-2(1-alpha) if alpha > 1/2. The constants depend on alpha, n_b, n_r and Tr(Sigma). The paper gives the rate for the mean only; for the covariance it shows numerically that the same transition appears §3.3.
  • Transient rise app. A. For constant alpha, define alpha* from n_r and n_b as the root in [0, 1] of (n_b - 1) a^2 + 2 n_r a - (n_r + n_b) (or (n_r + 1)/(2 n_r) if n_b = 1) app. A. If alpha > alpha*, the error rises at the first new batch; for alpha <= alpha* it does not. For alpha < 1 there is a single peak, and the error never exceeds twice its initial value.
  • Value of perfect detection app. B. With N_oracle = (1 - alpha) N_t + alpha n_r and N_eff = Tr(Sigma) / E||mu^(t) - mu||_2^2, the gain N_oracle / N_eff is of order 1 for alpha < 1/2, ln N_t at alpha = 1/2, and N_t^(2 alpha -
    1. for 1/2 < alpha < 1 app. B eq. 17 context.

Mean recursion and scalar form app. A. The mean updates as mu^(t) = rho_{t-1} mu^(t-1) + (1/N_t) times the sum of the new batch app. A. The covariance of mu^(t) is a scalar multiple of Sigma: C_t = c_t Sigma, with c_1 = 1/n_r and c_{t+1} = gamma_t c_t + beta_t, where lambda_t = (N_t + n_b(t) alpha_t) / N_{t+1}, beta_t = n_b(t) / N_{t+1}^2, and gamma_t = lambda_t^2 - alpha_t^2 beta_t app. A. Trace gives E||mu^(t) - mu||^2 = c_t Tr(Sigma).

Pseudocode
# Iterated estimation with indistinguishable mixed batches  [§3.1] [eq. 2]
# Input:  mu, Sigma (true); n_r; n_b[s]; alphas[s] in [0, 1]; T rounds
# Output: fitted mu_hat, Sigma_hat after round T

def fit(points):
    # Pool mean and covariance divided by N, not N - 1  [eq. 4]
    mean = average over points
    cov = average of (point - mean) outer (point - mean)
    return mean, cov

def run(mu, Sigma, n_r, n_b, alphas, T):
    # Round 1: real samples only  [eq. 1]
    pool = sample_gaussian(mu, Sigma, n_r)
    mu_hat, Sigma_hat = fit(pool)
    for s in range(1, T):
        # Batch s: each sample is drawn from the current fit with
        # probability alphas[s], otherwise from the true law; the
        # labels are dropped  [§3.1] [eq. 2]
        batch = draw_mixture(mu_hat, Sigma_hat, mu, Sigma,
                             n_b[s], alphas[s])
        pool = concatenate(pool, batch)
        # Refit on the whole pool, real and synthetic alike  [eq. 4]
        mu_hat, Sigma_hat = fit(pool)
    return mu_hat, Sigma_hat
Pseudocode
# Partial sum of the sharp criterion for a schedule  [§3.2.2] [eq. 12]
# Input:  n_r; n_b[t] and alphas[t] for t = 1..T (1-indexed)
# Output: partial sum of (1 - alpha_t)(1 - rho_t)

def criterion_partial_sum(n_r, n_b, alphas, T):
    # Pool size N_1 = n_r  [eq. 3]
    pool_size = n_r
    total = 0.0
    for t in range(1, T + 1):
        # Pool size after batch t, N_{t+1} = N_t + n_b(t)  [eq. 3]
        next_size = pool_size + n_b[t]
        # 1 - rho_t = n_b(t) / N_{t+1}; weight by the real share  [§3.2.2]
        total += (1 - alphas[t]) * n_b[t] / next_size
        pool_size = next_size
    return total

Hyperparameters

Theory experiments §3.3:

NameValueWhereNotes
dimension d4§3.3
true mean mu(-1, -1/3, 1/3, 1)§3.3
true covarianceSigma_ij = s_i s_j 2^-abs(i-j), s = (1, 4/3, 5/3, 2)§3.3Tr = 9.56, Frobenius norm 6.18, condition number 9.58
real batch size n_r20§3.3
batch size n_b(t)10§3.3constant
constant alpha grid15 values from 0 to 1§3.3includes 0.45, 0.55, 0.99 and 1
time-varying alpha_t1-1/t, 1-1/sqrt(t), 1-1/ln(t+2), 1-1/ln ln(t+16)§3.3sup = 1 for all
rounds T10^5 (N_T about 10^6)§3.3
Monte Carlo runs M20,000§3.3MSE standard error about 1%
W_2^2 evaluation10,000 runs, 55 log-spaced rounds§3.3closed form for Gaussians
alpha* example0.654 for n_r = 20, n_b = 10app. Aerror at alpha = 1 rises to 1.74 times its start

Diffusion experiment, CIFAR-10 app. D:

NameValueWhereNotes
data split5,000 validation (class-balanced), 45,000 train, 10,000 testapp. Dtest set never used for selection
n_r, n_b5,000 eachapp. Dpool N_t = n_r + (t-1) n_b
rounds t_max9app. Dfinal pool 45,000
alpha values0, 0.25, 0.75, 1app. Done training seed
synthetic per batchalpha n_b generated, (1 - alpha) n_b new realapp. Dfixed counts, not independent draws
diffusion steps1,000; linear beta from 1e-4 to 0.02app. Depsilon-prediction
optimiserAdam, lr 2e-4, 5,000-step linear warm-upapp. D
batch size, clipping128; gradient clip 1app. D
dropout0.1app. D
EMA decay0.9999app. Dsampling uses EMA weights
samplerDDIM, 200 stepsapp. D
training budgetfixed steps; default 200,000; up to 800,000 for alpha in {0, 0.25}app. Dcheckpoint every 50,000 steps
warm startalpha = 0.75, rounds t >= 4, lower learning rateapp. Dthe lower learning rate value is not stated
checkpoint choicelowest FID on validation (10,000 samples)app. D
FIDclean-fid, 10,000 DDIM samples vs test setapp. Dnot comparable to published FIDs
hardware8 NVIDIA RTX 6000 Ada GPUs, about 400 GPU-hoursapp. D
architectureU-Net from a third-party DDPM repositoryapp. Dwidths, depth and attention not stated in the paper

Results

Author-reported. Theory is proved, not measured; the numerical checks are simulations of the exact setup §3.3. Diffusion setup as in app. D.

SettingMetricBaselineThis paperSource
any fixed alpha in [0, 1)consistency (mu, Sigma, W_2)only real data (alpha = 0)consistent§3.2.2 eq. 12 eq. 16
alpha_t = 1 - 1/t or 1 - 1/sqrt(t)consistencynot applicablenot consistent (errors approach a positive limit)§3.2.2 fig. 1
alpha_t = 1 - 1/ln(t+2) or 1 - 1/ln ln(t+16)consistencynot applicableconsistent (errors keep decreasing)§3.2.2 fig. 1
alpha = 1Emu^(t) - mu
constant alpha, mean error rateorder of Emu^(t) - mu
gain from perfect detectionN_oracle / N_eff1 at alpha = 0bounded below 1/2; grows like N^(2 alpha - 1) above; about 10^3 at N = 10^6 for alpha in [0.9, 0.95]app. B fig. 2
CIFAR-10 DDPM, FID vs ttest FIDinitial FID 33.8alpha in {0, 0.25, 0.75} below 33.8 at every round t >= 2; alpha = 1 rises to 39.7§4 fig. 6 app. D
CIFAR-10 DDPM, alpha = 0.25 vs 0.75 at t = 9test FIDalpha = 0 reference 9.7 (45,000 real images)about 13.5 for alpha = 0.25; above 27 for alpha = 0.75app. D

Paper vs code

No official code. The paper does not link any code of its own. Its only code link is a third-party DDPM U-Net repository it cites for the diffusion runs app. D, which is not the authors' code and is not used here. Searches done for this card: the arXiv abstract page (no code link); GitHub repository search for "model collapse gaussian turgeman", "tirer collapse" and "iterative gaussian estimation collapse" (no results); a personal GitHub account matching one author's name (identity not verified; seven repositories on unrelated topics, none linked to the paper); a guessed GitHub login for the other author (no account). The paper has no project page.

Pitfalls

  • The criterion concerns an infinite series. A finite run cannot decide consistency. The convergent and divergent schedules differ only in how fast the real share vanishes, and the divergent ones keep decreasing only slowly §3.3. Do not infer collapse or non-collapse from a short run.
  • Per-sample versus fixed-count draws. The theory draws each new sample from the mixture independently §3.1 eq. 2. The diffusion runs use fixed counts of generated and real images per batch app. D. The paper says the difference is negligible for n_b = 5,000 app. D. A replication that follows the theory should draw per sample.
  • Index convention. The body draws the batch for iteration t with alpha_{t-1} and n_b(t-1) eq. 2. The appendix writes the batch of round s as alpha_s and n_b(s), which enters the pool N_{s+1} eq. 3. Both describe the same process; mixing the two shifts the criterion by one round.
  • Covariance normalisation. The estimator divides by N_t, not N_t - 1 eq. 4. The bias result eq. 9 holds for this divisor. The card's results do not cover the unbiased divisor, and the paper does not analyse it.
  • alpha = 1 is not collapse to zero error. The error is bounded but never shrinks. The bias of the covariance is permanent §3.2.2.
  • Transient increase. For alpha above alpha*, the mean error first rises before it falls (for alpha < 1) app. A. For n_r = 20 and n_b = 10 the threshold is about 0.654. Evaluating only at large t hides this.
  • Rate costs. Halving the error at alpha = 0.9 needs about 32 times more samples, and at alpha = 0.99 about 10^15 times more §3.2.3. The rate in eq. 17 is up to constants that the paper does not compute.
  • Covariance rate is empirical only. The rate is proved for the mean. The covariance follows the same transition in simulation §3.3; the card should not be cited as proof for the covariance.
  • Gaussian model and the diffusion analogy. The theorems hold for the Gaussian model only. The paper argues that FID is a Wasserstein-2 type distance between Gaussian fits to Inception features, which motivates the comparison app. D. That is a heuristic link, not a proof.
  • Diffusion experiment limits. One training seed; differences below about 0.5 FID between consecutive rounds are within run-to-run variation app. D. FIDs use 10,000 samples against the test set, so they are not comparable to published CIFAR-10 numbers app. D. The warm-start learning rate for alpha = 0.75 is not stated app. D. The paper notes that for large alpha below 1 the FID stops improving at some round and attributes this possibly to the simple training strategy and model size §4.

Lineage

  • Builds on: the Replace and Accumulate framing of iterative retraining §1 §2; the Gaussian model used in earlier model-collapse theory (Shumailov et al., Alemohammad et al., Kazdan et al., He et al.) §1 §3.1; the DDPM framework (Ho et al. 2020), DDIM sampling (Song et al. 2021) and clean-fid (Parmar et al. 2022) for the diffusion runs app. D.
  • Contrasts with: no experiment compares estimators. The paper contrasts its setting with work that separates real from generated data and weights them (He et al. 2025, Alemohammad et al. 2024, Garg et al. 2026) and with stability results that hold only locally (Bertrand et al. 2023) or only up to a finite number of rounds (Barzilai et al. 2025) §2.
  • Superseded by: not stated.

Source and license

Adapted from “Model Collapse Is Avoidable Even as Real Data Vanishes: A Sharp Criterion for Iterative Gaussian Estimation” by Roy Turgeman, Tom Tirer (arXiv:2611.00259), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.