Concept · 1 paper · updated 2026-11-21
Model collapse
Refitting a model on a pool that mixes real data with its own earlier output can drift away from the truth. For Gaussian mean and covariance, the fit converges if and only if the summed real share of fresh data diverges. A fixed real share above zero suffices; a shrinking share must shrink slowly. Theory from one paper, Gaussian only; a CIFAR-10 diffusion check is one seed.
model-collapsesynthetic-dataself-consuming-traininggaussian-estimationtheory
Markdown5 min read
AI-compiled overview, updated 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
Overview
Model collapse is the drift a model shows when it is repeatedly refit on data that contains its own earlier output. The question is whether real data keeps the fit anchored when real and generated samples are pooled and the model cannot tell them apart. The setting studied here is "Accumulate": earlier output is kept next to new real data, not discarded 2611.00259 §1 2611.00259 §2.
Each round draws a batch in which every sample comes from the current fit with probability α_t and from the true distribution otherwise. Labels are dropped. The whole pool is then refit 2611.00259 §3.1 eq. 2 eq. 4. The paper's central result is a single condition on the real share of each batch. The estimates converge to the truth if and only if the sum of (1 - α_t)(1 - ρ_t) diverges 2611.00259 §3.2.2 eq. 12. Here 1 - ρ_t is the weight of batch t in the pool, so each summand is the share of fresh real information that batch adds.
The analysis covers Gaussian models only, where the real data are Gaussian and the fit is a mean and a covariance 2611.00259 §1 §3.1. The paper does not claim the result for neural networks. Its diffusion check on CIFAR-10 is a single training seed and agrees only qualitatively with the theory §4 app. D. Treat it as a guide to the shape of the answer, not a proof for a large generative model.
Variants
These are regimes of the synthetic share, not competing methods. The criterion decides which regime is consistent.
Fixed synthetic share below one
A constant α in [0, 1) keeps the sum divergent, so the mean, the covariance and the 2-Wasserstein distance to the truth all go to zero 2611.00259 §3.2.2 eq. 16. The real-only case, α = 0, is the baseline. For the mean, the error falls as N^-1 when α is below 1/2, as N^-1 ln N at α = 1/2, and as N^-2(1 - α) above it, where N is the pool size 2611.00259 §3.2.3 eq. 17. The paper shows the same transition for the covariance in simulation only §3.3. Card: Model Collapse Is Avoidable.
Vanishing real share, logarithmic schedule
With a constant batch size, 1 - ρ_t is of order 1/t, so the criterion reduces to the sum of (1 - α_t)/t. The schedules α_t = 1 - 1/ln(t + 2) and α_t = 1 - 1/ln ln(t + 16) give a divergent sum, and the fit is consistent. The errors keep falling, but slowly 2611.00259 §3.2.2 §3.3. Card: Model Collapse Is Avoidable.
Vanishing real share, polynomial schedule
The schedules α_t = 1 - 1/t and α_t = 1 - 1/√t give a convergent sum. The fit is not consistent, and the errors approach a positive limit 2611.00259 §3.2.2 fig. 1. Card: Model Collapse Is Avoidable.
Purely synthetic rounds after the first
If α_t = 1 for every round after the first, the criterion sum is zero. The mean error nonetheless stays between Tr(Σ)/n_r and 2 Tr(Σ)/n_r and does not shrink. The mean stays unbiased but inconsistent. The covariance keeps a permanent downward bias 2611.00259 §3.2.2. The paper also finds that for a constant α above a threshold α*, the error first rises at the first new batch before it falls. For its example, n_r = 20 and n_b = 10, α* is about 0.654 2611.00259 app. A. Card: Model Collapse Is Avoidable.
When to use which
| Situation | Regime | What the cited results support |
|---|---|---|
| A fixed share of real data can be kept each round | Constant α < 1 | Consistency for any α in [0, 1) 2611.00259 §3.2.2 eq. 16. The rate exponent is 1 below α = 1/2 and falls toward 0 as α approaches 1 §3.2.3 eq. 17 |
| Real data is only available in the first round | α_t = 1 after round 1 | Error stays bounded away from zero, between Tr(Σ)/n_r and 2 Tr(Σ)/n_r §3.2.2 |
| The real share must shrink over rounds | Shrink it logarithmically, not as 1/t or 1/√t | Only the logarithmic schedules give a divergent sum §3.2.2 §3.3 |
| Synthetic share is above one half | Expect a slower rate and possibly a transient rise in error | Rate N^-2(1 - α) §3.2.3; transient rise above α* app. A |
| Generated samples can be reliably identified and set aside | Filter, then judge by the gain | The paper's gain from perfect detection is of order 1 below α = 1/2 and grows above it app. B |
| The covariance estimate must be reported | Correct for the known downward bias | E[Σ^(t)] = Σ - C_t, and C_t never exceeds (2/n_r) Σ §3.2.1 |
Real samples still matter for speed even when the limit is the same. The convergence constants depend on n_r, so a smaller real set means slower convergence §3.2.3. Curating real data therefore still pays, even under a consistent schedule.
Implementation sketch
The sketch follows the paper's Gaussian setup §3.1. The statistics are averaged over the pool and divided by its size, not by its size minus one eq. 4. The criterion is a check on the schedule, not on the fit.
# Iterated refit on a mixed pool [§3.1] [eq. 2] [eq. 4]
# Input: true mu, Sigma; n_r real samples; n_b[t] per round; alpha[t] in [0, 1]
# Output: pool and final fit
def refit_loop(mu, Sigma, n_r, n_b, alpha, T):
# Round 1: real samples only [eq. 1]
pool = draw_gaussian(mu, Sigma, n_r)
fit_mu, fit_cov = mean_and_cov_over_n(pool)
for t in range(2, T + 1):
# Each new sample: current fit with prob. alpha[t-1], else true law
# Labels are dropped, so the pool is not split by source
batch = []
for _ in range(n_b[t - 1]):
if random() < alpha[t - 1]:
batch.append(draw_gaussian(fit_mu, fit_cov, 1))
else:
batch.append(draw_gaussian(mu, Sigma, 1))
pool = concatenate(pool, batch)
# Refit on the whole pool, dividing by N, not N - 1 [eq. 4]
fit_mu, fit_cov = mean_and_cov_over_n(pool)
return fit_mu, fit_cov
# Partial sum of the criterion for a schedule [§3.2.2] [eq. 12]
def criterion_partial_sum(n_r, n_b, alpha, T):
pool_size = n_r
total = 0.0
for t in range(1, T + 1):
next_size = pool_size + n_b[t]
# Real share added by batch t is n_b[t] / next_size [§3.2.2]
total += (1 - alpha[t]) * n_b[t] / next_size
pool_size = next_size
return totalA replication that follows the theory should draw each sample independently from the mixture. The diffusion runs used fixed counts per batch instead app. D. The index convention also differs between the body and the appendix §3.1 app. A, so check the shift before comparing a schedule to the paper.
Contradictions
None known among the cited results. The paper argues against earlier analyses that assume new real data can be cleanly separated from generated data, and against results that hold only locally or for a finite number of rounds 2611.00259 §2. That is a difference of setting, not a conflict of numbers. The paper also cites earlier work that finds Replace can degrade without bound 2611.00259 §1. Its own analysis does not cover Replace.
Papers
- Model Collapse Is Avoidable: the sum-divergence criterion for consistency of iterated mean and covariance estimation with indistinguishable real and synthetic samples, plus the rates and a CIFAR-10 diffusion check.
Source and license
Compiled from the papers listed on this page, all published under CC BY, CC BY-SA or CC0, with citations. See each paper's page for its authors and license.