wikillm

Concept · 1 paper · updated 2026-11-21

Model collapse

Refitting a model on a pool that mixes real data with its own earlier output can drift away from the truth. For Gaussian mean and covariance, the fit converges if and only if the summed real share of fresh data diverges. A fixed real share above zero suffices; a shrinking share must shrink slowly. Theory from one paper, Gaussian only; a CIFAR-10 diffusion check is one seed.

model-collapsesynthetic-dataself-consuming-traininggaussian-estimationtheory

Markdown5 min read

AI-compiled overview, updated 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

Overview

Model collapse is the drift a model shows when it is repeatedly refit on data that contains its own earlier output. The question is whether real data keeps the fit anchored when real and generated samples are pooled and the model cannot tell them apart. The setting studied here is "Accumulate": earlier output is kept next to new real data, not discarded 2611.00259 §1 2611.00259 §2.

Each round draws a batch in which every sample comes from the current fit with probability α_t and from the true distribution otherwise. Labels are dropped. The whole pool is then refit 2611.00259 §3.1 eq. 2 eq. 4. The paper's central result is a single condition on the real share of each batch. The estimates converge to the truth if and only if the sum of (1 - α_t)(1 - ρ_t) diverges 2611.00259 §3.2.2 eq. 12. Here 1 - ρ_t is the weight of batch t in the pool, so each summand is the share of fresh real information that batch adds.

The analysis covers Gaussian models only, where the real data are Gaussian and the fit is a mean and a covariance 2611.00259 §1 §3.1. The paper does not claim the result for neural networks. Its diffusion check on CIFAR-10 is a single training seed and agrees only qualitatively with the theory §4 app. D. Treat it as a guide to the shape of the answer, not a proof for a large generative model.

Variants

These are regimes of the synthetic share, not competing methods. The criterion decides which regime is consistent.

Fixed synthetic share below one

A constant α in [0, 1) keeps the sum divergent, so the mean, the covariance and the 2-Wasserstein distance to the truth all go to zero 2611.00259 §3.2.2 eq. 16. The real-only case, α = 0, is the baseline. For the mean, the error falls as N^-1 when α is below 1/2, as N^-1 ln N at α = 1/2, and as N^-2(1 - α) above it, where N is the pool size 2611.00259 §3.2.3 eq. 17. The paper shows the same transition for the covariance in simulation only §3.3. Card: Model Collapse Is Avoidable.

Vanishing real share, logarithmic schedule

With a constant batch size, 1 - ρ_t is of order 1/t, so the criterion reduces to the sum of (1 - α_t)/t. The schedules α_t = 1 - 1/ln(t + 2) and α_t = 1 - 1/ln ln(t + 16) give a divergent sum, and the fit is consistent. The errors keep falling, but slowly 2611.00259 §3.2.2 §3.3. Card: Model Collapse Is Avoidable.

Vanishing real share, polynomial schedule

The schedules α_t = 1 - 1/t and α_t = 1 - 1/√t give a convergent sum. The fit is not consistent, and the errors approach a positive limit 2611.00259 §3.2.2 fig. 1. Card: Model Collapse Is Avoidable.

Purely synthetic rounds after the first

If α_t = 1 for every round after the first, the criterion sum is zero. The mean error nonetheless stays between Tr(Σ)/n_r and 2 Tr(Σ)/n_r and does not shrink. The mean stays unbiased but inconsistent. The covariance keeps a permanent downward bias 2611.00259 §3.2.2. The paper also finds that for a constant α above a threshold α*, the error first rises at the first new batch before it falls. For its example, n_r = 20 and n_b = 10, α* is about 0.654 2611.00259 app. A. Card: Model Collapse Is Avoidable.

When to use which

SituationRegimeWhat the cited results support
A fixed share of real data can be kept each roundConstant α < 1Consistency for any α in [0, 1) 2611.00259 §3.2.2 eq. 16. The rate exponent is 1 below α = 1/2 and falls toward 0 as α approaches 1 §3.2.3 eq. 17
Real data is only available in the first roundα_t = 1 after round 1Error stays bounded away from zero, between Tr(Σ)/n_r and 2 Tr(Σ)/n_r §3.2.2
The real share must shrink over roundsShrink it logarithmically, not as 1/t or 1/√tOnly the logarithmic schedules give a divergent sum §3.2.2 §3.3
Synthetic share is above one halfExpect a slower rate and possibly a transient rise in errorRate N^-2(1 - α) §3.2.3; transient rise above α* app. A
Generated samples can be reliably identified and set asideFilter, then judge by the gainThe paper's gain from perfect detection is of order 1 below α = 1/2 and grows above it app. B
The covariance estimate must be reportedCorrect for the known downward biasE[Σ^(t)] = Σ - C_t, and C_t never exceeds (2/n_r) Σ §3.2.1

Real samples still matter for speed even when the limit is the same. The convergence constants depend on n_r, so a smaller real set means slower convergence §3.2.3. Curating real data therefore still pays, even under a consistent schedule.

Implementation sketch

The sketch follows the paper's Gaussian setup §3.1. The statistics are averaged over the pool and divided by its size, not by its size minus one eq. 4. The criterion is a check on the schedule, not on the fit.

Pseudocode
# Iterated refit on a mixed pool  [§3.1] [eq. 2] [eq. 4]
# Input:  true mu, Sigma; n_r real samples; n_b[t] per round; alpha[t] in [0, 1]
# Output: pool and final fit

def refit_loop(mu, Sigma, n_r, n_b, alpha, T):
    # Round 1: real samples only  [eq. 1]
    pool = draw_gaussian(mu, Sigma, n_r)
    fit_mu, fit_cov = mean_and_cov_over_n(pool)
    for t in range(2, T + 1):
        # Each new sample: current fit with prob. alpha[t-1], else true law
        # Labels are dropped, so the pool is not split by source
        batch = []
        for _ in range(n_b[t - 1]):
            if random() < alpha[t - 1]:
                batch.append(draw_gaussian(fit_mu, fit_cov, 1))
            else:
                batch.append(draw_gaussian(mu, Sigma, 1))
        pool = concatenate(pool, batch)
        # Refit on the whole pool, dividing by N, not N - 1  [eq. 4]
        fit_mu, fit_cov = mean_and_cov_over_n(pool)
    return fit_mu, fit_cov


# Partial sum of the criterion for a schedule  [§3.2.2] [eq. 12]
def criterion_partial_sum(n_r, n_b, alpha, T):
    pool_size = n_r
    total = 0.0
    for t in range(1, T + 1):
        next_size = pool_size + n_b[t]
        # Real share added by batch t is n_b[t] / next_size  [§3.2.2]
        total += (1 - alpha[t]) * n_b[t] / next_size
        pool_size = next_size
    return total

A replication that follows the theory should draw each sample independently from the mixture. The diffusion runs used fixed counts per batch instead app. D. The index convention also differs between the body and the appendix §3.1 app. A, so check the shift before comparing a schedule to the paper.

Contradictions

None known among the cited results. The paper argues against earlier analyses that assume new real data can be cleanly separated from generated data, and against results that hold only locally or for a finite number of rounds 2611.00259 §2. That is a difference of setting, not a conflict of numbers. The paper also cites earlier work that finds Replace can degrade without bound 2611.00259 §1. Its own analysis does not cover Replace.

Papers

  • Model Collapse Is Avoidable: the sum-divergence criterion for consistency of iterated mean and covariance estimation with indistinguishable real and synthetic samples, plus the rates and a CIFAR-10 diffusion check.

Source and license

Compiled from the papers listed on this page, all published under CC BY, CC BY-SA or CC0, with citations. See each paper's page for its authors and license.