---
kind: concept
slug: model-collapse
title: Model collapse
summary: >-
  Refitting a model on a pool that mixes real data with its own earlier output can drift away from the truth. For Gaussian mean and covariance, the fit converges if and only if the summed real share of fresh data diverges. A fixed real share above zero suffices; a shrinking share must shrink slowly. Theory from one paper, Gaussian only; a CIFAR-10 diffusion check is one seed.
tags: [model-collapse, synthetic-data, self-consuming-training, gaussian-estimation, theory]
papers: ["2611.00259"]
related: [synthetic-training-data, self-training, dataset-curation, diffusion-models, frechet-inception-distance]
status: draft
updated_at: 2026-11-21
---

## Overview

Model collapse is the drift a model shows when it is repeatedly refit on data
that contains its own earlier output. The question is whether real data keeps
the fit anchored when real and generated samples are pooled and the model
cannot tell them apart. The setting studied here is "Accumulate": earlier
output is kept next to new real data, not discarded [arxiv:2611.00259 §1]
[arxiv:2611.00259 §2].

Each round draws a batch in which every sample comes from the current fit with
probability α_t and from the true distribution otherwise. Labels are dropped.
The whole pool is then refit [arxiv:2611.00259 §3.1] [eq. 2] [eq. 4]. The
paper's central result is a single condition on the real share of each batch.
The estimates converge to the truth if and only if the sum of
(1 - α_t)(1 - ρ_t) diverges [arxiv:2611.00259 §3.2.2] [eq. 12]. Here 1 - ρ_t is
the weight of batch t in the pool, so each summand is the share of fresh real
information that batch adds.

The analysis covers Gaussian models only, where the real data are Gaussian and
the fit is a mean and a covariance [arxiv:2611.00259 §1] [§3.1]. The paper does
not claim the result for neural networks. Its diffusion check on CIFAR-10 is a
single training seed and agrees only qualitatively with the theory [§4]
[app. D]. Treat it as a guide to the shape of the answer, not a proof for a
large generative model.

## Variants

These are regimes of the synthetic share, not competing methods. The criterion
decides which regime is consistent.

### Fixed synthetic share below one

A constant α in [0, 1) keeps the sum divergent, so the mean, the covariance
and the 2-Wasserstein distance to the truth all go to zero
[arxiv:2611.00259 §3.2.2] [eq. 16]. The real-only case, α = 0, is the baseline.
For the mean, the error falls as N^-1 when α is below 1/2, as N^-1 ln N at
α = 1/2, and as N^-2(1 - α) above it, where N is the pool size
[arxiv:2611.00259 §3.2.3] [eq. 17]. The paper shows the same transition for
the covariance in simulation only [§3.3]. Card: [Model Collapse Is Avoidable](/papers/2611.00259).

### Vanishing real share, logarithmic schedule

With a constant batch size, 1 - ρ_t is of order 1/t, so the criterion reduces
to the sum of (1 - α_t)/t. The schedules α_t = 1 - 1/ln(t + 2) and
α_t = 1 - 1/ln ln(t + 16) give a divergent sum, and the fit is consistent. The
errors keep falling, but slowly [arxiv:2611.00259 §3.2.2] [§3.3]. Card:
[Model Collapse Is Avoidable](/papers/2611.00259).

### Vanishing real share, polynomial schedule

The schedules α_t = 1 - 1/t and α_t = 1 - 1/√t give a convergent sum. The fit
is not consistent, and the errors approach a positive limit
[arxiv:2611.00259 §3.2.2] [fig. 1]. Card:
[Model Collapse Is Avoidable](/papers/2611.00259).

### Purely synthetic rounds after the first

If α_t = 1 for every round after the first, the criterion sum is zero. The
mean error nonetheless stays between Tr(Σ)/n_r and 2 Tr(Σ)/n_r and does not
shrink. The mean stays unbiased but inconsistent. The covariance keeps a
permanent downward bias [arxiv:2611.00259 §3.2.2]. The paper also finds that
for a constant α above a threshold α*, the error first rises at the first new
batch before it falls. For its example, n_r = 20 and n_b = 10, α* is about
0.654 [arxiv:2611.00259 app. A]. Card:
[Model Collapse Is Avoidable](/papers/2611.00259).

## When to use which

| Situation | Regime | What the cited results support |
|---|---|---|
| A fixed share of real data can be kept each round | Constant α < 1 | Consistency for any α in [0, 1) [arxiv:2611.00259 §3.2.2] [eq. 16]. The rate exponent is 1 below α = 1/2 and falls toward 0 as α approaches 1 [§3.2.3] [eq. 17] |
| Real data is only available in the first round | α_t = 1 after round 1 | Error stays bounded away from zero, between Tr(Σ)/n_r and 2 Tr(Σ)/n_r [§3.2.2] |
| The real share must shrink over rounds | Shrink it logarithmically, not as 1/t or 1/√t | Only the logarithmic schedules give a divergent sum [§3.2.2] [§3.3] |
| Synthetic share is above one half | Expect a slower rate and possibly a transient rise in error | Rate N^-2(1 - α) [§3.2.3]; transient rise above α* [app. A] |
| Generated samples can be reliably identified and set aside | Filter, then judge by the gain | The paper's gain from perfect detection is of order 1 below α = 1/2 and grows above it [app. B] |
| The covariance estimate must be reported | Correct for the known downward bias | E[Σ^(t)] = Σ - C_t, and C_t never exceeds (2/n_r) Σ [§3.2.1] |

Real samples still matter for speed even when the limit is the same. The
convergence constants depend on n_r, so a smaller real set means slower
convergence [§3.2.3]. Curating real data therefore still pays, even under a
consistent schedule.

## Implementation sketch

The sketch follows the paper's Gaussian setup [§3.1]. The statistics are
averaged over the pool and divided by its size, not by its size minus one
[eq. 4]. The criterion is a check on the schedule, not on the fit.

```pseudocode
# Iterated refit on a mixed pool  [§3.1] [eq. 2] [eq. 4]
# Input:  true mu, Sigma; n_r real samples; n_b[t] per round; alpha[t] in [0, 1]
# Output: pool and final fit

def refit_loop(mu, Sigma, n_r, n_b, alpha, T):
    # Round 1: real samples only  [eq. 1]
    pool = draw_gaussian(mu, Sigma, n_r)
    fit_mu, fit_cov = mean_and_cov_over_n(pool)
    for t in range(2, T + 1):
        # Each new sample: current fit with prob. alpha[t-1], else true law
        # Labels are dropped, so the pool is not split by source
        batch = []
        for _ in range(n_b[t - 1]):
            if random() < alpha[t - 1]:
                batch.append(draw_gaussian(fit_mu, fit_cov, 1))
            else:
                batch.append(draw_gaussian(mu, Sigma, 1))
        pool = concatenate(pool, batch)
        # Refit on the whole pool, dividing by N, not N - 1  [eq. 4]
        fit_mu, fit_cov = mean_and_cov_over_n(pool)
    return fit_mu, fit_cov


# Partial sum of the criterion for a schedule  [§3.2.2] [eq. 12]
def criterion_partial_sum(n_r, n_b, alpha, T):
    pool_size = n_r
    total = 0.0
    for t in range(1, T + 1):
        next_size = pool_size + n_b[t]
        # Real share added by batch t is n_b[t] / next_size  [§3.2.2]
        total += (1 - alpha[t]) * n_b[t] / next_size
        pool_size = next_size
    return total
```

A replication that follows the theory should draw each sample independently
from the mixture. The diffusion runs used fixed counts per batch instead
[app. D]. The index convention also differs between the body and the appendix
[§3.1] [app. A], so check the shift before comparing a schedule to the paper.

## Contradictions

None known among the cited results. The paper argues against earlier
analyses that assume new real data can be cleanly separated from generated
data, and against results that hold only locally or for a finite number of
rounds [arxiv:2611.00259 §2]. That is a difference of setting, not a conflict
of numbers. The paper also cites earlier work that finds Replace can degrade
without bound [arxiv:2611.00259 §1]. Its own analysis does not cover Replace.

## Papers

- [Model Collapse Is Avoidable](/papers/2611.00259): the sum-divergence criterion for consistency of iterated mean and covariance estimation with indistinguishable real and synthetic samples, plus the rates and a CIFAR-10 diffusion check.
