Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training

In plain words

A deep network is hard to analyse because all of its layers interact in a tangled way. This paper takes a shortcut: it keeps the final layer, which turns features into class scores, and treats everything below it as a set of free feature vectors, one per training example, subject to a size limit. This simpler model is still a faithful stand-in for well-trained networks. On balanced data it predicts a tidy geometry: each class's features shrink to a single point, and these points spread out evenly. On imbalanced data it predicts something new. When one group of classes has far more examples than another, the classifier vectors for the rare classes merge into one, so the network can no longer tell those classes apart. The authors check this against small image networks, and they test one common fix, copying the rare examples more often.

TL;DR

• The Layer-Peeled Model replaces all layers below the last one with per-example feature variables under norm budgets §2.
• With balanced classes and cross-entropy loss, every global minimiser has zero within-class feature spread and a simplex equiangular tight frame (ETF) geometry §3.1.
• With two class sizes, the classifiers of the minority classes become identical as the imbalance ratio goes to infinity. At finite ratios a threshold appears that is seen numerically §4.2.
• Small VGG and ResNet runs on FashionMNIST and CIFAR10 show the same merging §4.3. Oversampling raises minority test accuracy when the best of four rates is chosen tab. 2, and the paper finds that the largest rate tested hurts tab. 2 caption.

Problem

Training a deep network minimises a weight-decayed loss over all layers at once, which is nonconvex and hard to analyse §1. Neural collapse, a pattern reported for well-trained networks on balanced data, has no derivation from the full network §1.1. Imbalanced training is common, and its effect on the classifier of rare classes was, according to the authors, not understood from theory §4. The paper asks whether a tractable surrogate that keeps only the top layer can explain the first and predict the second §1.

Method

Surrogate model. Start from the usual objective: average cross-entropy over examples plus (λ/2)·‖W‖², where λ is the weight decay §1. Single out the last layer W, and bound the norm of the rest of the network instead of optimising it §2. The bias terms are dropped for simplicity §2. The paper says this step is a heuristic that lacks rigour §2. An ellipsoid ansatz is then assumed for the set of reachable features, which turns the constraint on the lower layers into a plain ℓ2 budget on the feature vectors §2. The resulting Layer-Peeled Model (LPM) has, for each training example, a free feature vector θ_{k,i} and a shared classifier W with K rows §2:

\min_{W,\Theta}\ \frac{1}{N}\sum_{k=1}^{K}\sum_{i=1}^{n_k} \mathcal{L}\big(W\theta_{k,i},\,y_k\big) \quad\text{s.t.}\quad \frac{1}{K}\sum_{k=1}^{K}\|w_k\|^2\le E_W,\quad \frac{1}{K}\sum_{k=1}^{K}\frac{1}{n_k}\sum_{i=1}^{n_k}\|\theta_{k,i}\|^2\le E_\Theta §2

Here n_k is the number of examples of class k, N the total, and E_W and E_Θ are positive budgets §2. The problem is nonconvex §2.

Simplex ETF. A K-point simplex ETF in R^p is the set of columns of M = sqrt(K/(K−1)) · P · (I_K − (1/K)·1·1ᵀ), where P ∈ R^{p×K} has orthonormal columns and p ≥ K (p ≥ K−1 suffices) §1, ETF definition. Its pairwise cosine is −1/(K−1) §3.1.

Balanced case, cross-entropy §3.1. Assume p ≥ K−1 §3. Every global minimiser (W*, H*) of the LPM with cross-entropy satisfies θ*_{k,i} = C·w*_k = C'·m*_k for all k and i, where C = sqrt(E_Θ/E_W), C' = sqrt(E_Θ), and the m*_k form a simplex ETF §3.1. So within-class features coincide, class means form a simplex ETF, classifiers align with class means, and the prediction is the nearest class mean §3.1. The minimisers are equivalent up to rotation §3.1, remark. This matches the four neural-collapse properties NC1 to NC4 reported by Papyan et al. §1.2.

Other losses (balanced case) §3.2.

• Supervised contrastive loss with temperature τ > 0 §3.2: every global minimiser of the LPM with it has h*_{k,i} = sqrt(E_Θ)·m*_k, a simplex ETF, so collapse holds even though the label is only used implicitly §3.2.
• A family of softmax losses, with g1 nonincreasing and convex and g2 nondecreasing and convex, on (0,1) §3.2. Cross-entropy is g1 = −log, g2 = 0. Another example is g1(x) = (1−x)^q and g2(x) = x^q for q > 1 §3.2. Under the condition sqrt(E_Θ·E_W) > ((K−1)/K)·log(K²·sqrt(E_Θ·E_W) + (2K−1)(K−1)), the collapsed configuration is a global minimiser; if g2 is strictly convex and g1 or g2 is strictly monotone, it is the only one §3.2. The paper leaves experimental checks of these losses to future work §3.2.

Convex relaxation for imbalance §4.1. Let μ_k be the mean feature of class k. Define the Gram matrix of the stacked class-mean features and classifiers, X = [θ_1, …, θ_K, Wᵀ]ᵀ·[θ_1, …, θ_K, Wᵀ] ∈ R^{2K×2K} §4.1. Then X is positive semidefinite, and the budgets become bounds on the average of diagonal entries (the feature bound uses Cauchy–Schwarz) §4.1. The relaxed problem is:

\min_{X\in\mathbb{R}^{2K\times 2K}}\ \sum_{k=1}^{K}\frac{n_k}{N}\, \mathcal{L}(z_k,y_k) \quad\text{s.t.}\quad X\succeq 0,\quad \frac{1}{K}\sum_{k=1}^{K}X(k,k)\le E_\Theta,\quad \frac{1}{K}\sum_{k=K+1}^{2K}X(k,k)\le E_W, with z_k = [X(k,K+1), …, X(k,2K)]ᵀ §4.1. Lemma 1 §4.1: if the loss is convex in its first argument and p ≥ 2K, any minimiser X* gives a minimiser of the LPM through a factorisation of X*. Here the features of each class are equal (θ_{k,i} = θ_k), and all minimisers have this form if the feature budget is met with equality §4.1. The size of the program does not depend on the number of training examples §4.1. The paper says that the saturation condition holds for cross-entropy and proves it in its appendix §4.1, SI. A nuclear-norm relaxation is given in the SI [SI, "More Discussions on Convex Relaxation"].

Minority Collapse §4.2. Consider K_A majority classes with n_A examples each and K_B minority classes with n_B examples each, and set the imbalance ratio R = n_A/n_B > 1 §4.2. Solving the relaxed program numerically gives a two-phase picture fig. 3. Below a threshold R_0, the average cosine between minority classifiers falls as R grows. Above it, the cosines become 1 and the minority classifiers have equal length, so they collapse to one vector [§4.2, fig. 3]. R_0 depends on K_A, K_B, E_W and E_Θ §4.2. The paper states that R_0 gets smaller when E_W or E_Θ gets smaller, or when K_B gets smaller §4.2. The text writes "E_H" for the second budget; the only budget in the model is E_Θ. The formal theorem is asymptotic: for any global minimiser of the cross-entropy LPM, the difference between two minority classifiers goes to zero as R → ∞ [§4.2, theorem]. The mechanism is that the majority classes dominate the risk, which spreads the majority classifiers and squeezes the minority ones together §4.2. A consequence is that the network predicts equal probabilities for all minority classes, so it cannot fit those training examples with argmax §4.2.

Experiments with deep networks §4.3. Architectures: VGG (with its dropout replaced by batch normalisation) and ResNet18, on FashionMNIST and CIFAR10 §4.3. Class splits (K_A, K_B) ∈ {(3,7), (5,5), (7,3)} §4.3. Majority classes keep all 6000 (FashionMNIST) or 5000 (CIFAR10) training examples of their class; minority classes keep 6000/R or 5000/R examples sampled at random §4.3. The weight decay is λ = 5·10⁻⁴ and the training recipe follows the neural-collapse setup of Papyan et al., with the learning rate fixed at 0.1 §4.3. Measured quantity: average cosine between minority classifiers, as in fig. 4 §4.3. The classifier lengths are compared with the variability measure in tab. 1 §4.3. Figure 2 compares the LPM prediction for the phase threshold with VGG13 on CIFAR10 under two weight decay values §1, fig. 2.

Oversampling §5. Duplicating each minority example w_r times (a positive integer) and keeping the penalty leaves the objective §5:

\frac{1}{\bar N_A K_A+w_r\bar N_B K_B} \Big[\sum_{k\le K_A}\sum_{i\le \bar N_A}\mathcal{L}(\cdot) + w_r\sum_{k>K_A}\sum_{i\le \bar N_B}\mathcal{L}(\cdot)\Big] §5

Within the LPM this acts as if the minority classes had w_r·n_B examples §5. When w_r = R, all pairwise classifier angles become equal §5. In the deep networks at R = 1000, larger oversampling rates enlarge the angles between minority classifiers fig. 6. The SI argues that duplicating examples can have much smaller update variance than reweighting the loss [SI, "Comparison of Oversampling and Weighted Adjusting"].

Other norms §6. For K ≥ 3 and p ≥ K, replacing the ℓ2 feature budget by an ℓq budget with q ∈ (0,2) ∪ (2,∞) stops neural collapse from appearing §6. The paper uses this as partial support for the ℓ2 ansatz §6.

Pseudocode 1: relaxed program (for a solver). Written from §4.1; the program is convex only after the relaxation, and it is solved with a generic SDP solver.

# Relaxed Layer-Peeled Model as a semidefinite program  [§4.1]
# Input:  class sizes n (length K), budgets E_theta and E_w, convex loss L
# Output: Gram matrix X of shape (2K, 2K)
def solve_relaxed_lpm(n, E_theta, E_w, loss):
    # Rows and columns 0..K-1 index features, K..2K-1 index classifiers  [§4.1]
    X = psd_variable(2 * K, 2 * K)               # (2K, 2K), X >= 0 in PSD sense
    # Average of the feature diagonal is at most E_theta  [§4.1]
    constraints = [mean(X[k, k] for k in range(K)) <= E_theta]
    # Average of the classifier diagonal is at most E_w  [§4.1]
    constraints.append(mean(X[k, k] for k in range(K, 2 * K)) <= E_w)
    # Each class k scores the classifier block of its own row  [§4.1]
    objective = 0
    for k in range(K):
        z = X[k, K:2 * K]                        # (K,) logits for class k
        # Weight each class loss by n_k / N  [§4.1]
        objective = objective + (n[k] / sum(n)) * loss(z, k)
    return minimise(objective, constraints + [X >> 0])

Pseudocode 2: oversampling by duplication. Written from §5 and the duplication step in the public code utils.py.

# Epoch index list with minority examples repeated  [§5]
# Input:  number of examples, boolean minority mask, integer rate w_r >= 1
# Output: shuffled indices for one epoch; batches of 128 follow [§4.3]
def oversampled_epoch_indices(num_examples, is_minority, w_r):
    # Every example appears once  [§5]
    indices = list(range(num_examples))
    # Minority examples get w_r - 1 extra copies  [§5]
    for i in range(num_examples):
        if is_minority[i]:
            indices += [i] * (w_r - 1)
    # Random order for the epoch  [§4.3]
    shuffle(indices)
    return indices

Pseudocode 3: minority-collapse metric. The measurement in fig. 4 and minority_collapse.py.

# Average pairwise cosine between minority classifiers  [§4.3, fig. 4]
# Input:  classifier matrix W of shape (K, p); indices of minority classes
# Output: scalar in [-1, 1]; 1 means full minority collapse
def minority_cosine(W, minority):
    # Keep only the minority rows  [§4.3]
    rows = W[minority]                           # (K_B, p)
    # Normalise each row to unit length  [§4.3]
    unit = rows / norm(rows, axis=1)             # (K_B, p)
    # Average the off-diagonal entries of the Gram matrix  [§4.3]
    gram = unit @ unit.T                         # (K_B, K_B)
    return mean(off_diagonal(gram))

Hyperparameters

Results

Author-reported. Setup: the networks and data in §4.3; the LPM simulations in fig. 3; the oversampling runs at R = 1000 on FashionMNIST tab. 2. The paper gives no error bars or number of runs.

Balanced case. The result is analytic, so there is no number to report. The claim is that minimisers of the LPM show neural collapse §3.1.

Minority Collapse in the LPM. The cosine between minority classifiers reaches 1 above a threshold ratio that depends on the budgets and class sizes fig. 3. The paper also reports that the threshold is in good agreement with experiments §1, fig. 2.

Minority Collapse in trained networks fig. 4 tab. 1. The average cosine between minority classifiers rises towards 1 as R grows. With K_A + K_B = 10 and ResNet18, the cosine reaches 1 at R = 5000 for K_A = 7 on CIFAR10 and at R = 3000 on FashionMNIST fig. 4 caption. The lengths of minority classifiers are almost equal at R = ∞ tab. 1.

Values are std/mean of minority classifier lengths at R = ∞ tab. 1.

Test accuracy on minority classes fig. 5. At R = 1000 the minority-class test accuracy is in some runs below 10 %, since minority test examples are assigned to majority classes §4.3, fig. 5 caption. The paper says that this is worse than at R = 1 §4.3.

Oversampling at R = 1000, FashionMNIST tab. 2. Test accuracy in %, best over w_r ∈ {1, 10, 100, 1000}:

The paper notes that the best rate was never 1000, so a very large rate hurts test accuracy tab. 2 caption. The overall gain is smaller for K_A = 7 tab. 2.

Pretrained ImageNet models [SI, "Additional Experimental Results"]. The authors report that the official PyTorch VGG19 and ResNet152 classifiers do not form a simplex ETF. The minimum and maximum between-class angles are 43° and 103° for VGG19, and 37° and 102° for ResNet152 [SI].

Paper vs code

Official code: https://github.com/HornHehhf/LPM, linked in the paper's footnote §4.3, at commit c8b5bb194a6d80bd3cfbddeec7fb183b4b16a116. The README names this paper and cites it README.md. The owner handle (HornHehhf) appears to match author Hangfeng He, but no author statement says so; the paper's own link is the basis for using it. The repository has no LICENSE file, so code_license is recorded as none. Only pseudocode and citations from it are used here.

The paper's footnote gave the link, so no further code search was needed.

Discrepancies found:

• Minority sample size rounding. Paper: 5000/R or 6000/R §4.3. Code uses integer division, so R = 5000 gives one example per minority class on CIFAR10 train_models.py.
• Weight decay values. Figure 2 uses λ = 5·10⁻³ and 5·10⁻⁴ §1, fig. 2. The shipped scripts hard-code 5·10⁻⁴, with no option to change it train_models.py.
• Figure 3 settings. The paper plots E_Θ = 5 and 10 fig. 3. The script runs only E_Θ = 5 and E_W = 1 over R from 1 to 10⁴ optimization.py.
• Feature centring for analysis. The class-mean features are computed after subtracting the global feature mean [code: minority_collapse.py#L48-L50, #L65]. The uncommented text does not state this step.
• Classifier analysis ignores biases. Only weight rows are compared minority_collapse.py. The paper omits biases in the model §2.
• Weight decay scope. The paper's penalty is on W (biases omitted for simplicity) §2. The code passes weight decay to SGD for all parameters, including batch-norm parameters train_models.py.
• Oversampling step count. Duplicated minority examples are appended to the epoch, so each epoch has more batches and the 350-epoch budget gives more SGD steps utils.py. The paper does not discuss this.

Consistent with the paper:

• The learning-rate drops, momentum, batch size, epochs, and dropout-to-batch-norm replacement agree train_models.py.
• The convex program matches Lemma 1, with the budgets on the diagonal blocks. The per-class loss weights are n_k/N up to a constant factor: the code uses 1 for majority and 1/R for minority terms [code: optimization.py#L13-L26, #L29-L35].

Pitfalls

• The main claims are about global minimisers. Real training finds approximate minimisers. The reduction from the full network to the LPM rests on an ansatz that the paper calls approximate and heuristic §2; the paper gives no bound on the gap §6.
• The minority theorem is asymptotic. It holds as R → ∞ §4.2. The finite threshold R_0 is seen only numerically fig. 3, and the paper gives no formula for it.
• Budgets are not tied to λ. The paper does not state how E_W and E_Θ follow from a weight decay value. For the trained-network comparison it picks values that make both constraints active in the trained network [fig. 4 caption]. An implementer cannot predict R_0 from λ alone.
• Convex relaxation depends on the solver. The script calls a generic solver and swallows exceptions: if solving fails, the variable is None and the script stops with an error optimization.py. Results near the threshold are sensitive to solver tolerance; the paper gives no tolerance.
• Test-set selection in tab. 2. The table reports the best of four oversampling rates, chosen on test accuracy tab. 2 caption. No validation split is described, so the gain is optimistic.
• Single runs, no error bars. The paper reports no run counts or variance for the deep-network results §4.3. The code fixes one seed train_models.py.
• Two different small-sample setups. The main script keeps 5000 (or 6000) majority examples and 5000//R minority examples train_models.py. The oversampling script uses n_A = 5R and n_B = 5 for CIFAR10 (6R and 6 for FashionMNIST) train_models_oversampling.py. The two agree at R = 1000, but differ at R = 1: the main script would give 5000 examples per class, while the fig. 5 caption (n_A = 5R, n_B = 5) corresponds to the oversampling script's setting. Use the caption's setting when comparing R = 1 with R = 1000 fig. 5 caption.
• Undisclosed preprocessing. The code uses per-channel normalisation constants and no data augmentation data.py. The paper states neither, so results may not transfer to a different pipeline.
• Losses other than cross-entropy are not tested. Contrastive and softmax results are analytic only §3.2.
• Two-size imbalance only. The theorem covers two class sizes. Extension to general imbalance is left open §6.
• Oversampling is not free. Large w_r means many more updates per epoch (see above). Comparing at equal epochs is not the same as comparing at equal compute.
• Typos in the source. The text writes E_H where E_Θ is meant §4.2. Table 1's ResNet18 entry is written with a typo in the LaTeX source; the value 5.0×10⁻⁸ is read from the table row.

Lineage

• Builds on: Papyan, Han and Donoho (2020) on neural collapse in balanced training §1. That paper has no card in this wiki and its arXiv id is not given in the source bibliography, so it is named, not linked.
• Compared against: other surrogate models for neural collapse that peel off the top layer or use per-class norm constraints, cited as concurrent work §1.2. None has a card here.
• Superseded by: none in the wiki.