Essentially No Barriers in Neural Network Energy Landscape

In plain words

Training a neural network means walking downhill on a huge, bumpy surface of possible settings until the error stops falling. People usually picture each trained network as sitting alone at the bottom of its own deep valley. This paper asks whether those valleys are really separate.

It takes two networks trained from different random starts and looks for a route between them through the space of settings, along which the error stays about as low as it is at the two ends. The method borrows an idea from chemistry: it searches for the lowest mountain pass between two stable molecular shapes, and it finds only an approximation of that pass. For deep residual and densely connected image classifiers on CIFAR-10 and CIFAR-100, such routes exist and the error barely rises along them. The authors read this as a sign that good solutions may lie on one connected low-error region rather than in isolated valleys.

TL;DR

Minima of CIFAR-10/100 networks (CNNs, ResNets, DenseNets) are connected by paths along which the training loss stays close to the loss at the minima §4.3 app. B. The path is found by AutoNEB: a nudged elastic band (NEB) with adaptive insertion of pivots, run for 14 cycles §3.1 alg. 2 §4.1. The loss barrier is small for deep ResNets and DenseNets, and larger for CIFAR-100 than for CIFAR-10 §4.3. Results are author-reported and from a small set of 10 minima per architecture. The method costs about as much as training the network once per minimum pair §3.2. Use this card when you need the method, the schedule, or the known gaps between the text and the code.

Problem

Minimum energy paths (MEPs) are the lowest-maximum-loss routes between two minima. In a few dimensions they can be computed exactly on a dense grid, but neural networks have millions of parameters, so only local heuristics are feasible §3.1.

Earlier work found flat straight lines between minima that start close together in parameter space §2. Other work pointed out that ReLU networks have trivial degeneracies from rescaling §2. NEB had been applied to a multi-layer perceptron with one hidden layer §2. The paper's question is whether minima of modern CNNs, ResNets and DenseNets on CIFAR images can be connected by paths without a barrier §1 §2.

The authors state the conjecture as: the region of parameter space where the loss stays below a low threshold forms one connected component §1. Their evidence is an upper bound on the barrier, not a proof, and they say they lack a formal characterisation of when it holds §5.

Method

Overview. Four steps, from the paper's experiment section §4:

1. Train 10 minima per architecture from random initialisations. The minima are distinct: on the test set, the misclassified images of two minima overlap by at most 70% for ResNets and DenseNets §4.
2. Order the pairs of minima to connect (landscape exploration, alg. 3).
3. For each chosen pair, run AutoNEB for 14 cycles alg. 2 §4.1. Each cycle runs NEB and then inserts pivots where the path is poorly sampled.
4. Keep the saddle of every connected pair. Reduce the graph of pairs to a minimum spanning tree (MST), using the saddle training loss as the edge weight §4.

Minimum energy path. The goal is a continuous path from one minimum to the other that minimises the largest loss on the path §3.1. The loss of the path is only an upper bound on the true saddle, because the path is sampled at a finite set of points §3.1 §3.2.

The paper models the path as a chain of pivots (parameter vectors) with springs between neighbours. The total energy is the sum of pivot losses plus spring energies eq. 1. The pivots at the two ends stay fixed at the minima.

E(p) = \sum_{i=1}^{N} L(p_i) + \sum_{i=0}^{N} \frac12 k \lVert p_{i+1} - p_i \rVert^2 eq. 1

Nudged elastic band (NEB) keeps only the loss force perpendicular to the path and only the spring force along it. The local tangent \hat\tau_i points at the neighbouring pivot that has the higher loss §3.1. The forces are:

F_i^L\big|_\perp = -\big(\nabla L(p_i) - (\nabla L(p_i)\cdot\hat\tau_i)\,\hat\tau_i\big) eq. 2

F_i^S = -k\big(\lVert p_i - p_{i-1} \rVert - \lVert p_{i+1} - p_i \rVert\big) eq. 3

The spring force along the tangent is F_i^S\big|_\parallel = (F_i^S\cdot\hat\tau_i)\,\hat\tau_i, and the NEB force is F_i^L\big|_\perp + F_i^S\big|_\parallel §3.1 eq. 2 eq. 3.

The authors do not use a finite spring constant. They re-space the pivots at every step and set the spring force to zero. This is the "string method" variant §3.1. The paper does not state the target spacing; the code keeps fixed fractions of arc length (see Paper vs code).

Algorithm 1 (NEB). Each iteration re-spaces the pivots, computes the perpendicular loss force at each interior pivot, and moves the pivot along it alg. 1 §3.1. The paper's plain update is replaced in the experiments by SGD with momentum 0.9 §4.1.

# One NEB update of the interior pivots  [alg. 1] [§3.1]
# pivots: (N+2, d), both endpoints fixed; momenta: (N, d)
# target_fractions: (N+1,) fixed share of arc length per gap  [§3.1]

def neb_update(pivots, momenta, target_fractions, lr, weight_decay):
    # Re-space the pivots along the path; spring constant k = inf  [§3.1]
    # [code: torch_autoneb/neb_model.py#L63-L66]
    pivots = respace_by_fractions(pivots, target_fractions)
    # Loss and gradient per pivot, each on its own mini-batch  [§4.1]
    # [code: torch_autoneb/neb_model.py#L73-L79]
    losses, loss_grads = evaluate_pivots(pivots)
    for i in range(1, len(pivots) - 1):
        # Tangent from the neighbouring losses  [§3.1]
        # [code: torch_autoneb/neb_model.py#L109-L126]
        tangent = path_tangent(pivots, losses, i)
        # L2 weight decay, added before the projection  [§4.1]
        # [code: torch_autoneb/neb_model.py#L81-L83]
        grad = loss_grads[i] + weight_decay * pivots[i]
        # Keep only the part perpendicular to the path  [alg. 1] [§3.1]
        # [code: torch_autoneb/neb_model.py#L100]
        grad = grad - dot(grad, tangent) * tangent
        # SGD with momentum 0.9  [§4.1]
        momenta[i] = 0.9 * momenta[i] + grad
        pivots[i] = pivots[i] - lr * momenta[i]
    return pivots, momenta

Algorithm 2 (AutoNEB, one pair). NEB runs for a short cycle, then new pivots are inserted where the loss between pivots is poorly approximated by linear interpolation alg. 2 §3.1 app. A. Inserted pivots are placed at the largest deviation in each gap, and at most one per gap per cycle app. A. The residual that decides insertion is eq. 4, normalised by the range of the pivot losses:

\vartheta > \frac{\Delta L^{i,i+1}(\alpha)}{\max_i L(p_i) - \min_i L(p_i)} eq. 4

Here \Delta L^{i,i+1}(\alpha) is the true loss at position \alpha in the gap minus the linear interpolation of the two pivot losses app. A. A pivot is inserted when the residual exceeds the threshold \vartheta. The paper gives the threshold in prose as 20% §4.1; see Paper vs code.

# AutoNEB for one pair of minima  [alg. 2] [§3.1] [app. A]
# cycles: list of (steps, lr) per cycle  [§4.1]

def auto_neb(first_minimum, second_minimum, cycles):
    # Start on a straight line with 3 interior pivots  [code: configs]
    # N is not stated in the paper
    pivots = interpolate_line(first_minimum, second_minimum, interior=3)
    fractions = equal_fractions(len(pivots) - 1)
    dense_losses = None
    for cycle_index, cycle in enumerate(cycles):
        if cycle_index > 0:
            # Insert up to 2 pivots, at most one per gap  [app. A]
            # [code: configs/cifar10-resnet20.yaml#L49-L50]
            # Paper: 20% [§4.1]. Code default: 0.1
            # [code: torch_autoneb/fill.py#L20]
            pivots, fractions = insert_at_largest_residuals(
                pivots, fractions, dense_losses, count=2, threshold=0.1
            )
        # New optimiser each cycle, so momentum restarts  [code: torch_autoneb/__init__.py#L66-L67]
        momenta = zeros_like(pivots[1:-1])
        for _ in range(cycle.steps):
            pivots, momenta = neb_update(
                pivots, momenta, fractions, cycle.lr, weight_decay=0.0001
            )
        # Dense evaluation: 9 points per gap, full train and test sets  [§4.1]
        # [code: torch_autoneb/neb_model.py#L147-L165]
        dense_losses = evaluate_densely(pivots, points_per_gap=9)
    # Saddle = maximum over the dense points  [§3.1] [app. B]
    saddle = max(dense_losses)
    return pivots, saddle

Cycle schedule. The 14 cycles are the same for all architectures; only the batch sizes differ §4.1. Each cycle runs NEB for the number of steps shown below, then inserts pivots before the next cycle:

• Cycles 1 to 4: 1000 steps each, learning rate 0.1 §4.1.
• Cycles 5 and 6: 2000 steps each, learning rate 0.1 §4.1.
• Cycles 7 to 10: 1000 steps each, learning rate 0.01 §4.1.
• Cycles 11 to 14: 1000 steps each, learning rate 0.001 §4.1.

The paper says the number of steps was increased in cycles 5 and 6 because inserting pivots after 1000 steps did not prove necessary §4.1.

Algorithm 3 (which pairs to connect). The paper describes the steps in prose and as alg. 3 §3.2. The implementation uses three suggesters in order suggest.py:

1. unfinished: a pair whose AutoNEB run is missing its last cycle.
2. disconnected: while the graph has several components, connect the lowest-loss minimum to the lowest-loss minimum of each other component.
3. mst: take the highest-saddle edge of the current MST, remove it, and look for a new pair across the cut that has no edge yet.

Each new path is kept as an extra edge; the simple graph keeps the lowest-saddle edge per pair [code: torch_autoneb/init.py#L164-L183]. When no suggester returns a pair, landscape exploration stops [code: torch_autoneb/init.py#L139-L148].

Architectures §4.2:

• CNN-W×D: D convolutions with 5×5 kernels and padding 2. Each is followed by a 2×2 max-pool and batch normalisation. Then one hidden fully connected layer of 256 units with batch normalisation, then the output. W is the number of channels. The paper uses W = 12, 24, 36, 48, 96 with D = 1, and CNN-48×2 and CNN-48×3 §4.2 cnn.py cifar10-cnn12.yaml.
• ResNet-(6n+2): CIFAR-style ResNet with 16, 32 and 64 filters per stage, stride 2 at the start of stages 2 and 3 §4.2 resnet.py. The paper trains ResNet-20, -32, -44, -56 §4.2; the table also lists ResNet-8 app. B.
• DenseNet-40-12 (no bottleneck) and DenseNet-100-12-BC (bottleneck, compression 0.5) §4.2 cifar10-densenet40,12.yaml cifar10-densenet100,12,BC.yaml.

Minimum training. The paper refers to the training recipes of the original ResNet and DenseNet papers §4.2. The configs set the following, which the text does not state:

• SGD with momentum 0.9 and learning rate 0.1, with step decays cifar10-resnet20.yaml.
• CNN and ResNet configs: 64000 steps, batch 128 cifar10-resnet20.yaml.
• DenseNet-40-12 config: 250000 steps, batch 64, Nesterov momentum, decays at 125000 and 187500 steps cifar10-densenet40,12.yaml.

Hyperparameters

Results

Author-reported. Setup: CIFAR-10 and CIFAR-100, 10 minima per architecture, with losses and errors reported on the training and test sets §4 app. B. Each row is an architecture. Minimum values are averaged over the minima; saddle values are the maximum along each path, averaged over the paths app. B. Values are copied from the paper's table. The "Epoch" column is the first epoch at which the training loss falls below the saddle loss. Rows marked † are bold in the paper, i.e. the crossing falls in the third training phase at learning rate 10⁻³ app. B.

Columns: train loss at minimum and saddle; "Factor" is saddle over minimum; test loss at minimum and saddle; test error at minimum and saddle (in %); Δ is the error increase in percentage points app. B.

Source: app. B (the table of quantitative results; the text of Appendix B explains the columns). The "+" in "C10+" and "C100+" is not explained in the paper.

Main claims (author-reported):

• For shallow CNNs the saddle loss is close to the test loss of the minima; for deep ResNets and DenseNets it is close to the training loss §4.3 fig. 5.
• Barriers fall as networks get deeper and wider and rise on CIFAR-100 §4.3.
• Test error rises only slightly along the paths: at most 0.5 points on CIFAR-10 and 2.2 points on CIFAR-100 for the deep architectures §4.3.
• The training loss crosses the saddle loss only after the first learning-rate decay and a further large drop; for several architectures this happens in the final phase §4.3 fig. 6.
• Along the paths each weight follows a smooth trajectory. The largest deviations from the straight line occur near the saddle §4.4 fig. 7.
• The paths are longer than the straight segment between minima by a factor the paper states as "between 50% to 2.5 times" §4.4; see Pitfalls.

Toy example §5.2 fig. 8: for a two-neuron XOR network, the paper argues that a path with zero extra loss exists if one helper neuron is added, and that depth can give the same effect. This is an argument, not an experiment.

Paper vs code

Code: https://github.com/fdraxler/PyTorch-AutoNEB, commit 13c8ba891bd55e4813f9fb7be69e09aca5a34cbd, MIT. The paper's experiment section links it §4. The repository is owned by the first author, so it belongs to the authors. Its licence file is dated 2022, which is later than the paper, so the code may have changed since. The card uses this commit.

Matches:

• Pivot spring constant: the paper re-spaces pivots and sets the spring force to zero §3.1; the configs set spring_constant: inf cifar10-resnet20.yaml.
• Cycle count, learning rates and step counts match the schedule in §4.1 cifar10-resnet20.yaml.
• NEB batch sizes for ResNet-20 and -32 (512), ResNet-44 and -56 (256), and both DenseNets (256) match §4.2 cifar10-resnet20.yaml cifar10-resnet56.yaml cifar10-densenet40,12.yaml.
• Tangent rule: the one-sided choice used by the paper §3.1 matches neb_model.py.
• Nine dense points per gap §4.1 match cifar10-resnet20.yaml.
• Insertion residual, normalised by the range of pivot losses app. A matches fill.py.
• MST weighted by the saddle training loss §4 matches [code: torch_autoneb/init.py#L152-L160] and suggest.py.

Differences:

• Insertion threshold. The paper says a pivot is inserted when the loss exceeds the interpolation by at least 20% of the total energy difference §4.1. The code's default threshold is 0.1 fill.py, and no config overrides it cifar10-resnet20.yaml. A lower threshold inserts more pivots, so the code does not match the text. The card cannot tell which value produced the reported tables.
• Extremum tangent. At a local maximum or minimum of the loss, the code interpolates the tangent from both neighbours, following the cited Henkelman method neb_model.py. The paper's equation for the tangent has only the two one-sided cases §3.1.
• Batch-norm mode. Interior pivots are evaluated in training mode, so batch normalisation uses batch statistics. Endpoints are evaluated in evaluation mode neb_model.py [code: torch_autoneb/models/init.py#L226-L234]. The paper does not mention this.
• Weight decay. The paper describes L2 regularisation in the optimiser §4.1. The code adds the decay term to the interior pivot gradients before the perpendicular projection, so the decay component along the tangent is removed neb_model.py. The optimiser's own weight decay is forbidden [code: torch_autoneb/init.py#L70-L71].
• Target spacing. The redistribution targets are the fractions that insertion produces. They are not updated from the converged path's arc length. Insertion splits a gap's fraction in the chosen ratio fill.py and the new fractions are returned unchanged after NEB [code: torch_autoneb/init.py#L57] [code: torch_autoneb/init.py#L88]. The paper does not say how the spacing is set.
• Momentum reset. Each cycle builds a new optimiser, so momentum restarts at zero [code: torch_autoneb/init.py#L66-L67]. The paper does not say this.
• Mini-batches. Each pivot's loss call draws the next batch from one shared shuffled loader with drop_last, so pivots in the same step see different batches [code: torch_autoneb/models/init.py#L272-L291].
• Training metrics. The dense analysis uses the full training set with the crop and flip augmentation still active datasets.py [code: torch_autoneb/models/init.py#L297-L311]. The paper does not mention this.
• Starting minimum. The paper says the first minimum θ₁ is connected to all others alg. 3 without naming it. The code's disconnected suggester starts from the lowest-loss minimum, so the starting point is the minimum with the lowest training loss suggest.py.
• Stopping. The paper stops when the spanning tree has similar saddle losses or when a compute budget runs out §3.2. The code has neither; it stops when no suggester returns a pair [code: torch_autoneb/init.py#L139-L148].
• Insertion timing. The paper inserts after each cycle §4.1. The code inserts at the start of each cycle from the second onwards, using the dense analysis of the previous cycle [code: torch_autoneb/init.py#L57] [code: torch_autoneb/init.py#L114-L116]. In effect the last cycle is not followed by an insertion. The paper does not say whether its last cycle is followed by one.
• Training length. The paper gives 136 and 266 epochs app. B. The configs give 64000 steps at batch 128 and 250000 steps at batch 64 [code: configs/cifar10-resnet20.yaml#L10-L21] cifar10-densenet40,12.yaml. Over 50,000 CIFAR-10 training images with drop_last, these are about 164 and about 320 epochs. The two sources disagree.

Pitfalls

• Threshold. The text gives 20% §4.1. The code uses 0.1 and no config overrides it fill.py. Pick one and state it, because the number of pivots and the saddle estimate both depend on it.
• Initial pivot count. The paper never gives N for the experiments §3.1. The code uses 3 interior pivots at the start cifar10-resnet20.yaml. With 2 pivots inserted per cycle over cycles 2 to 14, the code can reach at most 31 pivots by counting (5 + 2×13).
• Cost. Each NEB step evaluates every pivot, and each dense analysis passes over the full train and test sets at every dense point neb_model.py. The paper says a full AutoNEB run costs about as much as training the network once §3.2. Budget for that per minimum pair. There are C(10,2) = 45 pairs for 10 minima if you connect all of them.
• Saddle is an upper bound. The saddle is the maximum over 9 points per gap neb_model.py. A higher point between samples would be missed. The paper relies on dense sampling §3.1 and says the result is an upper bound §3.2.
• Train metrics are random. Because the train loader applies random crops and flips datasets.py, train losses in the analysis change from call to call. Fix the seed and the transform if you need reproducible numbers.
• Batch-norm mode mismatch. Interior pivots update batch-norm statistics in training mode while the endpoints do not [code: torch_autoneb/models/init.py#L226-L234]. Expect the endpoints and the interior pivots to be scored on different statistics.
• Fixed target fractions. Because the targets come from insertion history, the redistribution can keep a spacing that no longer matches the relaxed path. The paper says the spacing is adjustable but does not say how to set it §3.1. The card has no measurement of how much this matters.
• Appendix text vs table. Appendix B says the ResNet error rises by at most 0.7 points on CIFAR-10 and 2.9 points on CIFAR-100, and the DenseNet error by up to 0.4 and 1.5 points app. B. The table's own Δ column gives maxima of 0.5 and 2.2 for the ResNets, and 0.3 and 1.4 for the DenseNets. Section 4.3 gives 0.5 and 2.2 §4.3, which matches the table. Use the table.
• Path length. The paper gives the path length relative to the straight segment as a range from 50% to 2.5 times §4.4. The range is ambiguous: it could mean a 50% increase up to 2.5 times the length, or 1.5 to 2.5 times. Check the figure or the code before using it.
• Hessian claim. The arXiv abstract says every minimum has at least one vanishing Hessian eigenvalue beyond the trivial ones. The body contains no Hessian computation. The conclusion draws the implication that low eigenvalues exist §6, and the abstract wording is stronger ("vanishing", not "low").
• Epochs. The table's epoch column is defined as the first crossing of the training loss below the saddle loss app. B. The config-derived epoch counts are different from the caption's 136 and 266. Use the step counts from the config if you want the same training budget as the code.
• Dataset suffix. "C10+" and "C100+" are not explained in the paper. Do not infer their meaning from it. The code's training transform is random crop and horizontal flip datasets.py.
• Local MEPs only. AutoNEB is local and can get stuck §3.2. The paper expects that a fuller search would find lower paths §6. Do not present the reported saddles as true barriers.
• Weight decay is on coordinates. The decay pulls each pivot's coordinates toward zero, not the loss. Do not confuse it with the paper's optimiser-level regularisation §4.1.

Lineage

• Builds on: nudged elastic band §3.1, as the improved version of Henkelman and Jónsson (2000), and AutoNEB §3.1, from Kolsbjerg, Groves and Hammer (2016) README.md. The source contains no arXiv ids for these, so they are not linked here.
• Related: earlier work found flat straight paths between nearby minima §2. ReLU rescaling symmetries were shown to make minima degenerate §2. NEB had been applied to a perceptron with one hidden layer §2. Earlier work on the topology of CNN minima on MNIST and RNNs on PTB found low barriers, and a more frustrated landscape on CIFAR-10 §2.
• Concurrent work: a paper that appeared after this one's submission, reporting similar paths between minima §2. The paper does not name it in the source.
• Superseded by: not known from content on this wiki.