Mode connectivity

Overview

Two networks trained from different random starts usually end up in different minima of the training loss. Mode connectivity asks whether those minima are really separate: is there a path in parameter space between them along which the loss stays about as low as at the two ends 1803.00885 §1?

The invariant all variants share is the quantity they report. For a path between minima A and B, the saddle is the maximum loss on the path, and the barrier is that maximum minus the loss at the endpoints. A small barrier means the two minima sit on one connected low-loss region. The paper's conjecture is that the set of settings with loss below a low threshold is one connected component 1803.00885 §1.

The paper's evidence is an upper bound, not a proof. The path is sampled at a finite set of points, so the reported saddle is at most the true saddle, and the search is local 1803.00885 §3.1 §3.2. Read the reported barriers as upper bounds on what a path costs, not as the true barrier height.

Variants

Minimum energy path with nudged elastic band

The path is a chain of pivots (parameter vectors) with springs between neighbours. Each step keeps only the loss force perpendicular to the path and only the spring force along it, which is the nudged elastic band (NEB) idea 1803.00885 §3.1 eq. 2 eq. 3. The paper sets the spring constant to infinity and re-spaces the pivots at each step, which it calls the string method variant 1803.00885 §3.1.

Card: Essentially No Barriers in Neural Network Energy Landscape.

AutoNEB with adaptive pivot insertion

Plain NEB needs enough pivots to sample the path. AutoNEB runs NEB for a short cycle, then inserts new pivots in gaps where linear interpolation between neighbouring pivots misses the loss by a large margin 1803.00885 alg. 2 eq. 4. The paper runs 14 cycles per pair and uses a different schedule of steps and learning rates in each cycle 1803.00885 §4.1. This is the variant behind the paper's CIFAR results 1803.00885 app. B.

Card: Essentially No Barriers in Neural Network Energy Landscape.

Choosing which pairs to connect

With many minima, every pair is a candidate, and a full search is expensive. The paper's landscape exploration picks pairs in a fixed order: first finish unfinished runs, then connect disconnected components through their lowest-loss minima, then replace the highest-saddle edge of a minimum spanning tree 1803.00885 alg. 3 §3.2. The saddle training loss is the edge weight 1803.00885 §4.

Card: Essentially No Barriers in Neural Network Energy Landscape.

When to use which

The paper's results cover CIFAR-10 and CIFAR-100 with 10 minima per architecture. The card contains no results for other datasets or for models trained with other recipes, so the table does not extend to them.

Test error along the paths rises only slightly in the paper: at most 0.5 points on CIFAR-10 and 2.2 points on CIFAR-100 for the deep architectures 1803.00885 §4.3. Use these as author-reported figures for that setup.

Implementation sketch

The core is one NEB update of the interior pivots, followed by re-insertion of pivots between cycles. Pseudocode in our own words:

# One NEB update of the interior pivots [arxiv:1803.00885 alg. 1] [§3.1]
# pivots: (N+2, d), endpoints fixed; momenta: (N, d)

def neb_update(pivots, momenta, lr):
    # Re-space pivots so the spring force is zero [§3.1]
    pivots = respace(pivots)
    # Loss gradient at each pivot, on its own mini-batch [§4.1]
    grads = loss_gradients(pivots)
    for i in interior(pivots):
        # Keep only the component perpendicular to the path [§3.1]
        grad = remove_tangent_component(grads[i], tangent(pivots, i))
        # SGD with momentum [§4.1]
        momenta[i] = 0.9 * momenta[i] + grad
        pivots[i] = pivots[i] - lr * momenta[i]
    return pivots, momenta

# AutoNEB for one pair of minima [arxiv:1803.00885 alg. 2] [§3.1]

def auto_neb(first, second, cycles):
    # Start on a straight line between the two minima
    pivots = interpolate(first, second)
    for cycle_index, cycle in enumerate(cycles):
        if cycle_index > 0:
            # Insert pivots where linear interpolation misses the loss [app. A]
            pivots = insert_at_largest_residuals(pivots)
        for _ in range(cycle.steps):
            pivots, momenta = neb_update(pivots, momenta, cycle.lr)
    # Saddle: maximum loss over dense samples between pivots [§3.1]
    return pivots, max(dense_losses(pivots))

Where variants differ: the spring constant (infinite here), the insertion threshold, the step count and learning rate per cycle, and the dense-sampling density, which the paper sets at 9 points per gap §4.1. The paper's cycle schedule is 1000 steps at learning rate 0.1 for cycles 1 to 4, 2000 steps for cycles 5 and 6, then 1000 steps per cycle at 0.01 and 0.001 for the rest 1803.00885 §4.1.

Contradictions

The disagreements below are all inside the one source. No second paper in the corpus has a card for this concept.

• Insertion threshold. The text gives 20% of the total loss difference 1803.00885 §4.1. The paper's code uses 0.1 as its default, and no config overrides it fill.py. The text and the code therefore disagree, and the card cannot tell which value produced the reported tables 1803.00885 §4.1.
• Test-error increase. The appendix gives larger increases than the table: 0.7 and 2.9 points for ResNets, against the table's maxima of 0.5 and 2.2 1803.00885 app. B §4.3. The table is the source to use.
• Path length. The paper gives the path length relative to the straight segment as "between 50% to 2.5 times" 1803.00885 §4.4. The range can be read two ways, so check the figure before quoting it.
• Hessian claim. The abstract says each minimum has a vanishing Hessian eigenvalue beyond the trivial ones 1803.00885 §6. The body has no Hessian computation, and the conclusion states the weaker form, that low eigenvalues exist 1803.00885 §6.

Papers

• Essentially No Barriers in Neural Network Energy Landscape: applies AutoNEB, an NEB variant with adaptive pivot insertion, to connect independently trained CNN, ResNet and DenseNet minima on CIFAR-10 and CIFAR-100.