--- kind: concept slug: mode-connectivity title: Mode connectivity summary: >- Whether independently trained minima of a neural network's loss are joined by paths along which the loss stays low. The core idea is to search for a minimum energy path between two minima and report its highest loss, the saddle, minus the minima's loss, the barrier. The main trade-off is cost: each path needs a full AutoNEB run, and the result is only an upper bound on the true barrier. tags: [loss-landscape, optimization, analysis] papers: ["1803.00885"] related: [model-merging, ensembles] status: draft updated_at: 2026-10-24 --- ## Overview Two networks trained from different random starts usually end up in different minima of the training loss. Mode connectivity asks whether those minima are really separate: is there a path in parameter space between them along which the loss stays about as low as at the two ends [arxiv:1803.00885 §1]? The invariant all variants share is the quantity they report. For a path between minima A and B, the saddle is the maximum loss on the path, and the barrier is that maximum minus the loss at the endpoints. A small barrier means the two minima sit on one connected low-loss region. The paper's conjecture is that the set of settings with loss below a low threshold is one connected component [arxiv:1803.00885 §1]. The paper's evidence is an upper bound, not a proof. The path is sampled at a finite set of points, so the reported saddle is at most the true saddle, and the search is local [arxiv:1803.00885 §3.1] [§3.2]. Read the reported barriers as upper bounds on what a path costs, not as the true barrier height. ## Variants ### Minimum energy path with nudged elastic band The path is a chain of pivots (parameter vectors) with springs between neighbours. Each step keeps only the loss force perpendicular to the path and only the spring force along it, which is the nudged elastic band (NEB) idea [arxiv:1803.00885 §3.1] [eq. 2] [eq. 3]. The paper sets the spring constant to infinity and re-spaces the pivots at each step, which it calls the string method variant [arxiv:1803.00885 §3.1]. Card: [Essentially No Barriers in Neural Network Energy Landscape](/papers/1803.00885). ### AutoNEB with adaptive pivot insertion Plain NEB needs enough pivots to sample the path. AutoNEB runs NEB for a short cycle, then inserts new pivots in gaps where linear interpolation between neighbouring pivots misses the loss by a large margin [arxiv:1803.00885 alg. 2] [eq. 4]. The paper runs 14 cycles per pair and uses a different schedule of steps and learning rates in each cycle [arxiv:1803.00885 §4.1]. This is the variant behind the paper's CIFAR results [arxiv:1803.00885 app. B]. Card: [Essentially No Barriers in Neural Network Energy Landscape](/papers/1803.00885). ### Choosing which pairs to connect With many minima, every pair is a candidate, and a full search is expensive. The paper's landscape exploration picks pairs in a fixed order: first finish unfinished runs, then connect disconnected components through their lowest-loss minima, then replace the highest-saddle edge of a minimum spanning tree [arxiv:1803.00885 alg. 3] [§3.2]. The saddle training loss is the edge weight [arxiv:1803.00885 §4]. Card: [Essentially No Barriers in Neural Network Energy Landscape](/papers/1803.00885). ## When to use which | Situation | Use | Why | |---|---|---| | You need one path between two trained models and can afford one AutoNEB run per pair | AutoNEB with NEB pivots | The paper reports that a run costs about as much as training the network once per minimum pair [arxiv:1803.00885 §3.2] | | You have many minima and want a connectivity graph | Pair selection with a minimum spanning tree | Connects components first, then refines the tree, which avoids checking every pair [arxiv:1803.00885 alg. 3] | | Shallow CNNs on CIFAR-10 | Small training-loss factor, 1.1 to 1.3 in the paper | Saddle over minimum for the CNN rows of the table [arxiv:1803.00885 app. B]; the paper says the saddle sits near the minima's test loss for shallow CNNs [§4.3] | | Deep ResNets or DenseNets on CIFAR-10 | Small training-loss barriers, with large ratios for the deepest ResNet | DenseNet-100-12-BC saddle training loss 0.0026 against a minimum of 0.0005; ResNet-56 has a factor of 6.5 [arxiv:1803.00885 app. B] | | You need a guaranteed barrier height | Do not rely on this method | The saddle is an upper bound from sampling, and the search is local [arxiv:1803.00885 §3.2] | The paper's results cover CIFAR-10 and CIFAR-100 with 10 minima per architecture. The card contains no results for other datasets or for models trained with other recipes, so the table does not extend to them. Test error along the paths rises only slightly in the paper: at most 0.5 points on CIFAR-10 and 2.2 points on CIFAR-100 for the deep architectures [arxiv:1803.00885 §4.3]. Use these as author-reported figures for that setup. ## Implementation sketch The core is one NEB update of the interior pivots, followed by re-insertion of pivots between cycles. Pseudocode in our own words: ```pseudocode # One NEB update of the interior pivots [arxiv:1803.00885 alg. 1] [§3.1] # pivots: (N+2, d), endpoints fixed; momenta: (N, d) def neb_update(pivots, momenta, lr): # Re-space pivots so the spring force is zero [§3.1] pivots = respace(pivots) # Loss gradient at each pivot, on its own mini-batch [§4.1] grads = loss_gradients(pivots) for i in interior(pivots): # Keep only the component perpendicular to the path [§3.1] grad = remove_tangent_component(grads[i], tangent(pivots, i)) # SGD with momentum [§4.1] momenta[i] = 0.9 * momenta[i] + grad pivots[i] = pivots[i] - lr * momenta[i] return pivots, momenta ``` ```pseudocode # AutoNEB for one pair of minima [arxiv:1803.00885 alg. 2] [§3.1] def auto_neb(first, second, cycles): # Start on a straight line between the two minima pivots = interpolate(first, second) for cycle_index, cycle in enumerate(cycles): if cycle_index > 0: # Insert pivots where linear interpolation misses the loss [app. A] pivots = insert_at_largest_residuals(pivots) for _ in range(cycle.steps): pivots, momenta = neb_update(pivots, momenta, cycle.lr) # Saddle: maximum loss over dense samples between pivots [§3.1] return pivots, max(dense_losses(pivots)) ``` Where variants differ: the spring constant (infinite here), the insertion threshold, the step count and learning rate per cycle, and the dense-sampling density, which the paper sets at 9 points per gap [§4.1]. The paper's cycle schedule is 1000 steps at learning rate 0.1 for cycles 1 to 4, 2000 steps for cycles 5 and 6, then 1000 steps per cycle at 0.01 and 0.001 for the rest [arxiv:1803.00885 §4.1]. ## Contradictions The disagreements below are all inside the one source. No second paper in the corpus has a card for this concept. - **Insertion threshold.** The text gives 20% of the total loss difference [arxiv:1803.00885 §4.1]. The paper's code uses 0.1 as its default, and no config overrides it [code: torch_autoneb/fill.py#L20]. The text and the code therefore disagree, and the card cannot tell which value produced the reported tables [arxiv:1803.00885 §4.1]. - **Test-error increase.** The appendix gives larger increases than the table: 0.7 and 2.9 points for ResNets, against the table's maxima of 0.5 and 2.2 [arxiv:1803.00885 app. B] [§4.3]. The table is the source to use. - **Path length.** The paper gives the path length relative to the straight segment as "between 50% to 2.5 times" [arxiv:1803.00885 §4.4]. The range can be read two ways, so check the figure before quoting it. - **Hessian claim.** The abstract says each minimum has a vanishing Hessian eigenvalue beyond the trivial ones [arxiv:1803.00885 §6]. The body has no Hessian computation, and the conclusion states the weaker form, that low eigenvalues exist [arxiv:1803.00885 §6]. ## Papers - [Essentially No Barriers in Neural Network Energy Landscape](/papers/1803.00885): applies AutoNEB, an NEB variant with adaptive pivot insertion, to connect independently trained CNN, ResNet and DenseNet minima on CIFAR-10 and CIFAR-100.