---
kind: concept
slug: nesterov-momentum
title: Nesterov momentum
summary: >-
  Nesterov momentum as used in the outer loop of low-communication training: NoLoCo adds a pull toward its sync-group mean to the momentum update, and reports lower perplexity than DiLoCo in most runs. The corpus has one card for this concept so far, so the page covers one variant. Read the contradictions on the sign and the pull coefficient first.
tags: [optimization, distributed-training, momentum]
papers: ["2506.10911"]
related: [local-sgd, gossip-averaging, learning-rate-schedule, decentralized-federated-learning]
status: draft
updated_at: 2026-11-08
---

## Overview

This page tracks where a momentum coefficient that its authors call Nesterov
momentum is used. In the current corpus, only one card lists this concept:
NoLoCo, which applies it to an outer update that runs once per block of inner
steps [arxiv:2506.10911 §3.2]. In that update, a buffer with coefficient alpha
is added to the slow weights [arxiv:2506.10911 §3.2]. Other cards in the corpus use Nesterov momentum as an
inner-loop optimiser setting, but none lists this concept, so this page does
not cover them.

## Variants

### Outer Nesterov momentum with a pull toward the group mean (NoLoCo)

Each replica forms an outer gradient, the difference between its fast and slow
weights [arxiv:2506.10911 §3.2]. It then averages slow weights and outer
gradients inside a small group, which has size 2 in the experiments
[arxiv:2506.10911 §3.2] [arxiv:2506.10911 §4.1]. The momentum buffer is updated
with the alpha coefficient, the averaged outer gradient, and a term that pulls
the replica's slow weights toward the group mean [arxiv:2506.10911 §3.2]. The
paper's value for the outer momentum is 0.5 for NoLoCo and 0.3 for DiLoCo
[arxiv:2506.10911 §4.1]. The authors state that 0.3 beats the usual 0.9 in their
setting [arxiv:2506.10911 §4.1]. Card: [NoLoCo](/papers/2506.10911).

### Reduction to plain outer momentum

With the whole network in one group, the pull term vanishes, and the paper's
update reduces to the outer momentum used by DiLoCo [arxiv:2506.10911 §3.2].
This is a consistency check of the formula, not a separate method in the paper.

## When to use which

| Situation | Use | Why |
|---|---|---|
| Replicas can run a global all-reduce on a fast network | Plain outer momentum with the DiLoCo values, alpha 0.3 [arxiv:2506.10911 §4.1] | The paper reports DiLoCo results only in its own setting; on the 100 Gb/s setting, its two methods are close in wall-clock time [arxiv:2506.10911 §4.3] |
| Global sync is the bottleneck on a slow link | Pairwise group update with alpha 0.5 and the pull term [arxiv:2506.10911 §3.2] [arxiv:2506.10911 §4.1] | The authors report about 4x faster wall-clock time than DiLoCo at 100 Mb/s in their latency model [arxiv:2506.10911 §4.3] |
| Small perplexity gains matter | Check the gain against a repeat run first | Perplexity gains are up to about 4%, and the paper does not state the number of seeds [arxiv:2506.10911 §4.2] |
| The gamma term must be tuned | Start inside the printed stability range | For n = 2 and alpha = 0.5, the appendix bound gives 0.5 < gamma < 1.5 [arxiv:2506.10911 app. D.3] |

The paper's gains are measured on language-model pre-training with Reddit and
C4 data at 125M to 6.8B parameters [arxiv:2506.10911 §4.1]. The card does not
show results for other domains, so do not carry the gains over without a test.

## Implementation sketch

The outer step for one replica, in our own words. Symbols follow the paper:
phi are slow weights, theta fast weights, delta the momentum buffer
[arxiv:2506.10911 §3.2]. The gradient-term sign differs between the printed
equation and the code; see Contradictions.

```pseudocode
# Outer step for one replica, after m inner steps  [arxiv:2506.10911 §3.2]
# alpha: momentum coefficient; beta: outer learning rate; gamma: pull term
def outer_step(theta, phi, delta, group, alpha, beta, gamma):
    # Outer gradient: fast weights minus slow weights  [arxiv:2506.10911 §3.2]
    outer_grad = theta - phi
    # Average the slow weights and outer gradients over the group
    mean_phi = average(phi, group)
    mean_grad = average(outer_grad, group)
    # Momentum: decay, gradient term, pull toward the group mean
    # The sign of the gradient term follows the code, not the printed equation
    delta = alpha * delta + beta * mean_grad - gamma * (phi - mean_phi)
    # Slow weights move by the momentum; fast weights restart from them
    phi = phi + delta
    theta = phi
    return theta, phi, delta
```

## Contradictions

- **Sign of the gradient term.** The printed momentum update subtracts the
  averaged outer gradient [arxiv:2506.10911 §3.2]. The released code adds it
  with a plus sign, which the paper's pseudocode does not show. The card states
  that the printed sign taken literally moves slow weights away from fast
  weights [arxiv:2506.10911 §3.2]. Implement the code's sign only after reading
  the card's note on it.
- **Pull coefficient.** The paper names it gamma and sweeps it [arxiv:2506.10911
  §3.2] [arxiv:2506.10911 fig. 6]. The released code has no gamma and uses the
  outer learning rate, 0.7, in its place [arxiv:2506.10911 §4.1]. The value used
  in the paper is not stated, so the sweep cannot be reproduced from the code.
- **Symbol names.** Section 5 names alpha the local averaging parameter, eta the
  outer learning rate and beta the outer momentum [arxiv:2506.10911 §5]. The
  method section and figure 6 use alpha for momentum, beta for the outer
  learning rate and gamma for averaging [arxiv:2506.10911 §3.2]
  [arxiv:2506.10911 fig. 6]. Check which convention a source uses before you
  copy a value.

## Papers

- [NoLoCo](/papers/2506.10911): decentralised outer momentum with a pairwise
  group update and a pull toward the group mean, in place of a global sync.