Nesterov momentum

Overview

This page tracks where a momentum coefficient that its authors call Nesterov momentum is used. In the current corpus, only one card lists this concept: NoLoCo, which applies it to an outer update that runs once per block of inner steps 2506.10911 §3.2. In that update, a buffer with coefficient alpha is added to the slow weights 2506.10911 §3.2. Other cards in the corpus use Nesterov momentum as an inner-loop optimiser setting, but none lists this concept, so this page does not cover them.

Variants

Outer Nesterov momentum with a pull toward the group mean (NoLoCo)

Each replica forms an outer gradient, the difference between its fast and slow weights 2506.10911 §3.2. It then averages slow weights and outer gradients inside a small group, which has size 2 in the experiments 2506.10911 §3.2 2506.10911 §4.1. The momentum buffer is updated with the alpha coefficient, the averaged outer gradient, and a term that pulls the replica's slow weights toward the group mean 2506.10911 §3.2. The paper's value for the outer momentum is 0.5 for NoLoCo and 0.3 for DiLoCo 2506.10911 §4.1. The authors state that 0.3 beats the usual 0.9 in their setting 2506.10911 §4.1. Card: NoLoCo.

Reduction to plain outer momentum

With the whole network in one group, the pull term vanishes, and the paper's update reduces to the outer momentum used by DiLoCo 2506.10911 §3.2. This is a consistency check of the formula, not a separate method in the paper.

When to use which

The paper's gains are measured on language-model pre-training with Reddit and C4 data at 125M to 6.8B parameters 2506.10911 §4.1. The card does not show results for other domains, so do not carry the gains over without a test.

Implementation sketch

The outer step for one replica, in our own words. Symbols follow the paper: phi are slow weights, theta fast weights, delta the momentum buffer 2506.10911 §3.2. The gradient-term sign differs between the printed equation and the code; see Contradictions.

# Outer step for one replica, after m inner steps  [arxiv:2506.10911 §3.2]
# alpha: momentum coefficient; beta: outer learning rate; gamma: pull term
def outer_step(theta, phi, delta, group, alpha, beta, gamma):
    # Outer gradient: fast weights minus slow weights  [arxiv:2506.10911 §3.2]
    outer_grad = theta - phi
    # Average the slow weights and outer gradients over the group
    mean_phi = average(phi, group)
    mean_grad = average(outer_grad, group)
    # Momentum: decay, gradient term, pull toward the group mean
    # The sign of the gradient term follows the code, not the printed equation
    delta = alpha * delta + beta * mean_grad - gamma * (phi - mean_phi)
    # Slow weights move by the momentum; fast weights restart from them
    phi = phi + delta
    theta = phi
    return theta, phi, delta

Contradictions

• Sign of the gradient term. The printed momentum update subtracts the averaged outer gradient 2506.10911 §3.2. The released code adds it with a plus sign, which the paper's pseudocode does not show. The card states that the printed sign taken literally moves slow weights away from fast weights 2506.10911 §3.2. Implement the code's sign only after reading the card's note on it.
• Pull coefficient. The paper names it gamma and sweeps it [arxiv:2506.10911 §3.2] 2506.10911 fig. 6. The released code has no gamma and uses the outer learning rate, 0.7, in its place 2506.10911 §4.1. The value used in the paper is not stated, so the sweep cannot be reproduced from the code.
• Symbol names. Section 5 names alpha the local averaging parameter, eta the outer learning rate and beta the outer momentum 2506.10911 §5. The method section and figure 6 use alpha for momentum, beta for the outer learning rate and gamma for averaging 2506.10911 §3.2 2506.10911 fig. 6. Check which convention a source uses before you copy a value.

Papers

• NoLoCo: decentralised outer momentum with a pairwise group update and a pull toward the group mean, in place of a global sync.