Gossip averaging

Overview

Data-parallel training keeps model replicas identical with a global exchange, often an all-reduce. That exchange can dominate step time on slow or high-latency links 2506.10911 §1 2506.10911 §2. Gossip averaging replaces the global exchange with averaging inside small groups, so no sync waits on every replica 2506.10911 §3.2.

The invariant of the variant on this page: at each sync, a replica's new slow weights depend only on its own group. The groups are drawn again at the next sync, so the information spreads over many rounds. Replicas are not identical after a sync. The paper reports that their spread grows early in training and shrinks as the learning rate decays 2506.10911 §4.2.

The paper cites gossip averaging as motivation for pairwise mixing 2506.10911 §2. It gives no general theory of gossip averaging, and no other card in this corpus covers the topic. Claims below are about the paper's method only.

Variants

Random pairwise groups with a modified Nesterov outer step

Each outer step, all replicas draw the same random permutation. The permutation is cut into groups of n, and the paper's experiments use n = 2 2506.10911 §3.2 2506.10911 §4.1. The paper states that the network should have more than 2n replicas 2506.10911 §4.1. Within a group, each replica uses the group mean of the slow weights and of the outer gradients in its momentum update. The outer gradient of a replica is its fast weights minus its slow weights 2506.10911 eq. 1. The paper's update adds a pull term that moves each replica's slow weights toward the group mean of the slow weights, scaled by a local-averaging coefficient gamma 2506.10911 eq. 2 2506.10911 §3.2.

The paper gives no value for gamma in the text 2506.10911 §4.1. For n = 2 and alpha = 0.5, the stability range it derives is 0.5 < gamma < 1.5 2506.10911 app. D.3. The paper's theorem uses a quadratic loss with a Gaussian shift of the optimum and a constant inner learning rate, and it assumes infinitely many outer steps. Under these assumptions the expected slow weights go to zero and their variance scales with the square of the inner learning rate 2506.10911 §3.2. The variance step relies on a conjecture stated in appendix D [arxiv:2506.10911 app. D.2].

Card: NoLoCo: No-all-reduce Low Communication Training Method for Large Models.

Hyperparameters

Values for the main language-model runs 2506.10911 §4.1. The paper gives the architectures in its table 3 2506.10911 tab. 3.

• Group size n = 2 for the pairwise variant 2506.10911 §4.1.
• Outer learning rate beta = 0.7 and outer momentum alpha = 0.5 for the pairwise variant, against alpha = 0.3 for DiLoCo 2506.10911 §4.1.
• 50 inner steps per outer step for the pairwise variant, against 100 for DiLoCo 2506.10911 §4.1.
• Inner optimiser Adam, gradient clipping at 1.0, a cosine schedule with a linear warm-up of 1000 steps, and 25,000 optimiser steps [arxiv:2506.10911 §4.1] 2506.10911 app. B.
• The paper does not state Adam's betas or epsilon, nor how replicas start, and it does not say what happens to optimiser moments at a sync 2506.10911 app. B. The card lists the implementation gaps.

Results

Author-reported. The experiments run decentralised data parallelism combined with pipeline parallelism, with Llama-style models of 125M, 1.3B and 6.8B parameters, on Reddit and C4 data. The baselines are synchronous FSDP and DiLoCo 2506.10911 §4.1. The paper gives no number of random seeds 2506.10911 §4.2.

Validation perplexity 2506.10911 tab. 1:

• Against DiLoCo, the pairwise variant has lower perplexity in 8 of 8 Reddit runs and 6 of 8 C4 runs. The differences are small, up to about 4% 2506.10911 §4.2 2506.10911 tab. 1.
• Against FSDP, read from the same table, the pairwise variant has higher perplexity in every row. The paper's comparison focuses on DiLoCo, and this table does not show a case where the pairwise variant beats FSDP.

Wall-clock time per 100 inner steps on 32 workers, medium model, pure data parallelism 2506.10911 tab. 2 2506.10911 §4.3:

The pairwise variant is about 4.05 times faster than DiLoCo at 100 Mb/s, and the two are nearly equal at 100 Gb/s 2506.10911 tab. 2. The authors combine this with per-step convergence to claim about 4 times faster time-to-target on consumer networks 2506.10911 §4.3.

A simulation over world sizes from 4 to 1024 finds that the expected ratio of tree all-reduce time to pairwise-averaging time grows with world size and with the variance of message times 2506.10911 §4.3 [arxiv:2506.10911 fig. 5]. The authors state that DiLoCo falls about 20% behind at 1024 accelerators, counting only the blocking wait for the slowest worker 2506.10911 §4.3. The simulation assumes log-normal inner-step latency 2506.10911 app. G.

Group size, on the 125M model with 0.5M training tokens: perplexity improves only weakly as n goes from 2 to 4 and 8 2506.10911 §5 [arxiv:2506.10911 fig. 7]. Sweeps of alpha, beta, gamma and H are shown only as figures on the 1.3B C4 model 2506.10911 §5 2506.10911 fig. 6.

When to use which

Caveats the paper raises or the numbers show:

• Results are author-reported on one model family. The paper gives no seed count 2506.10911 §4.2.
• The paper attributes the perplexity gain to regularisation from weight noise and does not test this 2506.10911 §4.2.
• Per sync, the pairwise variant moves about the same volume of data as DiLoCo. The saving comes from removing the global wait, not from fewer bytes 2506.10911 app. C.
• The speed-ups are measured on H100 nodes with a throttled network interface 2506.10911 app. B. They are not a general result.
• The convergence result is for a quadratic toy problem with a constant learning rate, and it is not a guarantee for transformer training 2506.10911 §3.2 2506.10911 app. D.2.
• Implementation gaps between the paper and the official code are listed on the card. They include the sign of the gradient term and the value of gamma.

Implementation sketch

Common core of the pairwise variant, following the paper's algorithm and update. The official code differs in two places; see the card before implementing.

# Draw groups for one outer step  [arxiv:2506.10911 §3.2]
# Every replica computes the same groups from the step counter,
# so no extra communication is needed.

def make_groups(num_replicas, step, n):
    # Same random permutation on all replicas  [arxiv:2506.10911 §3.2]
    order = random_permutation(num_replicas, seed = step)
    # Cut the permutation into groups of n  [arxiv:2506.10911 §3.2]
    return [order[i:i + n] for i in range(0, num_replicas, n)]

# Pairwise outer step for one replica  [arxiv:2506.10911 eq. 1] [eq. 2]
# Inputs: slow weights phi, fast weights theta, outer momentum delta,
#         the replica's group, alpha, beta (outer rate), gamma

def outer_step(theta, phi, delta, group, alpha, beta, gamma):
    # Outer gradient of this replica  [arxiv:2506.10911 eq. 1]
    outer_grad = theta - phi
    # Group means of slow weights and outer gradients  [arxiv:2506.10911 §3.2]
    mean_phi = mean(phi of each member of group)
    mean_grad = mean(outer_grad of each member of group)
    # Momentum with a pull toward the group mean  [arxiv:2506.10911 eq. 2]
    delta = alpha * delta - beta * mean_grad - gamma * (phi - mean_phi)
    # Move slow weights by the momentum  [arxiv:2506.10911 eq. 3]
    phi = phi + delta
    # Restart the inner phase from the new slow weights  [arxiv:2506.10911 §3.2]
    return phi, delta, phi

Notes:

• The sign on the outer-gradient term is printed as negative in eq. 2 2506.10911 eq. 2. The sketch keeps the printed sign. The official code uses the opposite sign Card.
• The paper does not give gamma's value. Pick one inside the stability range for your n and alpha 2506.10911 app. D.3.
• Swap the all-reduce for the group exchange at each sync. Each member sends its slow weights and outer gradient to the group, and receives their means.
• The paper reports that random routing of samples through pipeline stages is a separate mixing step during inner steps. The sketch does not include it 2506.10911 §3.1.

Contradictions

Within the paper:

• Eq. 2 prints the outer-gradient term with a negative sign, while the official code adds it 2506.10911 eq. 2 Card.
• Table 2 labels its times "per inner step", but its throughput column matches 100 divided by the time, which implies time per 100-step outer round 2506.10911 tab. 2.
• The text says the step-time ratio approaches 5 at 100 Mb/s, while table 2 gives about 4.05 2506.10911 §4.3 2506.10911 tab. 2.

No other card in this corpus reports gossip averaging, so no cross-paper disagreement is known.

Papers

• NoLoCo: No-all-reduce Low Communication Training Method for Large Models: replaces the global outer sync with random pairwise groups, re-drawn each outer step, and reports lower perplexity than DiLoCo in most runs.