---
kind: concept
slug: gossip-averaging
title: Gossip averaging
summary: >-
  Gossip averaging replaces a global all-reduce with averaging inside small, randomly drawn groups. Only one card covers it so far. NoLoCo re-draws random pairs at each outer step. Author-reported, it beats DiLoCo in most runs, but its own table shows synchronous FSDP with lower perplexity in every run. The page covers that one variant.
tags: [distributed-training, gossip, decentralized, communication-efficiency]
papers: ["2506.10911"]
related: [local-sgd, decentralized-federated-learning, federated-aggregation]
status: draft
updated_at: 2026-11-08
---

## Overview

Data-parallel training keeps model replicas identical with a global exchange,
often an all-reduce. That exchange can dominate step time on slow or
high-latency links [arxiv:2506.10911 §1] [arxiv:2506.10911 §2]. Gossip
averaging replaces the global exchange with averaging inside small groups, so
no sync waits on every replica [arxiv:2506.10911 §3.2].

The invariant of the variant on this page: at each sync, a replica's new slow
weights depend only on its own group. The groups are drawn again at the next
sync, so the information spreads over many rounds. Replicas are not identical
after a sync. The paper reports that their spread grows early in training and
shrinks as the learning rate decays [arxiv:2506.10911 §4.2].

The paper cites gossip averaging as motivation for pairwise mixing
[arxiv:2506.10911 §2]. It gives no general theory of gossip averaging, and no
other card in this corpus covers the topic. Claims below are about the paper's
method only.

## Variants

### Random pairwise groups with a modified Nesterov outer step

Each outer step, all replicas draw the same random permutation. The
permutation is cut into groups of n, and the paper's experiments use n = 2
[arxiv:2506.10911 §3.2] [arxiv:2506.10911 §4.1]. The paper states that the
network should have more than 2n replicas [arxiv:2506.10911 §4.1]. Within a
group, each replica uses the group mean of the slow weights and of the outer
gradients in its momentum update. The outer gradient of a replica is its fast
weights minus its slow weights [arxiv:2506.10911 eq. 1]. The paper's update
adds a pull term that moves each replica's slow weights toward the group mean
of the slow weights, scaled by a local-averaging coefficient gamma
[arxiv:2506.10911 eq. 2] [arxiv:2506.10911 §3.2].

The paper gives no value for gamma in the text [arxiv:2506.10911 §4.1].
For n = 2 and alpha = 0.5, the stability range it derives is
0.5 < gamma < 1.5 [arxiv:2506.10911 app. D.3]. The paper's theorem uses a
quadratic loss with a Gaussian shift of the optimum and a constant inner
learning rate, and it assumes infinitely many outer steps. Under these
assumptions the expected slow weights go to zero and their variance scales
with the square of the inner learning rate [arxiv:2506.10911 §3.2]. The
variance step relies on a conjecture stated in appendix D [arxiv:2506.10911
app. D.2].

Card: [NoLoCo: No-all-reduce Low Communication Training Method for Large
Models](/papers/2506.10911).

## Hyperparameters

Values for the main language-model runs [arxiv:2506.10911 §4.1]. The paper
gives the architectures in its table 3 [arxiv:2506.10911 tab. 3].

- Group size n = 2 for the pairwise variant [arxiv:2506.10911 §4.1].
- Outer learning rate beta = 0.7 and outer momentum alpha = 0.5 for the
  pairwise variant, against alpha = 0.3 for DiLoCo [arxiv:2506.10911 §4.1].
- 50 inner steps per outer step for the pairwise variant, against 100 for
  DiLoCo [arxiv:2506.10911 §4.1].
- Inner optimiser Adam, gradient clipping at 1.0, a cosine schedule with a
  linear warm-up of 1000 steps, and 25,000 optimiser steps [arxiv:2506.10911
  §4.1] [arxiv:2506.10911 app. B].
- The paper does not state Adam's betas or epsilon, nor how replicas start,
  and it does not say what happens to optimiser moments at a sync
  [arxiv:2506.10911 app. B]. The card lists the implementation gaps.

## Results

Author-reported. The experiments run decentralised data parallelism combined
with pipeline parallelism, with Llama-style models of 125M, 1.3B and 6.8B
parameters, on Reddit and C4 data. The baselines are synchronous FSDP and
DiLoCo [arxiv:2506.10911 §4.1]. The paper gives no number of random seeds
[arxiv:2506.10911 §4.2].

Validation perplexity [arxiv:2506.10911 tab. 1]:

- Against DiLoCo, the pairwise variant has lower perplexity in 8 of 8 Reddit
  runs and 6 of 8 C4 runs. The differences are small, up to about 4%
  [arxiv:2506.10911 §4.2] [arxiv:2506.10911 tab. 1].
- Against FSDP, read from the same table, the pairwise variant has higher
  perplexity in every row. The paper's comparison focuses on DiLoCo, and
  this table does not show a case where the pairwise variant beats FSDP.

Wall-clock time per 100 inner steps on 32 workers, medium model, pure data
parallelism [arxiv:2506.10911 tab. 2] [arxiv:2506.10911 §4.3]:

| Method | 100 Mb/s (s) | 1 Gb/s (s) | 100 Gb/s (s) |
|---|---|---|---|
| FSDP | 5368.41 | 548.97 | 41.39 |
| DiLoCo | 61.80 | 6.55 | 1.70 |
| Pairwise variant | 15.26 | 2.80 | 1.64 |

The pairwise variant is about 4.05 times faster than DiLoCo at 100 Mb/s, and
the two are nearly equal at 100 Gb/s [arxiv:2506.10911 tab. 2]. The authors
combine this with per-step convergence to claim about 4 times faster
time-to-target on consumer networks [arxiv:2506.10911 §4.3].

A simulation over world sizes from 4 to 1024 finds that the expected ratio of
tree all-reduce time to pairwise-averaging time grows with world size and with
the variance of message times [arxiv:2506.10911 §4.3] [arxiv:2506.10911 fig.
5]. The authors state that DiLoCo falls about 20% behind at 1024
accelerators, counting only the blocking wait for the slowest worker
[arxiv:2506.10911 §4.3]. The simulation assumes log-normal inner-step latency
[arxiv:2506.10911 app. G].

Group size, on the 125M model with 0.5M training tokens: perplexity improves
only weakly as n goes from 2 to 4 and 8 [arxiv:2506.10911 §5] [arxiv:2506.10911
fig. 7]. Sweeps of alpha, beta, gamma and H are shown only as figures on the
1.3B C4 model [arxiv:2506.10911 §5] [arxiv:2506.10911 fig. 6].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Fast links and few workers, where synchronous all-reduce is affordable | Keep the all-reduce (FSDP) | FSDP has the lowest perplexity in every run of the paper's table [arxiv:2506.10911 tab. 1] |
| Slow links, and the global outer sync is the bottleneck | Pairwise group sync | Author-reported speed-up over DiLoCo at 100 Mb/s [arxiv:2506.10911 tab. 2] |
| Fast interconnect, where the speed-up over DiLoCo disappears | Either; the gain is small | Nearly equal wall-clock at 100 Gb/s [arxiv:2506.10911 tab. 2] |
| Many workers with variable message latency | Pairwise group sync | Simulated gap that grows with world size [arxiv:2506.10911 §4.3] [arxiv:2506.10911 fig. 5] |
| You need a value for gamma you can justify | Wait, or sweep it | The text gives no value, and the sweep is figure-only [arxiv:2506.10911 §4.1] [arxiv:2506.10911 fig. 6] |

Caveats the paper raises or the numbers show:

- Results are author-reported on one model family. The paper gives no seed
  count [arxiv:2506.10911 §4.2].
- The paper attributes the perplexity gain to regularisation from weight noise
  and does not test this [arxiv:2506.10911 §4.2].
- Per sync, the pairwise variant moves about the same volume of data as
  DiLoCo. The saving comes from removing the global wait, not from fewer bytes
  [arxiv:2506.10911 app. C].
- The speed-ups are measured on H100 nodes with a throttled network interface
  [arxiv:2506.10911 app. B]. They are not a general result.
- The convergence result is for a quadratic toy problem with a constant
  learning rate, and it is not a guarantee for transformer training
  [arxiv:2506.10911 §3.2] [arxiv:2506.10911 app. D.2].
- Implementation gaps between the paper and the official code are listed on
  the [card](/papers/2506.10911). They include the sign of the gradient term
  and the value of gamma.

## Implementation sketch

Common core of the pairwise variant, following the paper's algorithm and
update. The official code differs in two places; see the card before
implementing.

```pseudocode
# Draw groups for one outer step  [arxiv:2506.10911 §3.2]
# Every replica computes the same groups from the step counter,
# so no extra communication is needed.

def make_groups(num_replicas, step, n):
    # Same random permutation on all replicas  [arxiv:2506.10911 §3.2]
    order = random_permutation(num_replicas, seed = step)
    # Cut the permutation into groups of n  [arxiv:2506.10911 §3.2]
    return [order[i:i + n] for i in range(0, num_replicas, n)]
```

```pseudocode
# Pairwise outer step for one replica  [arxiv:2506.10911 eq. 1] [eq. 2]
# Inputs: slow weights phi, fast weights theta, outer momentum delta,
#         the replica's group, alpha, beta (outer rate), gamma

def outer_step(theta, phi, delta, group, alpha, beta, gamma):
    # Outer gradient of this replica  [arxiv:2506.10911 eq. 1]
    outer_grad = theta - phi
    # Group means of slow weights and outer gradients  [arxiv:2506.10911 §3.2]
    mean_phi = mean(phi of each member of group)
    mean_grad = mean(outer_grad of each member of group)
    # Momentum with a pull toward the group mean  [arxiv:2506.10911 eq. 2]
    delta = alpha * delta - beta * mean_grad - gamma * (phi - mean_phi)
    # Move slow weights by the momentum  [arxiv:2506.10911 eq. 3]
    phi = phi + delta
    # Restart the inner phase from the new slow weights  [arxiv:2506.10911 §3.2]
    return phi, delta, phi
```

Notes:

- The sign on the outer-gradient term is printed as negative in eq. 2
  [arxiv:2506.10911 eq. 2]. The sketch keeps the printed sign. The official
  code uses the opposite sign [Card](/papers/2506.10911).
- The paper does not give gamma's value. Pick one inside the stability range
  for your n and alpha [arxiv:2506.10911 app. D.3].
- Swap the all-reduce for the group exchange at each sync. Each member sends
  its slow weights and outer gradient to the group, and receives their means.
- The paper reports that random routing of samples through pipeline stages is
  a separate mixing step during inner steps. The sketch does not include it
  [arxiv:2506.10911 §3.1].

## Contradictions

Within the paper:

- Eq. 2 prints the outer-gradient term with a negative sign, while the
  official code adds it [arxiv:2506.10911 eq. 2] [Card](/papers/2506.10911).
- Table 2 labels its times "per inner step", but its throughput column matches
  100 divided by the time, which implies time per 100-step outer round
  [arxiv:2506.10911 tab. 2].
- The text says the step-time ratio approaches 5 at 100 Mb/s, while table 2
  gives about 4.05 [arxiv:2506.10911 §4.3] [arxiv:2506.10911 tab. 2].

No other card in this corpus reports gossip averaging, so no cross-paper
disagreement is known.

## Papers

- [NoLoCo: No-all-reduce Low Communication Training Method for Large Models](/papers/2506.10911): replaces the global outer sync with random pairwise groups, re-drawn each outer step, and reports lower perplexity than DiLoCo in most runs.