Pipeline parallelism

Overview

The paper splits the model into pipeline stages, and a sample passes through them in order. Gradients return along the same path 2506.10911 §3.1. It combines this with data parallelism (DP), so every stage is replicated across several DP ranks 2506.10911 §3.

The invariant of the variant on this page: a sample's path through the replicas of each stage is a random one-to-one pairing between neighbouring stages, and the pairing is drawn again for each sample. The paper's experiments use this together with a low-communication outer sync, so that replicas of a stage are never fully aligned 2506.10911 §3.1 2506.10911 §3.2.

Only one card in this corpus covers the topic. The page does not compare pipeline schedules, such as fill-and-drain or interleaved schedules, because no card covers them. Claims below are about the paper's method and its author-reported results.

Variants

Random per-sample routing across replicas

Each replica of a stage sends to exactly one replica of the next stage. Before a sample crosses a stage boundary, the ranks of the DP group are permuted, and the permutation decides the receiver. The receiver inverts the same permutation to find its sender 2506.10911 §3.1 [arxiv:2506.10911 alg. 1] dipaco_pipeline.py [code: src/noloco/dipaco_pipeline.py#L71-L75].

The paper describes the routing as re-drawn at each step [arxiv:2506.10911 §3.1]. The official code draws a new permutation for each sample instead. This is a contradiction, and it is listed below. Its pseudocode also gives a random neighbour for each direction, using a function it does not define 2506.10911 app. A. The code uses a single permutation instead, so the routing is a one-to-one matching.

The paper ties this routing to the outer sync, and it reports results with pipeline stages of 1, 2 and 4 2506.10911 tab. 1. Its ablation of random routing against fixed routing, with the outer step switched off, reports a lower spread of replica weights, by about 15% for the small model and about 10% for the medium model 2506.10911 §5 [arxiv:2506.10911 fig. 8]. The card lists no perplexity result for fixed routing.

Card: NoLoCo: No-all-reduce Low Communication Training Method for Large Models.

When to use which

The page has one variant, so the choice is between this variant and a pipeline without random routing. No card reports the second option under the same conditions, so the table below gives only what the paper supports.

The paper's perplexity results do not favour the method in every run. On C4, the medium model at 64 accelerators with four stages gives 22.9 for NoLoCo against 22.6 for DiLoCo. The medium model at 32 accelerators with two stages gives 23.4 against 23.2 2506.10911 tab. 1. Use the table rather than the summary when choosing a configuration.

Implementation sketch

The routing is the common core of this variant. It is written here in our own words, following the code's choice of one permutation per sample. The citations point to the code that does this.

# Random routing for one sample  [arxiv:2506.10911 §3.1]
# Source: dipaco_pipeline.py#L41-L75
# Every rank derives the same permutation from a shared seed, so the sender
# and the receiver agree without extra communication.

def route_sample(dp_rank, stage, sample_seed, num_stages, num_replicas):
    # The seed differs per sample and per stage boundary
    # [code: dipaco_pipeline.py#L155-L158]
    seed = sample_seed * num_stages + stage
    # One-to-one map from sender rank to receiver rank
    # [code: dipaco_pipeline.py#L71-L75]
    permutation = random_permutation(num_replicas, seed)
    next_rank = permutation[dp_rank]
    # The receiver at stage + 1 inverts the same map
    # [code: dipaco_pipeline.py#L41-L47]
    prev_rank = permutation.index(dp_rank)
    return next_rank, prev_rank

Where variants differ: the paper's text re-draws the routing per step, and its Algorithm 1 draws a random neighbour in each direction. The code is different on both points, as the next section explains.

Contradictions

• Routing granularity. The paper says the routing is re-randomised at each step 2506.10911 §3.1. The official code draws a new permutation for every sample, using a seed that advances with the batch pipeline_llama.py dipaco_pipeline.py.
• Routing pseudocode. Algorithm 1 draws a random rank for the previous and the next replica, and the function it uses is not defined [arxiv:2506.10911 app. A] 2506.10911 alg. 1. The code uses one permutation, so each replica sends to exactly one replica and receives from exactly one dipaco_pipeline.py.

Papers

• NoLoCo: No-all-reduce Low Communication Training Method for Large Models: random per-sample routing between pipeline stages, combined with a pairwise outer sync. Author-reported results are for 8 to 64 accelerators.