---
kind: concept
slug: pipeline-parallelism
title: Pipeline parallelism
summary: >-
  Pipeline parallelism splits a model into stages and sends each sample through them in order. Only one card covers it so far: NoLoCo combines it with data parallelism and sends each sample through a randomly drawn chain of replicas. Its routing and sync are author-reported, and the page covers that one variant only.
tags: [distributed-training, pipeline-parallelism]
papers: ["2506.10911"]
related: [gossip-averaging, local-sgd]
status: draft
updated_at: 2026-11-08
---

## Overview

The paper splits the model into pipeline stages, and a sample passes through
them in order. Gradients return along the same path [arxiv:2506.10911 §3.1].
It combines this with data parallelism (DP), so every stage is replicated
across several DP ranks [arxiv:2506.10911 §3].

The invariant of the variant on this page: a sample's path through the
replicas of each stage is a random one-to-one pairing between neighbouring
stages, and the pairing is drawn again for each sample. The paper's
experiments use this together with a low-communication outer sync, so that
replicas of a stage are never fully aligned [arxiv:2506.10911 §3.1]
[arxiv:2506.10911 §3.2].

Only one card in this corpus covers the topic. The page does not compare
pipeline schedules, such as fill-and-drain or interleaved schedules, because
no card covers them. Claims below are about the paper's method and its
author-reported results.

## Variants

### Random per-sample routing across replicas

Each replica of a stage sends to exactly one replica of the next stage. Before
a sample crosses a stage boundary, the ranks of the DP group are permuted, and
the permutation decides the receiver. The receiver inverts the same
permutation to find its sender [arxiv:2506.10911 §3.1] [arxiv:2506.10911
alg. 1] [code: src/noloco/dipaco_pipeline.py#L41-L47] [code:
src/noloco/dipaco_pipeline.py#L71-L75].

The paper describes the routing as re-drawn at each step [arxiv:2506.10911
§3.1]. The official code draws a new permutation for each sample instead.
This is a contradiction, and it is listed below. Its pseudocode also gives a
random neighbour for each direction, using a function it does not define
[arxiv:2506.10911 app. A]. The code uses a single permutation instead, so the
routing is a one-to-one matching.

The paper ties this routing to the outer sync, and it reports results with
pipeline stages of 1, 2 and 4 [arxiv:2506.10911 tab. 1]. Its ablation of
random routing against fixed routing, with the outer step switched off,
reports a lower spread of replica weights, by about 15% for the small model
and about 10% for the medium model [arxiv:2506.10911 §5] [arxiv:2506.10911
fig. 8]. The card lists no perplexity result for fixed routing.

Card: [NoLoCo: No-all-reduce Low Communication Training Method for Large
Models](/papers/2506.10911).

## When to use which

The page has one variant, so the choice is between this variant and a
pipeline without random routing. No card reports the second option under the
same conditions, so the table below gives only what the paper supports.

| Situation | Use | Why |
|---|---|---|
| Replicas of each stage sync over slow or high-latency links | Random per-sample routing with the paper's pairwise outer sync | Lower perplexity than DiLoCo in 14 of the 16 runs in the table [arxiv:2506.10911 tab. 1]. Faster wall-clock time at low bandwidth, measured in a pure-DP setting [arxiv:2506.10911 tab. 2] |
| Fast, tightly linked cluster | Not settled by these cards | The wall-clock table is for pure DP, with no pipeline stages. At 100 Gb/s the two methods are almost equal [arxiv:2506.10911 tab. 2] |
| Single stage, or a small number of replicas | Check the table first | The paper's results use 8 to 64 accelerators. Single-stage rows are in the table, but the gains there are small [arxiv:2506.10911 tab. 1] |
| Need the same result as a reference pipeline run | Not settled by these cards | Random routing changes which replica sees each sample. The main comparisons are with FSDP and DiLoCo, and the card lists no fixed-path pipeline baseline [arxiv:2506.10911 §4.1] |
| Limited time to implement | Use the official code as the reference | The code is MIT-licensed and shows the routing and the outer step. Sections of the paper and the code disagree, so follow the code where they differ [arxiv:2506.10911 §3.2] |

The paper's perplexity results do not favour the method in every run. On C4,
the medium model at 64 accelerators with four stages gives 22.9 for NoLoCo
against 22.6 for DiLoCo. The medium model at 32 accelerators with two stages
gives 23.4 against 23.2 [arxiv:2506.10911 tab. 1]. Use the table rather than
the summary when choosing a configuration.

## Implementation sketch

The routing is the common core of this variant. It is written here in our
own words, following the code's choice of one permutation per sample. The
citations point to the code that does this.

```pseudocode
# Random routing for one sample  [arxiv:2506.10911 §3.1]
# Source: dipaco_pipeline.py#L41-L75
# Every rank derives the same permutation from a shared seed, so the sender
# and the receiver agree without extra communication.

def route_sample(dp_rank, stage, sample_seed, num_stages, num_replicas):
    # The seed differs per sample and per stage boundary
    # [code: dipaco_pipeline.py#L155-L158]
    seed = sample_seed * num_stages + stage
    # One-to-one map from sender rank to receiver rank
    # [code: dipaco_pipeline.py#L71-L75]
    permutation = random_permutation(num_replicas, seed)
    next_rank = permutation[dp_rank]
    # The receiver at stage + 1 inverts the same map
    # [code: dipaco_pipeline.py#L41-L47]
    prev_rank = permutation.index(dp_rank)
    return next_rank, prev_rank
```

Where variants differ: the paper's text re-draws the routing per step, and
its Algorithm 1 draws a random neighbour in each direction. The code is
different on both points, as the next section explains.

## Contradictions

- **Routing granularity.** The paper says the routing is re-randomised at
  each step [arxiv:2506.10911 §3.1]. The official code draws a new permutation
  for every sample, using a seed that advances with the batch
  [code: src/noloco/pipeline_llama.py#L62] [code: src/noloco/dipaco_pipeline.py#L155-L158].
- **Routing pseudocode.** Algorithm 1 draws a random rank for the previous and
  the next replica, and the function it uses is not defined [arxiv:2506.10911
  app. A] [arxiv:2506.10911 alg. 1]. The code uses one permutation, so each
  replica sends to exactly one replica and receives from exactly one
  [code: src/noloco/dipaco_pipeline.py#L71-L75].

## Papers

- [NoLoCo: No-all-reduce Low Communication Training Method for Large
  Models](/papers/2506.10911): random per-sample routing between pipeline
  stages, combined with a pairwise outer sync. Author-reported results are
  for 8 to 64 accelerators.