Decentralized federated learning

Overview

In decentralized federated learning (DFL) there is no server that averages the client models. Each client sends its model to other clients, and the models move through the network until the clients stop 2306.01603 §I. The motivation is the centralized case (CFL): one server can be a single point of failure, a bottleneck for communication, and a load for storage and computation when many clients take part 2306.01603 §II-A. The authors argue that removing the server removes that communication and makes the system easier to customize 2306.01603 §I. They do not measure the savings.

The core question is who sends to whom, in what order, and how a client combines what it receives. The first survey answers with five taxonomies: iteration order, communication protocol, network topology, paradigm and temporal variability 2306.01603 §III 2306.01603 tab. 2. Every variant in the taxonomy part of this page is one choice on each axis.

The second survey starts from the literature instead. It reviews 35 DFL works selected by a systematic search and splits them into two families: peer-to-peer methods (TD-FL) and blockchain-based methods (BC-FL) 2308.04604 §3.2 2308.04604 §5. It then groups each family by the challenge its works mainly address 2308.04604 tab. 2. The two surveys use different labels, so a comparison needs a mapping between them.

Two of the three papers are surveys, and neither reports experiments. The trade-offs they state are predictions or literature counts, and several are untested 2306.01603 §III-D 2308.04604 §3. The third paper, WFAgg, is the only one with experiments. It reports one benchmark (MNIST with 20 nodes), and the numbers are its authors' own 2409.17754 §5. Read the survey variants as a map, and the WFAgg results as a first look.

Variants

Family: peer-to-peer or blockchain

The second survey splits DFL by how the models are exchanged 2308.04604 §5:

• TD-FL (peer-to-peer). Peers exchange models with standard distributed computing, not a ledger. The survey counts 23 works in this family 2308.04604 tab. 2 2308.04604 §3.4.
• BC-FL (blockchain). Models or updates are recorded on a blockchain. The survey counts 12 works 2308.04604 tab. 2. It argues that a ledger should be an optional part, used only for a concrete need such as auditability or incentive settlement 2308.04604 §2.3. In the works it reviews, the ledger often moves trust to miners or validators, which resembles a hierarchy like centralized FL 2308.04604 §6.1 2308.04604 §6.3.

The same survey groups the works by the challenge they mainly target 2308.04604 tab. 2. Works named in its table, by group:

• Fault tolerance: TD-FL IPLS, DFL-UN and DSpodFL; BC-FL FED-BC and GFL.
• Privacy: TD-FL ProxyFL, PVD-FL, f-DP and ERIS; BC-FL Privacy-Preserving FL for IoT, DeepChain and Biscotti.
• Security: TD-FL Trusted DFL, BALANCE, BR-DFL and SelfishAttack; BC-FL BytoChain, PIRATE and Trustworthy FL.
• Incentives: only BC-FL works, BlockFL and FedCoin.
• Bandwidth: TD-FL Combo and GossipFL.
• Data heterogeneity: TD-FL Def-KT, P2P FL over graphs, PANM, DeSA and NTK-DFL.
• Resource efficiency: BC-FL BAFL and BLADE-FL.

The table names the works and their group. It does not report how any of them performed, so use it to find candidates, then read each work's own results.

Paradigm: Continual or Aggregate

The paradigm decides what a client does with the models it receives 2306.01603 §III-D 2306.01603 tab. 3.

• Continual. A client takes one model from one peer, trains on it, and passes the result on. Storage and aggregation cost are low. The named risk is catastrophic forgetting 2306.01603 tab. 3. The paper describes this as a sequential hand-off and cites continual learning as one of its bases 2306.01603 §III-D.
• Aggregate. A client averages all the models it has received, trains on the average, and passes its model on together with the older ones. It needs more storage and communication. The authors expect better generalization 2306.01603 §III-D 2306.01603 tab. 3.

Communication protocol

The protocol sets how many peers receive each model 2306.01603 §III-B:

• Pointing: one specified peer.
• Gossip: one random peer.
• Broadcast: all peers.
• Broadcast-gossip: several peers.

Iteration order

The order in which clients take their turns 2306.01603 §III-A:

• Sequential, random or cycle: clients take turns one at a time, in a fixed order, a random order, or round after round around a loop.
• Parallel: all clients communicate asynchronously.

The authors say the order matters in DFL, unlike in CFL 2306.01603 §III-A.

Topology

The network shape 2306.01603 §III-C 2306.01603 §IV:

• Line (§IV-B): the base case, a sequential pointing line. The paper notes that the first clients in the queue end with worse models.
• Ring (§IV-C): a cyclic line that can keep iterating.
• Mesh (§IV-D): every client sends to every other client. The paper names Stochastic Gradient Push and BrainTorrent as examples.
• Star (§IV-E): one client acts as a central aggregator. The paper names split learning in a star and Swarm Learning, where the leader changes.
• Hybrid (§IV-F): combinations, such as two rings joined by central clients, or meshes inside organisations whose leaders form a ring. Matcha is the named example.

The paper also defines bus and tree shapes in its taxonomy, but §IV develops neither 2306.01603 §III-C.

Temporal variability

The topology is either static or dynamic. The paper gives three reasons to change it: external interference, communication savings and fairness 2306.01603 §III-E.

Robust aggregation: WFAgg

WFAgg is a rule for when some neighbours send bad models. It is built for decentralized learning, where no server can check the models first 2409.17754 §1. For the wider family of such rules, see Byzantine-robust aggregation. A node runs three filters on the models it received in a round 2409.17754 §4.2 2409.17754 alg. 1:

• Distance (WFAgg-D). Measures each neighbour's squared distance to the coordinate-wise median, and keeps the closest K − f − 1 models, where f is the assumed number of malicious nodes 2409.17754 alg. 2 2409.17754 §4.3.
• Cosine (WFAgg-C). Clips each neighbour's norm to the median norm, then keeps the neighbours with the smallest cosine distance to the median 2409.17754 alg. 3 2409.17754 §4.4.
• Temporal (WFAgg-T). Accepts a neighbour when its change since its last model stays within one standard deviation of its recent history. It accepts nothing during an initial transient period 2409.17754 alg. 4 2409.17754 §4.5.

Each filter that accepts a model adds its vote. The experiments use votes of 0.4, 0.4 and 0.2 2409.17754 §5.1. A model whose total is below 0.6 gets zero weight. The node then mixes its own model with the normalised mean of the rest, with smoothing α = 0.8 2409.17754 eq. 3 2409.17754 alg. 1.

The paper's results are author-reported. On MNIST with 20 nodes and two malicious neighbours, WFAgg keeps 94.8–95.8% benign accuracy across the attack settings in its table. Multi-Krum falls to 87.1% under IPM-0.5 2409.17754 tab. 1. With no attack and no malicious neighbours, WFAgg scores 97.40% and Mean scores 97.47% 2409.17754 tab. 1. Its experiments use one dataset, IID data and 10 rounds 2409.17754 §5.1. The released code differs from the paper in several details, so read the paper card before you reproduce the numbers.

When to use which

No survey gives a measured comparison. The table keeps to what the papers state, and marks where a row is only a prediction or an open problem. The WFAgg rows rest on the authors' own experiments.

Costs the first survey flags. Communication overhead is addressed with model compression and topology choice 2306.01603 §V-A. Under Aggregate, storage and computation grow because clients keep past models. The proposed remedy is to freeze lower layers, as transfer learning does 2306.01603 §V-B.

Implementation sketch

One pass of the generic loop. Each client takes its turn, receives models from the peers its protocol and topology name, combines them by its paradigm, trains, and sends the result on 2306.01603 alg. 1 2306.01603 alg. 2.

# One pass of decentralized FL, generic form  [alg. 1] [alg. 2]
# Input:  clients, peers_of (protocol + topology), paradigm,
#         epochs E, learning rate eta, loss L, models (one per client)
# Output: updated models

def dfl_pass(clients, peers_of, paradigm, epochs, eta, loss, models):
    # Visit clients in the chosen iteration order  [§III-A]
    for client in clients:
        # Peers that send to this client, set by protocol and topology  [§III-B]
        inbox = [models[p] for p in peers_of(client)]
        if paradigm == "Continual":
            # Continual: start from one incoming model  [tab. 3]
            start = inbox[0]
        else:
            # Aggregate: start from the average of all incoming models  [tab. 3]
            # The paper does not say what the first client averages  [§III-D]
            start = average(inbox)
        model = start
        # Local training: the paper repeats the update E-1 times  [alg. 1]
        for _ in range(epochs - 1):
            model = model - eta * gradient(loss, model, client)
        models[client] = model
        # Send the model to the peers that the protocol names  [§III-B]
    return models

Notes for implementers:

• The loop has no stop rule. Algorithm 2 runs while the cycle continues, and Table 3 names convergence as the end point. Stop the loop yourself alg. 2 tab. 3.
• The paper gives no value for E, eta, the loss, the number of clients, the batch size or the aggregation weights. Use your own values and record them.
• Aggregate needs at least two models to mean something §III-D. The paper does not say what the first client averages, so decide that rule yourself.
• Under the Aggregate mesh, the aggregate for client c uses models from clients before c only, while the send step goes to all clients alg. 2. Choose one rule and state it.
• Continual names a hand-off. Do not assume the replay or regularization methods of continual learning carry over to it §III-D.
• The second survey gives a three-step round (local training, exchange with neighbours, aggregation) as a figure summary, not as an algorithm. It has no stop rule either 2308.04604 fig. 1 2308.04604 §2.1.

Contradictions

Inside each paper, these points do not line up:

• First survey: Table 3 names convergence as the end point, but Algorithm 2 has no stop rule tab. 3 alg. 2.
• First survey: §III-D says Aggregate needs at least two models, but Algorithm 1 averages the received models for every client, including the first §III-D alg. 1.
• First survey: §III-D predicts faster convergence for Aggregate and more rounds for Continual. These predictions are untested, and the paper names its own assumptions for them §III-D.

Between the papers, the reports differ on these points:

• The word "gossip". The first survey's gossip protocol sends each model to one random peer 2306.01603 §III-B. WFAgg sends each model to all neighbours of a fixed graph, which it calls gossip-style [arxiv:2409.17754 §4.1]. Check the definition in each paper before you compare protocols.
• Removing the server and cost. The first survey presents dropping the server as a saving in communication, without numbers 2306.01603 §I. The second survey says a fully connected topology scales no better than standard FL and gives an O(n²) vs O(n) message count with no source 2308.04604 §1 2308.04604 §4.2. Both are assertions. They disagree on the direction for dense (mesh) designs, and neither measures it.
• Blockchain as a defense. The first survey lists blockchain-based privacy among the named defenses 2306.01603 §V-C. The second survey argues that a ledger should be optional and that in the surveyed works it often shifts trust to miners or validators 2308.04604 §2.3 2308.04604 §6.1. Neither paper compares the two positions with data.

Papers

• Decentralized Federated Learning: A Survey and Perspective: a survey that sorts DFL designs into five taxonomies, defines the Continual and Aggregate paradigms with pseudocode, and lists five topology variants. It has no experiments.
• A Survey on Decentralized Federated Learning: a systematic review of 35 DFL works. It splits them into peer-to-peer (TD-FL) and blockchain (BC-FL) families and maps them to the challenges in its table 2, then lists open gaps. It has no experiments of its own.
• Byzantine-Robust Aggregation for Securing Decentralized Federated Learning: WFAgg, a rule with three filters (distance, cosine, temporal) for decentralized nodes. It reports results on MNIST with 20 nodes, against centralized and decentralized baselines, all from its authors.