Decentralized Federated Learning: A Survey and Perspective

In plain words

Federated learning lets many devices or organizations train one shared model without sending their raw data to a central place. Usually a central server combines the updates each participant computes. This paper looks at the version with no server, where participants pass models directly to each other.

The authors sort these designs into five questions: in what order participants take turns, how they send messages, what shape the network has, how each one combines the models it receives, and whether the network changes over time. They describe two basic ways to pass models along. One learns from the single model it just received. The other first averages every model it has received and then learns from that average. The paper then shows example network shapes and lists open problems. It is a map of the field, not a test of a new method, so it reports no experiments of its own.

TL;DR

• Survey and perspective on decentralized federated learning (DFL): clients share model parameters directly, with no server §I.
• Five taxonomies: iteration order, communication protocol, network topology, paradigm and temporal variability §III tab. 2.
• Two paradigms, Continual (one incoming model) and Aggregate (average all incoming models), with pseudocode §III-D tab. 3 alg. 1 alg. 2.
• Five topology variants: line, ring, mesh, star and hybrid §IV.
• No experiments: the expected trade-offs in §III-D are untested predictions.

Problem

Centralized federated learning (CFL) runs on one server that averages client models. The paper treats that server as a single point of failure, as a bottleneck for communication and as a load for storage and computation when the number of clients is very large §II-A. The authors argue that dropping the server removes the communication the server needs and makes the system more customizable §I. They do not quantify these savings. They also note that most recent surveys cover CFL and that no survey before theirs covered the full DFL design chain §I tab. 1.

Method

This is a taxonomy, not a training method. The paper first reviews CFL: FedAvg averages client models weighted by sample count §II §II-A, and related variants cover proximal terms, differential-privacy noise, hierarchical, personalized, split, graph and asynchronous FL §II-B.

The paper's five DFL taxonomies §III tab. 2:

1. Iteration order (§III-A): sequential, random, cycle or parallel. In parallel, all clients communicate asynchronously. The authors say the order matters in DFL, unlike CFL §III-A.
2. Communication protocol (§III-B): pointing (one specified peer), gossip (a random peer), broadcast (all peers) or broadcast-gossip (several peers).
3. Network topology (§III-C): line, bus, ring, mesh, star, tree or hybrid. Bus and tree are defined in the table but no variant in §IV develops them.
4. Paradigm (§III-D) tab. 3: Continual or Aggregate, as described below.
5. Temporal variability (§III-E): static or dynamic. The paper lists three reasons to change topology: external interference, communication savings and fairness.

The two paradigms §III-D tab. 3:

• Continual: a client receives one model from one peer, trains on it and passes the result on. Storage and aggregation cost are low. Catastrophic forgetting is the named risk tab. 3.
• Aggregate: a client receives several models, averages them, trains on the average and passes on its model together with the older ones. It needs more storage and communication, and the authors expect better generalization §III-D tab. 3.

Algorithm 1 gives the sequential line form. Its two highlighted steps (copy the previous model, or aggregate received models) are alternatives, one per paradigm alg. 1. The local update is a gradient step with learning rate eta, repeated E-1 times alg. 1.

# Sequential pointing line DFL  [alg. 1]
# Input:  clients (ordered), epochs E, initial model w0, loss L, rate eta
# Output: one local model per client

def sequential_line_dfl(clients, epochs, w0, loss, eta, paradigm):
    # Models received so far; only Aggregate reads them  [alg. 1]
    received = []
    # Client 1 starts from the initial model  [alg. 1]
    previous = w0
    local_models = []
    for client in clients:  # in sequence  [alg. 1]
        if paradigm == "Continual":
            # Continual: copy the previous client's model  [alg. 1]
            model = previous
        else:
            # Aggregate: average the received models  [alg. 1]
            # Client 1 has none; the paper says Aggregate needs two  [§III-D]
            model = average(received)
        for _ in range(epochs - 1):  # local updates  [alg. 1]
            # Gradient step on the local loss  [alg. 1]
            model = model - eta * gradient(loss, model, client)
        local_models.append(model)
        # Pass the trained model on; Aggregate also passes older ones  [alg. 1]
        received = received + [model]
        previous = model
    return local_models

Algorithm 2 gives the cyclic form: a ring for Continual, a broadcast mesh for Aggregate alg. 2. The ring variant passes the model to the next client, and the mesh variant sends the trained model to all clients alg. 2.

# Cycle pointing ring (Continual) or broadcast mesh (Aggregate)  [alg. 2]
# Input:  clients (ordered), epochs E, initial model w0, loss L, rate eta
# Output: local models, updated while the cycle runs

def cyclic_dfl(clients, epochs, w0, loss, eta, paradigm):
    # Visit clients in a cycle; the paper gives no stop rule  [alg. 2]
    received = []
    previous = w0
    for client in cycle(clients):
        if paradigm == "Continual":
            # Continual: copy the previous client's model  [alg. 2]
            model = previous
        else:
            # Aggregate: average the models received so far  [alg. 2]
            model = average(received)
        for _ in range(epochs - 1):  # local updates  [alg. 2]
            model = model - eta * gradient(loss, model, client)
        # Pointing: next client only. Broadcast: all clients  [alg. 2]
        previous = model
        received = received + [model]

Topology variants §IV fig. 4 fig. 6:

• Line (§IV-B): the base case, a sequential pointing line. Used as a baseline in several cited works. The first clients in the queue get worse models.
• Ring (§IV-C): a cyclic pointing line, which can keep iterating. Examples: Chang et al. and Sheller et al. §IV-C.
• Mesh (§IV-D): every client sends to every other client. Stochastic Gradient Push (SGP) and BrainTorrent are the named examples §IV-D.
• Star (§IV-E): one client acts as a central aggregator. Pappas et al. (split learning in a star) and Swarm Learning (dynamic leader) are the named examples §IV-E.
• Hybrid (§IV-F): combinations, for example two rings joined through central clients, or meshes inside organizations whose leaders form a ring §IV-F. Matcha is the named example §IV-F.

Table 4 lists 12 cited DFL works with their paradigm, topology and one highlight each tab. 4.

Challenges and proposed directions §V:

• Communication overhead (§V-A): model compression and topology choice; Wang et al. (CoCo) is the named example §V-A.
• Computation and storage (§V-B): under Aggregate, clients keep past models and aggregate them. The proposal is to freeze lower layers, as transfer learning does §V-B.
• Cybersecurity (§V-C): attacks, poisoning and free-riding are harder to control without a server. Named defenses include blockchain-based privacy, differential privacy and Multi-Krum with secure aggregation §V-C.
• Incentives (§V-D): reputation scores are one proposal; the authors call incentive design an open problem §V-D.
• Management (§V-E): pre-requesting peer status (BrainTorrent), electing a leader per round (BDFL, Swarm Learning), and contingency plans §V-E.

Figures (no images are published here): Fig. 1 compares CFL and DFL on nine metrics. Fig. 2 contrasts local learning, centralized learning, CFL and DFL. Fig. 3 is the paper roadmap. Fig. 4 shows topology diagrams. Fig. 5 shows the two paradigms in parameter space, with arrow length standing for update size measured by the l2 norm. Fig. 6 shows four imagined real-world topologies (a line/ring, b mesh, c star, d hybrid) fig. 1 fig. 2 fig. 3 fig. 4 fig. 5 fig. 6.

Hyperparameters

The paper gives no values. Algorithms 1 and 2 use symbols only.

Results

Not applicable. The paper reports no experiments. Its numbers are literature counts, author-reported.

Paper vs code

No official code. The arXiv metadata has no code link, the LaTeX source (main.tex, reference.bib) has no URL, and no project page is linked. A GitHub repository search for "decentralized federated learning survey" returned no results.

Pitfalls

• No experiments. The five expectations in §III-D (loss oscillation, faster convergence for Aggregate, more rounds for Continual, stronger personalization for Continual) are speculation under stated assumptions. The paper's assumptions are that local optima are multivariate normal, some clients are biased, and models are not always trained to optimum §III-D.
• Communication savings are asserted, not measured. The paper gives no bandwidth or round counts for DFL against CFL §I §V-A.
• Algorithms mix two paradigms. In Algorithms 1 and 2 the Continual and Aggregate steps are colour-coded alternatives, not sequential steps alg. 1 alg. 2. The pseudocode above chooses one by a paradigm argument.
• Client 1 has no received models. Aggregate needs at least two models to be meaningful §III-D, but Algorithm 1 averages {w_1..w_{c-1}} for every client, including client 1. The paper does not say what client 1 uses.
• Algorithm 2 has no stop rule. It runs While over a cycle, while Table 3 says "until convergence" alg. 2 tab. 3. Stop the loop yourself.
• Algorithm 2 is ambiguous on the mesh. Its Aggregate branch sends to all clients, but the aggregate for client c uses models 1..c-1 only alg. 2.
• "Continual" is not standard continual learning. The paper names it a sequential hand-off and cites continual learning as a basis §III-D. Do not assume the usual replay or regularization methods apply.
• Figure labels in the text do not all match. §IV-F cites Fig. 4(g), but the LaTeX figure has no panel (g); the hybrid example is in Fig. 6(d). The panel letters in Fig. 4 come from the rendered PDF, which the source does not show. Check it before relying on them.
• Roman section numbers. IEEEtran numbers sections I to VI and subsections A to E, so cite §III-D as in the text, not as 3.4.
• Self-assessed comparison. Table 1 marks "Ours" as covering all five taxonomies; the authors assessed this themselves tab. 1.
• Mixed definitions. The authors say the five taxonomies "may exhibit overlapping meanings as well as conflicting aspects" and include "inferences regarding potential definitions" §III.

Lineage

• Builds on: FedAvg, the first mature CFL algorithm, reviewed in §II-A §II. Its weighting by sample count is the aggregation the DFL paradigms are compared with.
• Compared against: 13 related surveys, listed in tab. 1.
• Superseded by: not known from this wiki.