Byzantine-robust aggregation

Overview

A node that averages the models it receives can be pulled off course by a sender that is faulty or malicious. A Byzantine-robust aggregation rule is the step that combines the received models and limits that damage. The problem is sharper without a central server: each node sees only its neighbours, may have few of them, and a large share of them may be malicious 2409.17754 §1 2409.17754 §4.2. The round loop is on the federated learning page, and the aggregation step in general is on the federated aggregation page.

One card covers this so far. The shared idea in it is to score each received model with several tests, keep the models that pass enough of them, and blend their average with the node's own model 2409.17754 §4.2 alg. 1.

Variants

WFAgg: three filters, then a blend

A rule for decentralized federated learning, where nodes exchange models with their neighbours and no server exists 2409.17754 §1. Each node runs three filters on the same received set:

• Distance (WFAgg-D): distance to the coordinate-wise median; keeps the K − f − 1 closest models, f being the assumed number of malicious nodes 2409.17754 §4.3 alg. 2.
• Cosine (WFAgg-C): each model's norm is clipped to the median norm, then cosine distance to the median is measured; the same K − f − 1 rule applies 2409.17754 §4.4 alg. 3.
• Temporal (WFAgg-T): each neighbour's change from its previous model is compared with its own recent history; a model is accepted only when both its distance change and its cosine change lie within one standard deviation of their means. Nothing is accepted in the first rounds 2409.17754 §4.5 alg. 4.

Each filter that accepts a model adds its vote τ. In the experiments τ = (0.4, 0.4, 0.2) for (D, C, T). A model whose votes sum to less than 0.6 gets weight zero. The remaining models are averaged with normalised weights and blended with the node's own model, using smoothing factor α = 0.8 2409.17754 §5.1 alg. 1 eq. 3. Card: Byzantine-Robust Aggregation for Securing Decentralized Federated Learning.

Single filters and the blend alone

The paper's table also reports WFAgg-D, WFAgg-C, WFAgg-T and WFAgg-E (the blend without the filters) on their own, and a combined variant named Alt-WFAgg, built from Multi-Krum and Clustering 2409.17754 tab. 1 §6.2.2. Single filters fail on some attacks (see When to use which).

Reference rules compared in the paper

The paper compares against Mean, Trimmed-Mean, Median, Krum, Multi-Krum and Clustering, in centralized and decentralized settings 2409.17754 §5.1 2409.17754 tab. 1. This page does not describe them further, because the only source here is the paper's own comparison.

When to use which

The evidence covers one setting: 20 nodes on an 8-regular graph, at most two malicious neighbours per node, MNIST with IID data, 10 rounds, one small CNN 2409.17754 §5.1. Choose by constraints, and treat anything outside that setting as unverified.

Implementation sketch

The common core, in our own words. Each filter returns the models it accepts; the blend uses the votes. The distance and cosine filters are described in the Variants section and are not repeated here.

# One node's aggregation step, WFAgg-style  [arxiv:2409.17754 §4.2] [alg. 1]
def aggregate(own, received, f, taus, alpha, history):
    # Each filter returns the received models it accepts
    # [arxiv:2409.17754 §4.3] [§4.4] [§4.5]
    accepted = [
        distance_filter(received, f),
        cosine_filter(received, f),
        temporal_filter(received, history),
    ]
    # A model's vote is the sum of taus of the filters that accept it
    # [arxiv:2409.17754 alg. 1]
    votes = [sum(t for t, a in zip(taus, accepted) if m in a)
             for m in received]
    # Votes below the smallest sum of two taus count as zero
    # [arxiv:2409.17754 alg. 1]
    cut = min(taus[i] + taus[j] for i in range(3) for j in range(i + 1, 3))
    weights = [v if v >= cut else 0.0 for v in votes]
    # Eq. 3 is undefined when every weight is zero; keeping the local
    # model is a choice of ours, not the paper's
    if sum(weights) == 0:
        return own
    # Normalised weighted mean, then blend with the own model  [eq. 3]
    total = sum(weights)
    mean = sum(w / total * m for w, m in zip(weights, received))
    return (1 - alpha) * own + alpha * mean

Contradictions

Only one card lists this technique, so there is no disagreement between papers. Inside the paper, the text and the table give different values for Alt-WFAgg under Noise with two malicious neighbours: 46.78% in the text 2409.17754 §6.2.2 and 56.78% in the table 2409.17754 tab. 1. Both are reported here as written.

Pitfalls

• Trimmed-Mean in a small neighbourhood. The paper trims βK values from each end, with β = 0.1 2409.17754 §3.2 §5.1. With K = 8 or 9 models, that is less than one value, so a floor rule removes nothing. The Mean and Trimmed-Mean rows in the decentralized columns are close [arxiv:2409.17754 tab. 1]. State how the cut is rounded if you reproduce it.
• Noise scale. The paper says the noise has mean and variance 0.1, then writes σ = 0.1 2409.17754 §5.2. The two readings differ by a factor of about 3 in standard deviation. Pick one and document it.
• Temporal window. The paper's formula covers t−1 to t−W−1 with W = 3 and gives no further extraction rule 2409.17754 alg. 4 §4.5. The formula gives W + 1 values, which the text does not reconcile with the comparison against t−1. Define the window before you implement it.
• Vote threshold depends on τ. The "two or more filters" rule holds only with the threshold from Algorithm 1 2409.17754 alg. 1 §4.2. With τ = (0.5, 0.3, 0.2), the threshold is 0.5, so a model accepted only by the distance filter passes. Implement the threshold, not the verbal rule.
• All-zero weights. Eq. 3 divides by the sum of weights, which is zero if no model is accepted. The paper does not define this case 2409.17754 eq. 3.
• Krum distance. The paper's text describes Krum with Euclidean distances 2409.17754 §3.2. The released code uses squared distances; see the card's paper vs code notes before reproducing tab. 1.
• Missing details. Seeds, repetitions per table cell and hardware are not stated 2409.17754 §5. Differences of a few tenths of a point carry little evidence without repeated runs.
• Preprint version. The arXiv v2 is the accepted manuscript, not the version of record 2409.17754 p. 1. Check the published version before citing numbers.

Papers

• Byzantine-Robust Aggregation for Securing Decentralized Federated Learning: WFAgg, a three-filter Byzantine-robust rule and blend for decentralized federated learning, with a seven-attack benchmark on MNIST.