Federated aggregation

Overview

Each round, the server has to turn the client updates into one new global state. The rule for that step is the aggregation. It matters most when the clients hold differently distributed data (non-IID) 2108.00974 §4.4 2110.00857 §1 2106.03097 §1 2006.07242 §1 2105.10056 §1. The round loop around it is covered on the federated learning page.

Seven cards cover this so far. Four approaches appear:

• Average the weights, with different weights or a different local objective: FedAvg, FairFed, FedNTD, Fed+.
• Fuse predictions instead of weights: FedDF builds the new model by distilling the averaged logits of the client models 2006.07242 §3.
• Average the weights, but add a server-trained generator: FedGen trains a generator from the clients' prediction layers and sends it back to the clients as extra training data 2105.10056 §3.1 2105.10056 §3.2.
• Average the weights and the local gradients, with a local surrogate that adds a global gradient correction: FEDL 1910.13067 §IV.

One card compares FedAvg and Fed+ in a simulated federation of IoT intrusion-detection clients 2108.00974 §5. The second keeps FedAvg as the starting point and changes the weights each round to reduce group unfairness 2110.00857 §4.2. The third keeps the averaging step and adds a distillation term to each client's local loss 2106.03097 §3. The fourth keeps the weighted averaging of models, adds a second weighted average of local gradients, and changes each client's local objective to use them 1910.13067 §IV eq. 4 eq. 5.

Variants

Weighted averaging (FedAvg)

Each client's weights count in proportion to its share of the total data size d_i / D 2108.00974 §4.4. The paper treats FedAvg as the baseline and credits it to McMahan et al. (2017) 2108.00974 §2. The paper notes that skewed client data can slow or degrade convergence under FedAvg 2108.00974 §4.4. The FairFed paper also starts from these weights 2110.00857 §4.2. FedDF uses the same data-size weights for the starting model of its fusion step 2006.07242 alg. 1. The FedNTD paper's Algorithm 1 uses uniform weights over the sampled clients instead 2106.03097 alg. 1. The FedGen paper's round also takes a uniform mean over the active users 2105.10056 alg. 1. The FedML library implements FedAvg with the same data-size weights as a runnable reference 2007.13518 app. B.1. Card: FedML: A Research Library and Benchmark for Federated Machine Learning.

Per-client penalty (Fed+)

Fed+ does not require all clients to converge to one model. Its objective is the mean local loss plus, for each client, a weight α_i times a distance B between the shared model x and a central point C(X) computed from x 2108.00974 §4.4.

The paper gives no update rule, no choice of B or C, and no value for α_i. It refers to the Fed+ paper for these 2108.00974 §4.4. That paper, arXiv:2009.06303, has no card in this wiki, so nothing from it is used here. Fed+ cannot be implemented from this page alone. Card: IoT intrusion-detection review with FedAvg and Fed+.

Fairness-gap reweighting (FairFed)

The server keeps FedAvg weights as the start point and then moves them each round. Each client computes a gap Δ_k. When its local equal opportunity difference (EOD) is defined, the gap is the distance between the global EOD and the local EOD. Otherwise it is the distance between the client's accuracy and the mean accuracy 2110.00857 §4.2 eq. 6. Clients whose gap is above the mean gap lose weight, and clients below it gain weight. The step size is β, and with β = 0 the rule is FedAvg 2110.00857 §4.2 alg. 1.

The server does not see per-group data. The global EOD and the mean accuracy come from a secure aggregation over the clients' local metrics [arxiv:2110.00857 §4.3] alg. 1. The local debiasing method is a free choice per client; the paper uses reweighting, FairRep and FairBatch 2110.00857 §5.1.4. The paper gives no formal privacy analysis of the secure aggregation step 2110.00857 §4.3. The weight recursion also leaves some details open, such as the starting value of the unnormalised weights and whether they are clipped 2110.00857 §4.2. Card: FairFed: Enabling Group Fairness in Federated Learning.

Uniform averaging with a distillation loss (FedNTD)

FedNTD does not change the server's averaging. It changes each client's local loss. The loss adds a term that matches the local model's softmax over the not-true classes to the frozen round-start global model's softmax over the same classes, with the true class removed from both sides 2106.03097 §3. The authors read non-IID losses as forgetting of knowledge about classes a client does not hold, and they state that they measure this rather than prove the causal link 2106.03097 §2.

Algorithm 1 averages the uploads with weight 1/|K^(t)|, which is uniform over the sampled clients 2106.03097 alg. 1. The released code weights each client by its local dataset size instead. The paper does not say which rule produced its reported numbers. Card: FedNTD: Preservation of the Global Knowledge by Not-True Distillation in Federated Learning, section Paper vs code.

Nothing extra is sent over the network 2106.03097 §4.2. The loss does need the frozen global model's logits on every client batch [arxiv:2106.03097 §3], so local compute goes up. The paper does not report that cost.

Logit fusion by distillation (FedDF)

FedDF keeps the round loop but replaces the parameter average with a distillation step. The server first forms the data-size-weighted average of the returned client models, and uses it as the starting point of the fused model 2006.07242 alg. 1. It then runs a number of distillation steps. In each step, a mini-batch of unlabeled or generated data goes through every sampled client model. The target is the softmax of the mean of their logits, and the fused model is trained to match it with a KL divergence 2006.07242 eq. 1 2006.07242 §3. The fused model becomes the global model for the next round 2006.07242 alg. 1.

Because the target is an average of predictions, the client models may differ in architecture, size or numerical precision. The server keeps one prototype per group and distils the average of all sampled clients' logits into each prototype 2006.07242 §3 2006.07242 app. B. Clients do no extra work 2006.07242 §3.

The distillation data is a choice. Real unlabeled images from another domain work, and so do samples from a frozen generator; pure random noise gives a clear drop in fusion quality 2006.07242 §5 2006.07242 fig. 5. The paper gives no rule for picking a domain 2006.07242 §5. The fusion needs no labels for the distillation loss, but early stopping and the drop-worst rule use labelled server validation accuracy 2006.07242 §4.1 2006.07242 §4.2. Card: Ensemble Distillation for Robust Model Fusion in Federated Learning.

Generator-based data-free distillation (FedGen)

FedGen needs no proxy dataset on the server. The server trains a small conditional generator G(z | y) so that the clients' averaged prediction layers classify its samples as the label y 2105.10056 §3.1 [arxiv:2105.10056 eq. 4]. Only the predictor parameters of each client are needed for this 2105.10056 §3.1. The server sends the generator to the clients. Each client adds a term to its local loss so that its own predictor also classifies the generator's samples correctly 2105.10056 §3.2 [arxiv:2105.10056 eq. 5]. The paper shows this term is a KL divergence between the generator's and the client's conditional distributions in latent space [arxiv:2105.10056 eq. 6]. The authors call the generator samples an inductive bias for local training 2105.10056 §4.1.

The server still averages the uploaded client models 2105.10056 alg. 1. The generator is an addition to that loop, not a replacement. The paper also describes an optional variant in which clients share only the last layer and keep the feature extractor local 2105.10056 §3.3 [arxiv:2105.10056 alg. 2]. The paper says this lowers communication and exposure of the feature extractor 2105.10056 §3.3. The authors describe the generator as lightweight but do not report its size 2105.10056 §6.2. Card: Data-Free Knowledge Distillation for Heterogeneous Federated Learning.

Gradient-corrected local surrogate (FEDL)

FEDL keeps the model average and adds a second average. Each client also uploads its full local gradient at the new model, and the server averages these gradients with the same weights p_n = D_n / D as the models 1910.13067 alg. 1 eq. 4 eq. 5. The average gradient is then sent back and used in the next round's local problem 1910.13067 §IV.

Each client minimises a surrogate J_n(w) = F_n(w) + <η·ḡ − ∇F_n(w^(t−1)), w>, where ḡ is the averaged gradient and η is a hyper-learning rate 1910.13067 eq. 2. The client does not need an exact solve. It needs the gradient norm of the surrogate at its answer to be at most θ times the norm at the starting point, for a local accuracy θ in (0, 1) 1910.13067 eq. 3. Under the paper's assumption that each local loss is L-smooth and β-strongly convex, the authors prove a linear rate Θ for the global loss, which depends on η, θ and the condition number ρ = L/β 1910.13067 §III eq. 10 eq. 11. The analysis covers full-batch gradient descent with full participation only 1910.13067 §IV remark 5.

Two parameters need tuning. The authors say Θ ∈ (0, 1) is sufficient for convergence, not necessary 1910.13067 §IV remark 1. A large η can push Θ out of that range, and the paper says this can make the method diverge 1910.13067 §IV remark 3. A small θ means more local rounds and fewer global ones 1910.13067 §IV remark 2. The paper gives no rule for estimating L or β from data, so a practical choice of η and θ is not stated 1910.13067 §IV.

FEDL also has a wireless half. The paper models the wall-clock time and the energy of the UEs on a shared uplink, and splits the joint problem into three sub-problems for CPU frequency, uplink time shares, and θ with η 1910.13067 §V. These sub-problem solutions do not change the aggregation rule above. They choose its inputs under a resource model. Card: Federated Learning over Wireless Networks: Convergence Analysis and Resource Allocation.

Reported results

Mean client accuracy, as read from the text of §5.4 and fig. 11. Author-reported, with one run per setting and no variance 2108.00974 §5.

Source: 2108.00974 §5.4 fig. 11. The authors state Fed+ gives the higher average accuracy in all federated settings 2108.00974 §5.4. In the basic setting, the reported means are almost equal.

Mean accuracy hides per-client failures. Under Fed+ in the basic setting, two clients have zero precision and recall, with false positive rate 1, despite high accuracy 2108.00974 §5.1 fig. 4. Under Fed+ in the mixed setting, the best two clients reach F1 0.91 2108.00974 §5.3 fig. 10.

FairFed results on Adult, averaged over 20 seeds, with the best EOD per method chosen by grid search 2110.00857 §5.1.5 tab. 1:

α is the Dirichlet heterogeneity parameter; smaller values mean more skewed clients 2110.00857 §5.1.3. The selection uses the test-set EOD, and the paper does not state a separate validation split 2110.00857 §5.1.5. The reported gains may therefore be optimistic.

FedNTD against FedAvg, test accuracy (%), author-reported, one run per cell. A selection of the 14 non-IID settings in the main table [arxiv:2106.03097 tab. 1]:

FedNTD is above FedAvg in all 14 settings of the table [arxiv:2106.03097 tab. 1]. The paper's forgetting measure and standard deviations are in the paper's tables and appendix 2106.03097 tab. 1 [app. Experiment Table with Standard Deviation]. The paper says FedNTD is best or near-best in most settings 2106.03097 §4.2. The authors also say it "consistently outperforms the baselines on all setups" 2106.03097 §4.2. The table does not support that claim; see Contradictions.

FedDF, CIFAR-10 with ResNet-8, 20 clients, 40 local epochs, participation C = 0.4. Communication rounds needed to reach a target accuracy, mean ± std over three seeds, author-reported. Lower is better 2006.07242 tab. 1:

The authors report FedDF as best in every setting of that table, with a larger advantage at more local epochs 2006.07242 tab. 1 2006.07242 §4.2. With batch normalization, the final accuracy at 100 rounds is 71.36 ± 1.07 for FedDF against 62.22 ± 3.88 for FedAvg at α = 0.1 on CIFAR-10 [arxiv:2006.07242 tab. 2]. The FedDF and FedAvg numbers here come from one paper and one setup.

FedGen, selected rows of the main table. Twenty users, 200 rounds, local steps T = 20, top-1 test accuracy (%), mean ± a spread the paper does not define 2105.10056 tab. 1:

FedGen is not best in every row. FedDFusion, a baseline the paper describes as proxy-data distillation 2105.10056 §6.1, leads at EMNIST α = 0.05 2105.10056 tab. 1. The FedGen paper's Table 1 gives a full comparison with more baselines. This page does not assume that FedDFusion is the method on the FedDF card. The FedDF, FedNTD and FedGen settings differ in dataset, client count and rounds, so their numbers should not be read against one another.

FedML's benchmark reports FedAvg with ResNet-56 on CIFAR-10 under a latent-Dirichlet split: 93.19 with IID data and 87.12 with non-IID data, using 10 workers 2007.13518 tab. 5. The paper states no training recipe for these runs (rounds, epochs, learning rate, client sampling), so the numbers cannot be matched from the paper alone 2007.13518 §5. These figures come from one paper and one setup, and they are not comparable with the tables above.

FEDL against FedAvg, MNIST with a multinomial logistic model, 100 UEs, author-reported 1910.13067 §VI fig. 4. The paper gives the gains as read from its figure, with no spread across runs. Test accuracy gains over FedAvg are about +1.3% at batch 20, +0.7% at batch 40 and +0.8% at full batch. The training loss changes by −0.2% at batch 40. The training-loss change is reported as about 9.1% at batch 20 and about 14% at full batch, with no sign or base stated 1910.13067 §VI fig. 4. The gain is not uniform across batch sizes. The paper's DNN check shows a smaller gain that is less stable at small batch 1910.13067 §VI fig. 6. A synthetic-data check reports the gap to the exact global solution as negligible, without a number 1910.13067 fig. 3. The wireless results are total-cost curves against computation and communication weights, with a Pareto curve the authors read as better at low heterogeneity 1910.13067 §VII fig. 7 fig. 8. No FedAvg baseline is given for them.

When to use which

Whichever you pick, report per-class precision, recall and F1 next to accuracy. The authors make that point about their own metrics 2108.00974 §5.1.

Implementation sketch

FedAvg is the base step for most rules on this page. FEDL's server step is the same weighted average, applied to models and to gradients [arxiv:1910.13067 eq. 4] eq. 5. The round loop around these is the same as in the federated learning page.

# FedAvg aggregation step  [arxiv:2108.00974 §4.4]
# Input:  local_weights (one vector per client); sizes (data per client)
# Output: new global weights

def fedavg(local_weights, sizes):
    # total data size D across the clients taking part  [arxiv:2108.00974 §4.4]
    total = sum(sizes)
    # weight each client by its share d_k / D of the data  [arxiv:2108.00974 §4.4]
    return sum(
        (size / total) * weights
        for size, weights in zip(sizes, local_weights)
    )

Uniform averaging, as in FedNTD's Algorithm 1 2106.03097 alg. 1, is the same function called with every size set to 1.

FairFed changes only the weights. The aggregation step is the same weighted average, with the weights taken from the update below.

# FairFed weight update for one round  [arxiv:2110.00857 §4.2] [eq. 6]
# Input:  w_bar (unnormalised weights, one per client); gaps (Delta_k)
# Output: normalised weights for the weighted average

def fairfed_weights(w_bar, gaps, beta):
    # Mean of the per-client gaps, not weighted by data size  [eq. 6]
    mean_gap = sum(gaps) / len(gaps)
    # Clients above the mean gap lose weight, those below gain  [eq. 6]
    w_bar = [w - beta * (g - mean_gap) for w, g in zip(w_bar, gaps)]
    # Normalise so the weights sum to one  [arxiv:2110.00857 §4.2]
    total = sum(w_bar)
    return [w / total for w in w_bar]

FedDF replaces the parameter average with a distillation loop. The starting point is still the weighted average, so the first step is the FedAvg one above.

# FedDF fusion step  [arxiv:2006.07242 alg. 1] [eq. 1]
# Input:  client_models, sizes, distill_data, labelled val_set
# Output: fused model, which becomes the next global model

def feddf_fuse(client_models, sizes, distill_data, val_set, steps):
    # Start from the data-size-weighted average  [arxiv:2006.07242 alg. 1]
    student = fedavg(client_models, sizes)
    for _ in range(steps):
        # Unlabeled batch; labels are not read in the distillation loss  [§4.3]
        x = sample_batch(distill_data)
        # Target: softmax of the mean of all clients' logits  [eq. 1]
        teacher = softmax(mean([m(x) for m in client_models]))
        # Match the student to the teacher with a KL divergence  [eq. 1]
        student = kl_step(student, x, teacher)
        # Early stop on labelled server validation accuracy  [§4.1]
        if plateaued(student, val_set):
            break
    return student

Fed+ has no sketch here, because the paper does not define its update rule 2108.00974 §4.4. FedGen has none either, because its client and server steps need a full generator and a predictor split that the card does not spell out for implementation.

Contradictions

No direct disagreement between the papers on a shared setting. Several conflicts are within single papers:

• FedNTD paper vs its code, averaging weights. Algorithm 1 uses uniform weights 2106.03097 alg. 1. The released code weights clients by dataset size. The paper does not say which produced its numbers. This matters under LDA, where client sizes differ. See the FedNTD section above.
• FedNTD claim vs its table. The text says FedNTD "consistently outperforms the baselines on all setups" 2106.03097 §4.2. In tab. 1, SCAFFOLD is higher on CIFAR-10 sharding s=5 (65.74 vs 64.93), CIFAR-10 sharding s=10 (68.97 vs 68.56), CIFAR-10 LDA α=0.5 (66.16 vs 66.12) and CIFAR-100 LDA (33.68 vs 32.37) 2106.03097 tab. 1.
• FairFed text vs its tables. The results text gives percentage gains that do not reproduce from its tables. For example, the text says EOD on Adult at α = 0.1 improves by 93%, while the table gives a 90% reduction (-0.174 to -0.017) 2110.00857 §5.2.1 tab. 1. This page cites the tables. The text's accuracy figure for TILES, 2.6%, does not match the table either 2110.00857 §6.2.2 tab. 3.
• FedDF distillation data for CIFAR-100 and ImageNet. The main text and Table 2 use ImageNet-32 for CIFAR-100 2006.07242 §4.1 [arxiv:2006.07242 tab. 2]. The Fig. 4 caption names STL-10 for CIFAR-100 instead. Treat the main text as the setting used, and the caption as unclear.
• FEDL rate formula, within one paper. Remark 1 writes the numerator of Θ as 2η(A − B) 1910.13067 §IV remark 1. Eq. (11) as printed has η(A − B) over 2ρ(·) 1910.13067 eq. 11. The card checks that eq. (11) matches the paper's numerical sub-problem table, and that the remark's version gives about twice the Θ 1910.13067 §V-C. This page uses eq. (11). The card's Pitfalls section covers the check.
• FEDL paper vs its released code. Algorithm 1 needs η > 0 and a local accuracy θ 1910.13067 alg. 1. The released code defaults η to 0, never checks θ, and has no global gradient in its first round, so the correction term starts at round 2. The card's Paper vs code section lists these with code paths: Federated Learning over Wireless Networks, section Paper vs code.
• FedDF "no labels" vs its server. The paper says the fusion needs no labels 2006.07242 §4.3. Its early stopping and drop-worst rule still read labelled server validation accuracy 2006.07242 §4.1 2006.07242 §4.2. Read "no labels" as applying to the distillation loss.
• FedGen numbers in Figure 2 vs Table 5. The main-text caption gives one user 81.2% to 98.4%, with the oracle at 98.6% 2105.10056 fig. 2. The appendix table gives 81.2 to 98.2 and an oracle of 98.4 [arxiv:2105.10056 tab. 5]. Use the table for per-user numbers.

Papers

• IoT intrusion-detection review with FedAvg and Fed+: compares FedAvg with Fed+ on IoT intrusion data split into three non-IID client settings.
• FairFed: Enabling Group Fairness in Federated Learning: server-side aggregation that reweights clients by how close their local fairness gap is to the global gap.
• FedNTD: Preservation of the Global Knowledge by Not-True Distillation in Federated Learning: keeps the averaging step and adds a not-true-class distillation term to each client's loss, with no extra communication.
• Ensemble Distillation for Robust Model Fusion in Federated Learning: fuses client models by distilling their averaged logits into one server model on unlabeled or generated data, allowing mixed architectures.
• Data-Free Knowledge Distillation for Heterogeneous Federated Learning: trains a server generator from the clients' prediction layers and uses its samples as a regulariser in local training, with no proxy dataset.
• Federated Learning over Wireless Networks: Convergence Analysis and Resource Allocation: averages models and gradients, with a local surrogate that has a global gradient correction; gives a linear rate under smoothness and strong convexity, and a wireless time-energy allocation.
• FedML: A Research Library and Benchmark for Federated Machine Learning: open library and benchmark whose reference FedAvg weights clients by local sample count, with a fixed non-IID benchmark.