Federated Learning: Opportunities and Challenges

In plain words

Federated learning trains one shared model across many devices, or across organisations such as hospitals, while each party's raw data stays where it is. A central server sends the current model out, each participant improves it using only its own data, and the participants send back their updates, which the server combines. The cycle repeats for as many rounds as needed. This matters in areas like healthcare and finance, where sharing private records is risky. The approach is not safe by default: updates can leak information about the training data, and dishonest participants can corrupt the shared model. This report surveys the setups, the applications, the training problems, the attacks and defences, and some recent research directions.

TL;DR

Federated learning (FL) keeps training data with the participants and shares only model parameters with a server §1, §2. This paper is a survey. It lays out the four-step training loop, the FL types, applications, training bottlenecks, privacy and poisoning attacks with their defences, and recent work §2–§7. It contains no experiments or benchmark numbers §1–§8, so use it as a map of the field and its open problems, not as a method to implement.

Problem

Pooling data in one place is risky when the data is sensitive: patient records, bank customer data, personal text typed on phones [§1, §4.1, §4.3, §4.4]. FL removes the central data store, but it brings training-time problems such as communication overhead and device and data heterogeneity §5. It also opens attack surfaces: malicious clients, a compromised server, or a participant with black-box access to the model §1, §6.

Method

This is a survey, so there is no algorithm to reproduce. The paper gives a generic FL training loop §2, a taxonomy of FL types §3, the applications it reviews §4, the training bottlenecks §5, attacks and defences §6, and recent developments §7. Figures 1 and 2 show FL across devices and across organisations fig. 1 fig. 2 §2.

Generic training loop §2

The paper lists four steps, repeated for n rounds. The number of rounds n is not stated §2:

• Client selection or sampling: the server picks participants at random or with an algorithm. The paper points to client-selection works §2.
• Parameter broadcasting: the server sends the global model parameters to the selected clients §2.
• Local model training: each selected client retrains the model on its own data. The paper says these clients train in parallel §2.
• Model aggregation: clients send back their local parameters, which are aggregated into the global model §2. The aggregation rule is not written out; FedAvg is named as the baseline method §5.2.

# Generic FL training loop from the four steps  [§2]
# Input:  global_params; client_pool; num_rounds (n, not stated)
# Output: global_params after the final round

def federated_training(global_params, client_pool, num_rounds):
    # Repeat the four steps for n rounds  [§2]
    for round_index in range(num_rounds):
        # Client selection: random pick or an algorithm  [§2]
        selected_clients = select_clients(client_pool)
        # Parameter broadcasting: the server sends the global parameters  [§2]
        for client in selected_clients:
            client.receive(global_params)
        # Local training: each selected client retrains on its own data  [§2]
        local_params = [client.train_locally() for client in selected_clients]
        # Model aggregation: combine local parameters into the global model
        # Rule not given; FedAvg is named as the baseline  [§5.2]
        global_params = aggregate(global_params, local_params)
    return global_params

FL types §3

• Horizontal FL: participants share the same feature space but hold different samples. The Google keyboard use case is given as an example §3 §1.
• Vertical FL: participants hold different features for the same samples. The paper's example is two organisations with different feature sets for the same people §3.
• Federated transfer learning: adds a new feature to a pre-trained model, and extends vertical FL to sample instances that not all organisations hold §3.
• Cross-silo FL: few participants, available in every round, mostly organisations. The data can be horizontal or vertical §3.
• Cross-device FL: many participants. Client selection and incentive designs are needed §3.

Applications §4

• Healthcare: electronic health records are the main source. A single hospital's data can bias a model, and sharing records between hospitals is often not feasible §4.1.
• Transportation: vehicles can train online across locations, and traffic data split across organisations can be used without exchanging it §4.2.
• Finance: loan risk assessment across banks, possibly with other institutions §4.3.
• Natural language processing: text on edge devices contains user information. The paper cites one work that builds NLP models in an FL framework §4.4.

Training bottlenecks §5

• Communication overhead: multiple rounds cost communication. The cited fixes are data compression and sending back only relevant outputs §5.1.
• Systems and data heterogeneity: non-identically distributed data and unequal devices hurt performance. FedAvg is said not to be robust to system heterogeneity. Some works change the aggregation method §5.2.

Attacks and defences §6

• Membership inference §6.1: model updates can reveal whether a data point was in the training set. Three defence families are described:
  • Secure computation: secure multi-party computation and homomorphic encryption §6.1.1.
  • Differential privacy: clip the model parameters, add noise, then aggregate. Some accuracy is lost §6.1.2.
  • Trusted execution environments: low overhead compared with secure computation, but for CPU devices only, as the paper states it §6.1.3.
• Data poisoning §6.2: an adversary poisons the training data of some devices, directly or through other devices. Attacks can be targeted or non-targeted. Defence: identify malicious participants from their model updates before averaging each round §6.2.1.
• Model poisoning §6.3: compromised devices alter their local model parameters. Defence: reject updates by their effect on the global model's error rate, its loss, or both §6.3.1.
• Backdoor attacks §6.4: secure averaging hides which device contributed, so a backdoor can be inserted that mislabels chosen inputs. Attack strength depends on the share of compromised devices and the model's capacity §6.4. Defences: weak differential privacy or norm thresholding of updates §6.4.1.

Recent developments §7

• One-shot FL: a single communication round. A distilled variant has devices send distilled data to the server, which trains on the combined data §7.1.
• Incentive mechanisms: reputation-based rewards tied to accuracy, data reliability and contribution. The paper notes these works do not model convergence or the added communication cost §7.2.
• FL as a service: a cloud service where third-party applications contribute to a shared model §7.3.
• Asynchronous FL: updates arrive at different times, and asynchronous averaging can serve more devices than synchronous averaging §7.4.
• Blockchain: a decentralised network replaces the central aggregator. A sample architecture is shown in fig. 3 §7.5.

Open question §8

The paper asks whether a Byzantine-tolerant FL model can keep user privacy while using schemes with low computational cost §8.

Hyperparameters

Not applicable: the paper trains no model. It states no values for:

• number of rounds n, or participants per round §2;
• learning rate, batch size or local epochs §2, §5;
• clipping bound and noise scale for differential privacy §6.1.2;
• norm threshold for norm thresholding §6.4.1;
• rejection thresholds for error-rate or loss-based rejection §6.3.1.

Results

Not applicable: the paper reports no experiments, benchmarks or quantitative results §1–§8. Its comparative claims are qualitative. For example, it says FL offers a better privacy guarantee than other ML frameworks, but gives no measurements for that §8.

Paper vs code

No official code. The paper is a survey with no implementation. The abstract, text and bibliography name no code repository. The only URL in the bibliography is a blog post cited for the COVID-19 example §1. Following the wiki's rule for survey papers, I did not run a GitHub search.

Pitfalls

• Not an implementation guide. Nothing here fixes the aggregation rule, the learning rate, the number of rounds or the stopping rule. The paper names FedAvg but does not write its update §5.2.
• Differential privacy is described two ways. The introduction calls it a "cryptographic technique" §1. Section 6.1.2 describes it as noise added to clipped parameters before aggregation. Follow §6.1.2; the clip bound, noise scale and privacy budget are not stated.
• Secure computation has no protocol. Its full description is that parties compute on inputs and reveal outputs only to a subset of participants §6.1.1.
• Defence rules are sketches. Malicious-participant detection from updates §6.2.1, error-rate and loss-based rejection §6.3.1 and norm thresholding §6.4.1 come without formulas or thresholds. The paper itself says it is hard to find malicious participants under secure aggregation §6.4.1.
• Cost claims are unquantified. Differential privacy costs accuracy §6.1.2. Trusted execution environments have low overhead and are CPU-only §6.1.3. Incentive schemes' convergence and communication costs are not modelled §7.2.
• Application claims are qualitative. Several claims about where FL helps, such as transportation and NLP, have no numbers in the paper §4. The COVID-19 example is cited to a web page §1.
• The one-shot description says devices distil their data and send it to the server §7.1. It gives no distillation procedure and no privacy analysis.
• Some figure numbers are inferred. Figures are numbered in order of appearance in the LaTeX source, so the blockchain figure is cited as fig. 3 §7.5.

Lineage

• Builds on: the paper names no method it extends. It attributes the FL concept to Google and cites mcmahan2017federated for it §1. FedAvg is the baseline it discusses §5.2.
• Compared against: the paper says four cited surveys focus on architectures or on challenges: kairouz2019advances, yang2019federated, mothukuri2020survey and aledhari2020federated §1. Their content is not used here.