Federated Reinforcement Learning: Techniques, Applications, and Open Challenges

In plain words

Machine learning systems often need data from many places, but that data is private: regional hospitals cannot share patient records with each other. Federated learning gets around this by letting each party train on its own data and send only what it learned, such as model parameters, to a shared server that combines them.

Reinforcement learning is a different kind of learning. An agent tries actions in an environment, gets rewards, and gradually improves its behaviour. It usually needs a lot of trial and error, and each agent only sees its own small part of the world. Federated reinforcement learning puts the two together: many agents learn from their own experience and share what they learned, not the experience itself.

The paper is a survey. It explains the basics of both fields, sorts this combined approach into two families depending on how the agents' environments relate, and lists the fields where it has been tried. It ends with questions that remain open, such as communication cost, attacks, how agents join and leave, and why agents should contribute.

TL;DR

• Federated reinforcement learning (FRL) lets RL agents improve a shared model by exchanging parameters or gradients, so raw experience stays with each agent §1 §4.1.
• Horizontal FRL (HFRL) covers agents with the same task in separate environments; vertical FRL (VFRL) covers agents in one shared environment that each see part of the state §4.2 §4.3.
• The paper reviews work on edge computing, communication, control and attack detection, and reports no experiments of its own §5 §5.6.
• Read it as a map of the field and its gaps; it gives no tested recipe to implement.

Problem

RL in practice needs many samples, and a single agent cannot explore a large state or action space alone §1. Pooling data from several agents would speed learning, but the usual route sends all data or gradients to a central server, which conflicts with privacy and trust §1. The paper also points to two further problems: policies trained in simulation may not transfer to the real world, and an agent may observe only part of the state §1. FRL is proposed to address these together §1 §4.1.

Method

The paper is organised as a tutorial followed by a survey. It first defines federated learning (FL): a set of parties jointly train a model without pooling their data, the server combines their parameters, and an aggregation rule decides how much each party counts §2.1. It then describes two FL architectures, client-server and peer-to-peer §2.2, and three FL categories by how the data is partitioned: horizontal FL (HFL), vertical FL (VFL) and federated transfer learning (FTL) §2.3 fig. 3 fig. 4 fig. 5.

On the RL side the paper covers the agent-environment loop, return and discounted return, value functions, and the policy §3.1 fig. 6. It names three families, value-based (for example Q-learning), policy-based (for example REINFORCE) and actor-critic, and the Bellman-style update that defines them §3.2 fig. 7. It then covers deep RL, which replaces tables with neural networks, and gives a taxonomy of representative algorithms §3.3 tab. 1.

The FRL part defines the two families and their procedures.

1. HFRL §4.2 fig. 8 fig. 9 fig. 10. Agents have similar state and action spaces, S_i = S_j and A_i = A_j, but interact with different environments, E_i ≠ E_j §4.2. The procedure is a server-client round:
   1. an agent joins by downloading the shared model, or by confirming its model type with the coordinator if it already has one §4.2 step 1;
   2. each agent acts and learns from its own state, action and reward sequence §4.2 step 2;
   3. each agent sends encrypted parameters to the coordinator whenever its trigger condition holds §4.2 step 3;
   4. the coordinator aggregates, and does not need to wait for every agent §4.2 step 4;
   5. the coordinator sends the aggregate back, and agents fuse it into their own models §4.2 steps 5-6.
2. VFRL §4.3 fig. 8 fig. 11 fig. 12. All agents act in one environment, E_i = E_j = E, but each observes only part of the state, so the observation spaces differ, O_i ≠ O_j, and the union of all O_i is the global state S §4.3. Decision-oriented agents act and receive rewards; support-oriented agents only observe and get neither actions nor rewards §4.3. The six-step procedure is: initialise the models, collect state-action-reward-state tuples, send encrypted intermediate products to a federated model, aggregate them, send encrypted parameters back, and update each local model §4.3 steps 1-6.
3. Other FRL §4.4. Federated reward shaping shares reward-shaping values or embeddings rather than policies §4.4. A multi-task variant smooths parameter averages to reach consensus, with agents allowed different MDPs §4.4.

The FL aggregation rule in the paper is the weighted average of the local models, with each weight equal to the agent's share of samples §2.1. The paper sets the global loss as a weighted sum of the local losses, with positive weights that sum to one; equal weights (1/N) are the natural choice §2.1. The survey notes that other work uses different weights: the average cumulative reward over the last ten episodes §5.6, an error rate of actions §5.6, or the training batch size §5.6.

# HFRL round, server-client model  [§4.2] [§2.1]
# Input:  global_model; agents, each with its own environment
# Output: updated global_model

def hfrl_round(global_model, agents):
    # 1. Agents start from the shared model  [§4.2 steps 1, 5]
    for agent in agents:
        agent.model = copy(global_model)

    # 2. Each agent learns with RL in its own environment  [§4.2 step 2]
    for agent in agents:
        agent.model = train_locally(agent.model, agent.environment)

    # 3. Agents upload encrypted parameters; not all must  [§4.2 steps 3-4]
    uploads = [(agent.weight, encrypt(agent.model)) for agent in agents]

    # 4. Coordinator forms the weighted average  [§2.1] [§5.6]
    # agent.weight = n_i / n is the FedAvg default  [§2.1]
    global_model = weighted_average(uploads)

    # 5. Agents fuse the returned model with their own  [§4.2 step 6]
    for agent in agents:
        agent.model = fuse(agent.model, global_model)

    return global_model

Hyperparameters

The survey does not train models, so it gives no tuned values. The table lists the symbols the paper defines and what it states about each.

Results

The paper reports no experiments of its own. The numbers below are author-reported in the works it cites, quoted from its application sections. They use different setups and should not be compared with one another.

Other applications report qualitative results only, such as "better results than the other baselines" for FedRL on Grid-World and Text2Action §4.3, or "superior performance to the non-federated local training" for a driving experiment §5.3.

Paper vs code

No official code. Searched the arXiv abstract page (no comments or code link) and GitHub repository search for "federated reinforcement learning survey". The one hit belongs to an unrelated user and is not an author repository, so it was not used.

Pitfalls

• Identical versus non-identical environments. Section 4.2 says the HFRL environments are "independent and ideally identically distributed" §4.2. Its advantages list and open issues say that in practice they are usually not identically distributed §4.2 §6.1. Implement for the non-IID case; the paper does not give a method that fixes it.
• Symbol reuse. γ denotes both the FL learning rate §2.1 and the RL discount factor §3.1. The paper does not disambiguate them.
• Index mismatch in FedAvg. The local step is written as a gradient step from the global model, but the gradient is evaluated at a different index, and the aggregate is named differently from the local models §2.1. Use the standard FedAvg form.
• Policy gradient as written. The formula in §3.2 multiplies each step's log-probability by the total episode reward, not by the reward from that step onward. The REINFORCE steps repeat this §3.2. Check the formula against a reference REINFORCE before implementing. Step 3 also contains a missing backslash in its update.
• Q-learning convergence. The paper states that Q-learning converges to the optimal Q-function if the agent keeps visiting all state-action pairs §3.2. It gives no exploration schedule or step-size rule, so these conditions are left to the implementer.
• Aggregation weights are a design choice. The paper says the sample-share weight from FedAvg can be replaced, and gives several alternatives §5.6. None of them is compared against the others in the survey.
• VFRL is not MARL. The paper stresses that VFRL agents may have no action or reward and that states cannot be shared §4.3. Multi-agent methods that share observations or rewards do not match this setting.
• Attacks. In FRL a poisoning attack can tamper with rewards rather than labels, because the data comes from interaction §6.4. Defences discussed are reputation or safety assessments and punishing participants below a threshold §6.4.
• Join and exit. Synchronous algorithms can stall if a participant goes offline, because the coordinator waits for every update §6.5.
• Equations and figures are unnumbered in the source. Equation citations are given by section. Figure numbers follow the order in the compiled paper.

Lineage

• Builds on: federated averaging (FedAvg), cited as McMahan et al. §2.1; classical RL methods, including Q-learning, REINFORCE and actor-critic §3.2; deep RL methods §3.3 tab. 1; and distributed or parallel RL such as A3C and Gorila §4.2.
• Related: multi-agent RL and decentralised partially observable MDPs as the closest existing RL setting to VFRL §4.3.
• Compared against: the paper contrasts FRL with parallel RL, which assumes data sharing and identical environments §4.2.
• Superseded by: not stated.