Federated reinforcement learning

Overview

Reinforcement learning (RL) needs many samples, and one agent exploring a large state or action space alone is slow 2108.11887 §1. Pooling experience from several agents would help, but sending all data to a central server conflicts with privacy and trust 2108.11887 §1. Federated reinforcement learning (FRL) keeps the experience where it is made. Each agent learns from its own states, actions and rewards, and only parameters, gradients or intermediate products are exchanged 2108.11887 §1 [arxiv:2108.11887 §4.1]. That exchange rule is the invariant every variant below shares.

The survey splits FRL into two families by how the agents' environments relate 2108.11887 §4.2 2108.11887 §4.3. Federated learning (FL) supplies the server-side pattern and the aggregation rule 2108.11887 §2.1. See federated-learning for the FL side.

Variants

Horizontal FRL (HFRL)

Agents have the same state and action spaces but act in different environments 2108.11887 §4.2. A round has six steps: an agent joins by downloading the shared model; it learns locally from its own trajectory; it uploads encrypted parameters when its trigger condition holds; the coordinator aggregates without waiting for every agent; the coordinator returns the aggregate; and each agent fuses it into its own model [arxiv:2108.11887 §4.2 steps 1-6] 2108.11887 fig. 8 2108.11887 fig. 10.

The paper treats HFRL as a fix for parallel RL, which assumes shared data and identical environments 2108.11887 §4.2. Card: Federated Reinforcement Learning: Techniques, Applications, and Open Challenges.

Vertical FRL (VFRL)

All agents act in one environment, but each observes only part of the global state, so their observation spaces differ and the union of the observations is the global state 2108.11887 §4.3. Decision-oriented agents take actions and receive rewards; support-oriented agents only observe [arxiv:2108.11887 §4.3]. The procedure initialises the local models, collects state-action-reward tuples, sends encrypted intermediate products to a federated model, aggregates them, sends encrypted parameters back and updates each local model 2108.11887 §4.3 steps 1-6 2108.11887 fig. 11.

The paper stresses that VFRL is not multi-agent RL: agents may have no action or reward of their own, and states cannot be shared 2108.11887 §4.3.

Other forms: reward shaping and multi-task consensus

Federated reward shaping shares reward-shaping values or embeddings, not policies 2108.11887 §4.4. A multi-task variant smooths parameter averages toward consensus and lets agents face different MDPs 2108.11887 §4.4. The survey gives no detailed results for either.

Application areas

The survey groups applications into edge computing, communication, control and attack detection 2108.11887 §5. Reported gains are the authors' claims for the cited works and use different setups, so they are not comparable. One example: a MADDPG-based FL scheme for underwater IoT throughput is reported as 80% better than standard actor-critic 2108.11887 §5.2. For the other numbers, see the card.

When to use which

The survey gives no tested recipe, so these rows follow from the definitions and open issues it states.

Expect attacks. In FRL, poisoning can tamper with rewards, not only with labels, because the data comes from interaction. The surveyed defences are reputation or safety assessment and penalising participants below a threshold 2108.11887 §6.4.

Implementation sketch

Shared core of an HFRL round, in our own words. VFRL changes what is sent and who acts, not this loop.

# One HFRL round, server-client form  [arxiv:2108.11887 §4.2]
# Assumes a weighted average of parameters  [arxiv:2108.11887 §2.1]

def hfrl_round(global_model, agents):
    # Agents start from the shared model  [§4.2 steps 1, 5]
    for agent in agents:
        agent.model = copy(global_model)

    # Each agent runs RL in its own environment  [§4.2 step 2]
    for agent in agents:
        agent.model = rl_update(agent.model, agent.env)

    # Upload when the agent's trigger fires; the server need not wait for all
    # [§4.2 steps 3-4]
    uploads = [(agent.weight, encrypt(agent.model))
               for agent in agents if agent.trigger()]

    # Weighted average; weights are a design choice  [§2.1] [§5.6]
    global_model = weighted_average(uploads)

    # Agents fuse the returned model into their own  [§4.2 steps 5-6]
    for agent in agents:
        agent.model = fuse(agent.model, global_model)

    return global_model

Contradictions

• Are HFRL environments identical? The paper calls the environments "independent and ideally identically distributed" 2108.11887 §4.2. Its advantages list and open issues say that in practice they are usually not identically distributed 2108.11887 §4.2 2108.11887 §6.1. The paper gives no method for the non-identical case; implement for it with care.

Papers

• Federated Reinforcement Learning: Techniques, Applications, and Open Challenges: survey of FRL that defines HFRL and VFRL, reviews applications and lists open issues; no experiments of its own.