---
kind: concept
slug: federated-reinforcement-learning
title: Federated reinforcement learning
summary: >-
  Reinforcement learning agents share parameters, gradients or reward signals instead of raw experience. Horizontal FRL fits agents that run the same task in separate environments; vertical FRL fits agents that share one environment but each see part of the state. The one source is a survey with no experiments, so the choice rests on the setting.
tags: [federated-learning, reinforcement-learning, privacy, multi-agent]
papers: ["2108.11887"]
related: [federated-learning, federated-aggregation, vertical-federated-learning, multi-agent-systems, actor-critic, policy-gradient, differential-privacy, computation-offloading]
status: draft
updated_at: 2026-10-18
---

## Overview

Reinforcement learning (RL) needs many samples, and one agent exploring a
large state or action space alone is slow [arxiv:2108.11887 §1]. Pooling
experience from several agents would help, but sending all data to a central
server conflicts with privacy and trust [arxiv:2108.11887 §1]. Federated
reinforcement learning (FRL) keeps the experience where it is made. Each agent
learns from its own states, actions and rewards, and only parameters, gradients
or intermediate products are exchanged [arxiv:2108.11887 §1] [arxiv:2108.11887
§4.1]. That exchange rule is the invariant every variant below shares.

The survey splits FRL into two families by how the agents' environments relate
[arxiv:2108.11887 §4.2] [arxiv:2108.11887 §4.3]. Federated learning (FL)
supplies the server-side pattern and the aggregation rule
[arxiv:2108.11887 §2.1]. See [federated-learning](/concepts/federated-learning)
for the FL side.

## Variants

### Horizontal FRL (HFRL)

Agents have the same state and action spaces but act in different environments
[arxiv:2108.11887 §4.2]. A round has six steps: an agent joins by downloading
the shared model; it learns locally from its own trajectory; it uploads
encrypted parameters when its trigger condition holds; the coordinator
aggregates without waiting for every agent; the coordinator returns the
aggregate; and each agent fuses it into its own model [arxiv:2108.11887 §4.2
steps 1-6] [arxiv:2108.11887 fig. 8] [arxiv:2108.11887 fig. 10].

The paper treats HFRL as a fix for parallel RL, which assumes shared data and
identical environments [arxiv:2108.11887 §4.2]. Card:
[Federated Reinforcement Learning: Techniques, Applications, and Open
Challenges](/papers/2108.11887).

### Vertical FRL (VFRL)

All agents act in one environment, but each observes only part of the global
state, so their observation spaces differ and the union of the observations is
the global state [arxiv:2108.11887 §4.3]. Decision-oriented agents take actions
and receive rewards; support-oriented agents only observe [arxiv:2108.11887
§4.3]. The procedure initialises the local models, collects state-action-reward
tuples, sends encrypted intermediate products to a federated model, aggregates
them, sends encrypted parameters back and updates each local model
[arxiv:2108.11887 §4.3 steps 1-6] [arxiv:2108.11887 fig. 11].

The paper stresses that VFRL is not multi-agent RL: agents may have no action
or reward of their own, and states cannot be shared [arxiv:2108.11887 §4.3].

### Other forms: reward shaping and multi-task consensus

Federated reward shaping shares reward-shaping values or embeddings, not
policies [arxiv:2108.11887 §4.4]. A multi-task variant smooths parameter
averages toward consensus and lets agents face different MDPs
[arxiv:2108.11887 §4.4]. The survey gives no detailed results for either.

### Application areas

The survey groups applications into edge computing, communication, control
and attack detection [arxiv:2108.11887 §5]. Reported gains are the authors'
claims for the cited works and use different setups, so they are not
comparable. One example: a MADDPG-based FL scheme for underwater IoT
throughput is reported as 80% better than standard actor-critic
[arxiv:2108.11887 §5.2]. For the other numbers, see the card.

## When to use which

The survey gives no tested recipe, so these rows follow from the definitions
and open issues it states.

| Situation | Use | Why |
|---|---|---|
| Same task, separate environment copies, shared state and action spaces | HFRL | HFRL is defined for this case [arxiv:2108.11887 §4.2] |
| One shared environment, each agent sees a different part of the state | VFRL | Observation spaces differ and the global state is their union [arxiv:2108.11887 §4.3] |
| Agents must not exchange states or policies, only reward-shaping signals | Federated reward shaping | Shares reward-shaping values or embeddings [arxiv:2108.11887 §4.4] |
| Agents' MDPs differ and they need a common model | Multi-task variant | Parameter averages move toward consensus; different MDPs are allowed [arxiv:2108.11887 §4.4] |
| Participants may drop out during training | HFRL-style partial aggregation, with a test | The coordinator need not wait for all agents [arxiv:2108.11887 §4.2 step 4]. Synchronous algorithms can stall when a participant goes offline [arxiv:2108.11887 §6.5]. The survey gives no test of this |
| Choosing aggregation weights | Sample share (FedAvg) as default | Alternatives include reward-based and batch-size weights [arxiv:2108.11887 §2.1] [arxiv:2108.11887 §5.6] |
| Communication budget is tight | Treat as open | Communication interval is named as an open problem [arxiv:2108.11887 §6.3]. No method is given |

Expect attacks. In FRL, poisoning can tamper with rewards, not only with
labels, because the data comes from interaction. The surveyed defences are
reputation or safety assessment and penalising participants below a threshold
[arxiv:2108.11887 §6.4].

## Implementation sketch

Shared core of an HFRL round, in our own words. VFRL changes what is sent and
who acts, not this loop.

```pseudocode
# One HFRL round, server-client form  [arxiv:2108.11887 §4.2]
# Assumes a weighted average of parameters  [arxiv:2108.11887 §2.1]

def hfrl_round(global_model, agents):
    # Agents start from the shared model  [§4.2 steps 1, 5]
    for agent in agents:
        agent.model = copy(global_model)

    # Each agent runs RL in its own environment  [§4.2 step 2]
    for agent in agents:
        agent.model = rl_update(agent.model, agent.env)

    # Upload when the agent's trigger fires; the server need not wait for all
    # [§4.2 steps 3-4]
    uploads = [(agent.weight, encrypt(agent.model))
               for agent in agents if agent.trigger()]

    # Weighted average; weights are a design choice  [§2.1] [§5.6]
    global_model = weighted_average(uploads)

    # Agents fuse the returned model into their own  [§4.2 steps 5-6]
    for agent in agents:
        agent.model = fuse(agent.model, global_model)

    return global_model
```

## Contradictions

- **Are HFRL environments identical?** The paper calls the environments
  "independent and ideally identically distributed" [arxiv:2108.11887 §4.2].
  Its advantages list and open issues say that in practice they are usually not
  identically distributed [arxiv:2108.11887 §4.2] [arxiv:2108.11887 §6.1]. The
  paper gives no method for the non-identical case; implement for it with care.

## Papers

- [Federated Reinforcement Learning: Techniques, Applications, and Open Challenges](/papers/2108.11887): survey of FRL that defines HFRL and VFRL, reviews applications and lists open issues; no experiments of its own.