---
kind: concept
slug: centralized-training-decentralized-execution
title: Centralised training with decentralised execution (CTDE)
summary: >-
  Cooperative multi-agent learning where training may use global state, but each agent acts only on its own history at run time. One card compares a centralised critic, joint-value methods and independent PPO on StarCraft maps. Independent PPO matched or beat the others on several hard maps in author-reported runs. Pick by the state available in training and the coordination risk you accept.
tags: [multi-agent-reinforcement-learning, centralised-critic, decentralised-execution, starcraft]
papers: ["2011.09533"]
related: [multi-agent-systems, actor-critic, policy-gradient]
status: draft
updated_at: 2026-10-24
---

## Overview

Several agents act together, and each one sees only part of the situation at
run time. CTDE lets training use global information, such as the full state or
the joint actions of all agents. At execution time each agent still acts from
its own local history [arxiv:2011.09533 §1]. The invariant all variants share is
that the deployed policies never need the global information, so they can run
on separate units.

Variants differ in what the training-time information is used for. Some learn a
joint value function over all agents. Others give a shared critic the full state.
At the far end, the training is fully independent, with no shared information at
all [arxiv:2011.09533 §1] [arxiv:2011.09533 §4]. The card-backed source for this
page is one paper, so the variants below are the ones it describes.

## Variants

### Joint-value CTDE methods (QMIX, MAVEN)

These learn a joint value function from global information during training. The
paper uses them as baselines and names them as the usual CTDE joint learners. It
does not reimplement them in its own method [arxiv:2011.09533 §1] [arxiv:2011.09533 §5.1].
Their joint action space grows with the number of agents [arxiv:2011.09533 §1].
Card: [Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?](/papers/2011.09533)

### Centralised critic with full state

The actor stays local, but the critic reads the full state during training. The
paper tests this as a parameter-sharing centralised critic and reports that it
performs substantially worse than local critics on 3m, 3s5z vs 3s6z, MMM2 and
corridor [arxiv:2011.09533 §5.2] [arxiv:2011.09533 fig. 3]. The authors say the
result is not a verdict on central state in general. They say it is still unclear
what central state is worth on SMAC [arxiv:2011.09533 §6].
Card: [Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?](/papers/2011.09533)

### Independent PPO (no shared information)

Each agent runs PPO with its own local critic, and the agents share actor and
critic weights. Each agent uses the team reward in place of its own reward
[arxiv:2011.09533 §4]. The authors report that this matches or beats MAPPO, QMIX
and other joint-value methods on several hard maps, with mild per-map tuning
[arxiv:2011.09533 §5.1]. Their ablations point to policy clipping as the key
ingredient, not a lower learning rate [arxiv:2011.09533 §5.3].
Card: [Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?](/papers/2011.09533)

## When to use which

| Situation | Use | Why |
|---|---|---|
| Global state is available in training, and you want a centralised critic | Test it against a local critic first | In the one card-backed comparison, the centralised critic was worse on the listed maps [arxiv:2011.09533 §5.2] |
| You want a simple baseline on SMAC-style maps with little tuning | Independent PPO with shared weights | Author-reported win rates at or above joint-value methods on several hard maps [arxiv:2011.09533 §5.1] |
| The task has many agents and joint actions would be large | Avoid joint action spaces, such as independent learners | The joint action space grows exponentially with agent count [arxiv:2011.09533 §1] |
| Agents must coordinate tightly on rare joint outcomes | Be cautious with independent learners | The paper shows relative overgeneralisation is possible in a small matrix-game example, though not on its SMAC maps [arxiv:2011.09533 §6] |
| You need guarantees from theory | Do not rely on independent learners alone | Independent learning loses the usual convergence guarantees under non-stationarity [arxiv:2011.09533 §1] |

The table reflects one paper. The author-reported numbers come from the authors'
own runs, and the MAPPO numbers come from another benchmark [arxiv:2011.09533 fig. 2].

## Implementation sketch

The shared core is one update per agent with a critic whose input is the only
thing that changes between variants. The sketch follows the independent variant,
which is the one the card gives in full. For the centralised-critic variant,
pass the full state to the critic instead of the local observation
[arxiv:2011.09533 §5.2].

```pseudocode
# Shared-weight actor and critic, one update per agent  [arxiv:2011.09533 §4]
# Inputs: local observations (N, T, obs), actions (N, T), team rewards (T,)

def ctde_update(observations, state, actions, team_rewards, theta, phi):
    # Independent variant: critic reads each agent's local history  [§4]
    critic_input = observations
    # Centralised variant: critic reads the full state instead  [§5.2]
    # critic_input = state
    values = critic(critic_input, phi)                  # (N, T)
    # Team reward stands in for each agent's own reward  [§4]
    advantages = truncated_gae(team_rewards, values, h, 0.99, 0.95)
    # Clipped policy objective per agent, on its local observation  [eq. 5]
    ratio = prob(theta, observations, actions) / old_prob
    policy_term = minimum(ratio * advantages,
                          clip(ratio, 1 - eps, 1 + eps) * advantages)
    return -policy_term.mean() + value_loss(values, returns)
```

Execution is the same for every variant: each agent runs its actor on its own
observations only.

## Contradictions

None known across cards. Only one card lists this concept. Within the paper, the
value of central state is left open. The authors say it is unclear what central
state is worth on SMAC, given IPPO's results against MAPPO on the hard maps
[arxiv:2011.09533 §6].

## Papers

- [Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?](/papers/2011.09533): compares independent PPO with a centralised critic and with joint-value baselines on SMAC, and finds policy clipping is the key ingredient.