Centralised training with decentralised execution (CTDE)

Overview

Several agents act together, and each one sees only part of the situation at run time. CTDE lets training use global information, such as the full state or the joint actions of all agents. At execution time each agent still acts from its own local history 2011.09533 §1. The invariant all variants share is that the deployed policies never need the global information, so they can run on separate units.

Variants differ in what the training-time information is used for. Some learn a joint value function over all agents. Others give a shared critic the full state. At the far end, the training is fully independent, with no shared information at all 2011.09533 §1 2011.09533 §4. The card-backed source for this page is one paper, so the variants below are the ones it describes.

Variants

Joint-value CTDE methods (QMIX, MAVEN)

These learn a joint value function from global information during training. The paper uses them as baselines and names them as the usual CTDE joint learners. It does not reimplement them in its own method 2011.09533 §1 2011.09533 §5.1. Their joint action space grows with the number of agents 2011.09533 §1. Card: Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?

Centralised critic with full state

The actor stays local, but the critic reads the full state during training. The paper tests this as a parameter-sharing centralised critic and reports that it performs substantially worse than local critics on 3m, 3s5z vs 3s6z, MMM2 and corridor 2011.09533 §5.2 2011.09533 fig. 3. The authors say the result is not a verdict on central state in general. They say it is still unclear what central state is worth on SMAC 2011.09533 §6. Card: Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?

Independent PPO (no shared information)

Each agent runs PPO with its own local critic, and the agents share actor and critic weights. Each agent uses the team reward in place of its own reward 2011.09533 §4. The authors report that this matches or beats MAPPO, QMIX and other joint-value methods on several hard maps, with mild per-map tuning 2011.09533 §5.1. Their ablations point to policy clipping as the key ingredient, not a lower learning rate 2011.09533 §5.3. Card: Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?

When to use which

The table reflects one paper. The author-reported numbers come from the authors' own runs, and the MAPPO numbers come from another benchmark 2011.09533 fig. 2.

Implementation sketch

The shared core is one update per agent with a critic whose input is the only thing that changes between variants. The sketch follows the independent variant, which is the one the card gives in full. For the centralised-critic variant, pass the full state to the critic instead of the local observation 2011.09533 §5.2.

# Shared-weight actor and critic, one update per agent  [arxiv:2011.09533 §4]
# Inputs: local observations (N, T, obs), actions (N, T), team rewards (T,)

def ctde_update(observations, state, actions, team_rewards, theta, phi):
    # Independent variant: critic reads each agent's local history  [§4]
    critic_input = observations
    # Centralised variant: critic reads the full state instead  [§5.2]
    # critic_input = state
    values = critic(critic_input, phi)                  # (N, T)
    # Team reward stands in for each agent's own reward  [§4]
    advantages = truncated_gae(team_rewards, values, h, 0.99, 0.95)
    # Clipped policy objective per agent, on its local observation  [eq. 5]
    ratio = prob(theta, observations, actions) / old_prob
    policy_term = minimum(ratio * advantages,
                          clip(ratio, 1 - eps, 1 + eps) * advantages)
    return -policy_term.mean() + value_loss(values, returns)

Execution is the same for every variant: each agent runs its actor on its own observations only.

Contradictions

None known across cards. Only one card lists this concept. Within the paper, the value of central state is left open. The authors say it is unclear what central state is worth on SMAC, given IPPO's results against MAPPO on the hard maps 2011.09533 §6.

Papers

• Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?: compares independent PPO with a centralised critic and with joint-value baselines on SMAC, and finds policy clipping is the key ingredient.