Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?

In plain words

Several units have to act as a team, but each one can only see its own part of the situation. A common way to train such a team is to give a shared "coach" access to everything during training, so it can judge the team's joint moves. This paper asks whether something much simpler works: give every unit its own learner, use the standard PPO algorithm for each one, and let each learner judge its own situation using the team's shared score.

On StarCraft micromanagement maps, that simple setup often matches or beats the coached methods. The result is surprising because independent learners are known to have theoretical problems. From one unit's point of view, the others keep changing, so the world seems unstable. The authors suggest that PPO's clipping keeps each update small enough to soften this problem. Their ablations point to clipping, not a lower learning rate, as the key ingredient.

The numbers come from the authors' own runs on a set of StarCraft maps. The comparison numbers for the coached methods are partly taken from another benchmark. The authors also say that the value of the shared state for a coach is still unclear.

TL;DR

• Independent PPO (IPPO): each agent runs PPO with its own local critic, and all agents share one actor network and one critic network §4.
• The agents use the team reward in place of their own reward and are trained on their own observation histories §4.
• On SMAC, the authors report IPPO at or above MAPPO, QMIX, MAVEN and IQL on several hard maps, with mild per-map tuning §5.1.
• Ablations: removing policy clipping hurts most; a lower learning rate for IAC does not reproduce the gain §5.3.
• Code: none located. Only the paper's LaTeX source was read.

Problem

Cooperative multi-agent reinforcement learning (MARL) is often posed as a Dec-POMDP: agents choose actions from local histories and share one team reward §3. Centralised training with decentralised execution (CTDE) methods use global information during training. Many of them learn a joint value function. These joint learners face a joint action space that grows exponentially with the number of agents, plus learning pathologies that can leave them at suboptimal policies §1.

Independent learning avoids the joint action space, but the authors list two theoretical limits. First, other agents' learning and exploration make the environment non-stationary, which removes the usual convergence guarantees. Second, an independent learner may not tell environment noise apart from another agent's exploration §1. The paper's hypothesis is that PPO's objective, in particular policy clipping, may reduce the effect of this non-stationarity in practice §1.

Method

IPPO uses PPO independently for each agent. Each agent has its own local-observation critic V_\phi(z^a_t), with critic weights \phi shared across agents. The actor weights \theta are shared across agents as well §4. Advantages come from generalised advantage estimation (GAE) with \gamma = 0.99 and \lambda = 0.95, computed for each agent from the team reward §4.

The method has four parts §4:

1. Advantage. A truncated GAE sum over h steps, with h set per map in tab. 1 (column "steps num") eq. 4.
2. Policy loss. The PPO clipped surrogate, applied per agent to its own observation and action eq. 5.
3. Value loss. A clipped critic loss, in the style of GAE, that keeps each update within \epsilon of the old critic eq. 6.
4. Total loss. The sum over agents of the policy and critic losses plus an entropy term with its own coefficient eq. 7.

The paper also states the architecture §4. Observations for the last few steps are stacked as frames tab. 1. Three 1-D convolution layers (kernel size 3, padding "same, valid, valid") feed two ReLU layers of 256 and 128 units. The channel counts vary by map tab. 1. Weights use a variance-scaling initializer with a truncated normal distribution and scale 2.0 §4.

Key equations, as written in the paper:

PPO clipped objective for a single policy, used as the base eq. 3:

\mathcal{L}(\theta) = \mathbb{E}\left[ \min\left( \frac{\pi_\theta(u_t \mid s_t)}{\pi_{\theta_{old}}(u_t \mid s_t)} A(s_t, u_t),\; \mathrm{clip}\left(\frac{\pi_\theta(u_t \mid s_t)}{\pi_{\theta_{old}}(u_t \mid s_t)}, 1-\epsilon, 1+\epsilon\right) A(s_t, u_t) \right)\right]

Per-agent advantage, with the team reward standing in for the agent's reward eq. 4 §4:

A^a_t = \sum_{l=0}^{h} (\gamma\lambda)^l \, \delta^a_{t+l}, \qquad \delta^a_t = r_t + \gamma V_\phi(z^a_{t+1}) - V_\phi(z^a_t)

Per-agent policy loss eq. 5:

\mathcal{L}^a(\theta) = \mathbb{E}\left[ \min\left( \frac{\pi_\theta(u^a_t \mid z^a_t)}{\pi_{\theta_{old}}(u^a_t \mid z^a_t)} A^a_t,\; \mathrm{clip}\left(\frac{\pi_\theta(u^a_t \mid z^a_t)}{\pi_{\theta_{old}}(u^a_t \mid z^a_t)}, 1-\epsilon, 1+\epsilon\right) A^a_t \right)\right]

Clipped value loss, with \hat V^a_t = A^a_t + V_\phi(z^a_t) eq. 6:

\mathcal{L}^a(\phi) = \mathbb{E}\left[\min\left\{ \left(V_\phi(z^a_t) - \hat V^a_t\right)^2,\; \left(V_{\phi_{old}}(z^a_t) + \mathrm{clip}\left(V_\phi(z^a_t) - V_{\phi_{old}}(z^a_t), -\epsilon, +\epsilon\right) - \hat V^a_t\right)^2 \right\}\right]

Total loss eq. 7:

\mathcal{L}(\theta, \phi) = \sum_{a} \mathcal{L}^a(\theta) + \lambda_{critic}\, \mathcal{L}^a(\phi) + \lambda_{entropy}\, \mathcal{H}(\pi^a)

The paper gives the update in prose and equations only. It has no algorithm box. The pseudocode below is written from §4 and eq. 3 to eq. 7:

# Independent PPO update for N agents, shared actor and critic weights  [§4]
# Inputs:  observations (N, T, obs), actions (N, T), team rewards (T,)
# Output:  updated actor weights theta and critic weights phi

def ippo_update(observations, actions, team_rewards, theta, phi):
    # Each agent's local critic estimates value from its own history  [§4]
    values = critic(observations, phi)                    # (N, T)
    # Team reward stands in for each agent's own reward  [§4]
    # Truncated GAE over h steps, gamma 0.99, lambda 0.95  [eq. 4]
    advantages = truncated_gae(team_rewards, values, h, 0.99, 0.95)
    # Normalise advantages once, before training, not per minibatch  [§4]
    advantages = (advantages - mean(advantages)) / std(advantages)
    # Targets: V_hat = A + V_phi, as in the value loss  [eq. 6]
    returns = advantages + values                         # (N, T)
    # Probability ratio of the current to the old policy  [eq. 5]
    ratio = prob(theta, observations, actions) / old_prob
    # Pessimistic minimum of unclipped and clipped surrogate  [eq. 5]
    policy_term = minimum(
        ratio * advantages,
        clip(ratio, 1 - eps, 1 + eps) * advantages,
    )
    # Clipped value loss, eps for value clipping  [eq. 6]
    value_term = clipped_value_loss(values, old_values, returns, eps)
    # Entropy of each agent's policy  [eq. 7]
    entropy = policy_entropy(theta, observations)         # (N, T)
    # Eq. 7 is a sum; the paper's signs mix maximised and minimised terms,
    # so the signs here are chosen for minimisation  [eq. 7]
    loss = (
        -policy_term.sum_over_agents().mean()
        + critic_coef * value_term.sum_over_agents().mean()
        - entropy_coef * entropy.sum_over_agents().mean()
    )
    return loss

Hyperparameters

Values from tab. 1 and §4 unless marked. The paper states that the main hyperparameters were chosen with only mild tuning per map §5.

Per-map values from tab. 1 (the table's "norm input" column is not defined in the paper):

Source: tab. 1.

Results

Author-reported. Setup: SMAC game version 4.6 at the hardest built-in AI difficulty, 16 maps per the text (the table lists 17; see Pitfalls) §5. The main comparison is median test win rate, with uncertainty regions showing the [0.25, 0.75] interval of the median fig. 1. Baselines are QMIX and IQL fig. 1 §5.1. The MAPPO numbers come from a benchmarking study that the paper cites, not from runs in this paper fig. 2.

Median test win rate at 10M steps (values in percent) fig. 2:

Other reported comparisons, given only in prose or in figures (no numbers in the text):

• IPPO exceeds MAPPO and QMIX on 3s5z vs 3s6z, corridor and 6h vs 8z §5.1 fig. 2. On corridor it also exceeds MAVEN §5.1.
• IPPO outperforms QMIX on the easy maps 2m vs 1z and 3m, and is competitive on other maps §5.1 fig. 5.
• IPPO beats IQL in performance and is more stable across maps §5.1.
• Replacing IPPO's local critics with centralised parameter-sharing critics, using full state, performs substantially worse on 3m, 3s5z vs 3s6z, MMM2 and corridor §5.2 fig. 3.
• Ablation (6 maps: corridor, bane vs bane, 6h vs 8z, MMM2, 3s5z vs 3s6z, 2s3z): removing both clipping terms (an IAC variant) performs poorly everywhere; policy clipping is essential; value clipping helps on corridor and MMM2; a lower learning rate for IAC does not match IPPO §5.3 fig. 4.
• Relative overgeneralisation (Stag Hunt, penalty -2): the paper shows IPPO is in principle prone to it, while the SMAC maps do not show the problem §6 fig. 6.
• Open question: the paper says it "remains unclear" what central state information is worth on SMAC, given IPPO's results against MAPPO on the hard maps §6.

Paper vs code

No official code.

The paper has no code link in its text, abstract or arXiv comments. No project page is linked. GitHub searches found no repository from the authors: ippo (only third-party repos); ippo org:oxwhirl (none); independent ppo smac (none); marl starcraft ippo sun (none). The oxwhirl org repo list has pymarl, smac and smacv2, but no IPPO repo, and a direct oxwhirl/ippo lookup returned 404. The first 18 repos of the Schroeder de Witt account were listed; the rest were truncated, and no IPPO or SMAC repo appeared among the 18 shown. Third-party IPPO reimplementations were excluded.

Pitfalls

• Sign convention in eq. 5 and eq. 7. The policy term and the entropy term are written as quantities to maximise, but the value loss as a loss to minimise. Eq. 7 adds all three with plus signs. Pick signs explicitly; the pseudocode above uses one consistent convention.
• Value clipping direction in eq. 6. The equation takes the min of the two squared errors. The text does not say whether this is the intended form, and no code was located to check it. State the choice in your implementation.
• Value \hat V depends on \phi. Eq. 6 defines \hat V^a_t as A^a_t + V_\phi(z^a_t) with the current weights. Whether this target is detached from the gradient is not stated.
• Epsilon for value clipping. The text uses the same symbol \epsilon for policy and value clipping. The table caption lists both e\_clip = 0.2 and \epsilon = 0.2. Whether the value range equals the policy range is not stated explicitly.
• Convolution strides conflict. The text gives strides (2, 1, 1) for the three conv layers §4. The table caption says all convolutions have stride 3 tab. 1. The two cannot both hold. The paper does not say which is used.
• Advantage normalisation scope. The text says normalise "once before training" and that per-minibatch normalisation is worse §4. The caption says "always normalize" tab. 1. "Once" is ambiguous: it may mean over the whole batch collected before the update, or once per run.
• Missing budget details. The optimizer, the rollout length per update, the relation between "mini batch" and the rollout, the number of seeds, the evaluation episode count, and the main training length are not stated. The "norm input" column is not defined.
• Map count. The text says 16 maps §5. tab. 1 lists 17 maps. The count is not reconciled.
• Caption mismatch. The appendix figure caption says "comparing IPPO and IPPO-C with QMIX, VDN and IQL" fig. 5. The main text compares IPPO with QMIX and IQL only, and IPPO-C is not defined in the paper's text.
• MAPPO numbers are not re-measured here. The table compares IPPO against MAPPO results taken from the benchmarking study cited under the key anonymous_benchmarking_2020, whose bibliography entry gives the author as "Anonymous" fig. 2. The paper says its own centralised-critic runs are "calibrated" against that study's performance §5.2. Treat cross-paper numbers as unverified here.
• Team reward, not per-agent reward. The per-agent advantage uses the team reward as the reward for z^a_t §4. The paper does not say how this affects credit assignment, and the discussion notes that multi-agent credit assignment is not clearly needed on these maps §6.
• Shared weights and agent identity. Actor and critic weights are shared across agents §4. The paper does not say whether agent identity is added to the observations. Without it, agents with the same observation act the same way.
• Frame stacking differs by map. Four stacked frames for most maps, one for three maps tab. 1. The 1-frame setting is not discussed in the text.
• Lowered learning rate is not an equivalent control. The ablation tests a lower learning rate for IAC only, and the authors report that it does not close the gap §5.3. Other baselines are not tested with the same control.
• Figures have no numeric values in the text. The QMIX, MAVEN and IQL comparisons are read from plots fig. 1 fig. 5. Use the plots, not the prose, for numbers, and treat them as approximate.

Lineage

• Builds on: PPO, the clipped surrogate objective eq. 3 (Schulman et al., 2017, bibliography key schulman2017proximal); GAE, for the advantage eq. 4 (schulman_high-dimensional_2018); TRPO as the motivating trust-region method §3.
• Compared against: QMIX [rashid_monotonic_2020]; MAVEN [mahajan_maven_2020]; IQL §5.1; IAC [foerster_counterfactual_2017]; and MAPPO, which the paper treats as concurrent work [anonymous_benchmarking_2020].
• Superseded by: not known from this wiki.