Multi-agent reinforcement learning

Overview

Several agents learn to act in the same environment, and each one's outcome depends on what the others do 2011.09533 §3. This page covers the cooperative case, where all agents share one team reward. The setting is a Dec-POMDP: each agent chooses actions from its own observation history, and the team gets a single reward 2011.09533 §3.

The invariant the first two variants share is that the agents act from local information at run time. Methods differ in what they may use during training and in how each agent's learning signal is formed. The third variant does not keep that invariant: its agents act on a shared global state 2011.09902 §4.1 2011.09902 eq. 21.

The card-backed sources for this page are four papers: one on independent learners, one on splitting a sparse team return into per-agent rewards, one on per-base-station DDPG for edge resource allocation, and one survey of RL on FPGAs and integrated circuits that covers MARL hardware only briefly. The variants below are the ones they describe.

Variants

Independent learners (IPPO)

Each agent runs its own PPO learner with a local critic. The actor and critic weights are shared across agents, and each agent uses the team reward in place of its own reward 2011.09533 §4. No global information is used in training. The authors report that this matches or beats MAPPO, QMIX, MAVEN and IQL on several hard maps, with mild per-map tuning 2011.09533 §5.1. Card: Independent PPO on SMAC (IPPO)

The authors suggest that PPO's clipped objective keeps each agent's updates small enough to soften the effect of the other agents changing. Their ablations point to clipping as the key ingredient, not a lower learning rate 2011.09533 §1 2011.09533 §5.3.

Credit redistribution (TAR²)

This variant does not replace the learner. It changes the reward the learner sees. When the team gets one reward at the end of an episode, a reward model splits that return into one reward per time step, and each step's reward into one part per agent. The parts are non-negative and add up to the episodic return 2412.14779 §4.1 2412.14779 §4.2. The reward model is an attention network that extends AREL, with alternating temporal and agent attention blocks 2412.14779 §4.5. A single-agent policy-gradient method then trains on the split rewards. The paper uses IPPO and MAPPO for this 2412.14779 §5.1.

The motivation is that with sparse team rewards, the advantage estimate for one agent carries the noise of every other agent's contribution. The paper argues that this noise grows roughly linearly with the number of agents 2412.14779 §3.4. The paper also argues that the redistribution is a form of potential-based reward shaping, so the optimal policy is kept 2412.14779 §4.3. That proof sets the discount to 1, while the paper's own setting has a discount γ 2412.14779 §3.1 2412.14779 §4.3. Read the claim as holding for γ = 1 only.

The paper reports that this redistribution learns faster and more stably than dense per-agent rewards, episodic-only rewards, and other redistribution baselines. These comparisons appear only in a figure, on one scenario (5m_vs_6m), with no numbers in the text 2412.14779 §5.1 2412.14779 §6 2412.14779 fig. 1. Card: Agent-Temporal Credit Assignment (TAR²)

Per-agent DDPG for edge resource allocation

Here each agent is a base station, and the agents choose how a federated learning system is run: which base station hosts which user's digital twin, the batch size per twin, and each base station's share of bandwidth 2011.09902 §4.1. Each agent runs DDPG, an actor and critic with a replay memory and target networks 2011.09902 §4.2. All agents receive the same reward, the negative of the time per training iteration, because the slowest base station sets that time 2011.09902 §4.1 2011.09902 eq. 19.

Two points differ from the other variants. First, each actor reads the global state, which holds base-station compute, twin counts, twin data sizes and channel states, not its own observation 2011.09902 §4.1 2011.09902 eq. 21. Second, the critic's input is unclear. The text describes a critic that sees all agents' actions, but the critic update is written as a TD error on each agent's own action, so the paper does not say whether each agent has its own critic 2011.09902 §4.2 2011.09902 eq. 23.

The paper reports that the learned policy cuts total time cost compared with random and average edge association, but only in figures; the text gives no numbers 2011.09902 §5 2011.09902 fig. 5. The setup is one simulated system: CIFAR-10, 100 users and 5 base stations, with IID data 2011.09902 §5. Card: Low-latency Federated Learning and Blockchain for Edge Association in Digital Twin empowered 6G Networks

Hardware-accelerated MARL

This is a survey view, not a method. The FPGA and integrated-circuit review groups MARL accelerators by their bottleneck rather than by algorithm. It names three: control latency, inter-agent communication, and deep-network compute 2610.28699 §4.1.2. Its Table 8 lists two MARL designs: a fuel-cell vehicle energy manager on a Zynq-7020, and a fault-tolerant network-on-chip router on a Kintex-7 2610.28699 tab. 8. The survey runs no experiments of its own, and its baselines differ between studies, so its numbers cannot be ranked against each other 2610.28699 §3.9. Card: Reinforcement learning for hardware acceleration and system-level optimization on FPGA and integrated-circuit platforms.

Centralised training methods

These use global information during training and act locally at run time. The family includes joint value functions and centralised critics. The paper uses such methods as baselines and treats them as the usual alternative to independent learning 2011.09533 §1. They are covered on their own page: Centralised training with decentralised execution (CTDE).

When to use which

The table reflects four papers. The numbers come from the authors' own runs, and the MAPPO numbers come from a benchmark the paper cites 2011.09533 fig. 2. The TAR² rows rest on one scenario and one figure. The DDPG row rests on one simulated setup and figures without numbers. The hardware row rests on the survey's own tables, which it does not check.

Implementation sketch

The core is one PPO-style update per agent, with the team reward as each agent's reward. The sketch follows the independent variant, which the card gives in full. Defaults from the card: discount 0.99, GAE lambda 0.95, clip 0.2, gradient norm clip 0.5, and learning rate 1e-4 on most maps 2011.09533 tab. 1.

# Shared-weight independent PPO, one update per batch  [arxiv:2011.09533 §4]
# Inputs: local observations (N, T, obs), actions (N, T), team rewards (T,)

def ippo_step(observations, actions, team_rewards, theta, phi):
    # Each agent's critic reads only its own history  [§4]
    values = critic(observations, phi)                    # (N, T)
    # Team reward stands in for each agent's reward  [§4]
    advantages = truncated_gae(team_rewards, values, h, 0.99, 0.95)
    # Probability ratio of new to old policy, per agent  [eq. 5]
    ratio = prob(theta, observations, actions) / old_prob
    # Clipped surrogate, applied to each agent's own action  [eq. 5]
    policy_term = minimum(ratio * advantages,
                          clip(ratio, 1 - 0.2, 1 + 0.2) * advantages)
    # Value loss and entropy bonus are added per eq. 7  [eq. 6] [eq. 7]
    return loss_from(policy_term, values, theta, phi)

Execution is the same for every variant: each agent runs the shared actor on its own observations only 2011.09533 §4. The DDPG variant is the exception, since its actor reads the global state 2011.09902 eq. 21.

To add credit redistribution, replace team_rewards with the per-agent, per-step credit from the reward model. The learner and its update stay the same 2412.14779 §5.1. The paper trains the reward model on episode returns and uses it only after a 2000-episode warm-up 2412.14779 §6. The reward-model loss and its architecture are described in the card's Method section and are not restated here.

For the DDPG variant, do not follow the card's Algorithm 1 as printed. It nests the episode loop inside a loop over agents, and its target update is degenerate as written 2011.09902 alg. 1 2011.09902 eq. 24. The card infers the intended soft update, which is not stated in the paper; the paper gives no value for its rate β.

Contradictions

The cards bear on the same question, how much credit assignment matters, in different settings. The IPPO paper says multi-agent credit assignment is not clearly needed on its SMAC maps 2011.09533 §6. The TAR² paper argues that credit assignment is central when the team reward is sparse and episodic 2412.14779 §3.4. Neither paper tests the other's setting, so this is a difference in scope, not a direct conflict. The edge-association paper does not address credit assignment at all; its agents share one time-based reward 2011.09902 §4.1.

Within the IPPO paper, the value of central state is left open: the authors say it is unclear what it is worth on SMAC, given the independent learner's results against MAPPO on the hard maps 2011.09533 §6.

The TAR² paper states a warm-up of 2000 episodes 2412.14779 §6. The released code's default is 200 episodes, according to the paper's card. Check which value the reported curves used before comparing against them.

Papers

• Independent PPO on SMAC (IPPO): independent PPO with shared weights matches or beats joint-value and centralised-critic baselines on several SMAC maps, and policy clipping is the key ingredient in its ablations.
• Agent-Temporal Credit Assignment (TAR²): learns a reward model that splits a sparse episodic team return into per-step, per-agent rewards for IPPO or MAPPO; its results are one figure on one scenario, with no numbers.
• Low-latency Federated Learning and Blockchain for Edge Association in Digital Twin empowered 6G Networks: one DDPG agent per base station chooses twin association, batch sizes and bandwidth to cut federated training time; results are figures only, and the update equations have several defects.
• Reinforcement learning for hardware acceleration and system-level optimization on FPGA and integrated-circuit platforms: A survey: surveys RL on FPGAs and integrated circuits; its MARL coverage is one bottleneck grouping and two rows in its results table, with no new experiments.