--- kind: concept slug: multi-agent-reinforcement-learning title: Multi-agent reinforcement learning summary: >- Several learning agents act in one environment. Independent PPO with shared weights matches or beats joint-value methods on several StarCraft maps. TAR² splits a sparse team return into per-agent, per-step rewards. A per-base-station DDPG allocates edge resources. Results are author-reported; this page has four card-backed sources. tags: [reinforcement-learning, multi-agent, cooperation, starcraft] papers: ["2011.09533", "2412.14779", "2011.09902", "2610.28699"] related: [centralized-training-decentralized-execution, actor-critic, policy-gradient, proximal-policy-optimization, credit-assignment, reward-modeling] status: draft updated_at: 2026-11-01 --- ## Overview Several agents learn to act in the same environment, and each one's outcome depends on what the others do [arxiv:2011.09533 §3]. This page covers the cooperative case, where all agents share one team reward. The setting is a Dec-POMDP: each agent chooses actions from its own observation history, and the team gets a single reward [arxiv:2011.09533 §3]. The invariant the first two variants share is that the agents act from local information at run time. Methods differ in what they may use during training and in how each agent's learning signal is formed. The third variant does not keep that invariant: its agents act on a shared global state [arxiv:2011.09902 §4.1] [arxiv:2011.09902 eq. 21]. The card-backed sources for this page are four papers: one on independent learners, one on splitting a sparse team return into per-agent rewards, one on per-base-station DDPG for edge resource allocation, and one survey of RL on FPGAs and integrated circuits that covers MARL hardware only briefly. The variants below are the ones they describe. ## Variants ### Independent learners (IPPO) Each agent runs its own PPO learner with a local critic. The actor and critic weights are shared across agents, and each agent uses the team reward in place of its own reward [arxiv:2011.09533 §4]. No global information is used in training. The authors report that this matches or beats MAPPO, QMIX, MAVEN and IQL on several hard maps, with mild per-map tuning [arxiv:2011.09533 §5.1]. Card: [Independent PPO on SMAC (IPPO)](/papers/2011.09533) The authors suggest that PPO's clipped objective keeps each agent's updates small enough to soften the effect of the other agents changing. Their ablations point to clipping as the key ingredient, not a lower learning rate [arxiv:2011.09533 §1] [arxiv:2011.09533 §5.3]. ### Credit redistribution (TAR²) This variant does not replace the learner. It changes the reward the learner sees. When the team gets one reward at the end of an episode, a reward model splits that return into one reward per time step, and each step's reward into one part per agent. The parts are non-negative and add up to the episodic return [arxiv:2412.14779 §4.1] [arxiv:2412.14779 §4.2]. The reward model is an attention network that extends AREL, with alternating temporal and agent attention blocks [arxiv:2412.14779 §4.5]. A single-agent policy-gradient method then trains on the split rewards. The paper uses IPPO and MAPPO for this [arxiv:2412.14779 §5.1]. The motivation is that with sparse team rewards, the advantage estimate for one agent carries the noise of every other agent's contribution. The paper argues that this noise grows roughly linearly with the number of agents [arxiv:2412.14779 §3.4]. The paper also argues that the redistribution is a form of potential-based reward shaping, so the optimal policy is kept [arxiv:2412.14779 §4.3]. That proof sets the discount to 1, while the paper's own setting has a discount γ [arxiv:2412.14779 §3.1] [arxiv:2412.14779 §4.3]. Read the claim as holding for γ = 1 only. The paper reports that this redistribution learns faster and more stably than dense per-agent rewards, episodic-only rewards, and other redistribution baselines. These comparisons appear only in a figure, on one scenario (5m_vs_6m), with no numbers in the text [arxiv:2412.14779 §5.1] [arxiv:2412.14779 §6] [arxiv:2412.14779 fig. 1]. Card: [Agent-Temporal Credit Assignment (TAR²)](/papers/2412.14779) ### Per-agent DDPG for edge resource allocation Here each agent is a base station, and the agents choose how a federated learning system is run: which base station hosts which user's digital twin, the batch size per twin, and each base station's share of bandwidth [arxiv:2011.09902 §4.1]. Each agent runs DDPG, an actor and critic with a replay memory and target networks [arxiv:2011.09902 §4.2]. All agents receive the same reward, the negative of the time per training iteration, because the slowest base station sets that time [arxiv:2011.09902 §4.1] [arxiv:2011.09902 eq. 19]. Two points differ from the other variants. First, each actor reads the global state, which holds base-station compute, twin counts, twin data sizes and channel states, not its own observation [arxiv:2011.09902 §4.1] [arxiv:2011.09902 eq. 21]. Second, the critic's input is unclear. The text describes a critic that sees all agents' actions, but the critic update is written as a TD error on each agent's own action, so the paper does not say whether each agent has its own critic [arxiv:2011.09902 §4.2] [arxiv:2011.09902 eq. 23]. The paper reports that the learned policy cuts total time cost compared with random and average edge association, but only in figures; the text gives no numbers [arxiv:2011.09902 §5] [arxiv:2011.09902 fig. 5]. The setup is one simulated system: CIFAR-10, 100 users and 5 base stations, with IID data [arxiv:2011.09902 §5]. Card: [Low-latency Federated Learning and Blockchain for Edge Association in Digital Twin empowered 6G Networks](/papers/2011.09902) ### Hardware-accelerated MARL This is a survey view, not a method. The FPGA and integrated-circuit review groups MARL accelerators by their bottleneck rather than by algorithm. It names three: control latency, inter-agent communication, and deep-network compute [arxiv:2610.28699 §4.1.2]. Its Table 8 lists two MARL designs: a fuel-cell vehicle energy manager on a Zynq-7020, and a fault-tolerant network-on-chip router on a Kintex-7 [arxiv:2610.28699 tab. 8]. The survey runs no experiments of its own, and its baselines differ between studies, so its numbers cannot be ranked against each other [arxiv:2610.28699 §3.9]. Card: [Reinforcement learning for hardware acceleration and system-level optimization on FPGA and integrated-circuit platforms](/papers/2610.28699). ### Centralised training methods These use global information during training and act locally at run time. The family includes joint value functions and centralised critics. The paper uses such methods as baselines and treats them as the usual alternative to independent learning [arxiv:2011.09533 §1]. They are covered on their own page: [Centralised training with decentralised execution (CTDE)](/concepts/centralized-training-decentralized-execution). ## When to use which | Situation | Use | Why | |---|---|---| | Many agents, so a joint action space would be very large | Independent learners | The joint action space grows exponentially with the number of agents, which is the cost the authors name for joint learners [arxiv:2011.09533 §1] | | Global state is available in training and you want to try a centralised critic | Test it against local critics first | In the card's comparison the centralised critic was substantially worse on several maps [arxiv:2011.09533 §5.2] | | You want a simple, lightly tuned baseline on SMAC-style maps | Independent PPO with shared weights | Author-reported win rates at or above the compared methods on several hard maps [arxiv:2011.09533 §5.1] | | Agents must coordinate tightly on rare joint outcomes | Be cautious with independent learners | The paper says independent learners are in principle prone to relative overgeneralisation, though its SMAC maps did not show it [arxiv:2011.09533 §6] | | You need convergence guarantees | Do not rely on independent learners alone | Other agents' learning makes the environment non-stationary, which removes the usual guarantees [arxiv:2011.09533 §1] | | The team reward arrives only at episode end and per-step credit is the bottleneck | Credit redistribution (TAR²) under IPPO or MAPPO | The paper's motivation is that sparse team rewards make the per-agent advantage noisy [arxiv:2412.14779 §3.4]. Its gains are shown only as a figure on one scenario [arxiv:2412.14779 fig. 1] | | You need the redistributed reward to keep the optimal policy with a discount below 1 | Do not rely on the TAR² equivalence claim alone | The proof sets the discount to 1 [arxiv:2412.14779 §4.3]; the paper's setting uses a general discount [arxiv:2412.14779 §3.1] | | Agents assign resources or tasks to each other under one shared time-based reward, and a simulator is available | Per-agent DDPG, as in the edge-association paper | The paper's actions are batch sizes, bandwidth shares and twin assignments, with DDPG as the learner [arxiv:2011.09902 §4.1] [arxiv:2011.09902 §4.2]. Its gains are shown only in figures on one simulated setup [arxiv:2011.09902 fig. 5] | | You need to reproduce the DDPG update exactly | Do not copy the paper's update equations | The target-network update is written with θ on both sides, and the target value is never defined [arxiv:2011.09902 §4.2] [arxiv:2011.09902 eq. 24] | | The agents must run on an FPGA or integrated circuit | Start from the survey's bottleneck grouping, then read the original MARL design | The survey lists only two MARL designs and gives no method comparison; its numbers are author-reported and not comparable across rows [arxiv:2610.28699 tab. 8] [arxiv:2610.28699 §3.9] | The table reflects four papers. The numbers come from the authors' own runs, and the MAPPO numbers come from a benchmark the paper cites [arxiv:2011.09533 fig. 2]. The TAR² rows rest on one scenario and one figure. The DDPG row rests on one simulated setup and figures without numbers. The hardware row rests on the survey's own tables, which it does not check. ## Implementation sketch The core is one PPO-style update per agent, with the team reward as each agent's reward. The sketch follows the independent variant, which the card gives in full. Defaults from the card: discount 0.99, GAE lambda 0.95, clip 0.2, gradient norm clip 0.5, and learning rate 1e-4 on most maps [arxiv:2011.09533 tab. 1]. ```pseudocode # Shared-weight independent PPO, one update per batch [arxiv:2011.09533 §4] # Inputs: local observations (N, T, obs), actions (N, T), team rewards (T,) def ippo_step(observations, actions, team_rewards, theta, phi): # Each agent's critic reads only its own history [§4] values = critic(observations, phi) # (N, T) # Team reward stands in for each agent's reward [§4] advantages = truncated_gae(team_rewards, values, h, 0.99, 0.95) # Probability ratio of new to old policy, per agent [eq. 5] ratio = prob(theta, observations, actions) / old_prob # Clipped surrogate, applied to each agent's own action [eq. 5] policy_term = minimum(ratio * advantages, clip(ratio, 1 - 0.2, 1 + 0.2) * advantages) # Value loss and entropy bonus are added per eq. 7 [eq. 6] [eq. 7] return loss_from(policy_term, values, theta, phi) ``` Execution is the same for every variant: each agent runs the shared actor on its own observations only [arxiv:2011.09533 §4]. The DDPG variant is the exception, since its actor reads the global state [arxiv:2011.09902 eq. 21]. To add credit redistribution, replace `team_rewards` with the per-agent, per-step credit from the reward model. The learner and its update stay the same [arxiv:2412.14779 §5.1]. The paper trains the reward model on episode returns and uses it only after a 2000-episode warm-up [arxiv:2412.14779 §6]. The reward-model loss and its architecture are described in the card's Method section and are not restated here. For the DDPG variant, do not follow the card's Algorithm 1 as printed. It nests the episode loop inside a loop over agents, and its target update is degenerate as written [arxiv:2011.09902 alg. 1] [arxiv:2011.09902 eq. 24]. The card infers the intended soft update, which is not stated in the paper; the paper gives no value for its rate β. ## Contradictions The cards bear on the same question, how much credit assignment matters, in different settings. The IPPO paper says multi-agent credit assignment is not clearly needed on its SMAC maps [arxiv:2011.09533 §6]. The TAR² paper argues that credit assignment is central when the team reward is sparse and episodic [arxiv:2412.14779 §3.4]. Neither paper tests the other's setting, so this is a difference in scope, not a direct conflict. The edge-association paper does not address credit assignment at all; its agents share one time-based reward [arxiv:2011.09902 §4.1]. Within the IPPO paper, the value of central state is left open: the authors say it is unclear what it is worth on SMAC, given the independent learner's results against MAPPO on the hard maps [arxiv:2011.09533 §6]. The TAR² paper states a warm-up of 2000 episodes [arxiv:2412.14779 §6]. The released code's default is 200 episodes, according to the paper's card. Check which value the reported curves used before comparing against them. ## Papers - [Independent PPO on SMAC (IPPO)](/papers/2011.09533): independent PPO with shared weights matches or beats joint-value and centralised-critic baselines on several SMAC maps, and policy clipping is the key ingredient in its ablations. - [Agent-Temporal Credit Assignment (TAR²)](/papers/2412.14779): learns a reward model that splits a sparse episodic team return into per-step, per-agent rewards for IPPO or MAPPO; its results are one figure on one scenario, with no numbers. - [Low-latency Federated Learning and Blockchain for Edge Association in Digital Twin empowered 6G Networks](/papers/2011.09902): one DDPG agent per base station chooses twin association, batch sizes and bandwidth to cut federated training time; results are figures only, and the update equations have several defects. - [Reinforcement learning for hardware acceleration and system-level optimization on FPGA and integrated-circuit platforms: A survey](/papers/2610.28699): surveys RL on FPGAs and integrated circuits; its MARL coverage is one bottleneck grouping and two rows in its results table, with no new experiments.