Agent-Temporal Credit Assignment for Optimal Policy Preservation in Sparse Multi-Agent Reinforcement Learning

In plain words

In a team game, a group of agents often only hears "we won" or "we lost" at the very end. That single signal is hard to learn from: the agents cannot tell which moment, or which teammate, made the difference. This paper trains a separate model that looks back over a whole finished game and hands out a small reward to each agent at each moment, so the total still adds up to the final result.

The agents can then be trained with ordinary single-agent learning methods, which only see these small rewards. The authors argue that this handout does not change which behaviour is best, because the extra rewards cancel out over time. The method is tested on one StarCraft-style battle, where it learns faster than other ways of handing out rewards, though the paper does not give numbers.

TL;DR

• Sparse, end-of-episode team rewards make multi-agent credit assignment hard.
• TAR² learns a reward model that splits the episodic return across time steps and agents, using learned temporal and agent weights §4.2.
• The paper argues the redistribution is equivalent to potential-based reward shaping, so the optimal policy is preserved §4.3. The proof has gaps (see Pitfalls).
• Only one scenario (5m_vs_6m) is evaluated, and only as a figure §5 fig. 1.

Problem

With sparse or delayed global rewards, a policy-gradient update for agent i carries the noise of every other agent's contribution. The paper argues that this noise makes the variance of the advantage estimate grow roughly linearly with the number of agents §3.4, so learning needs more samples §3.4. Methods that handle agent credit (VDN, QMIX, COMA and others) do not address temporal credit, and dense-reward methods from single-agent RL do not address agent credit §2.

Method

The method has two stages §4.1 §4.2:

1. A reward model turns the episodic return R into one reward per time step, r_global,t = w_t · R, with weights w_t that are non-negative and sum to 1 over the episode §4.2.
2. The per-step reward is split across agents, r_i,t = w'{t,i} · r_global,t, with weights w'{t,i} that are non-negative and sum to 1 over agents §4.2.

Together, r_i,t = w'_{t,i} · w_t · R, and summing over agents and time gives back R §4.1 §4.2. The authors propose to learn w and w' with an architecture that extends AREL's attention reward model: a temporal attention block, then an agent attention block, alternated over layers §1 §4.5. The learned redistribution then replaces the sparse reward, and any single-agent policy-gradient method (the paper uses IPPO and MAPPO) is trained on it §5.1.

Reward model input and output, as implemented in the code, are described in the pseudocode below. The paper only states the constraints; the normalisation that enforces them is not specified in the paper.

# TAR² reward redistribution  [§4.2] [§4.5]
# Input:  obs and actions of N agents over T steps; episodic return R
# Output: credit[i, t] with sum over i and t equal to R

def redistribute(obs, actions, episode_return):
    # Per (agent, step) scores from the reward model  [§4.5]
    scores = reward_model(obs, actions)              # (N, T)
    # Temporal weights: non-negative, sum to 1 over time  [§4.2]
    temporal_weights = normalise_over_time(scores)   # (T,)
    # Agent weights: non-negative, sum to 1 over agents  [§4.2]
    agent_weights = normalise_over_agents(scores)    # (N, T)
    # Credit for agent i at step t  [§4.2]
    credit = agent_weights * temporal_weights * episode_return
    return credit                                    # (N, T)

Training the reward model. The paper says the predicted per-step rewards must add up to the episodic return §4.1, but does not state a loss. The released code uses a squared error on that sum plus an auxiliary action-prediction term:

# Reward model loss  [§4.1]; action term is code-only, see Paper vs code
def reward_loss(batch):
    # Predicted per-step rewards must sum to the episodic return  [§4.1]
    predicted = reward_model(batch.obs, batch.actions)    # (N, T)
    prediction_loss = squared_error(predicted.sum(), batch.episode_return)
    # Auxiliary action prediction from the hidden state  [code: MAPPO/agent.py#L841-L853]
    action_loss = cross_entropy(action_head(batch), batch.actions)
    return prediction_loss + 0.05 * action_loss   # 0.05 from code default

Policy learning. The paper says a warm-up period of 2000 episodes trains the reward model before it is used, and that this worked better than none for on-policy learners §6. The paper does not say whether the policy is updated during warm-up. The released code does not update the policy during warm-up: it clears the buffer until the warm-up ends, then replaces the rewards with the redistributed ones and updates as usual train_agent_multi_proc.py.

Hyperparameters

The paper states few numeric values. The environment and reward definitions below are from the paper; the other values come from the released training script and are code defaults, not paper values.

Results

Author-reported. Setup: SMACLite 5m_vs_6m, IPPO and MAPPO as the single-agent learners, average episodic reward with standard deviation fig. 1.

The paper says TAR² "stabilizes and accelerates" learning §6 §7. Without the numbers, these are qualitative claims. The paper has no ablation tables and no results on any other environment.

Paper vs code

Official code: the first author's account, AdityaKapoor74, at the commit in the frontmatter. The repository has no LICENSE file, so the code is cited by path and line only and nothing is quoted. The repository's own README is one line. I read MAPPO/ (the TAR²_v2 model in MAPPO/TARR_v2/) and MAPPO/agent.py. I did not check the QMIX folder or any IPPO runner.

• Which variant is the paper's method. The paper describes one model. The code has three TAR² variants (TARR, TARR_v2, TARR_HindSight). The training script selects TAR^2_v2 train_agent_multi_proc.py. I could not confirm from the paper alone which variant produced its figure.
• Warm-up. Paper: 2000 episodes §6. Code default: 200 train_agent_multi_proc.py.
• Weight normalisation. The paper requires non-negative weights that sum to 1 §4.2 but does not say how. The code takes the minimum over the batch, subtracts it, and divides by the sum agent.py. A softmax version is commented out agent.py.
• Loss. The paper states the sum constraint only §4.1. The code trains with a squared error on the summed predictions plus a cross-entropy action prediction term with weight 0.05 agent.py train_agent_multi_proc.py. The paper does not mention an action-prediction term.
• Attention architecture. The paper says it extends AREL with alternating temporal and agent attention §4.5. The TAR^2_v2 model replaces the agent attention block with a Shapley-style block that averages attention over random coalition masks drawn with sample_num samples TARR_v2.py. The paper does not describe this.
• Output head. The TAR^2_v2 reward head is linear, with no ReLU TARR_v2.py. The earlier TARR variant applies a sign-gated ReLU TARR.py. The paper does not say which applies.
• Training schedule. Code: reward model updated every 200 episodes on 128 sampled episodes, 200 epochs each train_agent_multi_proc.py. The paper gives no schedule.
• Reward timing. Matches the paper: the episodic team reward is sent at episode end and the dense per-agent reward is accumulated train_agent_multi_proc.py §5.2.
• Environment and outcome. The paper's scenario and reward definitions §5.2 are not checked against the external smaclite package, which is not in this repository.

Pitfalls

• The equivalence proof sets γ = 1. The proof §4.3 matches the redistributed reward to potential-based shaping only with γ = 1. The paper's own setting uses a discount γ §3.1. The equivalence is therefore shown only for the undiscounted case, which the paper does not say.
• The potential depends on the whole episode. The proof defines the potential as the episodic return times the cumulative weights §4.3. Standard potential-based shaping §3.3 needs a function of the state. The proof does not show the episode-dependent potential fits that form. The per-step reward r_i,t also depends on the final joint history h_|τ| §4.1, so it is not a function of (s_t, a_t, s_{t+1}) alone as used in §4.2.
• The direction claim is not proven. The proposition says the policy gradient "shares the same direction but ... a smaller magnitude" §4.4. The proof defines a trajectory-dependent scalar δ(τ) in [0, 1] and shows only that the rewards sum to the same total. It does not show equal directions per parameter.
• Normalisation is unspecified and has a side effect. The paper does not say how the weights are kept non-negative and normalised. The released code subtracts the batch minimum before normalising agent.py. That gives zero credit to the lowest step or agent in each batch, which the paper does not mention.
• Warm-up mismatch. The paper says 2000 episodes §6; the code default is 200 train_agent_multi_proc.py. Check which one the reported curves used.
• Claims exceed the experiments. The contribution list mentions "Alice & Bob", "various battle environments in SMACLite" and "Google football" §1. The experimental section evaluates only 5m_vs_6m §5. The results are one figure with no numbers fig. 1.
• Seeds and nondeterminism. The coalition mask uses np.random.shuffle and torch.bernoulli without a local seed TARR_v2.py TARR_v2.py. Runs are reproducible only if the global seeds are fixed.
• Hard-coded tracking key. The training script initialises a comet_ml experiment with a hard-coded project key train_agent_multi_proc.py. Set save_comet_ml_plot to False to run without it train_agent_multi_proc.py.
• Cost. The Shapley-style block runs sample_num attention passes per layer TARR_v2.py. The paper does not report the cost or the value of sample_num.

Lineage

• Builds on: AREL (Xiao et al., 2022), an attention reward-redistribution model §2 §4.5. TAR² adds the agent-attention split and the policy-preservation argument.
• Other redistribution methods compared: IRCR (Gangwani et al., 2020), dense reward and episodic reward baselines §5.1.
• Learners used: IPPO and MAPPO, cited in §5.1.
• Environment: SMACLite, cited in §5.2.
• Superseded by: not known from this wiki.