Credit assignment

Overview

Credit assignment asks which earlier actions, and which agent, deserve the credit or blame for an outcome. With sparse or delayed reward, the learning signal for one agent carries the noise of every other agent's contribution 2412.14779 §3.4. For LLM agents the unit is a whole action, a message with reasoning and a tool call, because the environment only reacts once that message is complete 2604.18401 §3.2. The same question comes up when debugging: a failed run often ends in a harmful action that was only carrying out an earlier decision 2606.08275 §1.

The four papers on this page use the term in different roles. The first turns a total return into dense training rewards 2412.14779 §4.2. The second gives each complete LLM-agent action its own advantage, so that an action is credited or blamed for what it led to 2604.18401 §3.3. The third attributes a failure to a step by intervening on it 2606.08275 §4. The fourth is a survey of the Shapley value, a general rule for splitting a shared total fairly across players 2202.05594 §2.1. The first two are training signals and the third is diagnostic. The third also applies the Shapley rule to interacting steps.

Methods that decompose a team value handle agent credit but not timing, and dense-reward single-agent methods handle timing but not agents 2412.14779 §2.

Variants

Learned reward redistribution (TAR²)

For training. A reward model scores each agent at each step. Scores become two sets of weights: temporal weights over the episode, and agent weights over the team at each step. Each agent's reward is the product of its two weights and the episodic return 2412.14779 §4.2. The model extends the AREL attention reward model with temporal and agent attention blocks, alternated over layers 2412.14779 §4.5. The redistributed reward replaces the sparse one, and any single-agent policy-gradient learner can be trained on it. The paper uses IPPO and MAPPO 2412.14779 §5.1. The paper does not say how the weights are kept non-negative and normalised. The released code subtracts the batch minimum before normalising, which gives the lowest step or agent in each batch no credit; the card's Paper vs code section records this. Card: TAR² (arXiv:2412.14779)

Per-action advantage for LLM agents (CAPO)

For training an LLM agent with a critic. The critic reads the value at the boundary before each complete action, so the state is the interaction history and the action is the whole response 2604.18401 §3.2 §3.3. The reward of an action is intermediate feedback or a sum of token penalties. Generalised advantage estimation runs backwards along the steps, and the single advantage it produces is given to every token of that action 2604.18401 eq. 2. The critic-free alternatives, GRPO-style methods, give every token of a trajectory one shared advantage instead 2604.18401 §1.

In a WebShop ablation with Qwen3-4B, action-level advantages scored 78.12 at γ = 0.99 against 71.86 for token-level advantages, but only 74.83 against 73.74 at γ = 1.00 2604.18401 tab. 3. The gain therefore depends on the discount, and the table reports one setting. The same paper also changes how the token ratios are combined for the policy update. That is a separate mechanism from the advantage and is described on its card [arxiv:2604.18401 §3.4]. The released code does not contain that ratio, and its step-level scripts use a different ratio and a different clip value from the paper (see the card). The critic is still trained, so its cost remains 2604.18401 app. F. Card: CAPO (arXiv:2604.18401)

Intervention-based attribution of agent failures (CAR)

For debugging a recorded run, not for training. The method redraws the action at one step from the same policy, re-runs the rest of the trajectory, and compares the probability of the bad outcome with the observed run, reporting a confidence interval 2606.08275 §2 2606.08275 §4. The step chosen as the cause is the latest one whose effect is significant and negative 2606.08275 §4. A known weakness is that redrawing a step also redraws every later random step, so an early, irrelevant step can appear to matter 2606.08275 §4.

For steps that only cause the failure together, the same paper uses a Monte-Carlo Shapley estimator. A coalition is a set of steps held at their recorded actions, and the value is the bad-outcome probability over repeated rollouts. Permutations are sampled with antithetic reverse pairs, and sampling stops when a budget is spent 2606.08275 §4. Validation is only on synthetic structural causal models with known answers; no real agent traces are evaluated 2606.08275 §5 2606.08275 §7. Card: Causal Agent Replay (arXiv:2606.08275)

Shapley value as a fair-share rule

The general rule. Each player receives its average marginal contribution over all orders in which the players could join 2202.05594 §2.1, eq. 1. Four properties characterise it: null player, efficiency, symmetry and linearity 2202.05594 §2.2. In machine learning the players can be features, training points, federated data owners, models in an ensemble, or agents, and the payoff is typically a model score 2202.05594 §4. Used as credit for agents, the payoff is the global reward a coalition of agents achieves 2202.05594 §4.5.

Exact computation sums over all orderings and grows factorially with the number of players 2202.05594 §5.1.1. The usual remedy is permutation sampling, which averages the marginals of a few random orderings 2202.05594 §3.1 2202.05594 alg. 1. The survey names this approximation as the largest open issue for the fairness axioms: they often fail once the value is approximated 2202.05594 §5.1.3. Card: The Shapley Value in Machine Learning (arXiv:2202.05594)

Other redistribution baselines

The TAR² paper compares its method with AREL (an attention reward-redistribution model), IRCR, an episodic-only reward and a dense temporal reward 2412.14779 §5.1. Those methods have no card here, so this page does not describe them. The paper reports that TAR² does better than these, but only as a figure without numbers 2412.14779 §5.1 fig. 1.

When to use which

The choice depends on whether you are training or diagnosing, and on what you can replay or afford to sample. The rows state only what the cited results support.

Implementation sketch

Sketches for the two training families and for the fair-share rule. The redistribution step follows the TAR² paper's description and released model 2412.14779 §4.2 2412.14779 §4.5. The paper does not specify weight normalisation, so it is left open here.

# Split the episodic team return into per-agent, per-step rewards  [§4.2]
# Input:  observations and actions of N agents over T steps; return R
# Output: credit[i, t] with the sum over agents and steps equal to R

def redistribute(obs, actions, episode_return):
    # Score each (agent, step) pair with the learned reward model  [§4.5]
    scores = reward_model(obs, actions)              # (N, T)
    # Temporal weights: non-negative, summing to 1 over the episode  [§4.2]
    temporal_weights = normalise_over_time(scores)   # (T,)
    # Agent weights: non-negative, summing to 1 over the agents  [§4.2]
    agent_weights = normalise_over_agents(scores)    # (N, T)
    # Credit is the product of the two weights and the return  [§4.2]
    return agent_weights * temporal_weights * episode_return

The per-action sketch follows the paper's advantage equation for one trajectory of T actions 2604.18401 eq. 2. The paper's runs use γ = 0.99 and λ = 1.0 2604.18401 §4.1. The released step-level code normalises these advantages over the batch, which the equation does not include.

# Action-level advantage for LLM-agent steps  [arxiv:2604.18401 eq. 2]
# Input:  values (T + 1,) critic value at each action boundary, last set to 0
#         rewards (T,) reward of each complete action
# Output: advantages (T,), one per action; every token of action t uses it

def action_advantages(values, rewards, gamma, lam):
    advantages = [0.0] * T
    last = 0.0
    for t in reversed(range(T)):
        # TD residual from this action boundary to the next  [eq. 2]
        delta = rewards[t] + gamma * values[t + 1] - values[t]
        # GAE recursion along the step timeline, not the token timeline  [eq. 2]
        last = delta + gamma * lam * last
        advantages[t] = last
    return advantages

The fair-share sketch estimates Shapley values by permutation sampling 2202.05594 alg. 1. Each prefix of an ordering is evaluated once, so each ordering costs one payoff call per player. The survey's algorithm calls the payoff twice per player instead, and does not note this reuse.

# Permutation sampling for Shapley shares  [arxiv:2202.05594 §3.1] [alg. 1]
# Input:  n players, payoff v(coalition) -> float, k orderings
# Output: estimate phi[i] for every player i

def shapley_sample(n, v, k):
    phi = zeros(n)
    for _ in range(k):
        # One uniformly random ordering of the players  [alg. 1]
        order = random_permutation(n)
        # v(empty set) must be given a convention; the ML games leave it open
        # [arxiv:2202.05594 §2.1] [§4]
        coalition = frozenset()
        before = v(coalition)
        for player in order:
            coalition = coalition | {player}
            after = v(coalition)
            # Marginal contribution in this ordering, averaged over k  [eq. 1]
            phi[player] += (after - before) / k
            before = after
    return phi

CAR's Shapley estimator adds antithetic reverse orderings and reports a normal-approximation interval per step 2606.08275 §4. Do not cache coalition values across orderings: the paper reports that caching makes the marginals collapse to zero variance and gives false confidence 2606.08275 §4.

Contradictions

• Single-step effects against joint shares, within CAR. On a synthetic two-step interaction the single-step contrastive effects add up to more than the total effect of 0.91, so they over-count. The Shapley shares sum to 0.909 2606.08275 §5. The paper gives no number for the over-count.
• Policy preservation of TAR². The paper argues that the redistributed reward keeps the optimal policy 2412.14779 §4.3. Its proof sets γ = 1 2412.14779 §4.3, while its experiments use a discount 2412.14779 §3.1. The paper's own setting therefore falls outside the proof.
• Fairness axioms. The survey says the desired properties often fail under approximation 2202.05594 §5.1.3. CAR uses sampled Shapley values but checks them only against analytic answers on synthetic models 2606.08275 §5. Its checks do not test the axioms, so the two sources do not conflict on this point, but CAR does not settle it either.

Papers

• TAR² (arXiv:2412.14779): splits a sparse episodic team return into per-step, per-agent rewards with a learned attention model, and argues that this keeps the optimal policy.
• CAPO (arXiv:2604.18401): gives each complete LLM-agent action one advantage from a critic read at the action boundary, and adds an action-aware token-ratio aggregation; author-reported gains on four agent tasks, and the released code lacks that ratio.
• Causal Agent Replay (arXiv:2606.08275): attributes a failed LLM-agent run to a step by redrawing that step and re-running the rest, with a Monte-Carlo Shapley estimator for interacting steps; validated on synthetic models only.
• The Shapley Value in Machine Learning (arXiv:2202.05594): survey of the Shapley value, its axioms, sampling approximations and machine-learning uses, including multi-agent reinforcement learning; no experiments.