---
kind: concept
slug: credit-assignment
title: Credit assignment
summary: >-
  Deciding which steps, agents or inputs deserve credit or blame for an outcome. Covered: learned reward redistribution that splits an episodic team return into per-step, per-agent rewards; per-action advantages for LLM agents; intervention-based attribution of failed agent runs; and Shapley-style fair shares. Evidence is mostly author-reported, with synthetic-only and survey sources.
tags: [reinforcement-learning, multi-agent, reward-shaping, credit-assignment, attribution, llm-agents]
papers: ["2202.05594", "2412.14779", "2604.18401", "2606.08275"]
related: [multi-agent-reinforcement-learning, centralized-training-decentralized-execution, policy-gradient, proximal-policy-optimization, actor-critic, reward-design, reward-modeling, feature-attribution, causal-inference, counterfactual-explanation, llm-agent, shapley-value]
status: draft
updated_at: 2026-11-01
---

## Overview

Credit assignment asks which earlier actions, and which agent, deserve the
credit or blame for an outcome. With sparse or delayed reward, the learning
signal for one agent carries the noise of every other agent's contribution
[arxiv:2412.14779 §3.4]. For LLM agents the unit is a whole action, a message
with reasoning and a tool call, because the environment only reacts once that
message is complete [arxiv:2604.18401 §3.2]. The same question comes up when
debugging: a failed run often ends in a harmful action that was only carrying
out an earlier decision [arxiv:2606.08275 §1].

The four papers on this page use the term in different roles. The first turns
a total return into dense training rewards [arxiv:2412.14779 §4.2]. The second
gives each complete LLM-agent action its own advantage, so that an action is
credited or blamed for what it led to [arxiv:2604.18401 §3.3]. The third
attributes a failure to a step by intervening on it [arxiv:2606.08275 §4]. The
fourth is a survey of the Shapley value, a general rule for splitting a shared
total fairly across players [arxiv:2202.05594 §2.1]. The first two are training
signals and the third is diagnostic. The third also applies the Shapley rule to
interacting steps.

Methods that decompose a team value handle agent credit but not timing, and
dense-reward single-agent methods handle timing but not agents
[arxiv:2412.14779 §2].

## Variants

### Learned reward redistribution (TAR²)

For training. A reward model scores each agent at each step. Scores become two
sets of weights: temporal weights over the episode, and agent weights over the
team at each step. Each agent's reward is the product of its two weights and
the episodic return [arxiv:2412.14779 §4.2]. The model extends the AREL
attention reward model with temporal and agent attention blocks, alternated
over layers [arxiv:2412.14779 §4.5]. The redistributed reward replaces the
sparse one, and any single-agent policy-gradient learner can be trained on it.
The paper uses IPPO and MAPPO [arxiv:2412.14779 §5.1]. The paper does not say
how the weights are kept non-negative and normalised. The released code
subtracts the batch minimum before normalising, which gives the lowest step or
agent in each batch no credit; the card's Paper vs code section records this.
Card: [TAR² (arXiv:2412.14779)](/papers/2412.14779)

### Per-action advantage for LLM agents (CAPO)

For training an LLM agent with a critic. The critic reads the value at the
boundary before each complete action, so the state is the interaction history
and the action is the whole response [arxiv:2604.18401 §3.2] [§3.3]. The reward
of an action is intermediate feedback or a sum of token penalties. Generalised
advantage estimation runs backwards along the steps, and the single advantage it
produces is given to every token of that action [arxiv:2604.18401 eq. 2]. The
critic-free alternatives, GRPO-style methods, give every token of a trajectory
one shared advantage instead [arxiv:2604.18401 §1].

In a WebShop ablation with Qwen3-4B, action-level advantages scored 78.12 at
γ = 0.99 against 71.86 for token-level advantages, but only 74.83 against 73.74
at γ = 1.00 [arxiv:2604.18401 tab. 3]. The gain therefore depends on the
discount, and the table reports one setting. The same paper also changes how
the token ratios are combined for the policy update. That is a separate
mechanism from the advantage and is described on its card [arxiv:2604.18401
§3.4]. The released code does not contain that ratio, and its step-level
scripts use a different ratio and a different clip value from the paper
(see the card). The critic is still trained, so its cost remains
[arxiv:2604.18401 app. F]. Card:
[CAPO (arXiv:2604.18401)](/papers/2604.18401)

### Intervention-based attribution of agent failures (CAR)

For debugging a recorded run, not for training. The method redraws the action
at one step from the same policy, re-runs the rest of the trajectory, and
compares the probability of the bad outcome with the observed run, reporting
a confidence interval [arxiv:2606.08275 §2] [arxiv:2606.08275 §4]. The step
chosen as the cause is the latest one whose effect is significant and negative
[arxiv:2606.08275 §4]. A known weakness is that redrawing a step also redraws
every later random step, so an early, irrelevant step can appear to matter
[arxiv:2606.08275 §4].

For steps that only cause the failure together, the same paper uses a
Monte-Carlo Shapley estimator. A coalition is a set of steps held at their
recorded actions, and the value is the bad-outcome probability over repeated
rollouts. Permutations are sampled with antithetic reverse pairs, and sampling
stops when a budget is spent [arxiv:2606.08275 §4]. Validation is only on
synthetic structural causal models with known answers; no real agent traces
are evaluated [arxiv:2606.08275 §5] [arxiv:2606.08275 §7]. Card:
[Causal Agent Replay (arXiv:2606.08275)](/papers/2606.08275)

### Shapley value as a fair-share rule

The general rule. Each player receives its average marginal contribution over
all orders in which the players could join [arxiv:2202.05594 §2.1, eq. 1].
Four properties characterise it: null player, efficiency, symmetry and
linearity [arxiv:2202.05594 §2.2]. In machine learning the players can be
features, training points, federated data owners, models in an ensemble, or
agents, and the payoff is typically a model score [arxiv:2202.05594 §4].
Used as credit for agents, the payoff is the global reward a coalition of
agents achieves [arxiv:2202.05594 §4.5].

Exact computation sums over all orderings and grows factorially with the
number of players [arxiv:2202.05594 §5.1.1]. The usual remedy is permutation
sampling, which averages the marginals of a few random orderings
[arxiv:2202.05594 §3.1] [arxiv:2202.05594 alg. 1]. The survey names this
approximation as the largest open issue for the fairness axioms: they often
fail once the value is approximated [arxiv:2202.05594 §5.1.3]. Card:
[The Shapley Value in Machine Learning (arXiv:2202.05594)](/papers/2202.05594)

### Other redistribution baselines

The TAR² paper compares its method with AREL (an attention reward-redistribution
model), IRCR, an episodic-only reward and a dense temporal reward
[arxiv:2412.14779 §5.1]. Those methods have no card here, so this page does not
describe them. The paper reports that TAR² does better than these, but only as
a figure without numbers [arxiv:2412.14779 §5.1] [fig. 1].

## When to use which

The choice depends on whether you are training or diagnosing, and on what you
can replay or afford to sample. The rows state only what the cited results
support.

| Situation | Use | Why |
|---|---|---|
| Team reward arrives only at episode end, and the learner is on-policy (IPPO or MAPPO) | Learned redistribution (TAR²) | The paper reports faster learning than episodic-only and other redistribution baselines on one scenario, in a figure [arxiv:2412.14779 §5.1] [fig. 1] |
| Discounted returns (γ < 1) | Check the policy-preservation claim before relying on it | The equivalence argument is given for γ = 1 [arxiv:2412.14779 §4.3]. The paper's own setting uses a discount [arxiv:2412.14779 §3.1] |
| Enough data to warm up the reward model | Use a warm-up period before redistributing | The paper uses 2000 warm-up episodes and reports that warm-up beat none for on-policy learners [arxiv:2412.14779 §6] |
| Reward must depend only on the current transition | Not a direct fit | The paper's per-step reward depends on the joint history of the episode [arxiv:2412.14779 §4.1] |
| Need evidence beyond one scenario | Not yet available | Only 5m_vs_6m in SMACLite is evaluated [arxiv:2412.14779 §5] [fig. 1] |
| Multi-turn LLM agent with a critic affordable, and per-action credit wanted | Per-action advantage (CAPO) | Author-reported gains over PPO, GRPO, GiGPO and ST-PPO on four agent tasks with two Qwen3 backbones, mean over 3 seeds [arxiv:2604.18401 tab. 1] [app. E] |
| LLM agent with no room for a critic | Critic-free baselines (GRPO-style) | They give one trajectory-level advantage and need no critic [arxiv:2604.18401 §1]; the paper's own method keeps a critic [arxiv:2604.18401 app. F] |
| Per-action advantage with the paper's exact update | Implement the ratio yourself | The released repository does not contain it, so a reproduction needs new code [arxiv:2604.18401 §3.4]; see the card |
| A failed LLM-agent run, with the ability to replay steps and a function that scores the outcome | Single-step intervention (CAR) | The method needs replay from recorded state and a scoring function [arxiv:2606.08275 §3] [arxiv:2606.08275 §7] |
| Hosted model, where replay is not exact | Expect noise; report the action-match rate | The paper measures replay fidelity by action match rather than assuming exact reproducibility [arxiv:2606.08275 §3] |
| Failure needs two or more steps together | Monte-Carlo Shapley over steps (CAR) | On a synthetic AND interaction the paper reports 0.44 and 0.45 for the two steps, against 0.5 each analytically [arxiv:2606.08275 §5] |
| Cheap first pass on a single cause | Single-step contrastive test (CAR) | Its effects are total effects, not direct ones, and can over-count [arxiv:2606.08275 §4] [arxiv:2606.08275 §5] |
| Fair shares with the axioms, few players | Exact Shapley | The exact sum is factorial in the number of players [arxiv:2202.05594 §5.1.1] |
| Many players, shares only needed approximately | Permutation sampling | Its error depends on the sample count, and the survey notes that the axioms may fail under approximation [arxiv:2202.05594 §3.1] [arxiv:2202.05594 §5.1.3] |

## Implementation sketch

Sketches for the two training families and for the fair-share rule. The
redistribution step follows the TAR² paper's description and released model
[arxiv:2412.14779 §4.2] [arxiv:2412.14779 §4.5]. The paper does not specify
weight normalisation, so it is left open here.

```pseudocode
# Split the episodic team return into per-agent, per-step rewards  [§4.2]
# Input:  observations and actions of N agents over T steps; return R
# Output: credit[i, t] with the sum over agents and steps equal to R

def redistribute(obs, actions, episode_return):
    # Score each (agent, step) pair with the learned reward model  [§4.5]
    scores = reward_model(obs, actions)              # (N, T)
    # Temporal weights: non-negative, summing to 1 over the episode  [§4.2]
    temporal_weights = normalise_over_time(scores)   # (T,)
    # Agent weights: non-negative, summing to 1 over the agents  [§4.2]
    agent_weights = normalise_over_agents(scores)    # (N, T)
    # Credit is the product of the two weights and the return  [§4.2]
    return agent_weights * temporal_weights * episode_return
```

The per-action sketch follows the paper's advantage equation for one
trajectory of T actions [arxiv:2604.18401 eq. 2]. The paper's runs use
γ = 0.99 and λ = 1.0 [arxiv:2604.18401 §4.1]. The released step-level code
normalises these advantages over the batch, which the equation does not include.

```pseudocode
# Action-level advantage for LLM-agent steps  [arxiv:2604.18401 eq. 2]
# Input:  values (T + 1,) critic value at each action boundary, last set to 0
#         rewards (T,) reward of each complete action
# Output: advantages (T,), one per action; every token of action t uses it

def action_advantages(values, rewards, gamma, lam):
    advantages = [0.0] * T
    last = 0.0
    for t in reversed(range(T)):
        # TD residual from this action boundary to the next  [eq. 2]
        delta = rewards[t] + gamma * values[t + 1] - values[t]
        # GAE recursion along the step timeline, not the token timeline  [eq. 2]
        last = delta + gamma * lam * last
        advantages[t] = last
    return advantages
```

The fair-share sketch estimates Shapley values by permutation sampling
[arxiv:2202.05594 alg. 1]. Each prefix of an ordering is evaluated once, so
each ordering costs one payoff call per player. The survey's algorithm calls
the payoff twice per player instead, and does not note this reuse.

```pseudocode
# Permutation sampling for Shapley shares  [arxiv:2202.05594 §3.1] [alg. 1]
# Input:  n players, payoff v(coalition) -> float, k orderings
# Output: estimate phi[i] for every player i

def shapley_sample(n, v, k):
    phi = zeros(n)
    for _ in range(k):
        # One uniformly random ordering of the players  [alg. 1]
        order = random_permutation(n)
        # v(empty set) must be given a convention; the ML games leave it open
        # [arxiv:2202.05594 §2.1] [§4]
        coalition = frozenset()
        before = v(coalition)
        for player in order:
            coalition = coalition | {player}
            after = v(coalition)
            # Marginal contribution in this ordering, averaged over k  [eq. 1]
            phi[player] += (after - before) / k
            before = after
    return phi
```

CAR's Shapley estimator adds antithetic reverse orderings and reports a
normal-approximation interval per step [arxiv:2606.08275 §4]. Do not cache
coalition values across orderings: the paper reports that caching makes the
marginals collapse to zero variance and gives false confidence
[arxiv:2606.08275 §4].

## Contradictions

- **Single-step effects against joint shares, within CAR.** On a synthetic two-step
  interaction the single-step contrastive effects add up to more than the
  total effect of 0.91, so they over-count. The Shapley shares sum to 0.909
  [arxiv:2606.08275 §5]. The paper gives no number for the over-count.
- **Policy preservation of TAR².** The paper argues that the redistributed
  reward keeps the optimal policy [arxiv:2412.14779 §4.3]. Its proof sets
  γ = 1 [arxiv:2412.14779 §4.3], while its experiments use a discount
  [arxiv:2412.14779 §3.1]. The paper's own setting therefore falls outside the
  proof.
- **Fairness axioms.** The survey says the desired properties often fail under
  approximation [arxiv:2202.05594 §5.1.3]. CAR uses sampled Shapley values but
  checks them only against analytic answers on synthetic models
  [arxiv:2606.08275 §5]. Its checks do not test the axioms, so the two sources
  do not conflict on this point, but CAR does not settle it either.

## Papers

- [TAR² (arXiv:2412.14779)](/papers/2412.14779): splits a sparse episodic team return into per-step, per-agent rewards with a learned attention model, and argues that this keeps the optimal policy.
- [CAPO (arXiv:2604.18401)](/papers/2604.18401): gives each complete LLM-agent action one advantage from a critic read at the action boundary, and adds an action-aware token-ratio aggregation; author-reported gains on four agent tasks, and the released code lacks that ratio.
- [Causal Agent Replay (arXiv:2606.08275)](/papers/2606.08275): attributes a failed LLM-agent run to a step by redrawing that step and re-running the rest, with a Monte-Carlo Shapley estimator for interacting steps; validated on synthetic models only.
- [The Shapley Value in Machine Learning (arXiv:2202.05594)](/papers/2202.05594): survey of the Shapley value, its axioms, sampling approximations and machine-learning uses, including multi-agent reinforcement learning; no experiments.