Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failures

In plain words

When an AI agent that calls tools and takes many steps ends up doing something bad, such as issuing a refund it should not have, developers can see what happened in the log but not which step was the real cause. The step that performs the harmful action is often just carrying out an earlier decision. Asking another language model to read the log and guess tends to be unreliable.

This paper answers the question by experiment. It rewinds the run to one step, lets the agent make that decision again, and lets the rest of the run play out. If the bad outcome becomes less likely when that step is redone, the step mattered. Because the agent is random, each rewind is repeated many times, and the results are reported with confidence intervals rather than as a single run.

The paper also handles cases where two steps only cause the failure together. A Shapley-value calculation shares the blame between them. The method is tested on small made-up agents where the right answer is known in advance, which is how the authors check that the method finds it.

TL;DR

Causal Agent Replay (CAR) attributes an agent failure to a step by intervening on that step: it redraws the step's action from the same policy, re-runs the trajectory forward, and compares outcome distributions with confidence intervals §1 §4. A single-step contrastive estimator finds the causal locus with a "latest step still rescuing the run" rule §4. A budget-bounded Monte-Carlo Shapley estimator splits credit across interacting steps §4. Validation is on synthetic SCMs with planted ground truth; there is no evaluation on real agent traces in this paper §5 §7. The code is open source (Apache-2.0 per pyproject.toml) pyproject.toml.

Problem

Observability tools show what an agent did, and evaluation tools show whether it passed. Neither identifies which step caused the failure §1. The paper argues that the harmful-action step is often not the decisive step, so blaming the step that executed the action is a category error §1. LLM-judge attribution is described as correlational, with about 14% step-level accuracy on the Who&When benchmark as the reported state of the art §1 §6. The method assumes the agent can be replayed faithfully from recorded state, and that an outcome can be scored by a function §3 §7.

Method

Model. A trajectory is a sequence of initial state, (action, observation) pairs, and a scored outcome in [0, 1] eq. 1 §2. Each action is drawn from a stochastic policy given the exact state §2. An intervention is a do-operation on one variable; everything downstream is re-decided by the agent §2.

Intervention algebra. Five operations are defined §2 [code: src/car/replay/intervene.py#L124-L150]:

1. do_resample: redraw the action at step k from the same policy. This is the null intervention and the basis of attribution §2.
2. do_action: force the action at step k intervene.py.
3. do_observation: keep the recorded action at step k and replace its tool result intervene.py.
4. do_context: edit the message history at step k intervene.py.
5. do_policy: swap the model from step k onward intervene.py.

Because the policy is stochastic, K re-runs give a distribution P(outcome | do), not one path §2.

Faithful replay. Replay reconstructs state at step k from the recording §3. The model call is the single nondeterministic input and is recorded. The paper reports an action-match rate for replay rather than assuming reproducibility §3.

Contrastive single-step attribution §4.

# Contrastive attribution, one step at a time  [§4]
# Input:  factual trajectory with n steps, bad outcome label, K samples
# Output: per-step effect with CI, causal locus

def contrastive_attribution(factual, bad_label, K):
    # Baseline is the observed run, scored once  [§4]
    # [code: contrastive.py#L89]
    observed = score(factual)
    per_step = []
    for k in range(n):
        # Hold steps before k at their factual actions; resample k onward  [§4]
        branch = run_forward_K_times(factual, held=range(k), K=K)
        # Effect = P(bad | intervened) minus the observed run  [§4]
        effect = bootstrap_difference_ci(observed, branch, bad_label)
        per_step.append((k, effect))
    # Causal locus = last step whose effect is significant and negative  [§4]
    locus = last(k for (k, e) in per_step if e.significant and e.point < 0)
    return per_step, locus

The paper names the subtlety: resampling step k also redraws every later stochastic step, so an early irrelevant step can show an effect §4. The point-of-commitment rule takes the latest step whose effect confidence interval still excludes zero as the locus §4. The paper uses different interval methods for the proportion and for the difference §4.

Shapley attribution §4. Coalitions are sets of steps held at their factual actions. The value v(S) is P(bad | held = S) over K rollouts. Shares are estimated by permutation sampling with antithetic reverse permutations §4. Each permutation is walked from the empty set to the full set, and a step's marginal is its change in v when that step is added §4 [code: src/car/attribute/shapley.py#L167-L213]. Values are not cached across permutations, so marginals stay independent and the CLT interval is valid §4 shapley.py.

# Monte-Carlo Shapley over steps, antithetic pairs  [§4]
# Input:  factual trajectory with n steps, number of permutations P
# Output: per-step value and confidence interval

def shapley_attribution(factual, P):
    marginals = {k: [] for k in range(n)}
    for p in range(P):
        perm = random_permutation(range(n))
        # Walk forward and backward; average the two marginals  [§4]
        forward = walk(perm)
        backward = walk(reversed(perm))
        for k in range(n):
            marginals[k].append((forward[k] + backward[k]) / 2)
    # Mean and normal-approximation interval per step  [§4]
    return {k: clt_interval(marginals[k]) for k in range(n)}

Pseudocode for walk is the same as the forward pass above, with each step added to the held set in turn, and it evaluates v on every prefix of the permutation shapley.py.

The estimator runs only on demand and stops when a budget is exhausted §4 shapley.py.

Hyperparameters

The paper does not state a number of rollouts K, a Shapley permutation count, or a bootstrap resample count. The values below are code defaults, not paper values.

Results

Author-reported. Setup: synthetic structural causal models with planted ground truth, mocked and deterministic tools, no real LLM traces in this paper §5 §7.

No comparison against LLM-judge or other attribution methods on real traces is reported §6 §7. The related-work discussion positions CAR against AgenTracer (oracle substitution) and Ma et al. (Shapley with causal discovery over static logs) without experiments §6.

Paper vs code

Checked at commit db90fa28b97164c35e7c524e4597b8cbdb3035af of https://github.com/jaineet17/causal-agent-replay, linked from the paper's conclusion §8. The repository is the authors' own.

• Causal locus rule. Paper: the locus is the latest step whose effect confidence interval still excludes zero §4. Code: a step counts only if it is significant and its point estimate is negative, meaning resampling reduces the bad probability contrastive.py contrastive.py. The sign test is not in the paper's wording.
• Interval method. Paper: different interval methods for the proportion and for the difference §4. Code: the attribution path uses only the bootstrap percentile interval on the difference [code: src/car/attribute/effects.py#L115-L136] effects.py. Wilson is defined but not called by the attribution code [code: src/car/attribute/effects.py#L103-L112].
• Baseline. The paper does not say how the baseline is formed. Code: the baseline is the single observed run, so its indicator is 0 or 1 and carries no variance contrastive.py [code: src/car/attribute/effects.py#L226-L230].
• Synthetic validation. The paper reports 0.44, 0.45 and 0.909 §5. The repository tests assert values near 0.455 with tolerance 0.15 per step, and an efficiency sum within 0.2 of 0.91 test_attribution.py. The paper's exact run settings are not stated.
• Licence file. No LICENSE file is in the repository. pyproject.toml and the README both name Apache-2.0 pyproject.toml README.md. The card uses the declared license.
• Demo. The paper's figure is a generated report, not a result table fig. 1. The repository ships a demo report under examples/ [code: examples/demo_report.html].

Searched for: links in the paper, the abstract and the arXiv comments (found the repository in the conclusion). No further search was needed.

Pitfalls

• The baseline is one run. The observed trajectory is scored once, so the comparison is P(bad | intervened) against 0 or 1. Its interval reflects only the intervened samples effects.py. Implementers may want an interval for the observed rate as well.
• Sign convention. A "rescue" is a negative effect with an interval below zero, not just any interval that excludes zero [code: src/car/attribute/contrastive.py#L47-L50]. A step that makes the bad outcome more likely is not counted.
• Total effect, not direct effect. Resampling step k also redraws every later stochastic step, so early irrelevant steps can look important §4. The paper names common random numbers as a possible refinement and does not implement it §7.
• Cost. Contrastive attribution runs about n times K forward rollouts [code: src/car/attribute/contrastive.py#L13]. One Shapley permutation costs two walks of n+1 coalition evaluations each, with K rollouts per evaluation [code: src/car/attribute/shapley.py#L244-L249]. Shapley is exponential in the worst case, and the Monte-Carlo variance depends on budget §7.
• Do not cache coalition values in Shapley. Caching would make per-step marginals collapse to zero variance and show false confidence §4 [code: src/car/attribute/shapley.py#L181].
• Outcome function. Rule-based outcome functions are preferred. Judge-based outcomes add their own noise §7.
• Live environment. Counterfactual suffixes call the environment fresh, so side effects of real tools are not modelled. The paper's demonstrations rely on mocked tools §7 forward.py.
• Replay is not exact for hosted models. Hosted inference varies even at temperature 0, so action-match rate, not reproducibility, is the measure §3. The paper says a seeded local single-stream model replays exactly §3. The card does not verify this claim.

Lineage

• Builds on: Pearl's do-calculus §6; counterfactual credit assignment in RL (Mesnard et al.) §4 §6; COMA (counterfactual multi-agent policy gradients) §4 §6; causal influence diagrams for agents (Everitt et al.; Kenton et al.) §2 §6; Shapley sampling (Castro et al.) §4.
• Compared against (in the paper's discussion, without experiments): Who&When LLM-judge attribution §1 §6; AgenTracer (oracle substitution) §6; Ma et al. (Shapley with causal discovery over logs) §6.
• Superseded by: not known.