---
kind: concept
slug: agent-trace-replay
title: Agent trace replay
summary: >-
  Re-running a recorded LLM agent execution to debug it or test a counterfactual, then checking that the replay matches the original: printed output, state changes and the final verdict. Trade-off: strict checks reject many replays that look right, while text-and-verdict checks can hide real state differences. Based on one author-reported study of one benchmark version.
tags: [agents, evaluation, reproducibility, debugging]
papers: ["2611.11216"]
related: [llm-agent, tool-use, benchmark-validity]
status: draft
updated_at: 2026-11-21
---

## Overview

A replay re-executes a recorded agent run, with the same code and the recorded
interactions, to see what happens again. Replays serve three uses: debugging a
failed attempt, failure analysis across many attempts and counterfactual
evaluation [arxiv:2611.11216 §1]. The shared problem is that a replay which
prints the same text is not automatically the same execution. Output can match
while the state underneath differs [arxiv:2611.11216 §4.2], and a replay can be
rejected on an output difference even when the state matches
[arxiv:2611.11216 §1].

The invariant the cited study uses is a conjunction. A replay is admitted only
when the printed output, the database changes and the native verdict all match
[arxiv:2611.11216 §2.2]. Each check can also be unknown, for example when an
endpoint was not measured. An unknown is not a difference, and the study keeps
match, difference and unknown counts apart rather than pooling them into a
failure rate [arxiv:2611.11216 §2.2] [tab. 2].

## Variants

The study defines three levels of checking and control. They are not competing
methods so much as points on one scale of strictness.

### Output and verdict only

Compare printed output and the verdict (success flag and test counts). This is
the cheapest check and needs no access to the environment state. In the
study's census it admits 70 of 71 replays for both controls, with one unknown
[arxiv:2611.11216 tab. 2]. Because both controls score the same here, this
rule does not separate them.

### Output, database deltas and verdict (strict admission)

Add a byte comparison of the state changes against the recorded delta files,
checked against immutable snapshots taken before the test oracle ran
[arxiv:2611.11216 §2.2]. Under the declared random seed, 42 of 71 replays pass
this rule. When the random-number generator is also reset before each recorded
interaction, 68 of 71 pass [arxiv:2611.11216 §3.1] [tab. 2]. Card:
[Replay Fidelity in LLM Agent Evaluation](/papers/2611.11216).

The gap between the two checks shows how much a state-only comparison can hide.
Adding the database and oracle check to the verdict rule leaves 4 cases
different and 1 unknown under the declared seed, against 0 different under the
reset control [arxiv:2611.11216 tab. 2].

### Export after the oracle, or from saved trackers

When the state is exported after the test oracle has run, the export can differ
from the pre-oracle state. In the study, a naive post-oracle export lowers
admission from 42 to 35 under the declared seed and from 68 to 58 under the
reset control [arxiv:2611.11216 §7.1]. Exporting from output trackers that were
saved before the oracle runs restores the match for all 10 affected reset-control
cases [arxiv:2611.11216 §7.2] [fig. 3]. The study presents this as a narrow fix
for the export source. It does not undo global ORM rebinding or prove that
execution can safely continue after a later step [arxiv:2611.11216 §7.3].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Only need to know whether the agent got the same answer | Output and verdict | Cheap, but both controls score the same, so it cannot show state drift [arxiv:2611.11216 tab. 2] |
| Replay results will support a claim about agent behaviour | Strict admission, with the three checks reported separately | Output-only agreement can hide different database rows [arxiv:2611.11216 §4.2] |
| The original random-number setup is unknown | Report both the declared-seed and the reset result as a sensitivity range | The reset is a uniform intervention, not a proof of the original setup [arxiv:2611.11216 §9.3] |
| Harness exports state after tests run | Snapshot or retain the trackers before the oracle, and validate after | A naive post-oracle export changed admission for many cases [arxiv:2611.11216 §7.1] |
| Matching tool calls by argument hash | Do not assume the same arguments give the same effect | The same call can change rows depending on later RNG state [arxiv:2611.11216 §5.2] |
| Comparing authentication tokens between runs | Compare token behaviour, not only token value | Equal token values can arise under controlled randomness, and logout blacklists values [arxiv:2611.11216 §6.1] |

Adopt the strict rule when the cost of a false match is high, and state which
checks were run. The study's counts depend on one benchmark version and one
model lineage, so a team on a different stack should measure its own rates
rather than borrow these ones [arxiv:2611.11216 §1.1] [§9.3].

## Implementation sketch

The common core is a three-part comparison with a tri-state result per part.
The two variants differ in which parts are checked and how the random state is
set before each step.

```pseudocode
# Replay one recorded run and admit it only on full agreement  [arxiv:2611.11216 §2.2]
# Input:  recorded run (output, state-change files, verdict), a replay of it
# Output: "match", "different" or "unknown"

def admit_replay(recorded, replayed):
    # Compare each observation; each result is match, different or unknown
    results = [
        compare_text(recorded.output, replayed.output),
        compare_state_files(recorded.deltas, replayed.deltas),
        compare_verdict(recorded.verdict, replayed.verdict),
    ]
    # One observed difference makes the replay different, even if another
    # check could not be measured  [arxiv:2611.11216 §2.2]
    if "different" in results:
        return "different"
    # Otherwise an unmeasured check keeps the result open
    if "unknown" in results:
        return "unknown"
    return "match"
```

```pseudocode
# Replay with a controlled random state  [arxiv:2611.11216 §2.1]
# Input:  recorded interactions, declared seed, reset flag
# Output: replayed observations for the checks above

def run_replay(interactions, declared_seed, reset_each_step):
    # Start from the seed the original run declared  [arxiv:2611.11216 §2.1]
    seed_random(declared_seed)
    for step in interactions:
        # Optional uniform reset before every step; a sensitivity control,
        # not a reconstruction of the original state  [arxiv:2611.11216 §9.3]
        if reset_each_step:
            seed_random(declared_seed)
        execute(step, timeout_seconds=60)
    return observations()
```

Where the variants differ: the output-and-verdict rule drops the state
comparison, the reset flag changes admission counts, and the export step
changes when state is read. Report results per check, with match, different
and unknown counts kept separate.

## Contradictions

None known. Only one cited paper covers this concept, and its results are
author-reported on one historical benchmark version and one model lineage
[arxiv:2611.11216 §9.3]. Independent replications have not been checked.

## Papers

- [Replay Fidelity in LLM Agent Evaluation: Randomness, Hidden State, and Test Oracles in AppWorld](/papers/2611.11216): audits 71 failed runs from one AppWorld corpus, compares output, database and verdict checks under two random-state controls, and proposes an export fix.
