wikillm

Concept · 1 paper · updated 2026-11-21

Agent trace replay

Re-running a recorded LLM agent execution to debug it or test a counterfactual, then checking that the replay matches the original: printed output, state changes and the final verdict. Trade-off: strict checks reject many replays that look right, while text-and-verdict checks can hide real state differences. Based on one author-reported study of one benchmark version.

agentsevaluationreproducibilitydebugging

Markdown4 min read

AI-compiled overview, updated 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

Overview

A replay re-executes a recorded agent run, with the same code and the recorded interactions, to see what happens again. Replays serve three uses: debugging a failed attempt, failure analysis across many attempts and counterfactual evaluation 2611.11216 §1. The shared problem is that a replay which prints the same text is not automatically the same execution. Output can match while the state underneath differs 2611.11216 §4.2, and a replay can be rejected on an output difference even when the state matches 2611.11216 §1.

The invariant the cited study uses is a conjunction. A replay is admitted only when the printed output, the database changes and the native verdict all match 2611.11216 §2.2. Each check can also be unknown, for example when an endpoint was not measured. An unknown is not a difference, and the study keeps match, difference and unknown counts apart rather than pooling them into a failure rate 2611.11216 §2.2 tab. 2.

Variants

The study defines three levels of checking and control. They are not competing methods so much as points on one scale of strictness.

Output and verdict only

Compare printed output and the verdict (success flag and test counts). This is the cheapest check and needs no access to the environment state. In the study's census it admits 70 of 71 replays for both controls, with one unknown 2611.11216 tab. 2. Because both controls score the same here, this rule does not separate them.

Output, database deltas and verdict (strict admission)

Add a byte comparison of the state changes against the recorded delta files, checked against immutable snapshots taken before the test oracle ran 2611.11216 §2.2. Under the declared random seed, 42 of 71 replays pass this rule. When the random-number generator is also reset before each recorded interaction, 68 of 71 pass 2611.11216 §3.1 tab. 2. Card: Replay Fidelity in LLM Agent Evaluation.

The gap between the two checks shows how much a state-only comparison can hide. Adding the database and oracle check to the verdict rule leaves 4 cases different and 1 unknown under the declared seed, against 0 different under the reset control 2611.11216 tab. 2.

Export after the oracle, or from saved trackers

When the state is exported after the test oracle has run, the export can differ from the pre-oracle state. In the study, a naive post-oracle export lowers admission from 42 to 35 under the declared seed and from 68 to 58 under the reset control 2611.11216 §7.1. Exporting from output trackers that were saved before the oracle runs restores the match for all 10 affected reset-control cases 2611.11216 §7.2 fig. 3. The study presents this as a narrow fix for the export source. It does not undo global ORM rebinding or prove that execution can safely continue after a later step 2611.11216 §7.3.

When to use which

SituationUseWhy
Only need to know whether the agent got the same answerOutput and verdictCheap, but both controls score the same, so it cannot show state drift 2611.11216 tab. 2
Replay results will support a claim about agent behaviourStrict admission, with the three checks reported separatelyOutput-only agreement can hide different database rows 2611.11216 §4.2
The original random-number setup is unknownReport both the declared-seed and the reset result as a sensitivity rangeThe reset is a uniform intervention, not a proof of the original setup 2611.11216 §9.3
Harness exports state after tests runSnapshot or retain the trackers before the oracle, and validate afterA naive post-oracle export changed admission for many cases 2611.11216 §7.1
Matching tool calls by argument hashDo not assume the same arguments give the same effectThe same call can change rows depending on later RNG state 2611.11216 §5.2
Comparing authentication tokens between runsCompare token behaviour, not only token valueEqual token values can arise under controlled randomness, and logout blacklists values 2611.11216 §6.1

Adopt the strict rule when the cost of a false match is high, and state which checks were run. The study's counts depend on one benchmark version and one model lineage, so a team on a different stack should measure its own rates rather than borrow these ones 2611.11216 §1.1 §9.3.

Implementation sketch

The common core is a three-part comparison with a tri-state result per part. The two variants differ in which parts are checked and how the random state is set before each step.

Pseudocode
# Replay one recorded run and admit it only on full agreement  [arxiv:2611.11216 §2.2]
# Input:  recorded run (output, state-change files, verdict), a replay of it
# Output: "match", "different" or "unknown"

def admit_replay(recorded, replayed):
    # Compare each observation; each result is match, different or unknown
    results = [
        compare_text(recorded.output, replayed.output),
        compare_state_files(recorded.deltas, replayed.deltas),
        compare_verdict(recorded.verdict, replayed.verdict),
    ]
    # One observed difference makes the replay different, even if another
    # check could not be measured  [arxiv:2611.11216 §2.2]
    if "different" in results:
        return "different"
    # Otherwise an unmeasured check keeps the result open
    if "unknown" in results:
        return "unknown"
    return "match"
Pseudocode
# Replay with a controlled random state  [arxiv:2611.11216 §2.1]
# Input:  recorded interactions, declared seed, reset flag
# Output: replayed observations for the checks above

def run_replay(interactions, declared_seed, reset_each_step):
    # Start from the seed the original run declared  [arxiv:2611.11216 §2.1]
    seed_random(declared_seed)
    for step in interactions:
        # Optional uniform reset before every step; a sensitivity control,
        # not a reconstruction of the original state  [arxiv:2611.11216 §9.3]
        if reset_each_step:
            seed_random(declared_seed)
        execute(step, timeout_seconds=60)
    return observations()

Where the variants differ: the output-and-verdict rule drops the state comparison, the reset flag changes admission counts, and the export step changes when state is read. Report results per check, with match, different and unknown counts kept separate.

Contradictions

None known. Only one cited paper covers this concept, and its results are author-reported on one historical benchmark version and one model lineage 2611.11216 §9.3. Independent replications have not been checked.

Papers

Source and license

Compiled from the papers listed on this page, all published under CC BY, CC BY-SA or CC0, with citations. See each paper's page for its authors and license.