Harness Handbook: Making Evolving Agent Harnesses Readable, Navigable, and Editable

In plain words

An AI coding agent is often asked to change how the agent program itself behaves, for example to add a check before the agent finishes a task. Before any edit, someone has to find every place in the code that produces that behavior. In a large agent program that code is spread over many files and functions, so a plain keyword search misses some of it.

The paper proposes a "handbook" for such a program. Instead of organizing the pages by file, it organizes them by what the program does, in stages, and links each behavior to the exact source. A planner reads the handbook from the overview down to the details, then checks each linked spot in the real code before writing an edit plan. When the code changes, the handbook is updated for the changed parts.

The authors tested this on two open-source agent programs and 60 change requests. Plans written with the handbook were judged better more often, and they used fewer planner tokens. The judges were language models, so the scores are the authors' own measurements.

TL;DR

• Builds a behavior-organized "Harness Handbook" from a harness's code: static analysis first, then LLM-written structure and summaries, with links to source §3.2 fig. 2.
• Localizes a requested change with Behavior-Guided Progressive Disclosure (BGPD): navigate from the overview to the relevant stages and entries, follow call edges, then verify each candidate against the current repository §3.3.1 alg. 1.
• Reports better plan-quality win rates and fewer planner tokens than a no-handbook baseline on Codex and Terminus-2, with the largest gains on scattered sites §4.2 tab. 1.

Problem

Before a harness behavior can be changed, every code location that implements it must be found. Repository maps, code search and long-context editing find pieces of code but do not show how those pieces produce a behavior, or whether every affected site has been found §1 §2.3. The authors call this behavior localization and treat it as the central bottleneck in evolving a harness §1.

Method

1. Representation

The handbook has three levels plus a side view §3.1 fig. 1:

• L1 is a system overview: architecture, execution model, main stages and global data flow.
• L2 is a component overview per stage: responsibilities, inputs, outputs, dependencies and local state.
• L3 is a unit deep dive: source-grounded entries, each with a locator into the repository.
• Z, the state-register view, records state that is read or written across stage boundaries §3.1.

Two rules apply. Progressive disclosure: readers go from L1 to L3 only as far as the task needs §3.1. Behavior–implementation alignment: an L3 locator must still resolve to the current repository. If it cannot be revalidated, the entry is frozen and excluded from localization until refreshed §3.1 app. A.3.

2. Construction

The leaf mode g is either function or file. It is fixed for the lifetime of a handbook and sets the granularity of L3 entries: a function (or contiguous regions of one) in function mode, a file in file mode §3.2.

• Function-as-leaf starts from a seed skeleton S0 of stages and state registers. It is used when a trustworthy seed exists and function-level entries fit the budget §3.2 app. A.2.1. Terminus-2 uses this mode §4.1.
• File-as-leaf has no seed. It infers the stage skeleton from file descriptions. It is used when no seed exists or function-level organization would exceed the budget §3.2 app. A.2.2. Codex uses this mode §4.1.

Construction has three phases §3.2 alg. 2 app. A:

• Phase I, static fact extraction. A per-language adapter parses the repository and records each function's identity, location, signature and callers and callees. The graph has an edge only where the callee is an internal function or a named boundary. Any call that cannot be resolved goes to an audit log, with no target guessed for it. This phase makes no LLM calls §3.2 app. A.1.
• Phase II, behavioral organization. Function mode assigns each function, or contiguous regions of it, to one or more stages, then runs review rounds that may add, remove, merge or split stages app. A.2.1. File mode first writes a card per file, then drafts stages and assigns each file to one primary stage, optionally naming one or two secondary stages for cross-cutting files. It then orders files within each stage app. A.2.2. Three variants exist for the stage draft: oneshot (default), doctor and agent app. A.2.2.
• Phase III, synthesis and packaging. Function mode builds top-down from the stage skeleton: L1, then each L2, then the L3 entries of that stage. File mode builds bottom-up: file cards become L3, then stage summaries roll up to L2 and L1 app. A.3. Every L3 locator is resolved against the repository and frozen if it fails. The package keeps the program graph, the skeleton, the organization state and the configuration, so later updates can be incremental app. A.3.

Phase I to III produce the handbook in both modes. Table 2 gives the static facts and resulting size for each harness tab. 2.

3. Handbook-guided modification and resynchronization

For a request q, the workflow is alg. 1 §3.3:

# Handbook-guided modification  [alg. 1]
# Input: request, handbook, repository; output: plan, new repo, new handbook
def modify(request, handbook, repo):
    # The leaf mode stays fixed for the whole run  [alg. 1] [§3.2]
    leaf_mode = leaf_mode_of(handbook)
    # Localize: navigate the handbook, then verify against the repo  [§3.3.1]
    evidence = bgpd(request, handbook, repo)
    # Plan edits and action declarations  [§3.3.2]
    plan, declarations = make_plan(request, evidence)
    # A separate executor applies the plan  [§3.3.2]
    new_repo = execute(repo, plan)
    diff = diff_of(repo, new_repo)
    # Resynchronize only when the diff is non-empty  [§3.3.3]
    if diff:
        new_handbook = resync(handbook, repo, new_repo, diff, declarations)
    else:
        new_handbook = handbook
    return plan, declarations, new_repo, diff, new_handbook

Localization with BGPD §3.3.1 app. B.1:

# Behavior-Guided Progressive Disclosure  [§3.3.1]
# Coarse to fine: stages first, then entries, then source
def bgpd(request, handbook, repo):
    # Pick stages whose behavior matches the request (L1, L2)  [app. B.1]
    stages = select_stages(request, handbook.overview, handbook.stages)
    # Add stages coupled to them through a shared state register (Z)  [§3.3.1]
    stages = add_coupled_stages(stages, handbook.registers)
    # Pick the most relevant L3 entries and take their locators  [§3.3.1]
    candidates = select_entries(request, stages, handbook.entries)
    # Expand along call edges; boundary nodes give context only  [§3.3.1]
    candidates = expand_calls(candidates, handbook.call_graph)
    # Resolve each locator in the current repo; keep only relevant sites  [§3.3.1]
    evidence = []
    for locator in candidates:
        site = resolve(locator, repo)
        if site is not None and is_relevant(site, request):
            evidence.append(site)
    return evidence

Planning and execution. Each edit block names a target file, an optional function or region anchor, the supporting source excerpt and the intended change §3.3.2. Each block also gets an action declaration: modify, add or remove. A rename is one removal plus one addition §3.3.2. In the evaluation implementation, each change is a verbatim old/new text pair app. B.2. The executor can read and write files, make targeted replacements and search within a file. It cannot list directories or use a shell. It applies each replacement directly, re-reads a file only if a replacement fails, and runs a syntax gate when one is configured app. B.3. The diff Δ drives handbook invalidation; the declarations Γ are used only to check whether execution followed the plan §3.3.2.

Resynchronization §3.3.3 app. B.4. Every non-empty diff triggers it. The program graph is rebuilt for the changed source and compared with the stored one. In function mode, functions are matched by body fingerprints that ignore line numbers, so a moved function keeps its locator. In file mode, file-set differences and content hashes identify changes. If the stage skeleton still fits, only the affected entries are refreshed. If it no longer fits, the construction algorithm is rerun with the same leaf mode and configuration. Unparseable or unclassifiable content is frozen or recorded as uncovered, not guessed. Model calls are limited to four kinds: classification, file assignment, within-stage organization and description revision §3.3.3 app. B.4.

4. Handbook size as built (reported, not a result)

Hyperparameters

Most values are not stated in the paper. Rows marked "code default" come from the repository, not the paper, and may differ from what the authors ran.

Results

Author-reported. Setup: Terminus-2 (Python terminal agent, function-as-leaf) and Codex (Rust coding agent, file-as-leaf), 30 requests each. A read-only DeepSeek-V4-Pro planner produces an edit plan in each arm. Baseline explores the repository directly. Handbook-Assisted also uses the handbook. Judges are GPT-5.5, Opus 4.8 and DeepSeek-V4-Pro §4.1 app. C.3.

Plan quality and planner cost §4.2.1 fig. 3:

The Codex gap is 10.0 points for every judge. On Terminus-2 it ranges from 13.3 to 26.7 points across judges §4.2.1.

Gain per dimension, Handbook minus Baseline, averaged over judges §4.2.1 fig. 4:

Localization against reference plans §4.2.2 tab. 1. Values are macro-averages over valid requests, in %. Wrong is the share of requests with zero overlap with the reference; lower is better.

The text reports that all 24 Recall, Precision and F1 comparisons favor Handbook-Assisted, with F1 gains from 5.0 to 18.8 points. Against GPT-5.5 on Terminus-2, Handbook-Assisted reaches 93.3% precision at both granularities §4.2.2.

Breakdown by request type and difficulty §4.2.3 fig. 5: all six harness-by-type comparisons favor Handbook-Assisted, with gains from 16.3 to 33.3 points. Codex's largest type gain is Query (+26.7). Terminus-2's largest is Search-Hostile (+33.3). The two Cross-file gains are 16.3 and 20.0; the text lists them in the order Codex, then Terminus-2. Every harness-by-difficulty comparison is positive, and the spread is 3.7 to 33.3 points. Codex's best level is Easy, and Terminus-2's is Medium. The remaining values appear only in fig. 5.

Paper vs code

Official repository: Ruhan-Wang/Harness_Handbook, linked from the paper's project page. Apache-2.0. Commit d913d8cd6a5305f1070d15db656a530fea28cdf1 (shallow clone). The owner name matches the first author, Ruhan Wang; the project page is the link that ties the repository to the paper.

• Evaluation code is not released. The repository says all eval, scoring and benchmark code was removed README.md. The score weights in eq. 1, the δ = 3 tie rule, the win-rate calculation and the judge prompts in app. D.3 are therefore paper-only. Table 1 and the win rates cannot be reproduced from the repository.
• Planner model differs. The paper's planner is DeepSeek-V4-Pro §4.1. The planner runner defaults to gpt-4o-mini unless OPENAI_MODEL is set code_agent.py. Its default temperature is 0.0 and output cap 32000 tokens code_agent.py. The paper says only that decoding was held equal across arms §4.1.
• Default organization mode matches. The default synthesis mode is oneshot, as the paper says run.py app. A.2.2.
• Budgets are set in code, not in the paper. The paper gives no numeric budget for file assignment. The code's default is 6 rounds synth_stages.py and 25 files per assignment call run.py.
• Reviewer bias matches. The paper says reviewers lean toward approval app. D.1. The shared critic prompt says the same: "Be GENEROUS with APPROVE" critic.py.
• Prompt wording differs. The paper says its prompts were transcribed from the implementation app. D. The classification rules match in substance, but the wording does not. The paper's cross-cutting rule app. D.1 matches pass_a_classify.py, while the other prompt llm_analyze.py says a function that only calls a utility is not cross-cutting. Which version the authors ran is not stated.
• Planner prompt matches. The shipped planner prompt has the same route, read, edit-block and declaration structure as app. D.2 planner_handbook.md.
• Resync matching matches. Function identity after an edit uses body fingerprints resync_handbook.py, as app. B.4 describes.
• Not checkable here. The clone contains no generated handbook, so the counts in tab. 2 cannot be verified. The clone also does not show the evaluation harness, so the reported token counts and win rates cannot be verified.

Pitfalls

• The judging is the main unverifiable part. There is no scoring code, and the judge pipeline is not described in enough detail to rebuild. The judges include DeepSeek-V4-Pro, which is also the planner §4.1. The paper does not discuss self-preference.
• Score scale is ambiguous. eq. 1 puts S on 0–100, but the judges output 0.0–5.0 per dimension app. D.3. The mapping from judge output to S is not stated, so the δ = 3 tie margin has no clear unit.
• Win-rate denominators are not stated. The Codex rates (38.3%, 28.3%) are multiples of 1/60, and the Terminus-2 rates (45.6%, 26.7%) of 1/90. That is consistent with 60 and 90 valid judge–request comparisons, but the paper does not say how many were valid. Check before quoting.
• Terminus-2 symbol-level numbers repeat file-level ones. In tab. 1, Terminus-2's symbol-level Wrong values equal its file-level values for both references. Its GPT-5.5 Handbook symbol values (recall 87.5, precision 93.3, F1 89.3, Wrong 6.7) also equal the file-level values. The paper does not explain this. It may be a copy error, or the two granularities may coincide on this data. Do not treat these rows as independent evidence.
• Reference plans are not ground truth. Localization accuracy is measured against plans written by Opus 4.8 and GPT-5.5 §4.1. Those plans are not checked against the code.
• Setup favours the handbook. The handbook and both arms share one repository snapshot §4.1 app. C.3. The experiments do not test a handbook that has drifted from its code, so the resync procedure is not evaluated.
• Build cost is not counted. The token figures cover planner tokens, including handbook and source context used during localization and planning §4.1. The LLM cost of building the handbook and resyncing it is not reported.
• Coverage can silently shrink. An L3 entry whose locator fails is frozen and excluded from localization until refreshed §3.1 app. A.3. A plan can therefore miss a site without any error. Check frozen entries after each code change.
• Call-graph recall is unknown. Unresolved calls are logged, not guessed app. A.1. Expansion along call edges can miss any site reached only through an unresolved call. The paper does not report how many calls are unresolved in either harness.
• Relevance check is unspecified. BGPD keeps only sites "relevant to q" §3.3.1, but the paper does not say whether relevance is judged by a model or by a rule.
• Function mode relies on a hand-written skeleton. Function-mode classification is described as using a "hand-authored data-flow skeleton" app. D.1. How much work that took is not stated, and it bears on whether the method carries over to a harness without one.
• Some budgets are missing. The organization, generation and register budgets are "stored in Θ" app. A.2.1 app. A.3, and no values are given. Reproducing the construction needs the authors' configuration.
• Only two harnesses, 60 requests. Both harnesses are open source and one is Rust. The results may not hold for other stacks or for requests written differently.

Lineage

• Builds on: no earlier method is named as its base. Its related work on repository representations §2.3 covers approaches that have no card here, so nothing from them is used.
• Compared against: the Baseline arm, which uses the same planner and tools without the handbook §4.1.
• Superseded by: not known from this card's sources.