wikillm

Paper · 2026-07-13 · cs.AI, cs.CL, cs.LG, cs.MA

StructAgent: Harness Long-horizon Digital Agents with Unified Causal Structure

StructAgent runs a planner, actor and verifier loop around a shared state of requirements, values and verified evidence. Only verifier decisions change requirement status. Author-reported, on OSWorld-Verified with self-run Qwen3.5-27B, success rises from 31.6% to 62.2%. The released code supports Qwen3.5 only, not the MiniMax-M3 run behind the 78.9% headline.

LLM agentLLM-as-a-judgeAgent memoryagentscomputer-useverificationlong-horizon

arXivPDFCodeMarkdown12 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Computer agents that work for a long time tend to lose track. They reread a growing pile of screenshots and actions, and they can declare a task finished when it is not. StructAgent keeps a small, separate record of the task instead: what must still be true, the useful details found on the way (such as a file path), and the evidence that a step really worked. A planner picks the next piece of work, an actor tries it on the screen, and a checker looks at what happened. Only the checker's verdict can mark progress as done or take it back. Claims made by the planner or the actor are treated as suggestions.

The same record makes it easier to pick up after an interruption, to work out why a step failed, and to choose a different fix. The paper tests this on desktop computer tasks and, as a side experiment, on a Minecraft task set.

TL;DR

StructAgent is a planner, actor and verifier loop built on a shared state of requirements, values and verified evidence. Only verifier decisions can mark a requirement verified or invalidated. The paper reports gains over single-model runs and over other agent frameworks on OSWorld-Verified, all author-reported. The released code supports the Qwen3.5 models only. The 78.9% MiniMax-M3 result cannot be reproduced from it, and several safeguards the paper describes are off by default or behave differently in code.

Problem

Long computer-use tasks pile up observations, failed attempts, partial edits and unsupported completion claims in the raw history §1. An agent that plans from that history has trouble telling what is really complete, verifying it, and recovering when a step fails §1 §2. The paper's claim is that progress has to be represented explicitly and changed only through verification §1.

Assumption: the environment must expose checkable evidence, such as files, shell output, URLs or application state. The paper says the gains are largest where progress can be structured and checked §4.2 §5.

Method

Notation: s_t is the state at turn t. It has three parts: requirements s^req, values s^val and verified evidence s^ver eq. 2.

Unified state

  • Requirements s^req: the conditions the current subgoal must meet and what the verifier should check. Each has status Pending, Verified or Invalidated §3.1.
  • Values s^val: details found during execution that later steps reuse, for example paths, links, chosen items and pulled-out fields §3.1.
  • Evidence s^ver: what the verifier collected while probing, kept so that later turns can audit and reuse it §3.1.

Status update rule eq. 3: a requirement becomes Verified if the verifier decision verifies it, becomes Invalidated if the decision invalidates it, and otherwise keeps its previous status §3.1. When the verifier identifies values, they are written into s^val together with their evidence in s^ver §3.1.

The implementation names these differently. Its unit is an outcome with state pending, verified or reverted (reverted is the implementation's Invalidated). Planner and actor outputs are logged but do not change outcome state app. C tab. 7.

Workflow

One turn is a fixed chain: state, then planner, then actor, then verifier, then update, then the next state eq. 4 §3.2.

  • Planner reads the state and selects the next subgoal g_t §3.2.
  • Actor receives the subgoal and the state, predicts an action trace τ_t and executes it in the environment §3.2.
  • Verifier uses structured and agentic probing of the evidence and outputs a decision d_t that verifies progress, invalidates earlier progress, or leaves the state unchanged §3.2.
  • Memory holds reusable experience that can suggest subgoals, action patterns or checks. It cannot change state §3.2 app. D.
Pseudocode
# One StructAgent turn: plan, act, verify, update  [§3.2] [eq. 4]
def structagent_turn(state, environment):
    # Planner chooses the next subgoal from the state only  [§3.2]
    subgoal = planner(state)
    # Actor proposes and runs the actions for that subgoal  [§3.2]
    action_trace = actor(subgoal, state, environment)
    # Verifier decides what the attempt has proven  [§3.2] [app. C]
    decision = verifier(subgoal, state, action_trace, environment)
    # Only the verifier decision can change requirement status  [§3.1] [eq. 3]
    return update_state(state, decision)
Pseudocode
# Requirement status update from one verifier decision  [§3.1] [eq. 3]
def update_state(state, decision):
    # Check each requirement against the decision  [§3.1] [eq. 3]
    for requirement in state.requirements:
        if decision.verifies(requirement):
            requirement.status = VERIFIED
        elif decision.invalidates(requirement):
            requirement.status = INVALIDATED
    # Store useful values with their supporting evidence  [§3.1]
    state.values.extend(decision.values)
    state.evidence.extend(decision.evidence)
    return state

Control features built on the state

  • Progress checkpointing: the state is a compact snapshot. The agent can resume without replaying the full history §3.3.
  • Failure recovery: the agent diagnoses a stalled state from the verification record and picks a targeted intervention: replanning, retrying, an extra check, repairing the environment, or handing the task to a human. The chosen intervention becomes a new subgoal, and progress still needs a verifier decision §3.3. The release router sends a diagnosis to one of five outcomes: re-plan, redo the actor step, override the verifier, fix the environment, or use a fallback planner app. H.
  • Tool-supported execution: procedures mined from past trajectories are exposed as typed action surfaces, each with the subgoals it supports, its constraints and the evidence that verifies it. These include GUI actions, browser navigation, shell commands, JSON edits, information extraction, app launch, and typed Calc, Writer and Impress actions §3.3 app. C tab. 9.

Verification at runtime (appendix)

  • Milestones. The initializer proposes a short ordered chain of gating milestones. A step counts as a milestone if its regression would force downstream work to be redone app. H.
  • Boundary check. It fires when the actor hands control back to the planner, and again when the planner declares the task finished. Its inputs are the current screenshot, earlier screenshots, filtered accessibility text, the last actor responses, the subgoal and the target outcomes app. C.
  • Probes. The boundary verifier may request one read-only check from seven kinds tab. 8. A determinate probe result overrides the visual judgment. An indeterminate result falls back to the visual judgment app. C.
  • Low-level verifier. A separate check judges one actor turn from before and after screenshots. Its verdicts say whether the subgoal is finished, whether progress is being made, whether the action did nothing, whether the agent went off course, or whether it is blocked. The verdict decides if the actor keeps control, and the check never changes milestone state app. C app. H.
  • DONE gate. A planner DONE claim goes through the verification gate and then a DONE auditor. The auditor returns PASS, FAIL or PARTIAL app. C app. H.
Pseudocode
# Milestone check: visual judgment first, optional probe  [app. C]
def check_milestone(outcome, screens, accessibility, probe):
    # Visual judgment from the current screen is the default  [app. C]
    verdict = observe(outcome, screens, accessibility)
    # A determinate probe result overrides the visual judgment  [app. C]
    if probe is not None and probe.result is not None:
        verdict = probe.result
    # Only an explicit verified verdict counts as progress  [app. H]
    return verdict == VERIFIED

Hyperparameters

The paper gives few numeric settings. Where it is silent the table says so, and the code defaults are listed next to it. Code paths are relative to the repository root, and mm_agents/structagent/ is shortened to structagent/.

NameValueWhereNotes
OSWorld-Verified step budget100§4.1Release script default is also 100 run.sh
Rollouts per task1§4.1Unless stated otherwise
Display1920×1080app. GFixed for controlled runs
BackbonesQwen3.5-9B, Qwen3.5-27B, MiniMax-M3§4.1Release ships Qwen3.5 aliases only, see Paper vs code
Verifier modelnot stated for the controlled rowsapp. GPaper default is the main model. Repo recipe uses Qwen3.5-27B REPRODUCE.md
Sampling, planner, actornot stated in paper—Code defaults: temperature 0.0, top_p 0.9, 1500 max tokens run_structagent.py
Planner screenshotsnot stated in paper—Code default 5 run_structagent.py
Boundary verifier window4 earlier screenshots, last 8 actor responsesapp. CLast 8 actor lines match boundary_verify.py
Low-level verifier window3 earlier screenshots, last 3 actionsapp. CMatches low_level_verify.py and low_level_verify.py
Verifier samplingnot stated in paper—Code: temperature 0.0, top_p 0.9, 700 tokens for low-level low_level_verify.py; 1500 tokens for boundary boundary_verify.py
DONE auditor budgetnot stated in paper—Script default 3 per task run.sh; class default 10 loop.py
DONE auditor samplingnot stated in paper—Code: temperature 0.0, top_p 1.0, 2048 tokens done_auditor.py
Memory embedding and indexall-MiniLM-L6-v2 with FAISSapp. DRetrieval depth default 10 run.sh
Memory on or offnot stated in paperapp. DOff by default in the script run.sh
Replans before judging infeasiblenot stated in paper—8, re-judged every 6 config.py
Actor turns per subgoalnot stated in paper—5 turns, stop after 2 no-effect verdicts config.py
Replans without progress before recoverynot stated in paper—3 loop.py
DONE rejections before LLM fallbacknot stated in paper—5 timeline.py
Verifier-spec rewritesnot stated in paper—Needs 2 strikes and 2 DONE rejections, up to 5 patches timeline.py
Reverts per outcomenot stated in paper—Cap of 3 timeline.py
Mind2Web judgeQwen3.5-27B, temperature 0app. BJudge output cap 400 tokens mind2web_eval.py
Mind2Web step budgetnot stated in paper—Repo recipe uses 50 REPRODUCE.md
Best-of-N (analysis only)N = 5, judge temperature 0app. E.3Test-time analysis, not part of the agent core §5
Minecraft environment128×128 RGB, 20 Hz, 12,000–18,000 stepsapp. F.1Minecraft code is not in this release

Results

Author-reported. Setup: OSWorld-Verified with a 100-step budget and a single rollout §4.1. Self-run rows use the same runner and scoring §4.1.

Controlled comparison, overall success rate (%) tab. 1:

BackboneSingle modelStrongest self-run baselineStructAgentSource
Qwen3.5-9B27.043.3 (OS-Symphony)46.9tab. 1
Qwen3.5-27B31.660.2 (Agent S3)62.2tab. 1
MiniMax-M375.2none self-run78.9tab. 1

Reported rows, not controlled, from the leaderboard and the baselines' own papers: Claude Sonnet 4.6 72.1, Kimi K2.6 73.1, Qwen3.7-Plus 73.3, Agent S3 (Opus/GPT, behavior best-of-N) 72.6, OS-Symphony (GPT-5) 65.8 and VLAA-GUI (Opus 4.5) 76.3 tab. 1.

On Qwen3.5-9B, StructAgent trails the best self-run baseline on Daily, Prof. and Multi-Apps tab. 1. The paper names Office tasks as where the 9B gain is clearest §4.2.

Mind2Web, WebJudge-style judge, success rate (%) tab. 2:

BackboneStructAgentStrongest baselineSource
Qwen3.5-9B47.346.0 (Agent S3)tab. 2
Qwen3.5-27B52.351.7 (Agent S3)tab. 2

Minecraft, GPT-4o as backbone, success rate (%) tab. 3. The Optimus-1 row is published, not rerun. StructAgent averages 76 and Optimus-1 averages 59.

TierStructAgentOptimus-1Source
Wooden10099tab. 3
Stone7092tab. 3
Iron6547tab. 3
Golden869tab. 3
Redstone5725tab. 3

Analysis numbers from the paper §5:

  • Verifier at the DONE gate: over 354 scored Qwen3.5-27B trajectories, the structured verifier raises recall from 0.67 to 0.81 and F1 from 0.70 to 0.76 against a screenshot-only verifier, with precision close to it §5 (Q2) fig. 4.
  • Disagreements: in 100 reviewed cases the structured verifier was correct 63% of the time, the visual-only verifier 31%, and 6% were unclear §5 (Q2).
  • Failure audit: 83 score-zero trajectories from Qwen3.5-27B, capped at 20 per domain. Actor 33%, planner 30%, verifier 30%, environment 7%. The most common single failure is a bad state check §5 (Q4) fig. 6.
  • Judge reliability for Mind2Web: 86.0% agreement with human labels on 150 examples, with a 4.7% false-positive rate and 9.3% false-negative rate tab. 6.

Paper vs code

Repository: github.com/WenyiWU0111/StructAgent, linked from the paper's header. The owner handle matches the first author's name. Cloned at commit bd9452afab6c9c472af8d9a1fff2d7f4c33d499e, license Apache-2.0. Code paths below use the prefix stated under Hyperparameters.

Minecraft: the README links a separate repository, AayushSalvi/StructAgent-Minecraft, whose owner handle matches an author's name. It is not in this release and was not read for this card.

  • MiniMax-M3 backbone. The paper reports 78.9% with MiniMax-M3 tab. 1. The release runs Qwen3.5 only, and its docs say MiniMax-M3 support is "coming soon" MODELS.md REPRODUCE.md. The grounding code has a MiniMax coordinate branch grounding_contract.py, but I found no MiniMax endpoint alias in the model endpoint file.
  • Experience memory. Appendix D describes layered memory with retrieval app. D. The script turns it off by default run.sh. The reproduce doc says all paper capabilities are on by default REPRODUCE.md, which contradicts that default. Memory needs prebuilt indexes, and without them retrieval silently does nothing CONFIG.md.
  • DONE gate scope. The planner prompt says DONE needs every required outcome verified app. H. The code gates only on leaf outcomes, meaning those no other outcome depends on. Intermediate outcomes are not checked. If there are no leaves, it checks all outcomes ledger.py ledger.py.
  • Gate disabled. With the ledger off, the DONE gate accepts without a check; the auditor still runs if enabled orchestrator.py. The script's switch DISABLE_LEDGER defaults to 0 run.sh.
  • Auditor is fail-open. Only an explicit FAIL refutes DONE. PARTIAL, errors and unparseable output keep the accepted DONE loop.py. The paper maps errors to PARTIAL app. H but does not say that PARTIAL keeps the claim. The budget is 3 per task in the script and 10 in the class default. Once the budget is spent, no audit runs loop.py run.sh.
  • Visual or probe fallback. After 5 DONE rejections on an outcome, the deterministic probe is dropped and an LLM judge decides [code: structagent/ledger/core/timeline.py#L156]. The paper says only indeterminate probes fall back to visual judgment app. C.
  • Invalidation guard. Eq. 3 lets any verifier decision invalidate progress. The code blocks an LLM judge from reverting an outcome that a deterministic check verified timeline.py. Reverts stop after 3 timeline.py.
  • LLM-written verifier specs. Verifier specs can be rewritten by an LLM call after repeated disagreement, up to 5 patches, and up to 5 rounds inside a step timeline.py. The paper mentions verifier rewrite logic tab. 7 but not these caps.
  • Verifier model for the controlled rows. Appendix G says the verifier follows the main model unless configured otherwise app. G. The repo recipe runs Qwen3.5-9B as planner with Qwen3.5-27B as verifier REPRODUCE.md. The paper does not say which setup produced the 9B and 27B rows.
  • Matching parts. Boundary and low-level verifier windows match appendix C boundary_verify.py low_level_verify.py. The actor's one retry before falling back to an impossible marker matches app. H decomposer_actor.py.
  • Thresholds the paper omits. The feasibility, burst, stuck and fallback thresholds listed in the Hyperparameters table are code only [code: structagent/config.py#L76-L80] loop.py.

Searched: the paper's header link; the README; scripts/, docs/ and mm_agents/structagent/ for the constants above. No third-party reimplementation was used.

Pitfalls

  • Headline not reproducible from the release. The 78.9% MiniMax-M3 row needs a backbone the code does not ship. A reproduction can only check the Qwen3.5 rows tab. 1 MODELS.md.
  • Turning memory on. Appendix D describes memory as part of the system, but the default run has it off. Enabling it without the banks gives no error and no retrieval CONFIG.md.
  • Small margins on one run. For MiniMax-M3, the gain over the single model is 3.7 points, from one rollout per task. The main tables show no spread or confidence intervals. Minecraft has 7 to 17 tasks per tier, so one task moves a tier by about 14 points (7 tasks) or about 6 points (17 tasks) app. F.2.
  • No component ablation on OSWorld. The paper has no table that switches off the state, the verifier gate, the auditor or failure attribution one at a time. The only isolation is the verifier comparison at the DONE gate §5 (Q2).
  • Hidden LLM calls and cost. Each boundary can trigger a summariser call [code: structagent/ledger/core/summarizer.py#L315] and a verifier call with several images. The code comment puts five frames at about 10k tokens [code: structagent/core/verifier/boundary_verify.py#L432]. The paper reports neither latency nor call counts, so the cost per step is not stated.
  • Mind2Web judge is the same model family. The judge is Qwen3.5-27B. It has a 9.3% false-negative rate on the validation set tab. 6. The step budget for Mind2Web is not stated in the paper; the repo recipe uses 50 [code: docs/REPRODUCE.md#L24].
  • Verifier-backed does not always mean deterministic. After repeated DONE rejections, or when a spec is rewritten, the judging step can be an LLM. Treat "verifier-backed" in an implementation as "verifier-decided", and log which path decided.
  • Unstated values. Sampling settings, planner screenshot count and most runtime thresholds are not in the paper. Use the code defaults listed above and say so in any reproduction.

Lineage

  • Builds on: the OSWorld and OSWorld-Verified benchmarks §4.1. The Minecraft setting follows Optimus-1's task setting, uses Optimus-1's AMEP module as one memory signal, and uses STEVE-1 and JARVIS-1 as actors app. F.1 app. F.3 app. F.4.
  • Compared against: Agent S3, OS-Symphony and VLAA-GUI, reproduced under matched settings §4.1. Optimus-1 numbers in Minecraft are published results tab. 3.
  • Related: tool-use procedures from past trajectories (ToolCUA) app. D.
  • Superseded by: not known from this wiki.

Source and license

Adapted from “StructAgent: Harness Long-horizon Digital Agents with Unified Causal Structure” by Wenyi Wu, Sibo Zhu, Kun Zhou, Aayush Salvi, Zixuan Song, Biwei Huang (arXiv:2607.11388), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/WenyiWU0111/StructAgent at commit bd9452afab6c (Apache-2.0). Any quoted code is shown under that license.