wikillm

Paper · 2026-10-16 · cs.SE, cs.AI

VERAS: Verification-Guided Adaptive Search for Reliable Operations Research

VERAS wraps a coding agent in a verify-then-revise loop: structural checks, per-task observers and a fresh-context challenger supply evidence, and a revision replaces the incumbent only if it does not regress. Author-reported: 42.99% pass rate on ORAgentBench (46 of 107) versus 35.51% for Codex with GPT-5.4; 10.34% lower peak bandwidth than native Codex on one archived run.

LLM agentSelf-refinement (draft, critique, rewrite loop)LLM self-evaluationProgram synthesisagentsverificationoperations-researchoptimizationcode-generation

arXivPDFCodeMarkdown14 min read

AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

An AI coding agent can solve an operations research problem, such as scheduling staff, routing trucks or planning a power grid, by writing a program that outputs a plan. The trouble is that the agent often checks its own plan with checks it wrote itself. If it misread a rule, the program and its checks miss the rule in the same way, and the plan looks correct when it breaks a constraint.

VERAS adds a second layer of checking. It runs basic validity checks on the plan, runs specific checkers where they exist, and asks a separate reviewer agent, which starts with a clean context, to look for rules the checks did not cover. The reviewer's findings then guide the next revision: first repair any broken constraint, then review coverage, then improve the objective. A new version of the plan replaces the old one only if it does not break what was already working.

On a benchmark of 107 such problems, the authors report a higher pass rate than a strong coding agent using the same language model. On two industrial problems, the reported plans beat the native agent and, in one case, a hand-built model, but the paper itself shows that these comparisons are retrospective and rest on single selected runs.

TL;DR

  • VERAS is a harness around a general-purpose coding agent. It checks each candidate with structural checks, task-specific observers and a fresh-context challenger, then revises and admits only non-regressing candidates §3.1.
  • Author-reported ORAgentBench result: 46 of 107 tasks pass (42.99%) against 38 (35.51%) for Codex with GPT-5.4 under the same official criteria. The largest gain is on Hard tasks, 12 of 34 against 7 of 34 §5.1 tab. 1.
  • Industrial results are single selected runs: a 10.34% lower peak bandwidth than native Codex, and a 6.28% lower annualised cost than an expert hourly model. The paper's own follow-up runs weaken both claims §5.3 §5.4 app. A.4.2 app. A.4.3.
  • Code is public at holimario/VERAS without a license file, so this card gives no code snippets. The release holds the controller, observers and case profiles, but not the benchmark tasks or the industrial inputs.

Problem

An LLM agent that writes an optimisation program and then checks its output with its own tests can pass those tests and still violate the task. The authors call this a correlated specification error: the code and the checks written for it both get one requirement wrong in the same way. They also name two further gaps: self-written verification often covers only part of the requirements, and feedback that says only "failed" gives too little to act on. A third gap is that a revision meant to fix one thing can break something that already worked §1 fig. 1.

Method

Problem and state

An OR instance I holds a task description and instance data. The problem is to minimise an objective f(x; I) over decision variables x, subject to constraint predicates c_j(x; I) = 1 for each j in a constraint set §3.1 eq. 1. At iteration t the incumbent state holds the program kept so far (P_t), the artifact it exports (a_t) and the evidence gathered on that artifact (E_t) §3.1.

The coding agent writes an executable program, runs it, and exports a decision artifact. VERAS wraps this producer with three mechanisms §3.1 fig. 2:

  1. Task-grounded verification produces evidence E_t = Verify(I, a_t) with three parts: whether the artifact is well formed, which constraints it demonstrably breaks, and how much of the requirement set was checked §3.2 eq. 2.
  2. Evidence-guided refinement chooses the next action from that evidence §3.3.
  3. Verification-filtered update decides whether a candidate replaces the incumbent §3.4.

Task-grounded verification §3.2

  • Structural review. Checks that the required files exist, that decision fields and data types are right, and that records are complete. Defects are reported with the affected fields or records.
  • Deterministic observer. Evaluates the candidate against the constraints it can reconstruct from the task and instance data. It reports witnessed violations with numerical residuals where they apply, and records its verification scope, the constraint families it covers. This record is what separates "no violation found" from "not checked" §3.2.
  • Isolated challenger. A separate agent in a fresh context reads the task, the instance data and the candidate, and searches for counterexamples to requirements the observers did not cover. Its findings add evidence but do not change the program §3.2 app. A.1.

Evidence-guided refinement §3.3

A controller runs three stages in a fixed order, restore, audit, improve. Each stage runs only if its activation condition holds for the current evidence and the remaining budget eq. 3.

  • Feasibility restoration. Runs when the artifact is structurally invalid or a witnessed violation exists. The producer receives the violated requirements and the available residuals and revises the formulation, heuristic or program. The revised program is executed and re-verified before admission §3.3.
  • Coverage-aware auditing. Runs for a viable candidate whose verification coverage is incomplete. The challenger's report goes to the producer as evidence. The auditing step does not edit the program §3.3.
  • Performance improvement. Runs for a viable candidate while budget remains. The producer uses the accumulated feedback to improve the original objective, and each revision is executed and verified before admission §3.3.

Verification-filtered update §3.4

Each proposed program is executed and verified to give a candidate artifact and evidence. Acceptance is a 0/1 decision that depends on the evidence before and after, and on the current phase eq. 4. The incumbent is replaced only on acceptance; otherwise the old program, artifact and evidence stay eq. 5. Restoration accepts incremental feasibility gains, and performance updates also need a valid artifact with no detected violation under the supported checks §3.4.

Pseudocode
# VERAS: one revision round for one task  [§3.1] [§3.3]
# Input:  task, incumbent (program, artifact, evidence), remaining budget
# Output: incumbent for the next round

def veras_round(task, incumbent, budget):
    # Structural review and deterministic observer  [§3.2]
    evidence = verify(task, incumbent.artifact)
    findings = None
    # Restore if the artifact is invalid or a violation exists  [§3.3]
    if needs_restoration(evidence):
        candidate = producer_fix(task, incumbent, evidence)
        incumbent = admit(incumbent, candidate, performance=False)
    # Audit a viable candidate with incomplete coverage  [§3.3]
    if is_viable(evidence) and not evidence.coverage_complete:
        findings = challenger(task, incumbent.artifact, evidence)
        # The challenger only observes, so the artifact is restored  [§3.3]
        incumbent = restore_snapshot(incumbent)
        evidence = verify(task, incumbent.artifact)
    # Improve the objective from a viable candidate  [§3.3]
    if is_viable(evidence):
        candidate = producer_improve(task, incumbent, evidence, findings)
        incumbent = admit(incumbent, candidate, performance=True)
    return incumbent
Pseudocode
# Candidate admission  [§3.4]
# Input:  incumbent, candidate, phase flag (performance or restoration)
# Output: the candidate if admitted, otherwise the incumbent

def admit(incumbent, candidate, performance):
    # Reject any regression in structure: validity, files, syntax  [§3.4]
    if structural_regression(incumbent, candidate):
        return incumbent
    # Reject a worse residual tuple on the supported checks  [§3.4]
    if residual_regression(incumbent, candidate):
        return incumbent
    # Performance steps also need a viable candidate  [§3.4]
    if performance and not is_viable(candidate):
        return incumbent
    return candidate

Note on the pseudocode: the admit and veras_round blocks are a condensed reading of the paper's Section 3 and of the controller code. They are not a verbatim transcription; see Paper vs code for where the code adds steps.

Hyperparameters

Benchmark values come from the paper's appendix table, and from the code where the paper is silent. The case-study values differ, as listed in the second table.

NameValueWhereNotes
producer model / reasoningGPT-5.4 / highapp. A.3 tab. 5the release leaves the model user-supplied CONFIGURATION.json
concurrent task workers2app. A.3 tab. 5the benchmark profile field reads 1 with a comment that the launcher uses two profiles.py
time limit per stage2,700 s§4.1 tab. 5
timeout scale1.00tab. 5 profiles.py
reserved recovery fraction0.30§4.1 tab. 5 CONFIGURATION.jsonthe code caps the reserve at total minus 60 s; not stated in the paper controller.py
minimum recovery allocation120 s§4.1 tab. 5 constants.py
challenger fraction of remaining budget0.22§4.1 constants.pycode also has a 45 s floor and a cap; see Paper vs code
challenger cap180 sprofiles.pynot in the paper's benchmark table; code default
minimum intervention time45 sconstants.pynot stated in the paper
recovery budget share0.72 of remaining, minimum 45 scontroller.pynot stated in the paper
harness-side local solver cap180 stab. 5 cli.pyused only when the stage names a solver script
transient transport attempts4tab. 5 cli.py
candidate SCIP call ceiling300 s§4.1 tab. 5not checkable here: the solver and benchmark live outside this release
target SCIP relative gap5 × 10⁻⁴§4.1 tab. 5not checkable here, as above
pass thresholdfeasibility > 0 and normalised quality > 0.4§4.1 eq. 6 scores.py

Case-study profiles, which replace the benchmark values tab. 6:

NameDatacenter placement54-bus network planningWhere
model / reasoningGPT-5.6 Sol, xhigh effortsame model and efforttab. 6 profiles.py profiles.py
agent wall-time limit19,800 s15,360 stab. 6
timeout scale3.02.0tab. 6 profiles.py profiles.py
recovery reserve fraction0.400.40tab. 6 profiles.py profiles.py
challenger cap180 s480 stab. 6 profiles.py profiles.py
candidate execution contract120-s output generator, one replay900-s solver, at most four threads§4.2 §4.3 tab. 6
replay solver limitnot stated (120-s output limit applies)750 s for source replayapp. A.4.2 app. A.4.3
input the agent can seetransfer records from 8–9 August 2025one fixed 54-bus instance§4.2 tab. 6
how results are scoredreplay on 10 August 2025independent constraint and objective audit§4.2 §4.3

Results

Author-reported. Setup: ORAgentBench, 107 tasks (32 Easy, 41 Medium, 34 Hard), GPT-5.4 with high reasoning effort, under the benchmark's official scoring. The Codex and Claude Code rows are published results, not reruns; the paper says they do not control realised inference compute §4.1 tab. 1. The feasibility and quality columns are task-count-weighted means across difficulty splits tab. 1. The pass rate is the share of tasks with feasibility above 0 and normalised quality above 0.4 §4.1 eq. 6.

HarnessModelFeas. overallQuality overallPass Easy (%)Pass Medium (%)Pass Hard (%)Pass overall (%)
CodexGPT-5.40.570.3246.8839.0220.5935.51
CodexGPT-5.3-Codex0.510.2931.2543.9020.5932.71
CodexGPT-5.4-mini0.460.2443.7529.275.8826.17
Claude CodeClaude Opus 4.60.490.3150.0039.0214.7134.58
Claude CodeDeepSeek V4 Pro0.410.2539.5828.4613.7327.10
Claude CodeKimi K2.60.430.2231.2536.5911.7627.10
Claude CodeGLM 5.10.380.2537.5031.718.8226.17
Claude CodeDeepSeek V4 Flash0.390.2446.8826.835.8826.17
Claude CodeClaude Sonnet 4.60.420.2331.2529.2714.7125.23
VERASGPT-5.40.670.3956.2539.0235.2942.99

tab. 1. VERAS passes 46 of 107 tasks; the GPT-5.4 Codex baseline passes 38. Feasibility for VERAS is 72 of 107 tasks CONFIGURATION.json. Four tasks did not complete all required stages, one Medium and three Hard CONFIGURATION.json.

Ablation §5.2 fig. 3. Six Hard tasks, GPT-5.4, one worker per variant, and separate runs for each variant. The VERAS row is the frozen original result. The text gives the following values; the feasibility count for the no-challenger variant is not stated in the text.

VariantFeasible (/6)Passed (/6)Mean normalised quality
VERAS (frozen original)550.621
No isolated challengernot stated00.054
No deterministic observer430.466
No feasibility-restoration routing220.237
No verification-filtered admission520.409

Inter-datacenter placement (625,191 tasks, August 2025 traces, 120-s output limit) §4.2 §5.3 tab. 2. The paper gives these as author-reported values from an archived VERAS run and a later native Codex run.

MethodPeak (Gbps) ↓Migration (PB) ↓
Operational mapping19,495.1670.000
Historical-majority heuristic18,276.0710.333
Native Codex1,238.62525.473
VERAS1,110.60325.901

A second VERAS run with the same performance instructions gives 1,567.303 Gbps, which reverses the ranking against native Codex app. A.4.2.

54-bus active distribution network planning (MISOCP, 4 scenarios, 24 periods) §4.3 §5.4 tab. 3. Costs in million USD per year.

MethodTotal ↓Investment ↓Operation ↓
Hourly MISOCP (expert)42.06518.98423.081
Varying-timestep MISOCP (expert)42.92319.01223.911
Native Codex45.69018.49027.200
VERAS39.42416.15423.269

The VERAS arm executed alongside native Codex gives 42.613 million USD, a separate selected VERAS run gives 39.424 million USD, and the selected run's best bound is 37.404 million USD with a 5.122% solver gap app. A.4.3.

Run-level mechanism counts, reported for the 117-stage benchmark: 15 structural restorations, three residual reductions, 103 tasks with a challenger run, and 114 of 132 performance revisions admitted §5.1. The release file that summarises these counts agrees with the 15 restorations, the three residual reductions and 103 challenger tasks component-evidence-summary.json.

Paper vs code

Official code: github.com/holimario/VERAS, which the abstract links. The checked commit is 39f1525 (full hash in the frontmatter). The code was read, not copied. The repository has no LICENSE file. Its NOTICE.md says no license is granted for third-party benchmark material or operational datasets, and that an explicit source license will follow separately. No snippets are quoted here.

Matches:

  • Benchmark defaults: high reasoning, timeout scale 1.0, reserve fraction 0.30, challenger fraction 0.22, minimum recovery 120 s CONFIGURATION.json constants.py.
  • Local solver cap 180 s and four transient transport attempts cli.py.
  • Case-study profiles: GPT-5.6 Sol at xhigh, timeout scales 3.0 and 2.0, reserve 0.40, challenger caps 180 s and 480 s profiles.py.
  • Pass rule: feasibility above 0 and normalised quality above 0.4 scores.py.
  • Headline counts (46 passed, 72 feasible, 103 completed of 107, quality 0.392) CONFIGURATION.json.
  • The challenger runs in a fresh context controller.py, and its candidate is restored after observation controller.py.

Differs or is unclear:

  • Extra stage not in the paper. The controller runs an existing solver script named by the stage before the producer's restoration, when the artifact is invalid and such a script exists controller.py audits.py. Eq. 3 lists three stages only §3.3 eq. 3.
  • Challenger budget. The paper says 0.22 of the remaining budget. The code takes the smaller of the cap and the larger of 45 s and 0.22 of the remaining budget controller.py. The cap is 180 s for the benchmark and datacenter, 480 s for the grid profiles.py profiles.py profiles.py. The benchmark cap is not stated in the paper.
  • Reserve. The paper gives a 0.30 fraction with a 120 s minimum. The code also caps the reserve at the total budget minus 60 s controller.py.
  • Recovery budget. Restoration gets 0.72 of the remaining budget, at least 45 s controller.py. Not in the paper.
  • Admission. The paper states non-regression. The code rejects a candidate when structural validity, required files or Python syntax regress, or when the supported residual tuple gets worse audits.py audits.py. The residual tuple is compared in this order: largest normalised residual, total violation mass, violation count. Performance steps also need a viable candidate controller.py. A restoration candidate that regresses nothing is kept even when it is not yet viable controller.py.
  • Observers are per task family. The release registers deterministic observers for nine named families registry.py. For any other task, the observer reports unsupported, so the candidate is admitted on structural checks alone and the challenger always runs registry.py. The paper describes observers in general terms §3.2.
  • Ablation code is absent. The release contains no switches or result files for the four ablation variants (no matches for "ablat" outside the report folder). The ablation numbers cannot be checked against code §5.2.
  • Quality normalisation. The release's summary divides the sum of stage qualities by twice the number of stages scores.py. The paper does not state this, and the code does not explain the factor. The official scorer is not in the release.
  • Datacenter performance instructions. The paper summarises the industrial instructions in general terms app. A.2. The code contains the full text, with specific targets: a 0.050 Gbps margin to the public floor, a balance of 85%, and a utilisation cap of 0.990 profiles.py. The paper does not give these numbers.
  • Grid performance instructions. The code requires that the final solve.py run cleanly from a fresh directory within the 900-s solver contract, with no imported outside solutions profiles.py.
  • Input hashes. The case profiles pin SHA-256 hashes of the task inputs and refuse to replay on a mismatch profiles.py profiles.py. The inputs themselves are not in the release app. A.5.
  • Benchmark root. The harness needs the external ORAgentBench checkout, found through ORAGENTBENCH_ROOT or a sibling folder constants.py.

Pitfalls

  • Development benchmark. The paper says ORAgentBench informed the harness, and that task failures were inspected while it was built § Limitations (Evaluation scope). The nine observer families in the code are benchmark task names registry.py. Treat 42.99% as a development-set figure, not an estimate for unseen problems.
  • Stage order is wider than Eq. 3. Implementers who copy only the three stages will miss the solver-script step controller.py.
  • Observer coverage decides a lot. Without a registered observer, admission reduces to structural checks, and the challenger's counterexamples are the only semantic evidence. A zero residual means "no violation in the checked scope", which the paper also states §3.2 § Limitations (Verification coverage).
  • Challenger output is advisory. Its text is not verified, and the candidate is reverted after it runs. Feed the report to the producer as text only.
  • Incomplete tasks score zero. A task that does not finish all stages counts as infeasible §4.1 scores.py.
  • Industrial comparisons are retrospective. August 10 records were visible during development, and the archived VERAS run was selected after earlier feedback on that date §4.2 app. A.1. The second run reverses the datacenter ranking app. A.4.2.
  • Grid comparison uses separate runs. The reported VERAS plan is a separately selected run. The arm executed beside native Codex scores 42.613 million USD, which is above the 42.065 million USD expert hourly model app. A.4.3 tab. 3. The best bound also leaves a 5.122% gap, so the plan is a feasible incumbent, not a proven optimum app. A.4.3.
  • Compute is not matched. Shared time ceilings do not mean equal inference compute, and the paper does not establish cost-effectiveness §4.1 § Limitations (Reproducibility).
  • Models are proprietary. GPT-5.4 and GPT-5.6 Sol are needed for the reported results. The paper notes API nondeterminism, so exact trajectories cannot be replayed § Limitations (Reproducibility).
  • Industrial results cannot be rerun from the release. The release withholds the operational input rows and the agent-written output files (Parquet, NPZ). The release has the solver code, the input hashes and replay audits app. A.5.
  • No licence for code reuse. Re-implement from the pseudocode and this card. Any copy of the repository's code needs written permission from the authors.
  • Small per-task counts. One benchmark task is 0.93 percentage points. The Hard split has 34 tasks, so one task moves that split by almost 3 points.

Lineage

  • Builds on: general-purpose coding agents, with the producer running on a coding-agent harness §3.1. Earlier OR agents named in the introduction are CoE, OptiMUS and NEMO. FunSearch and AlphaEvolve use evaluation feedback to evolve programs §1.
  • Benchmark: ORAgentBench §4.1 §1.
  • Compared against: Codex (GPT-5.4, GPT-5.3-Codex, GPT-5.4-mini) and Claude Code (Opus 4.6, DeepSeek V4 Pro, Kimi K2.6, GLM 5.1, DeepSeek V4 Flash, Sonnet 4.6) as published results tab. 1. In the industrial studies: an operational mapping and a historical-majority heuristic §4.2, and two expert MISOCP models §4.3.
  • Superseded by: not known.

Source and license

Adapted from “VERAS: Verification-Guided Adaptive Search for Reliable Operations Research” by Minbo Ma, Honglin Zhang, Bangtao Zhou, Chunguang Yu, Dasheng Ju, Yichen Shen, Fengli Xu, Yong Li (arXiv:2610.24329), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/holimario/VERAS at commit 39f15255bc95 (none (no LICENSE file in repository; NOTICE.md grants no license)). Any quoted code is shown under that license.