Paper · 2026-10-16 · cs.SE, cs.AI
VERAS: Verification-Guided Adaptive Search for Reliable Operations Research
VERAS wraps a coding agent in a verify-then-revise loop: structural checks, per-task observers and a fresh-context challenger supply evidence, and a revision replaces the incumbent only if it does not regress. Author-reported: 42.99% pass rate on ORAgentBench (46 of 107) versus 35.51% for Codex with GPT-5.4; 10.34% lower peak bandwidth than native Codex on one archived run.
LLM agentSelf-refinement (draft, critique, rewrite loop)LLM self-evaluationProgram synthesisagentsverificationoperations-researchoptimizationcode-generation
arXivPDFCodeMarkdown14 min read
AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
An AI coding agent can solve an operations research problem, such as scheduling staff, routing trucks or planning a power grid, by writing a program that outputs a plan. The trouble is that the agent often checks its own plan with checks it wrote itself. If it misread a rule, the program and its checks miss the rule in the same way, and the plan looks correct when it breaks a constraint.
VERAS adds a second layer of checking. It runs basic validity checks on the plan, runs specific checkers where they exist, and asks a separate reviewer agent, which starts with a clean context, to look for rules the checks did not cover. The reviewer's findings then guide the next revision: first repair any broken constraint, then review coverage, then improve the objective. A new version of the plan replaces the old one only if it does not break what was already working.
On a benchmark of 107 such problems, the authors report a higher pass rate than a strong coding agent using the same language model. On two industrial problems, the reported plans beat the native agent and, in one case, a hand-built model, but the paper itself shows that these comparisons are retrospective and rest on single selected runs.
TL;DR
- VERAS is a harness around a general-purpose coding agent. It checks each candidate with structural checks, task-specific observers and a fresh-context challenger, then revises and admits only non-regressing candidates §3.1.
- Author-reported ORAgentBench result: 46 of 107 tasks pass (42.99%) against 38 (35.51%) for Codex with GPT-5.4 under the same official criteria. The largest gain is on Hard tasks, 12 of 34 against 7 of 34 §5.1 tab. 1.
- Industrial results are single selected runs: a 10.34% lower peak bandwidth than native Codex, and a 6.28% lower annualised cost than an expert hourly model. The paper's own follow-up runs weaken both claims §5.3 §5.4 app. A.4.2 app. A.4.3.
- Code is public at
holimario/VERASwithout a license file, so this card gives no code snippets. The release holds the controller, observers and case profiles, but not the benchmark tasks or the industrial inputs.
Problem
An LLM agent that writes an optimisation program and then checks its output with its own tests can pass those tests and still violate the task. The authors call this a correlated specification error: the code and the checks written for it both get one requirement wrong in the same way. They also name two further gaps: self-written verification often covers only part of the requirements, and feedback that says only "failed" gives too little to act on. A third gap is that a revision meant to fix one thing can break something that already worked §1 fig. 1.
Method
Problem and state
An OR instance I holds a task description and instance data. The problem is to minimise an objective f(x; I) over decision variables x, subject to constraint predicates c_j(x; I) = 1 for each j in a constraint set §3.1 eq. 1. At iteration t the incumbent state holds the program kept so far (P_t), the artifact it exports (a_t) and the evidence gathered on that artifact (E_t) §3.1.
The coding agent writes an executable program, runs it, and exports a decision artifact. VERAS wraps this producer with three mechanisms §3.1 fig. 2:
- Task-grounded verification produces evidence E_t = Verify(I, a_t) with three parts: whether the artifact is well formed, which constraints it demonstrably breaks, and how much of the requirement set was checked §3.2 eq. 2.
- Evidence-guided refinement chooses the next action from that evidence §3.3.
- Verification-filtered update decides whether a candidate replaces the incumbent §3.4.
Task-grounded verification §3.2
- Structural review. Checks that the required files exist, that decision fields and data types are right, and that records are complete. Defects are reported with the affected fields or records.
- Deterministic observer. Evaluates the candidate against the constraints it can reconstruct from the task and instance data. It reports witnessed violations with numerical residuals where they apply, and records its verification scope, the constraint families it covers. This record is what separates "no violation found" from "not checked" §3.2.
- Isolated challenger. A separate agent in a fresh context reads the task, the instance data and the candidate, and searches for counterexamples to requirements the observers did not cover. Its findings add evidence but do not change the program §3.2 app. A.1.
Evidence-guided refinement §3.3
A controller runs three stages in a fixed order, restore, audit, improve. Each stage runs only if its activation condition holds for the current evidence and the remaining budget eq. 3.
- Feasibility restoration. Runs when the artifact is structurally invalid or a witnessed violation exists. The producer receives the violated requirements and the available residuals and revises the formulation, heuristic or program. The revised program is executed and re-verified before admission §3.3.
- Coverage-aware auditing. Runs for a viable candidate whose verification coverage is incomplete. The challenger's report goes to the producer as evidence. The auditing step does not edit the program §3.3.
- Performance improvement. Runs for a viable candidate while budget remains. The producer uses the accumulated feedback to improve the original objective, and each revision is executed and verified before admission §3.3.
Verification-filtered update §3.4
Each proposed program is executed and verified to give a candidate artifact and evidence. Acceptance is a 0/1 decision that depends on the evidence before and after, and on the current phase eq. 4. The incumbent is replaced only on acceptance; otherwise the old program, artifact and evidence stay eq. 5. Restoration accepts incremental feasibility gains, and performance updates also need a valid artifact with no detected violation under the supported checks §3.4.
# VERAS: one revision round for one task [§3.1] [§3.3]
# Input: task, incumbent (program, artifact, evidence), remaining budget
# Output: incumbent for the next round
def veras_round(task, incumbent, budget):
# Structural review and deterministic observer [§3.2]
evidence = verify(task, incumbent.artifact)
findings = None
# Restore if the artifact is invalid or a violation exists [§3.3]
if needs_restoration(evidence):
candidate = producer_fix(task, incumbent, evidence)
incumbent = admit(incumbent, candidate, performance=False)
# Audit a viable candidate with incomplete coverage [§3.3]
if is_viable(evidence) and not evidence.coverage_complete:
findings = challenger(task, incumbent.artifact, evidence)
# The challenger only observes, so the artifact is restored [§3.3]
incumbent = restore_snapshot(incumbent)
evidence = verify(task, incumbent.artifact)
# Improve the objective from a viable candidate [§3.3]
if is_viable(evidence):
candidate = producer_improve(task, incumbent, evidence, findings)
incumbent = admit(incumbent, candidate, performance=True)
return incumbent# Candidate admission [§3.4]
# Input: incumbent, candidate, phase flag (performance or restoration)
# Output: the candidate if admitted, otherwise the incumbent
def admit(incumbent, candidate, performance):
# Reject any regression in structure: validity, files, syntax [§3.4]
if structural_regression(incumbent, candidate):
return incumbent
# Reject a worse residual tuple on the supported checks [§3.4]
if residual_regression(incumbent, candidate):
return incumbent
# Performance steps also need a viable candidate [§3.4]
if performance and not is_viable(candidate):
return incumbent
return candidateNote on the pseudocode: the admit and veras_round blocks are a condensed
reading of the paper's Section 3 and of the controller code. They are not a
verbatim transcription; see Paper vs code for where the code adds steps.
Hyperparameters
Benchmark values come from the paper's appendix table, and from the code where the paper is silent. The case-study values differ, as listed in the second table.
| Name | Value | Where | Notes |
|---|---|---|---|
| producer model / reasoning | GPT-5.4 / high | app. A.3 tab. 5 | the release leaves the model user-supplied CONFIGURATION.json |
| concurrent task workers | 2 | app. A.3 tab. 5 | the benchmark profile field reads 1 with a comment that the launcher uses two profiles.py |
| time limit per stage | 2,700 s | §4.1 tab. 5 | |
| timeout scale | 1.00 | tab. 5 profiles.py | |
| reserved recovery fraction | 0.30 | §4.1 tab. 5 CONFIGURATION.json | the code caps the reserve at total minus 60 s; not stated in the paper controller.py |
| minimum recovery allocation | 120 s | §4.1 tab. 5 constants.py | |
| challenger fraction of remaining budget | 0.22 | §4.1 constants.py | code also has a 45 s floor and a cap; see Paper vs code |
| challenger cap | 180 s | profiles.py | not in the paper's benchmark table; code default |
| minimum intervention time | 45 s | constants.py | not stated in the paper |
| recovery budget share | 0.72 of remaining, minimum 45 s | controller.py | not stated in the paper |
| harness-side local solver cap | 180 s | tab. 5 cli.py | used only when the stage names a solver script |
| transient transport attempts | 4 | tab. 5 cli.py | |
| candidate SCIP call ceiling | 300 s | §4.1 tab. 5 | not checkable here: the solver and benchmark live outside this release |
| target SCIP relative gap | 5 × 10⁻⁴ | §4.1 tab. 5 | not checkable here, as above |
| pass threshold | feasibility > 0 and normalised quality > 0.4 | §4.1 eq. 6 scores.py |
Case-study profiles, which replace the benchmark values tab. 6:
| Name | Datacenter placement | 54-bus network planning | Where |
|---|---|---|---|
| model / reasoning | GPT-5.6 Sol, xhigh effort | same model and effort | tab. 6 profiles.py profiles.py |
| agent wall-time limit | 19,800 s | 15,360 s | tab. 6 |
| timeout scale | 3.0 | 2.0 | tab. 6 profiles.py profiles.py |
| recovery reserve fraction | 0.40 | 0.40 | tab. 6 profiles.py profiles.py |
| challenger cap | 180 s | 480 s | tab. 6 profiles.py profiles.py |
| candidate execution contract | 120-s output generator, one replay | 900-s solver, at most four threads | §4.2 §4.3 tab. 6 |
| replay solver limit | not stated (120-s output limit applies) | 750 s for source replay | app. A.4.2 app. A.4.3 |
| input the agent can see | transfer records from 8–9 August 2025 | one fixed 54-bus instance | §4.2 tab. 6 |
| how results are scored | replay on 10 August 2025 | independent constraint and objective audit | §4.2 §4.3 |
Results
Author-reported. Setup: ORAgentBench, 107 tasks (32 Easy, 41 Medium, 34 Hard), GPT-5.4 with high reasoning effort, under the benchmark's official scoring. The Codex and Claude Code rows are published results, not reruns; the paper says they do not control realised inference compute §4.1 tab. 1. The feasibility and quality columns are task-count-weighted means across difficulty splits tab. 1. The pass rate is the share of tasks with feasibility above 0 and normalised quality above 0.4 §4.1 eq. 6.
| Harness | Model | Feas. overall | Quality overall | Pass Easy (%) | Pass Medium (%) | Pass Hard (%) | Pass overall (%) |
|---|---|---|---|---|---|---|---|
| Codex | GPT-5.4 | 0.57 | 0.32 | 46.88 | 39.02 | 20.59 | 35.51 |
| Codex | GPT-5.3-Codex | 0.51 | 0.29 | 31.25 | 43.90 | 20.59 | 32.71 |
| Codex | GPT-5.4-mini | 0.46 | 0.24 | 43.75 | 29.27 | 5.88 | 26.17 |
| Claude Code | Claude Opus 4.6 | 0.49 | 0.31 | 50.00 | 39.02 | 14.71 | 34.58 |
| Claude Code | DeepSeek V4 Pro | 0.41 | 0.25 | 39.58 | 28.46 | 13.73 | 27.10 |
| Claude Code | Kimi K2.6 | 0.43 | 0.22 | 31.25 | 36.59 | 11.76 | 27.10 |
| Claude Code | GLM 5.1 | 0.38 | 0.25 | 37.50 | 31.71 | 8.82 | 26.17 |
| Claude Code | DeepSeek V4 Flash | 0.39 | 0.24 | 46.88 | 26.83 | 5.88 | 26.17 |
| Claude Code | Claude Sonnet 4.6 | 0.42 | 0.23 | 31.25 | 29.27 | 14.71 | 25.23 |
| VERAS | GPT-5.4 | 0.67 | 0.39 | 56.25 | 39.02 | 35.29 | 42.99 |
tab. 1. VERAS passes 46 of 107 tasks; the GPT-5.4 Codex baseline passes 38. Feasibility for VERAS is 72 of 107 tasks CONFIGURATION.json. Four tasks did not complete all required stages, one Medium and three Hard CONFIGURATION.json.
Ablation §5.2 fig. 3. Six Hard tasks, GPT-5.4, one worker per variant, and separate runs for each variant. The VERAS row is the frozen original result. The text gives the following values; the feasibility count for the no-challenger variant is not stated in the text.
| Variant | Feasible (/6) | Passed (/6) | Mean normalised quality |
|---|---|---|---|
| VERAS (frozen original) | 5 | 5 | 0.621 |
| No isolated challenger | not stated | 0 | 0.054 |
| No deterministic observer | 4 | 3 | 0.466 |
| No feasibility-restoration routing | 2 | 2 | 0.237 |
| No verification-filtered admission | 5 | 2 | 0.409 |
Inter-datacenter placement (625,191 tasks, August 2025 traces, 120-s output limit) §4.2 §5.3 tab. 2. The paper gives these as author-reported values from an archived VERAS run and a later native Codex run.
| Method | Peak (Gbps) ↓ | Migration (PB) ↓ |
|---|---|---|
| Operational mapping | 19,495.167 | 0.000 |
| Historical-majority heuristic | 18,276.071 | 0.333 |
| Native Codex | 1,238.625 | 25.473 |
| VERAS | 1,110.603 | 25.901 |
A second VERAS run with the same performance instructions gives 1,567.303 Gbps, which reverses the ranking against native Codex app. A.4.2.
54-bus active distribution network planning (MISOCP, 4 scenarios, 24 periods) §4.3 §5.4 tab. 3. Costs in million USD per year.
| Method | Total ↓ | Investment ↓ | Operation ↓ |
|---|---|---|---|
| Hourly MISOCP (expert) | 42.065 | 18.984 | 23.081 |
| Varying-timestep MISOCP (expert) | 42.923 | 19.012 | 23.911 |
| Native Codex | 45.690 | 18.490 | 27.200 |
| VERAS | 39.424 | 16.154 | 23.269 |
The VERAS arm executed alongside native Codex gives 42.613 million USD, a separate selected VERAS run gives 39.424 million USD, and the selected run's best bound is 37.404 million USD with a 5.122% solver gap app. A.4.3.
Run-level mechanism counts, reported for the 117-stage benchmark: 15 structural restorations, three residual reductions, 103 tasks with a challenger run, and 114 of 132 performance revisions admitted §5.1. The release file that summarises these counts agrees with the 15 restorations, the three residual reductions and 103 challenger tasks component-evidence-summary.json.
Paper vs code
Official code: github.com/holimario/VERAS, which the abstract links. The
checked commit is 39f1525 (full hash in the frontmatter). The code was read,
not copied. The repository has no LICENSE file. Its NOTICE.md says no
license is granted for third-party benchmark material or operational datasets,
and that an explicit source license will follow separately. No snippets are
quoted here.
Matches:
- Benchmark defaults: high reasoning, timeout scale 1.0, reserve fraction 0.30, challenger fraction 0.22, minimum recovery 120 s CONFIGURATION.json constants.py.
- Local solver cap 180 s and four transient transport attempts cli.py.
- Case-study profiles: GPT-5.6 Sol at xhigh, timeout scales 3.0 and 2.0, reserve 0.40, challenger caps 180 s and 480 s profiles.py.
- Pass rule: feasibility above 0 and normalised quality above 0.4 scores.py.
- Headline counts (46 passed, 72 feasible, 103 completed of 107, quality 0.392) CONFIGURATION.json.
- The challenger runs in a fresh context controller.py, and its candidate is restored after observation controller.py.
Differs or is unclear:
- Extra stage not in the paper. The controller runs an existing solver script named by the stage before the producer's restoration, when the artifact is invalid and such a script exists controller.py audits.py. Eq. 3 lists three stages only §3.3 eq. 3.
- Challenger budget. The paper says 0.22 of the remaining budget. The code takes the smaller of the cap and the larger of 45 s and 0.22 of the remaining budget controller.py. The cap is 180 s for the benchmark and datacenter, 480 s for the grid profiles.py profiles.py profiles.py. The benchmark cap is not stated in the paper.
- Reserve. The paper gives a 0.30 fraction with a 120 s minimum. The code also caps the reserve at the total budget minus 60 s controller.py.
- Recovery budget. Restoration gets 0.72 of the remaining budget, at least 45 s controller.py. Not in the paper.
- Admission. The paper states non-regression. The code rejects a candidate when structural validity, required files or Python syntax regress, or when the supported residual tuple gets worse audits.py audits.py. The residual tuple is compared in this order: largest normalised residual, total violation mass, violation count. Performance steps also need a viable candidate controller.py. A restoration candidate that regresses nothing is kept even when it is not yet viable controller.py.
- Observers are per task family. The release registers deterministic observers for nine named families registry.py. For any other task, the observer reports unsupported, so the candidate is admitted on structural checks alone and the challenger always runs registry.py. The paper describes observers in general terms §3.2.
- Ablation code is absent. The release contains no switches or result files for the four ablation variants (no matches for "ablat" outside the report folder). The ablation numbers cannot be checked against code §5.2.
- Quality normalisation. The release's summary divides the sum of stage qualities by twice the number of stages scores.py. The paper does not state this, and the code does not explain the factor. The official scorer is not in the release.
- Datacenter performance instructions. The paper summarises the industrial instructions in general terms app. A.2. The code contains the full text, with specific targets: a 0.050 Gbps margin to the public floor, a balance of 85%, and a utilisation cap of 0.990 profiles.py. The paper does not give these numbers.
- Grid performance instructions. The code requires that the final
solve.pyrun cleanly from a fresh directory within the 900-s solver contract, with no imported outside solutions profiles.py. - Input hashes. The case profiles pin SHA-256 hashes of the task inputs and refuse to replay on a mismatch profiles.py profiles.py. The inputs themselves are not in the release app. A.5.
- Benchmark root. The harness needs the external ORAgentBench checkout, found
through
ORAGENTBENCH_ROOTor a sibling folder constants.py.
Pitfalls
- Development benchmark. The paper says ORAgentBench informed the harness, and that task failures were inspected while it was built § Limitations (Evaluation scope). The nine observer families in the code are benchmark task names registry.py. Treat 42.99% as a development-set figure, not an estimate for unseen problems.
- Stage order is wider than Eq. 3. Implementers who copy only the three stages will miss the solver-script step controller.py.
- Observer coverage decides a lot. Without a registered observer, admission reduces to structural checks, and the challenger's counterexamples are the only semantic evidence. A zero residual means "no violation in the checked scope", which the paper also states §3.2 § Limitations (Verification coverage).
- Challenger output is advisory. Its text is not verified, and the candidate is reverted after it runs. Feed the report to the producer as text only.
- Incomplete tasks score zero. A task that does not finish all stages counts as infeasible §4.1 scores.py.
- Industrial comparisons are retrospective. August 10 records were visible during development, and the archived VERAS run was selected after earlier feedback on that date §4.2 app. A.1. The second run reverses the datacenter ranking app. A.4.2.
- Grid comparison uses separate runs. The reported VERAS plan is a separately selected run. The arm executed beside native Codex scores 42.613 million USD, which is above the 42.065 million USD expert hourly model app. A.4.3 tab. 3. The best bound also leaves a 5.122% gap, so the plan is a feasible incumbent, not a proven optimum app. A.4.3.
- Compute is not matched. Shared time ceilings do not mean equal inference compute, and the paper does not establish cost-effectiveness §4.1 § Limitations (Reproducibility).
- Models are proprietary. GPT-5.4 and GPT-5.6 Sol are needed for the reported results. The paper notes API nondeterminism, so exact trajectories cannot be replayed § Limitations (Reproducibility).
- Industrial results cannot be rerun from the release. The release withholds the operational input rows and the agent-written output files (Parquet, NPZ). The release has the solver code, the input hashes and replay audits app. A.5.
- No licence for code reuse. Re-implement from the pseudocode and this card. Any copy of the repository's code needs written permission from the authors.
- Small per-task counts. One benchmark task is 0.93 percentage points. The Hard split has 34 tasks, so one task moves that split by almost 3 points.
Lineage
- Builds on: general-purpose coding agents, with the producer running on a coding-agent harness §3.1. Earlier OR agents named in the introduction are CoE, OptiMUS and NEMO. FunSearch and AlphaEvolve use evaluation feedback to evolve programs §1.
- Benchmark: ORAgentBench §4.1 §1.
- Compared against: Codex (GPT-5.4, GPT-5.3-Codex, GPT-5.4-mini) and Claude Code (Opus 4.6, DeepSeek V4 Pro, Kimi K2.6, GLM 5.1, DeepSeek V4 Flash, Sonnet 4.6) as published results tab. 1. In the industrial studies: an operational mapping and a historical-majority heuristic §4.2, and two expert MISOCP models §4.3.
- Superseded by: not known.
Source and license
Adapted from “VERAS: Verification-Guided Adaptive Search for Reliable Operations Research” by Minbo Ma, Honglin Zhang, Bangtao Zhou, Chunguang Yu, Dasheng Ju, Yichen Shen, Fengli Xu, Yong Li (arXiv:2610.24329), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/holimario/VERAS at commit 39f15255bc95 (none (no LICENSE file in repository; NOTICE.md grants no license)). Any quoted code is shown under that license.