VeriEvolve: Evaluator-First Evolution for Tool-Using Agents

Yueshen Li; Kamer Ali Yuksel; Gokhan Tur; Dilek Hakkani-Tür; João Pedro Maia

cs.SEOct 21, 2026

VeriEvolve: Evaluator-First Evolution for Tool-Using Agents

Authors: Yueshen Li, Kamer Ali Yuksel, Gokhan Tur, Dilek Hakkani-Tür, João Pedro Maia

Organizations: University of Illinois Urbana-Champaign · aiXplain

Abstract

A central obstacle to self-improving agents is the absence of a reliable fitness signal for realistic stateful tasks. We introduce VeriEvolve, an evaluator-first evolution framework that constructs this missing evaluation layer before optimizing the agent. From task instruction, VeriEvolve synthesizes instruction-grounded challenge states and an executable evaluator that checks them. During each round of workflow evolution, the evaluator is frozen and serves as the fitness signal. Between rounds, VeriEvolve revises the evaluator by converting the end states of strong and weak runs into a pair of counterfactual test cases. Across all 30 MCPMark Filesystem tasks, VeriEvolve constructs 30 evaluators with 141 challenge states, achieving 141/141 agreement on the generated suites and 113/122 agreement on held-out states produced by executor agents. Evaluator-guided evolution increases externally verified success from 6/30 to 18/30 for GLM-5.2 and from 27/30 to 30/30 for Claude; refinement removes all three false positives that the weak executor exposes. Cross-environment studies on PostgreSQL, Playwright, Notion, and GitHub further expose complementary limits from observability, state representation, and search/execution capacity. These results show that executable evaluators can be synthesized and used to guide search when task success is observable and an external verifier is available for calibration.

Figures & tables

Figure 1: Overview of VeriEvolve. Stage 1 constructs challenge states and an executable evaluator; Stage 2 freezes that evaluator while evolving workflows; and Stage 3 mines strong–weak execution-state differences to extend the challenge set and revise the evaluator. One Stage 3 update followed by a Stage 2 rerun is a refinement round . Purple nodes are model-backed roles, green nodes are persisted artifacts or states, and gray nodes are deterministic operations.

Environment	Constructed	Cases (P/N)	Ext. agreement	Stage-2 study
Filesystem	30/30	141 (34/107)	141/141	30/30
PostgreSQL	18/21	120 (22/98)	115/120	15/21

Table 1: Primary RQ1 evidence. Cases are generated challenge states; P/N reports their post-hoc external-verifier labels. External agreement compares the frozen internal evaluator with those labels. The external verifier is consulted only after evaluator construction and is not exposed to any model-backed role or used as evaluator-repair feedback. For the Stage 2 study, we include tasks whose frozen evaluator reaches at least

0.85

post-hoc external agreement.

Executor	States	$\checkmark$ / $\checkmark$	$\checkmark$ / $\times$	$\times$ / $\checkmark$	$\times$ / $\times$	Agr.	Prec.	Rec.	Spec.
		Int/Ext
GLM-5.2	85	18	3	1	63	95.3	85.7	94.7	95.5
Claude	44	30	0	5	9	88.6	100.0	85.7	100.0
All (distinct)	122	41	3	6	72	92.6	93.2	87.2	96.0

Table 2: Held-out evaluation on Filesystem: frozen Stage 1 evaluators versus the external verifier on distinct agent-produced states from both Stage 2 campaigns (Int/Ext columns count states; Agr., Prec., Rec., and Spec. are agreement, precision, recall, and specificity in %, with “pass” as the positive class). Seven states produced by both executors are counted once in the last row. Details in Appendix D.3 .

Executor	Workflow	Int. pass	Ext. pass	FP	FN
Claude	Seed	24/30	27/30	0	3
Claude	Evolved	30/30	30/30	0	0
GLM-5.2	Seed	8/30	6/30	2	0
GLM-5.2	Evolved	21/30	18/30	3	0

Table 3: Filesystem seed and evolved workflow performance. Seed denotes the internally selected best candidate at round 0, before workflow evolution, under each campaign’s recorded path budget. FP and FN denote

\mathrm{Int}\checkmark/\mathrm{Ext}\times

and

\mathrm{Int}\times/\mathrm{Ext}\checkmark

, respectively.

Figure 2: Worked example on the MCPMark Filesystem task structure_mirror . Orange labels give what the mirror must do with each directory. Stage 1 builds

E^{(0)}

from five challenge states;

E^{(0)}

checks placeholders in only 5 of the 8 empty leaf directories. In Stage 2, GLM-5.2 reaches a state

F_{0}^{-}

that

E^{(0)}

accepts but the external verifier rejects, which selects the task for Stage 3. Stage 3 turns the difference from the Claude state

F^{+}

into paired cases and a revised evaluator

E^{(1)}

, under which the weak executor’s next state

F_{1}^{-}

passes both checks. Int and Ext denote the internal evaluator and the external verifier.

Run	Exec. tok.	Evolver	Wall	Normalized cost
Filesystem
Stage 2, GLM-5.2	33.18M	$31.24	43.5 h	2.76M, $2.60 per gained task (12)
Stage 2, Claude	0.58M a	$1.74	5.3 h	0.19M, $0.58 per gained task (3)
Stage 3	8.10M	$3.49	4.6 h b	2.70M + 21.4K TSG/VE, $1.16 per repaired evaluator (3)
PostgreSQL (15 tasks)
Stage 2, GLM-5.2	27.82M	$22.74	37.4 h	9.27M per solved task (3)

Table 4: Recorded cost of the main campaigns. Evolver is the Workflow Evolver’s dollar cost; wall time is summed over tasks or runs. Normalized cost divides the whole campaign by the tasks that pass externally only after evolution (Filesystem), by repaired evaluators (Stage 3), or by externally verified successes (PostgreSQL). a Excludes prompt-cache reads. b 2.5 h elapsed with runs in parallel. Stage 1 cost was not recorded.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Environment	Universe	RQ1 attempted	S2 eligible	Valid pairs	RQ3 cases	Disposition
Filesystem	30	30	30	30-task campaigns	3	Primary standardized study; no exclusions.
PostgreSQL	21	21	15	15	0	Full Stage 1 disposition and Stage 2 sweep over all eligible tasks; three additional evaluators fall below the admission threshold and three tasks are blocked by the observation interface.
Playwright	4	4	2	2	0	Two formal exclusions: missing independent ground truth and unobservable success state.
Notion	28	23	2	1 historical	0	One historical matched pair; broader standardized execution remains limited by the live-state representation.
GitHub	23	23	1	0 standardized	0	Incomplete feasibility campaign; one externally calibrated evaluator and one corrected strong-executor $F^{+}$ , with the corrected weak-executor pair incomplete.

Appendix

Table 5: Audited task scope. “S2 eligible” counts evaluators or tasks admitted to the corresponding Stage 2 study. Historical, diagnostic, and incomplete pilot results are reported separately from standardized matched pairs.

Task	Evaluation regime	Representative requirements
uppercase	Nearly direct specification	Create explicitly named uppercase file copies and a fixed-format word-count artifact. Major instruction clauses map nearly one-to-one to deterministic evaluator checks.
structure_mirror	Implicit state specification	Reproduce a transformed hierarchy while satisfying interacting existence, naming, placeholder-content, discard, preservation, and absence constraints. The prohibition on invalid extra structure is exposed only through strong–weak execution-state comparison in the development and controlled runs (Appendix C ).

Appendix

Table 6: Examples of differing evaluation regimes within stateful tool-use tasks. Some instructions admit a nearly direct translation into executable checks, whereas others distribute correctness across interacting properties of the final environment state.

Role	Base model	Invocation	Context and visible inputs	Output
Test-Suite Generator, Stage 1	Claude	Interactive Claude Code CLI	Persistent task session; task instruction, observable initial state, and self-generated artifacts	Declarative labeled challenge-state specification
Verifier Evolver, Stage 1	Claude	Same interactive session	Shares context with the Test-Suite Generator; receives challenge states and deterministic internal scoring feedback	Sandboxed verify(context) evaluator
Test-Suite Generator, Stage 3	Claude Sonnet	Headless retained CLI session	Packet-restricted task instruction, current evaluator, sanitized cases, state-difference facts, and output contract	Anchor-based paired challenge cases
Verifier Evolver, Stage 3	Claude Sonnet	Resumes Stage 3 TSG session	Current evaluator, cumulative cases, factual paired contrasts, and deterministic repair feedback	Localized evaluator edits or replacement code
Workflow Evolver	Claude Sonnet	claude -p ; one call per round	Fresh call with alternative statistics, score trajectory, failed checks, and artifact-inspection diagnosis	At most one new alternative per selected node
Strong Workflow Executor	Claude Sonnet	Headless CLI subprocess per path	Candidate workflow prompts and isolated task environment	Final environment state and execution trace

Appendix

Table 7: Model roles and invocation modes. Stage 1 TSG and VE share one interactive session; Stage 3 TSG and VE share one retained packet-restricted session per refinement round. The Workflow Evolver and both executors are invoked separately for each round or candidate.

Figure 3: Persistent multi-alternative workflow used in Stage 2. A candidate path selects one prompt or behavior alternative from each active node, producing an artifact and execution trace. Alternatives persist across rounds, enabling non-destructive reuse and recombination.

Stage	Component	Gate or budget
1	Internal construction	Positive acceptance $\geq 0.90$ ; adversarial rejection $\geq 0.80$ .
1	Post-hoc admission	External agreement $\geq 0.85$ for admission to the standardized Stage 2 study.
2	Workflow search	Five paths per round; candidate concurrency $1$ ; at most $15$ rounds.
2	Candidate resources	Candidate wall-time limit $300$ seconds; token budget $250$ K; ReAct limit at most $20$ by default and $24$ at the framework cap.
2	Strong executor	At most $40$ turns and $600$ seconds per candidate.
2	Round timeout	One hour per round subprocess (Filesystem GLM-5.2 and PostgreSQL campaigns); a round that exceeds it ends the run, which is counted as unsolved.

Appendix

Table 8: Standardized experimental gates and resource caps.

Environment	Materialized candidate	Evaluator-visible state	Principal limitation
Filesystem	Isolated directory tree	Paths, file contents, directory structure, and supported metadata	Volatile backup paths and duplicate artifacts require conservative canonicalization.
PostgreSQL	Fresh isolated database clone	Canonical schema, relation, and row snapshots	Some object classes and large relations are not fully represented in the internal snapshot.
Playwright	Isolated browser execution	Tool transcript and selected browser-state signals	Some tasks lack independent factual ground truth or an observable post-condition.
Notion	Duplicated workspace	Canonical page, block, database, and row snapshots	Linked views and serialization changes can obscure or alter the graded state.
GitHub	Isolated repository	Repository contents and supported issue, pull-request, and history state	Tool-surface and verifier-environment limitations block broad standardized coverage.

Appendix

Table 9: Category-specific materialization and observation interfaces. Evaluator construction is meaningful only when task-relevant success conditions are represented in the permitted observation interface.

Case	Label	Type	Targeted requirement
correct_full_mirror	$\checkmark$	Positive	Complete transformed mirror.
mirror_dir_missing	$\times$	Clean negative	Required mirror root is absent.
no_processed_suffix	$\times$	Adversarial	Digit-containing names are not transformed.
discarded_dir_present	$\times$	Adversarial	A directory with more than two direct files is retained.
missing_placeholder	$\times$	Adversarial	A mirrored empty directory lacks its placeholder.

Appendix

Table 10: Initial structure_mirror challenge states.

Step	Evaluator	Newly exposed difference	New evaluation evidence	Promoted check	Weak-state outcome
Initial	$E^{(0)}$	–	Five initial Stage 1 cases	Presence, renaming, discard, and placeholder checks (5 listed directories)	$F_{0}^{-}$ : $\mathrm{Int}\checkmark/\mathrm{Ext}\times$
Round 1	$E^{(1)}$	Empty-directory, missing-directory, and missing-file clusters	Six paired counterfactual cases	Three placeholder paths added; regression rejects $F_{0}^{-}$	$F_{1}^{-}$ : $\mathrm{Int}\checkmark/\mathrm{Ext}\checkmark$
Round 2 (excluded)	$E^{(2)}$	Duplicate directory created by file synchronization	Reproduce/remove pair	Extra-structure absence check	$\mathrm{Int}\times/\mathrm{Ext}\times$ (not counted)

Appendix

Table 11: Primary structure_mirror evaluator lineage. One revision suffices: the weak state reached under

E^{(1)}

passes the external verifier. The second round was triggered by a file-synchronization artifact and is excluded from our results.

Task	Cases (P/N/A)	External agreement	Evaluator SHA	Refinement
author_folders	4 (1/3/3)	4/4	cadf0721	–
budget_computation	6 (1/5/4)	6/6	3ffc73ef	–
code_locating	4 (2/2/2)	4/4	af458c70	–
contact_information	6 (1/5/2)	6/6	31564641	–
dataset_comparison	4 (1/3/3)	4/4	71bb6f84	–
debugging	4 (1/3/3)	4/4	06c927a2	E1

Appendix

Table 12: Complete per-task Filesystem Stage 1 results. P, N, and A denote positive, negative, and adversarial challenge states. External agreement is measured only on the generated challenge suite.

	GLM-5.2 (weak)		Claude CLI (strong)		GLM-5.2 loop
Task	Seed (Int/Ext)	Evolved (Int/Ext)	Seed (Int/Ext)	Evolved (Int/Ext)	Final round	Best score	Tokens (M)
author_folders	$\times$ / $\times$	$\times$ / $\times$	✓/✓	✓/✓	14	0.00	2.70
budget_computation	$\times$ / $\times$	$\times$ / $\times$	$\times$ / $\times$	✓/✓	14	0.40	2.80
code_locating	✓/✓	✓/✓	✓/✓	✓/✓	0	1.00	0.09
contact_information	$\times$ / $\times$	$\times$ / $\times$	✓/✓	✓/✓	14	0.40	2.47
dataset_comparison	$\times$ / $\times$	✓/✓	✓/✓	✓/✓	12	1.00	2.61

Appendix

Table 13: Complete per-task Filesystem Stage 2 outcomes. Each cell under Seed and Evolved reports internal/external acceptance. Seed is the internally selected best round-0 candidate before workflow evolution under the corresponding campaign’s recorded path budget (five paths for GLM-5.2, three for Claude).

\mathrm{Int}\checkmark/\mathrm{Ext}\times

and

\mathrm{Int}\times/\mathrm{Ext}\checkmark

counts: GLM seed 2 and 0; GLM evolved 3 and 0; Claude seed 0 and 3; Claude evolved 0 and 0. Final round, best score, and tokens refer to the GLM-5.2 loop; best score is 1.00 for converged tasks and otherwise the maximum per-round internal score. ‡ Run ended early when a round subprocess exceeded a one-hour timeout; counted as unsolved.

Executor	Verdict	Seed	95% CI	Evolved	95% CI	Gained	$p$
GLM-5.2	external	6/30	[0.10, 0.37]	18/30	[0.42, 0.75]	12	<0.001
GLM-5.2	internal	8/30	[0.14, 0.44]	21/30	[0.52, 0.83]	13	<0.001
Claude	external	27/30	[0.74, 0.97]	30/30	[0.89, 1.00]	3	0.25
Claude	internal	24/30	[0.63, 0.90]	30/30	[0.89, 1.00]	6	0.031

Appendix

Table 14: Paired uncertainty of the Filesystem seed-to-evolved gains on the same 30 tasks: Wilson 95% intervals and the exact two-sided McNemar test on tasks whose outcome changed.

		Int/Ext
Executor	States	$\checkmark$ / $\checkmark$	$\checkmark$ / $\times$	$\times$ / $\checkmark$	$\times$ / $\times$	Agr.	Prec.	Rec.	Spec.
All retained round states
GLM-5.2 (all)	203	18	3	1	181	98.0	85.7	94.7	98.4
GLM-5.2 (distinct)	85	18	3	1	63	95.3	85.7	94.7	95.5
Claude (all)	55	30	0	16	9	70.9	100.0	65.2	100.0
Claude (distinct)	44	30	0	5	9	88.6	100.0	85.7	100.0

Appendix

Table 15: Held-out evaluation of the Stage 1 frozen evaluators on agent-produced Filesystem states, by executor and pool. “All” counts every retained state; “distinct” counts identical states once. Metrics in %.

Task	States	$\checkmark$ / $\checkmark$	$\checkmark$ / $\times$	$\times$ / $\times$
author_folders	2	1	0	1
budget_computation	4	1	0	3
code_locating	2	2	0	0
contact_information	4	1	0	3
dataset_comparison	10	2	0	8
debugging	2	1	1	0

Appendix

Table 16: Held-out evaluation per task: distinct retained states from both executors, by internal/external verdict.

Task	Diff-exposed constraint	Revision	Outcome
uppercase	Exact text and terminal-newline fidelity	$E^{(0)}\!\rightarrow E^{(1)}$ (1)	FP removed; final $\mathrm{Int}\times/\mathrm{Ext}\times$
debugging	Presence of the required fixed code pattern	$E^{(0)}\!\rightarrow E^{(1)}$ (3)	FP removed; final $\mathrm{Int}\times/\mathrm{Ext}\times$
structure_mirror	Missing nested directories and leaf placeholders	$E^{(0)}\!\rightarrow E^{(1)}$ (1)	FP removed; next state $\mathrm{Int}\checkmark/\mathrm{Ext}\checkmark$

Appendix

Table 17: Execution-state refinement on the three Filesystem false positives. Parentheses give Verifier-Evolver calls for each accepted revision. On uppercase and debugging , the final weak states are

\mathrm{Int}\times/\mathrm{Ext}\times

; on structure_mirror , the weak state reached under

E^{(1)}

passes both. A second structure_mirror round, triggered by a file-synchronization artifact, is excluded (Appendix C ).

Task	Provenance class	Evaluators	Final Int./Ext.	Terminal	Key result
uppercase	Primary refinement run	E0 $\to$ E1	$\times$ / $\times$	stabilized (r1)	Original false positive removed (exact-match check replaces .strip() tolerance).
debugging	Primary refinement run	E0 $\to$ E1	$\times$ / $\times$	stabilized (r1)	Original false positive removed (require-fix-present check via paired contrast).
structure_mirror	Primary refinement run	E0 $\to$ E1 (E2 excluded)	✓/✓ after r1	stabilized (r2)	Original false positive removed; the weak state under E1 is externally valid (hash 80aeae4e ). Round 2 was triggered by a file-synchronization artifact and is excluded.
structure_mirror	FP exposure (development case study)	E0 $\to$ E1	✓/ $\times$	semantic convergence	Extra-structure blind spot: E1 accepts a state with eleven extra placeholder files, which fails external verification.
structure_mirror	Separate controlled run (external pass)	E0 $\to$ E1 $\to$ E2	✓/✓	stabilized (r2)	Round 2 targets genuine extra structure (plus a synchronization artifact); final state hash 80aeae4e , identical to the primary run’s round-1 state.

Appendix

Table 18: Primary and case-study Filesystem Stage 3 trajectories. Primary refinement runs, the false-positive-exposure run, and the separate controlled run are separated by provenance class. The final weak states of the uppercase and debugging primary runs are rejected internally and externally (calibrated true negatives); the structure_mirror primary run’s state after round 1 passes both. The controlled run is not pooled with the primary runs.

Environment	Stage 1 evidence	Stage 2 evidence	Stage 3 disposition
PostgreSQL	18/21 evaluators successfully constructed over 120 calibration states (22 positive / 98 negative), with 115/120 internal–external agreement; 15/21 meet the Stage 2 admission threshold	15 matched pairs run to the configured limits; both executors solve three tasks. The strong executor succeeds at round 0 on all three, while the weak executor succeeds after rounds 11, 2, and 7; neither converges on the remaining 12 tasks (8 weak-executor runs end at the one-hour round timeout)	No agent-produced $\mathrm{Int}\checkmark/\mathrm{Ext}\times$ state across the 30 Stage 2 conditions; refinement not triggered
Playwright	Four evaluators over 88 generated cases; two correctness evaluators and two format-contract-only evaluators	Two valid pairs: both executors pass extraction_table ; both fail r1_arxiv	No eligible false positive; two tasks formally excluded
Notion	Two loop-native evaluators calibrate at agreement $1.0$ ; one additional evaluator is blocked by the observation interface	One historical matched pair on remove_osaka : strong condition $\mathrm{Int}\checkmark/\mathrm{Ext}\times$ , weak condition $\mathrm{Int}\times/\mathrm{Ext}\times$	No completed eligible refinement pair; outer refinement not run
GitHub	One full calibration at $0.857$ ; additional Stage 1 evaluators remain incomplete, divergent, or blocked by the observation interface	Corrected canonical-snapshot scoring yields one provenance-valid strong-executor $F^{+}$ on find_salient_file ; a corrected weak-executor matched run was not completed	No corrected $\mathrm{Int}\checkmark/\mathrm{Ext}\times$ pair; refinement not triggered

Appendix

Table 19: Cross-environment evidence. PostgreSQL provides the broadest cross-environment Stage 1/Stage 2 study; Playwright, Notion, and GitHub remain smaller or incomplete studies. These results are not pooled into the Filesystem task-level aggregate.

Campaign	Executor	Workflow and evolver	Seed Ext.	Final Ext.
Historical	Claude	Task understanding plus ReAct execution; only the ReAct prompt evolved by an interactive Claude Code outer loop	17/30	30/30
Primary	Claude	Five-node persistent multi-alternative workflow; headless Workflow Evolver	27/30	30/30
Primary	GLM-5.2	Same five-node representation and task-specific evaluators	6/30	18/30

Appendix

Table 20: Workflow representation and evolution results. The historical and primary Claude campaigns differ in representation, seed, source version, and evolver invocation, so the comparison is supporting rather than a controlled single-variable ablation.

Environment	Tasks	Usable $F^{+}$	$\mathrm{Int}\checkmark/\mathrm{Ext}\times$	Int $\times$
Filesystem	30	30	0	0
PostgreSQL	15	3	0	12
Playwright	2	1	0	1
Notion	1	0	1	0
GitHub	1	1	0	0
Total	49	35	1	13

Appendix

Table 21: Strong-executor (Claude) outcomes in the paper’s runs, one final run per task. A usable reference

F^{+}

is a state accepted both internally and externally (

\mathrm{Int}\checkmark/\mathrm{Ext}\checkmark

); Int

\times

counts states not accepted internally.

Trigger rule (weak vs. strong accepted state)	Flagged	FPs caught	Correct states flagged
Any difference	13/21	3/3	10/18
Structural difference (missing/extra/empty)	4/21	1/3	3/18
Missing structure only (strong has it, weak lacks it)	2/21	1/3	1/18
Content difference (excluding newline-only)	10/21	1/3	9/18
Any difference, newline-only differences ignored	12/21	2/3	10/18

Appendix

Table 22: Retrospective verifier-free triggers on the 21 internally accepted GLM-5.2 final states (3 false positives, 18 correct), each compared with the strong executor’s accepted state on the same task.

Analysis	Finding	Interpretation
PostgreSQL numeric type	An evaluator rejects valid Decimal values because it tests only int and float ; the external verifier accepts the state	Evaluator type mismatch rather than executor failure; corrected by a type-equivalence update before the final campaign.
PostgreSQL Stage 2 non-convergence	Neither executor converges on 12/15 Stage 2-eligible tasks despite evaluators that cleared the admission threshold	A usable fitness signal does not guarantee task completion; workflow search, executor capability, and interaction budget remain limiting.
Playwright observability	Turnstile exposes no stable transcript-visible success signal; another task lacks independently grounded factual truth	Formal exclusions rather than task failures.
Notion serialization	Agreement falls from $0.75$ to $0.50$ when people and date fields change serialization	A real evaluator disagreement compounded by backward-incompatible state representation.
GitHub scoring interface	Historical in-loop evaluator calls queried the live backend inside a short sandbox timeout, producing vacuous zero scores; canonical snapshot scoring restores valid internal evaluation	Historical non-convergence is excluded from performance claims; corrected strong-executor execution demonstrates feasibility, while the standardized matched pair remains incomplete.

Appendix

Table 23: Environment-specific diagnostic findings.

Experiment	Tasks/runs	Executor tokens	Evolver cost	Recorded wall	Qualification
Filesystem Stage 1	30	–	–	–	Interactive accounting not recorded.
Filesystem Stage 2, Claude	30	575,739	$1.74	5.3 task-hours	Primary strong campaign; executor tokens exclude prompt-cache reads.
Filesystem Stage 2, GLM-5.2	30	33,176,550	$31.24	43.5 task-hours	Primary weak campaign; tasks executed across parallel lanes.
Filesystem Stage 3	3	8,100,868	$3.49	$\approx 2.5$ h actual	Primary refinement runs; TSG and VE additionally use 37,031 and 27,235 Sonnet tokens.
PostgreSQL matched pairs	30 conditions	$\approx 46.9$ M	$41.88	$\approx 71$ h loop wall	Full 15-pair Stage 2 campaign over all eligible tasks.
Playwright matched pairs	4 conditions	1,062,274	$1.71	94 min	Two valid matched pairs.

Appendix

Table 24: Primary and qualified experimental compute accounting. Executor tokens and evolver cost are reported separately. Missing measurements are left unestimated; historical or incomplete pilot rows are explicitly labeled.

Phase	Technically enforced controls	Procedural restrictions	Audit evidence
Interactive Stage 1	No hard filesystem-level isolation is proven by retained artifacts	Workspace policy prohibits reading verifier source; external calibration occurs after evaluator construction	External fields appear only in post-construction calibration artifacts. Claims for this phase are procedural, not technical.
Automated Stage 2	Workflow-Evolver inputs are constructed only from internal scores, failed checks, traces, and alternative statistics; executors are restricted to isolated task environments	Prompts prohibit reference to external verifier logic	Source/input hashes, routing policies, and run manifests are retained.
Stage 3 TSG and VE	Packet-only temporary working directory outside the repository; shell, search, write, web, and task tools disabled; packet allowlists and forbidden-token scans; tool-access logging and run-aborting outside-access checks	Read access is additionally restricted by the packet contract	Per-round packet hashes and access-decision records; primary runs contain no information-boundary violation.
Post-hoc verification	Separate subprocess with provider credentials stripped; bounded output capture; invoked only after internal termination	External result cannot be routed back into a model role	External records contain invocation status and return code and are stored outside model packets.

Appendix

Table 25: External-verifier isolation by phase. Initial interactive Stage 1 relies on a documented procedural boundary; Stage 2 and especially Stage 3 add technically enforced input and tool restrictions.

Self-Evolving Agents

Self-evolving agents improve their skills, memory, tools, policies, or environments by learning from interaction and evaluation feedback.

See all 421 papers

Oct 8, 2026

Who Verifies the Verifier? Co-Evolving Inspectable Graders with Self-Improving Agents

We changed the agent: did it actually get better? Every self-improving agent loop answers this hundreds of times, and every answer comes from a verifier. On open-ended tasks none exists, so the loop is handed a hand-written rubric or a bare LLM judge grading output from a model like itself, inviting reward hacking and shared blind spots. We make the verifier the evolving object: an inspectable expression over small, mostly deterministic drawback detectors, synthesized from clustered failures, gated at birth, and selected for agreement with a ten-item anchored reference set plus consensus over unlabeled outputs, never for the agent's score. On MBPP+ it gains +0.21 held-out agreement over the hand-authored seed composition, on every seed, and ends ahead of the bare LLM judge it contains. One finding should change how co-evolved verifiers are validated: removing the anchor guards collapses the verifier into a vacuous always-pass grader, yet that collapsed verifier trains skills just as well. Downstream task score cannot certify a self-evolved verifier. Score does answer sufficiency, and there an evolved verifier can substitute: Double Ratchet, pairing the verifier with a lifecycle-managed skill loop, retains 88-110% of the lift that ground truth or a rubric buys the same loop, across code generation, enterprise text-to-SQL, and reference-free report generation. When evolved skills gamed the report rubric, an outer judge caught it and one added detector repaired it; the judge itself was wrong until given the task contract.

Xing Zhang, Guanghui Wang, Yanwei Cui +4

AWS Forward Deployed Engineering · HSBC Holdings Plc., HSBC Technology Center, China

Automated Evaluation LLM Agent Self-Improvement Automated Grading LLM-as-a-Judge

Ask PDF

Oct 2, 2026

VERSE: Verified Self-Evolving Optimizer for Agent Harnesses

Harness evolution improves an LLM agent's prompts, tools, and workflow, while the optimizer's own tools and procedures often remain fixed. We study whether an optimizer can improve another agent more effectively by also improving how it diagnoses failures, develops edits, and tests their effects. Two observations guide our design. In a controlled study, optimizer self-evolution fails to improve performance without execution-based verification, but achieves the best result of that study when verification is available. Across five executors, self-evolving optimizers build their own tools for failure analysis, verification, training audits, and workflow control. Motivated by these findings, we introduce VERSE, a Verified Self-Evolving optimizer for agent harnesses. VERSE lets the optimizer test draft edits, replay failures, and perturb suspected steps before submission, while tracking fixes and regressions across rounds. Using this feedback, the optimizer revises both the executor harness and its own prompts, skills, tools, hooks, and notes, while the weights of the optimizer and executor models stay fixed. Under a shared protocol with disjoint training, validation, and test tasks, VERSE improves all four evaluated harness optimizers on held-out SWE-rebench tasks and newer out-of-distribution tasks in five languages. Its best validation-selected harness reaches 42.3% and 37.7% accuracy, respectively, against 39.2% and 29.3% for the strongest baselines. Code is available at https://github.com/wzekai/VERSE.

Zekai Wang, Yingqiang Ge, Zekun Wang +6

MIT · Amazon

Agent Harness Optimization Self-Evolving Agents Software Engineering Agents LLM Agent Verification

Ask PDF

Oct 13, 2026

Verifier-Harness Co-Evolution for Training-Free Self-Improvement on Open-Ended Tasks

Training-free self-improvement lets an agent use feedback to revise the response it is about to return and to update the harness state that shapes how it handles later tasks. On open-ended tasks that feedback rarely comes from reference answers or executable tests, so the agent has to rely on a model-based verifier. When the verifier is wrong, the current response can get worse, and misleading experience can persist across the task stream. A verifier that never changes also grows stale as the agent's responses improve and task demands shift. We introduce Unified Co-Evolution (UCoE), a training-free method that evolves an executable verifier program together with the agent's harness state while all model parameters stay fixed. UCoE synthesizes candidate verifier programs and tests them on response contrasts, pairs of responses that differ in one declared property and have a known preferred side. It selects the candidate whose discrimination on its weakest contrast family is highest, a quantity we call driving capacity. The selected verifier diagnoses one weakness in each draft, guides a single repair, decides which response is served, and determines whether the accepted repair may enter the harness state. As tasks complete, their diagnostics seed new candidate verifiers, which must beat the current verifier on the same contrasts before adoption. Across six open-ended benchmarks, UCoE averages 59.00% preference against matched direct generation with Qwen3-8B and 57.19% with Qwen3.8-27B, margins of 9.00 and 7.19 percentage points. On three verifiable benchmarks, the same method raises the mean from 36.06% for direct generation to 38.48%. Code is available at https://github.com/tianyi0216/Unified-Co-Evolution.

Tianyi Xu, Yaolun Zhang, Yiran Wu +6

Oregon State University · AG2 AI · Pennsylvania State University +1

LLM Agent Self-Improvement LLM Self-Refinement Agent Harness Optimization

Ask PDF

Abstract

Self-Evolving Agents

Explore similar work

Who Verifies the Verifier? Co-Evolving Inspectable Graders with Self-Improving Agents

VERSE: Verified Self-Evolving Optimizer for Agent Harnesses

Verifier-Harness Co-Evolution for Training-Free Self-Improvement on Open-Ended Tasks