Verifier-Harness Co-Evolution for Training-Free Self-Improvement on Open-Ended Tasks

Tianyi Xu; Yaolun Zhang; Yiran Wu; Joe Nguyen; Shengyu Dai; Saba Moeinizade; Zhenwen Shao; Qingyun Wu

cs.AIOct 13, 2026

Verifier-Harness Co-Evolution for Training-Free Self-Improvement on Open-Ended Tasks

Authors: Tianyi Xu, Yaolun Zhang, Yiran Wu, Joe Nguyen, Shengyu Dai, Saba Moeinizade, Zhenwen Shao, Qingyun Wu,

Organizations: Oregon State University · AG2 AI · Pennsylvania State University · Johnson & Johnson

Abstract

Training-free self-improvement lets an agent use feedback to revise the response it is about to return and to update the harness state that shapes how it handles later tasks. On open-ended tasks that feedback rarely comes from reference answers or executable tests, so the agent has to rely on a model-based verifier. When the verifier is wrong, the current response can get worse, and misleading experience can persist across the task stream. A verifier that never changes also grows stale as the agent's responses improve and task demands shift. We introduce Unified Co-Evolution (UCoE), a training-free method that evolves an executable verifier program together with the agent's harness state while all model parameters stay fixed. UCoE synthesizes candidate verifier programs and tests them on response contrasts, pairs of responses that differ in one declared property and have a known preferred side. It selects the candidate whose discrimination on its weakest contrast family is highest, a quantity we call driving capacity. The selected verifier diagnoses one weakness in each draft, guides a single repair, decides which response is served, and determines whether the accepted repair may enter the harness state. As tasks complete, their diagnostics seed new candidate verifiers, which must beat the current verifier on the same contrasts before adoption. Across six open-ended benchmarks, UCoE averages 59.00% preference against matched direct generation with Qwen3-8B and 57.19% with Qwen3.8-27B, margins of 9.00 and 7.19 percentage points. On three verifiable benchmarks, the same method raises the mean from 36.06% for direct generation to 38.48%. Code is available at https://github.com/tianyi0216/Unified-Co-Evolution.

Figures & tables

Figure 1: Feedback for open-ended self-improvement. (a) On verifiable tasks, a known answer marks each response right or wrong and guides agent improvement. (b) On open-ended tasks there is no ground truth, so the agent must judge for itself which response is better, and a wrong judgment harms both the served response and the experience kept for later tasks. (c) UCoE mines quality cues from blinded response pairs, builds response contrasts in which a base response is preferred over an edited copy that lacks one property, tests candidate verifier programs on them, and uses the selected verifier to guide revision while experience from completed tasks updates it over time.

Figure 2: UCoE workflow. Top, the verifier side: (1) Synthesize proposes candidate verifier programs from the task requirements and from quality cues discovered in blinded response pairs. (2) Select edits calibration responses one property at a time to form response contrasts and deploys the candidate that is strongest on its weakest family. (3) Evolve combines the current verifier with diagnostics from completed tasks into a refined candidate, which is re-tested on the same contrasts before it can replace the current verifier. Bottom, the agent side: the fixed actor drafts with the current harness state, the deployed verifier guides one revision and chooses between draft and revision, and accepted repairs are distilled into experience that the updated harness state reuses on later tasks.

Qwen3-8B
Method	Arena- Hard	Writing Prompts	WildBench	Writing Bench	Multi Challenge	Refine Bench	Overall
Direct Generation	$50.00$	$50.00$	$50.00$	$50.00$	$50.00$	$50.00$	$50.00$
LLM as a Judge	$50.60$	$50.54$	$46.34$	$49.50$	$38.87$	$51.98$	$47.97$
RaR ( Gunjal et al., 2026 )	$51.20$	$50.54$	$48.54$	$51.90$	$38.48$	$51.26$	$48.65$
DREAM ( Anonymous, 2026b )	$45.60$	$43.90$	$43.12$	$52.60$	$31.05$	$53.78$	$45.01$
SpyRL ( Wang et al., 2026 )	$51.10$	$49.27$	$35.74$	$47.00$	$33.98$	$53.06$	$45.03$

Table 1: Open-ended preference against matched direct generation. Each entry is the tie-adjusted percentage of tasks on which a method’s response is preferred to the Direct Generation response of the same actor, as judged by Qwen3.5-397B-A17B in both presentation orders with ties given half credit, so

50

is parity and higher is better. All methods share the actor, the task order, and the fixed-model, one-revision protocol. Overall is the unweighted mean over the six benchmarks, and bold marks the best method in each column.

Method	IFBench	Hard-MATH	BigCodeBench-Hard	Overall
Direct Generation	$30.00$	$56.54$	$\mathbf{21.62}$	$36.06$
LLM as a Judge	$33.33$	$55.24$	$16.89$	$35.15$
RaR	$32.00$	$57.33$	$18.92$	$36.08$
DREAM	$28.67$	$\mathbf{59.42}$	$18.24$	$35.44$
SpyRL	$30.67$	$58.12$	$18.24$	$35.68$
UCoE	$\mathbf{34.67}$	$59.16$	$\mathbf{21.62}$	$\mathbf{38.48}$

Table 2: Verifiable tasks with Qwen3-8B. Accuracy for IFBench and Hard-MATH, Pass@1 for BigCodeBench-Hard.

Configuration	vs. direct %
UCoE	$\mathbf{58.30}$
Random initial verifier	$56.70$
Fixed verifier	$56.60$
Harness-state updates only	$50.80$
No memory updates	$56.70$

Table 3: Arena-Hard ablations with Qwen3-8B. Tie-adjusted preference against direct generation.

Figure 3: UCoE inside the Codex agent harness on SpreadsheetBench-2. Exact-match accuracy on a 75-task stream, overall and by task category, with GPT-5.6-luna as the actor.

Panel of Figure 4 — Figure 4: Mechanism diagnostics. (a) Share of repair targets per criterion group. Gray marks groups absent from the deployed verifier. (b) Two verifier updates in WritingPrompts. Each arrow runs from the current verifier to the candidate selected by driving capacity, in signed driving capacity against selection gain on frozen response pairs. (c) Reward margins of 360 Arena-Hard proposals under the initial and updated verifier. Dashed lines mark the acceptance margin, and colors give the external judgment for the 94 proposals the updated verifier selects. (d) Reuse lag of stored procedures at 381 Arena-Hard tasks, colored by outcome. The dashed line marks the switch to mathematics tasks.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Task type	Domain	Evaluated tasks
Arena-Hard	Open-ended	Challenging user requests to a general assistant	500
WritingPrompts	Open-ended	Story generation from writing prompts	1,024
WildBench	Open-ended	Real-user requests to a general assistant	1,024
WritingBench	Open-ended	Writing across diverse professional and creative domains	500
MultiChallenge	Open-ended	Multi-turn conversations with instruction constraints	128
RefineBench	Open-ended	Free-form refinement of an existing response	278

Appendix

Table 4: Benchmarks, task types, domains, and evaluated tasks.

Method	Feedback and parsed output	Revision decision
Direct Generation	No feedback call. The shared draft is served unchanged.	No revision and no procedural memory.
LLM as a Judge	A fixed prompt reads the task and response. It returns {"success": true\|false} and a separate actionable critique, {"critique": "..."} .	Binary reward; a revision is accepted only for a $0\!\rightarrow\!1$ change.
RaR	The task produces 7–15 criteria labeled Essential, Important, Optional, or Pitfall. Grading returns one Boolean per criterion as {"passes": [...]} ; failed criteria form the feedback.	Reward is the passed-criterion fraction. Acceptance requires a gain above $0.01$ .
DREAM	The task produces 3–6 weighted criterion nodes and a root threshold as {"nodes": [...], "root": {"rule": "weighted_mean", "tau": $t$ }} . The three lowest-scoring nodes form the feedback.	Reward is the weighted node mean. Acceptance requires a gain above $0.01$ .
SpyRL	A detector compares responses generated from the full task and a partially masked task. Its vote counts yield a role-normalized civilian advantage; the counts and advantage form the feedback.	The revision plays a second game. It is accepted when its advantage improves by more than $0.01$ .

Appendix

Table 5: Training-free baseline interfaces used in our experiments.

Component	Hyperparameter	Value
Decoding	Generation / revision temperature	$0.7$ / $0.7$
	Scoring / synthesis / distillation temperature	$0$ / $0$ / $0$
Verifier synthesis	Initial candidates	2
	New candidates per checkpoint	1
	Maximum new criteria per candidate	2
Response contrasts	Reference contrast families	3

Appendix

Table 6: Default UCoE hyperparameters on Arena-Hard, shared by the Qwen3-8B and Qwen3.8-27B runs.

Benchmark	Native metric	Scorer
Arena-Hard	Style-adjusted win rate against the o3-mini-2025-01-31 anchor, both orders, five-level $w_{3}$ weighting	GPT-4.1 (gpt-4.1-2025-04-14)
WritingPrompts	None
WildBench	Mean rating from 1 to 10 with the full-response template	Qwen3.5-397B-A17B
WritingBench	Critic score from 1 to 10	Critic-7B
MultiChallenge	Hidden-rubric conversation accuracy	Qwen3.5-397B-A17B
RefineBench	Checklist accuracy	Qwen3.5-397B-A17B with the benchmark checklist

Appendix

Table 7: Native metric and scorer for each benchmark.

	Qwen3-8B						Qwen3.8-27B
Method	Arena- Hard	Wild Bench	Writing Bench	Multi Challenge	Refine Bench	Overall	Arena- Hard	Wild Bench	Writing Bench	Multi Challenge	Refine Bench	Overall
	%	1 to 10	1 to 10	%	%	mean	%	1 to 10	1 to 10	%	%	mean
Direct Generation	$7.0$	$5.78$	$6.908$	$33.59$	$43.26$	$42.15$	$51.6$	$7.68$	$8.179$	$56.25$	$67.44$	$66.78$
LLM as a Judge	$8.1$	$5.70$	$6.931$	$\mathbf{39.84}$	$42.85$	$43.42$	$53.5$	$7.84$	$8.206$	$53.12$	$68.10$	$67.04$
RaR	$7.8$	$5.73$	$6.972$	$35.94$	$45.02$	$43.16$	$57.0$	$7.68$	$\mathbf{8.238}$	$51.56$	$67.95$	$67.14$
DREAM	$7.1$	$5.57$	$\mathbf{6.992}$	$29.69$	$44.16$	$41.31$	$57.1$	$7.76$	$8.223$	$\mathbf{57.81}$	$68.14$	$68.58$

Appendix

Table 8: Benchmark-specific metrics for Qwen3-8B and Qwen3.8-27B. Overall is the unweighted mean after scaling the two 1-to-10 ratings to 0 to 100. Bold marks column maxima.

Panel of Figure 5 — Figure 5: Cost and preference on six Qwen3-8B benchmarks. (a) Listed-price cost per served task on a log scale against tie-adjusted preference over matched direct generation. Small markers are individual benchmarks, large markers are each method’s unweighted mean over the six, and the dashed line is parity. (b) UCoE’s cost per task on each benchmark broken down by model, with Qwen3.6-35B-A3B as the verifier model. Costs use list prices and exclude final evaluation.

	Qwen3-14B	GPT-5.4		GPT-5.4-mini
Method	Arena- Hard	Arena- Hard	Multi Challenge	Arena- Hard	WildBench	Refine Bench	Multi Challenge
Direct Generation	$50.00$	$50.00$	$50.00$	$50.00$	$50.00$	$50.00$	$50.00$
LLM as a Judge	$47.40$	$54.50$	$\mathbf{56.64}$	$52.70$	$54.39$	$50.00$	$50.59$
RaR	$45.80$	$49.20$	$53.12$	$50.80$	$58.74$	$50.72$	$49.22$
DREAM	$47.60$	$54.70$	$\mathbf{56.64}$	$52.30$	$51.61$	$49.10$	$\mathbf{53.32}$
SpyRL	$46.90$	$54.60$	$48.83$	$51.80$	$52.69$	$49.46$	$47.27$

Appendix

Table 9: Transfer to other actors: preference against matched direct generation. Same protocol and judge as Table 1 . Qwen3-14B uses Qwen3.6-35B-A3B as the verifier model, and both GPT actors use GPT-5.4. MultiChallenge is the 128-conversation development set. Bold marks the best method in each column.

Factor	Value	Worst $J$	Accepted	Updates	vs. direct	Default vs. variant
Decision margin $\epsilon$	$0$	$0.500$	$195$	$0$	$59.40$	$49.50$
	$0.01$	$0.417$	$123$	$2$	$57.90$	$50.00$
	$0.05$	$0.417$	$107$	$0$	$57.00$	$50.60$
Initial candidate count $K$	$2$	$0.417$	$123$	$2$	$57.90$	$50.00$
	$4$	$0.417$	$123$	$2$	$57.90$	$50.00$
	$8$	$0.417$	$123$	$2$	$57.90$	$50.00$

Appendix

Table 10: Arena-Hard sensitivity with Qwen3-8B. Each row changes one factor from

\epsilon=0.01

K=2

, and revision temperature

0.7

Qwen3-8B
Method	Arena- Hard	Writing Prompts	WildBench	Writing Bench	Multi Challenge	Refine Bench
Direct Generation	$50.00$	$50.00$	$50.00$	$50.00$	$50.00$	$50.00$
LLM as a Judge	$50.60$ [-1pt] $[47.17,\,53.80]$	$50.54$ [-1pt] $[50.20,\,50.93]$	$46.34$ [-1pt] $[42.72,\,50.00]$	$49.50$ [-1pt] $[46.80,\,52.20]$	$38.87$ [-1pt] $[31.84,\,45.90]$	$51.98$ [-1pt] $[48.38,\,55.58]$
RaR	$51.20$ [-1pt] $[48.44,\,54.50]$	$50.54$ [-1pt] $[50.24,\,50.88]$	$48.54$ [-1pt] $[45.85,\,51.42]$	$51.90$ [-1pt] $[49.00,\,54.90]$	$38.48$ [-1pt] $[31.64,\,45.51]$	$51.26$ [-1pt] $[46.22,\,56.29]$
DREAM	$45.60$ [-1pt] $[40.40,\,50.98]$	$43.90$ [-1pt] $[39.94,\,47.51]$	$43.12$ [-1pt] $[39.99,\,46.34]$	$52.60$ [-1pt] $[49.70,\,55.50]$	$31.05$ [-1pt] $[24.80,\,37.70]$	$53.78$ [-1pt] $[48.92,\,58.63]$
SpyRL	$51.10$ [-1pt] $[47.16,\,54.71]$	$49.27$ [-1pt] $[45.61,\,52.69]$	$35.74$ [-1pt] $[31.93,\,39.50]$	$47.00$ [-1pt] $[44.30,\,49.70]$	$33.98$ [-1pt] $[27.73,\,40.62]$	$53.06$ [-1pt] $[48.74,\,57.37]$

Appendix

Table 11: Open-ended preference with pointwise 95% intervals. Same entries as Table 1 , each with its bootstrap interval in brackets. Bold marks the best method in each column.

Actor	Decision	$\boldsymbol{n}$	Better:Worse:Tie	Critic preference (%)	Mean Critic change
Qwen3-8B	Accepted	158	99:33:26	70.89	$+0.228$
Qwen3-8B	Rejected	309	132:123:54	51.46	$-0.171$
Qwen3.8-27B	Accepted	95	54:17:24	69.47	$+0.187$
Qwen3.8-27B	Rejected	375	143:129:103	51.87	$-0.036$

Appendix

Table 12: Recorded UCoE decisions on natural WritingBench revisions. Critic preference compares each proposal with its matched draft.

Stage	Information available	Output	Information unavailable
Initial verifier synthesis	Public task-type description; definitions of the available criteria, including any relation criterion mined from blinded response pairs	One verifier program: criteria, at most two new criteria with anchors, one bounded aggregation expression	Task stream, drafts, benchmark answers, final-evaluator feedback
Contrast comparison	Calibration prompts with a base response and one targeted degradation; candidate verifiers	Per-pair success indicators; worst-family centered accuracy; the selected candidate	Transformation names and preference directions (hidden from the verifier model); benchmark answers; final evaluator
Criterion scoring	User request, one response, the definitions of one criterion block	One score in $[0,1]$ per criterion	Other responses, the aggregation, benchmark answers, final evaluator
Localized diagnosis	User request, draft, fixed criterion scores, the weakest criterion and its definition	A diagnosis grounded in the draft and one repair instruction	Any rescoring, benchmark answers, final evaluator
Response revision	User request, draft, the diagnosis and repair instruction, at most two retrieved procedures	One revised response	Criterion scores, the current verifier program, benchmark answers, final evaluator
Checkpoint candidate synthesis	Definitions of the current verifier; aggregate diagnostics of completed tasks (acceptance count, weak-criterion counts, procedural-memory size)	One candidate verifier that keeps the current criteria and adds one or two new criteria	Individual drafts or scores, held-out results, benchmark answers, final evaluator

Appendix

Table 13: Information available to and produced by each UCoE stage.

LLM Agent Self-Improvement

Large language model agent self-improvement studies how agents use experience and feedback to improve their behavior, tools, and workflows.

See all 273 papers

Who Verifies the Verifier? Co-Evolving Inspectable Graders with Self-Improving Agents
Oct 8, 2026Xing Zhang, Guanghui Wang, Yanwei Cui +4Automated Evaluation LLM Agent Self-Improvement
AWS Forward Deployed Engineering · HSBC Holdings Plc., HSBC Technology Center, China
VERSE: Verified Self-Evolving Optimizer for Agent Harnesses
Oct 2, 2026Zekai Wang, Yingqiang Ge, Zekun Wang +6Agent Harness Optimization Self-Evolving Agents
MIT · Amazon
VeriEvolve: Evaluator-First Evolution for Tool-Using Agents
Oct 21, 2026Yueshen Li, Kamer Ali Yuksel, Gokhan Tur +2Self-Evolving Agents Tool-Use Evaluation
University of Illinois Urbana-Champaign · aiXplain

Oct 8, 2026J/K move · Enter open · S save

Who Verifies the Verifier? Co-Evolving Inspectable Graders with Self-Improving Agents

Xing Zhang, Guanghui Wang, Yanwei Cui, Ziyuan Li+3

AWS Forward Deployed Engineering · HSBC Holdings Plc., HSBC Technology Center, China

Automated Evaluation LLM Agent Self-Improvement Automated Grading LLM-as-a-Judge

Open paper AskPDF arXiv

We changed the agent: did it actually get better? Every self-improving agent loop answers this hundreds of times, and every answer comes from a verifier. On open-ended tasks none exists, so the loop is handed a hand-written rubric or a bare LLM judge grading output from a model like itself, inviting reward hacking and shared blind spots. We make the verifier the evolving object: an inspectable expression over small, mostly deterministic drawback detectors, synthesized from clustered failures, gated at birth, and selected for agreement with a ten-item anchored reference set plus consensus over unlabeled outputs, never for the agent's score. On MBPP+ it gains +0.21 held-out agreement over the hand-authored seed composition, on every seed, and ends ahead of the bare LLM judge it contains. One finding should change how co-evolved verifiers are validated: removing the anchor guards collapses the verifier into a vacuous always-pass grader, yet that collapsed verifier trains skills just as well. Downstream task score cannot certify a self-evolved verifier. Score does answer sufficiency, and there an evolved verifier can substitute: Double Ratchet, pairing the verifier with a lifecycle-managed skill loop, retains 88-110% of the lift that ground truth or a rubric buys the same loop, across code generation, enterprise text-to-SQL, and reference-free report generation. When evolved skills gamed the report rubric, an outer judge caught it and one added detector repaired it; the judge itself was wrong until given the task contract.

Abstract

LLM Agent Self-Improvement

Explore similar work

Who Verifies the Verifier? Co-Evolving Inspectable Graders with Self-Improving Agents

VERSE: Verified Self-Evolving Optimizer for Agent Harnesses

VeriEvolve: Evaluator-First Evolution for Tool-Using Agents