VeRO: A Harness for Agents to Optimize Agents

Varun Ursekar; Apaar Shanker; Veronica Chatrath; Yuan Xue; Samuel Marc Denton

cs.AIFeb 25, 2026

VeRO: A Harness for Agents to Optimize Agents

Authors: Varun Ursekar, Apaar Shanker, Veronica Chatrath, Yuan Xue, Samuel Marc Denton

Organizations: Scale AI.

Abstract

An important emerging application of coding agents is agent harness optimization: the iterative improvement of a target agent by editing and evaluating its code. Despite its relevance, the community lacks a systematic understanding of coding agent performance on this task. Harness optimization differs from conventional software engineering: agent harnesses interleave deterministic code with stochastic LLM completions, requiring structured capture of both intermediate execution traces and downstream outcomes. To address these challenges, we introduce (1) VeRO (Versioning, Rewards, and Observations), an outer harness that provides versioned snapshots, budget-controlled evaluation, and structured execution traces of target harnesses, and (2) VeRO-Bench, a benchmark suite of target agents and tasks with reference evaluation procedures. Using VeRO, we conduct an empirical study comparing optimizers across tasks and analyzing which modifications reliably improve target agent harnesses. We release VeRO to support research on agent optimization as a core capability for coding agents. Code is available at https://github.com/scaleapi/vero.

Figures & tables

Figure 1: VeRO system architecture. Top (orange): example optimization trajectory. Bottom (green): system components. VeRO enforces versioning, reproducible execution, and controlled feedback, enabling systematic comparison of optimizers for agent optimization.

Task	Domain	Tr/Val/Te	Initial Tools
GAIA	Multi-step	50/87/—	—
GPQA	Science QA	98/—/100	—
MATH	Math	59/60/486	—
TAU-Bench Retail	Tool use	100/20/115	Original
SimpleQA	Factual QA	46/45/80	Wikipedia

Table 1: Benchmark study tasks with respective Train / Validation / Test data splits and initial base agent tools .

Optimizer	Variant	Model	GAIA	GPQA	MATH	Retail	SimpleQA	Avg.
Baseline	—	—	0.07	0.60	0.87	0.38	0.61	0.50
Claude Code	Pure	Sonnet	0.13 (0.29)	0.58 (0.63)	0.88 ( 0.90 )	0.43 (0.46)	0.65 (0.68)	0.53 (0.59)
Claude Code	VeRO Tools	Sonnet	0.14 (0.21)	0.64 ( 0.71 )	0.88 ( 0.90 )	0.39 (0.43)	0.67 (0.71)	0.55 (0.59)
VeRO-Agent	Default	Sonnet	0.26 ( 0.30 )	0.64 (0.65)	0.86 (0.87)	0.55 (0.66)	0.73 (0.76)	0.61 ( 0.65 )
VeRO-Agent	Orchestrator	Opus	0.18 (0.18)	0.62 (0.66)	0.88 (0.88)	0.55 (0.57)	0.74 ( 0.86 )	0.59 (0.63)
VeRO-Agent	Orchestrator	Sonnet	0.16 (0.20)	0.62 (0.65)	0.87 (0.88)	0.51 ( 0.72 )	0.71 (0.72)	0.57 (0.63)

Table 2: Benchmark Suite Results. Each cell reports the average best score over

N=3

iterations, with maxima in parentheses. The baseline shows average initial-commit performance over

t\times 3=21

runs. GPT-4.1 mini is used as the target agent model throughout.

Task	Optimizer	Target Model	Init.	Final	$\Delta$
GAIA	GPT-5.2	GPT-4.1	0.15	0.11	-0.03
	GPT-5.2	Gemini-Flash	0.26	0.24	-0.02
	GPT-5.2	Qwen3-30B	0.11	0.03	-0.08
	GPT-5.2	Qwen3-4B	0.08	0.07	-0.01
	Sonnet-4.5	GPT-4.1	0.15	0.22	+0.07
	Sonnet-4.5	Gemini-Flash	0.26	0.21	-0.06

Table 3: Robustness Study Results. Evaluation results on the best commits found by Orchestrator variants in Table 2 . Results show performance of various model checkpoints with target agents that were optimized with GPT-4.1 mini as the target model. Coverage is uneven across tasks by design: due to computational constraints we report complete optimizer

\times

target-model coverage on GAIA and extend to other tasks only for certain pairs. Deltas are computed on unrounded scores.

Panel of Figure 2 — Figure 2: Case study results highlighting optimization outcomes across optimizer instructions types for FACTS, GAIA, and SimpleQA. Orange circles: Pawn agent; green triangles: Knight agent. Dashed lines: baseline accuracies.

Figure 3: Accuracy ratio (iteration accuracy/baseline) across instruction types, aggregated over all three evaluation datasets (FACTS, GAIA, SimpleQA) for Pawn and Knight agents.

Figure 4: We plot the probability of changes made between successive evaluations in a trajectory falling into one of several types (e.g. prompt, tool, workflow changes) across all trajectories conducted using VeRO-Agent as the optimizer. Bold numbers capture the entropy of the probability distribution defined by each bar.

Interface	Budget	Pass rate	Error rate
Baseline	—	30.3% (27/89)	46.1% (41/89)
Tools	89	33.7% (30/89)	36.0% (32/89)
Filesystem	178	30.3% (27/89)	30.3% (27/89)
Tools	178	37.1% (33/89)	33.7% (30/89)

Table 4: TerminalBench-2 case study results across three optimizer runs and two interface modes. Pass rate = fraction of the 89 tasks for which the verifier returned a positive reward. Error rate = fraction of tasks in which the target agent crashed or timed out before reaching the verifier; the remainder of non-passing tasks complete without crashing but receive a zero reward.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Toolset	Capabilities	Core Abstractions
File System Tools
FileRead	Read files with line ranges and pagination	Filesystem
FileWrite	Write/edit files with search-replace; auto-commits changes	Filesystem , GitWorktree
BashTool	Restricted shell commands for filesystem search ( ls , find , tree , pwd )	Filesystem
Grep	Regex search using ripgrep with multiple output modes	Filesystem
Git Tools

Appendix

Table 5: Toolsets available in the VeRO harness. Each toolset is tied to core VeRO abstractions.

Agent	Variant	Toolset Configuration
VeRO-Agent	Default	Base toolset: FileRead , FileWrite , Grep , BashTool , GitViewer , GitControl , ExperimentRunner , ExperimentViewer , DatasetViewer , WebSearch , WebFetch , TodoList , think , ContextStore (Cookbook). Sub-agent delegation enabled.
VeRO-Agent	Orchestrator	Orchestrator tools restricted to: ExperimentRunner , GitControl , GitViewer , ContextStore , TodoList , think . Removes file read/write tools from orchestrator to force sub-agent usage. Sub-agents retain full toolset; orchestrator delegates code tasks.
VeRO-Agent	Resources-Only	Removes FileWrite ; adds ResourceControl . Optimizer can only modify @resource -decorated functions (prompts, tool descriptions, parameters). No direct file writes or shell mutations.
Claude Code	VeRO Tools	Base: Claude Code native tools (file editing, Bash, web search). Adds ExperimentRunner , ExperimentViewer , DatasetViewer , ContextStore via MCP. Auto-commit hook instruments file writes for observability.
Claude Code	Pure	Unmodified Claude Code. No VeRO tools exposed. Optimizer must invoke the target agent via shell and parse outputs manually. No auto-commit instrumentation.

Appendix

Table 6: Optimizer configurations evaluated. Toolset changes are relative to the base setting. The Cookbook is a pre-loaded ContextStore containing agent design patterns.

Figure 5: Optimizer performance across tasks. Each grouped bar shows the average lift (best score minus initial score) for each optimizer configuration. Red horizontal lines indicate the mean lift per task across all configurations.

Figure 6: We plot the probability of changes made between successive evaluations in a trajectory fall into one of several types (e.g. prompt, tool, workflow changes) across all trajectories per task. Bold numbers capture the entropy of the probability distribution defined by each bar.

Figure 7: Mean normalized phase at which the optimal commit is discovered across trajectories. We average over all VeRO trajectories conducted for the benchmark study.

Figure 8: Budget ablation on all five benchmark-study tasks ( $N=3$ iterations per $(B,\text{task})$ pair). Top row: tool-use tasks (GAIA, SimpleQA, TAU-Bench Retail) where holdout score is budget-sensitive. Bottom row: reasoning-heavy tasks (GPQA, MATH) where holdout score is flat across budget levels. Error bars: $\pm 1$ standard deviation across iterations. Y-axis scales differ across panels to highlight within-task variation.

Figure 9: Top sub-types for 3 types of modifications optimizers make to targets.

Figure 10: Entropy of change types as a function of optimization phase for different agent-variant-model triples.

Figure 11: UMAP-projected text embeddings of Git diffs of commits made in each optimization trajectory with respect to an empty commit. Initial and final commits show natural clusters.

Figure 12: An example trajectory of changes made by VeRO-Agent Orchestrator with Sonnet 4.5 on the MATH task.

Figure 13: An example trajectory of changes made by VeRO-Agent Resources Only with Sonnet 4.5 on the GPQA task.

Figure 14: An example trajectory of changes made by VeRO-Agent with Sonnet 4.5 on the GAIA task.

Figure 15: An example trajectory of changes made by VeRO-Agent Orchestrator with Opus 4.5 on the SimpleQA task.

Agent	GAIA	FACTS	SimpleQA
Pawn	19.5%	54.7%	68.9%
Knight	43.7%	64.7%	84.4%

Appendix

Table 7: Baseline accuracy for case study agents.

Agent	Template	Mean	Std Dev
Pawn	Cookbook+Reasoning	57.9%	6.79%
	Minimal	53.7%	5.15%
	Tool-Centric	55.8%	0.93%
	Evidence-Based	53.3%	1.05%
Knight	Cookbook+Reasoning	66.6%	2.08%
	Minimal	68.7%	1.45%

Appendix

Table 8: Mean accuracy and standard deviation on FACTS across templates (4 iterations each).

Template	Knight (s)	Pawn (s)
Cookbook+Reasoning	30.6	14.5
Minimal	56.0	20.4
Tool-Centric	56.6	32.8
Evidence-Based	26.2	12.6

Appendix

Table 9: Mean runtime (seconds) per sample by template.

Aspect	Pawn (Minimal)	Knight (Sophisticated)
Tools
Total tools	4	6
Web search	Basic (3 results, no retries)	Enhanced (5+ results, retry logic)
Wikipedia	✗	✓
Reflect tool	✗	✓ (LLM-powered self-evaluation)
File reading	Text only (.txt, .md, .json, .csv)	Complex formats (PDF, Excel, Word, ZIP)

Appendix

Table 10: Architectural comparison of Pawn and Knight agents.

Feature	Cookbook+Reasoning	Minimal	Tool-Centric	Evidence-Based
Lines of code	398	140	396	503
Cookbook access	✓	✗	✓	✓
Tool docstring guidance	✗	✗	✓ (detailed)	Tiered hierarchy
LLM-integrated tool examples	✗	✗	✓	✗ (discouraged)
Planning examples	✓	✗	✓	✓
Reasoning patterns	12 patterns	✗	✓	Condensed

Appendix

Table 11: Feature comparison of optimizer instruction templates.

Template	Iter	Worktree	FACTS	GAIA	SimpleQA	Time (s)
Baseline	–	–	64.72%	43.68%	84.44%	26.6
Cookbook+Reasoning	1	plain_pond	65.06%	43.68%	84.44%	27.2
	2	noisy_block	65.73%	49.43%	84.44%	26.3
	3	solitary_sky	65.84%	45.98%	86.67%	26.3
	4	quiet_sunset	69.66%	48.28%	84.44%	45.7
Minimal	1	yellow_bar	67.98%	50.57%	84.44%	39.7

Appendix

Table 12: Knight agent: per-iteration results. Bold indicates best result per template. ∗ Evaluation crashed

Template	Iter	Worktree	FACTS	GAIA	SimpleQA	Time (s)
Baseline	–	–	54.72%	19.54%	68.89%	9.0
Cookbook+Reasoning	1	proud_disk	62.36%	29.89%	68.89%	14.2
	2	quiet_glitter	54.83%	25.29%	71.11%	13.7
	3	black_lake	49.33%	25.29%	51.11%	10.7
	4	polished_band	65.17%	31.03%	82.22%	21.9
Minimal	1	snowy_leaf	48.76%	17.24%	51.11%	17.6

Appendix

Table 13: Pawn agent: per-iteration results. Bold indicates best per template; red indicates regression below baseline.

Agent	Dataset	Accuracy	Correct/Total	Avg Time (s)	Errors
Knight	FACTS	64.72%	576/890	29.2	60
	GAIA	43.68%	38/87	42.1	6
	SimpleQA	84.44%	38/45	8.5	0
Pawn	FACTS	54.72%	487/890	8.1	17
	GAIA	19.54%	17/87	13.6	3
	SimpleQA	68.89%	31/45	5.1	0

Appendix

Table 14: Baseline performance for case study agents before optimization.

Run	Turns	Wallclock	Cost	Input tokens	Commits	Budget used
Tools - $B{=}89$	87	2h38m	$2.41	2.3M	7	89/89
Filesystem - $B{=}178$	101	6h54m	$2.97	4.3M	15	178/178
Tools - $B{=}178$	133	12h38m	$3.13	3.5M	11	129/178

Appendix

Table 15: Per-run optimizer-side statistics for the three TerminalBench-2 runs. “Budget used” = proportion of sample budget consumed.

Error category	Baseline	Tools - $B{=}89$	Filesystem - $B{=}178$	Tools - $B{=}178$
AgentTimeoutError	16	19	23	17
ContextLengthExceededError	13	11	1	2
AttributeError	11	0	0	11
VerifierTimeoutError	1	0	1	0
ExecutionError	0	2	0	0
Other ( Internal , Runtime )	0	0	2	0

Appendix

Table 16: Per-error-category breakdown across the baseline and the three optimized commits on the full 89-task test split. “Failed (no error)” counts tasks that completed without crashing but received a zero reward from the verifier.

Run	File	Lines $+$	Lines $-$
Tools - $B{=}89$	terminus_kira/terminus_kira.py	27	3
	prompt-templates/terminus-kira.txt	7	0
Filesystem - $B{=}178$	terminus_kira/terminus_kira.py	58	18
	prompt-templates/terminus-kira.txt	23	6
	analyze_traces.py (new)	155	0
	detailed_analysis.py (new)	139	0

Appendix

Table 17: File-level diff statistics of each run’s final commit relative to the unoptimized Terminus-KIRA baseline. Lines are newline-delimited. Newly-created files are marked.

Run	Pass	Retained	New passes	Regressions
Tools - $B{=}89$	30	20/27	10	7
Filesystem - $B{=}178$	27	21/27	6	6
Tools - $B{=}178$	33	21/27	12	6

Appendix

Table 18: Per-sample outcome breakdown of each run’s final commit relative to the unoptimized Terminus-KIRA baseline (27 passing tasks out of 89). Retained counts baseline-passing tasks that remain passing in the optimized commit; New passes are previously-failing tasks newly passing; Regressions are baseline-passing tasks newly failing. Net pass-rate change = New passes

-

Regressions.

Figure 16: Sample-state transition matrices for the three optimization runs (all 89 task instances). Rows are the sample’s category under the unoptimized baseline; columns are its category under the run’s final commit. Cell value = count; color intensity uses a $\sqrt{\cdot}$ scale so the large diagonal values do not wash out off-diagonal transitions. Categories: Pass (positive verifier reward), Fail (no error, zero reward), four named error types, and Other ( ExecutionError , RuntimeError , InternalError , etc.).

Agent Harness Optimization

Agent harness optimization improves the prompts, tools, memory, workflows, and control code that shape how AI agents act.

See all 183 papers

Oct 2, 2026

VERSE: Verified Self-Evolving Optimizer for Agent Harnesses

Harness evolution improves an LLM agent's prompts, tools, and workflow, while the optimizer's own tools and procedures often remain fixed. We study whether an optimizer can improve another agent more effectively by also improving how it diagnoses failures, develops edits, and tests their effects. Two observations guide our design. In a controlled study, optimizer self-evolution fails to improve performance without execution-based verification, but achieves the best result of that study when verification is available. Across five executors, self-evolving optimizers build their own tools for failure analysis, verification, training audits, and workflow control. Motivated by these findings, we introduce VERSE, a Verified Self-Evolving optimizer for agent harnesses. VERSE lets the optimizer test draft edits, replay failures, and perturb suspected steps before submission, while tracking fixes and regressions across rounds. Using this feedback, the optimizer revises both the executor harness and its own prompts, skills, tools, hooks, and notes, while the weights of the optimizer and executor models stay fixed. Under a shared protocol with disjoint training, validation, and test tasks, VERSE improves all four evaluated harness optimizers on held-out SWE-rebench tasks and newer out-of-distribution tasks in five languages. Its best validation-selected harness reaches 42.3% and 37.7% accuracy, respectively, against 39.2% and 29.3% for the strongest baselines. Code is available at https://github.com/wzekai/VERSE.

Zekai Wang, Yingqiang Ge, Zekun Wang +6

MIT · Amazon

Agent Harness Optimization Self-Evolving Agents Software Engineering Agents LLM Agent Verification

Ask PDF

Feb 23, 2026

ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?

We introduce ISO-Bench, a benchmark for coding agents to test their capabilities on real-world inference optimization tasks. These tasks were taken from vLLM and SGLang, two of the most popular LLM serving frameworks. Each task provides an agent with a codebase and bottleneck description, whereby the agent must produce an optimization patch evaluated against expert human solutions. We curated 54 tasks from merged pull requests with measurable performance improvements. While existing benchmarks heavily use runtime-based metrics, such approaches can be gamed to pass tests without capturing the actual intent of the code changes. Therefore, we combine both hard (execution-based) and soft (LLM-based) metrics to show that both are necessary for complete evaluation. While evaluating both closed and open-source coding agents, we find no single agent dominates across codebases. Surprisingly, agents often identify correct bottlenecks but fail to execute working solutions. We also show that agents with identical underlying models differ substantially, suggesting scaffolding is as important as the model.

Ayush Nangia, Shikhar Mishra, Aman Gokrani +1

Coding Agent Benchmarks AI Coding Agents AI Agent Benchmarks Code Generation Evaluation Code Optimization LLM Inference Efficiency+1

Ask PDF

May 18, 2026

Code as Agent Harness

Recent large language models (LLMs) have demonstrated strong capabilities in understanding and generating code, from competitive programming to repository-level software engineering. In emerging agentic systems, code is no longer only a target output. It increasingly serves as an operational substrate for agent reasoning, acting, environment modeling, and execution-based verification. We frame this shift through the lens of agent harnesses and introduce code as agent harness: a unified view that centers code as the basis for agent infrastructure. To systematically study this perspective, we organize the survey around three connected layers. First, we study the harness interface, where code connects agents to reasoning, action, and environment modeling. Second, we examine harness mechanisms: planning, memory, and tool use for long-horizon execution, together with feedback-driven control and optimization that make harness reliable and adaptive. Third, we discuss scaling the harness from single-agent systems to multi-agent settings, where shared code artifacts support multi-agent coordination, review, and verification. Across these layers, we summarize representative methods and practical applications of code as agent harness, spanning coding assistants, GUI/OS automation, embodied agents, scientific discovery, personalization and recommendation, DevOps, and enterprise workflows. We further outline open challenges for harness engineering, including evaluation beyond final task success, verification under incomplete feedback, regression-free harness improvement, consistent shared state across multiple agents, human oversight for safety-critical actions, and extensions to multimodal environments. By centering code as the harness of agentic AI, this survey provides a unified roadmap toward executable, verifiable, and stateful AI agent systems.

Xuying Ning, Katherine Tieu, Dongqi Fu +39

LLM Agent Orchestration Long-Horizon LLM Agents LLM Agent Harnesses LLM Agent Reliability

Ask PDF

Abstract

Agent Harness Optimization

Explore similar work

VERSE: Verified Self-Evolving Optimizer for Agent Harnesses

ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?

Code as Agent Harness