Beneath the Surface: What We Overlook When We Talk About Agent Evolution

Xiaobo Wang; Zimu Wang; Siyu Ye; Jiaqi Li; Min Tang; Jun Bai; Yuling Shi; Ruobing Wang

cs.AINov 5, 2026

Beneath the Surface: What We Overlook When We Talk About Agent Evolution

Authors: Xiaobo Wang, Zimu Wang, Siyu Ye, Jiaqi Li, Min Tang, Jun Bai, Yuling Shi, Ruobing Wang,

Organizations: University of Science and Technology of China · Beijing Institute for General Artificial Intelligence · University of California, Berkeley · Simple Agent Lab ·

Abstract

Agent harness evolution enables recursive self-improvement (RSI) without updating model weights. We examine whether higher benchmark scores yield reliable, efficient, and transferable improvement. We evaluate four methods across harnesses, general and coding tasks. Repeated evaluations, controlled agent expansion, component interventions, and admission replays reveal recurring patterns. All sixteen evolved agents gain previously uncovered tasks while losing previously covered ones. In one general-task case, reliability rises by 5.2 points despite losing ten previously covered tasks. Most reliability gains also increase deployment cost. Specific changes usually outperform generic reminders, but their benefits can change when combined. The highest-scoring agent does not yield the largest mean improvement in any of our four pools, and stricter admission can discard useful alternatives. Transfer selectively preserves gains: coverage usually falls on the general-task target across both harness types and paradigms, but rises on the coding target with the outer-layer-only harness. These findings motivate preservation-aware evaluation, tested decision rules, and explicit applicability conditions. As a constructive follow-up, EvoAudit turns these diagnostics into revisions of select, analyze, mutate, and gate, achieving the best performance.

Figures & tables

Figure 1: The agent harness evolution procedure (left) and our analyses of Phenomenology, Generalization, and Mechanisms (right).

Benchmark	Harness	Variant	Generation	$\mathrm{Avg}@8\pm\mathrm{Std}$	$\Delta\mathrm{Avg}$	$\mathrm{Pass}@8$	$\Delta\mathrm{Pass}$
$\tau^{3}$ -Banking	MiniSWE	Seed	0	$27.4\!\pm\!3.1$	$-$	$54.6$	$-$
		AHE	2	$32.9\!\pm\!3.9$	$+5.4$	$66.0$	$+11.3$
		HyperAgents	9	$32.6\!\pm\!4.1$	$+5.2$	$57.7$	$+3.1$
		A-Evolve	1	$26.3\!\pm\!2.5$	$-1.2$	$55.7$	$+1.0$
		GEPA	11	$28.2\!\pm\!2.3$	$+0.8$	$58.8$	$+4.1$
	Codex	Seed	0	$30.3\!\pm\!4.3$	$-$	$56.7$	$-$

Table 1: Reliability and empirical coverage across eight passes. Generation 0 is the Seed; later entries give the selected Champion’s Generation. Scores are percentages and deltas are percentage points.

Behavior category	Observable distinction	Increase	Decrease
State and dependency management	Preserve inputs, respect action dependencies, and retain a self-contained executable result.	7	1
Exact grounding	Match product conditions, dates, or output characters to the evidence and task requirements.	1	4
Completion and handoff	Finish all requested work and deliver the required final state or actual handoff.	2	3
Tool discovery and recovery	Use an available interface and recover from tool errors using the current state.	2	1
Implementation debugging	Resolve implementation errors and deliver a working program within the execution window.	1	0

Table 2: Primary behavior categories in the 22 selected task–lineage comparisons. Increase and decrease refer to eight-pass success counts.

Figure 2: Task-level Seed-to-Champion transitions on

\tau^{3}

-Banking (blue) and Terminal-Bench 2 (orange). Each cell reports the number of tasks with the corresponding Seed and Champion success counts. Right-hand bars group tasks by whether the Champion succeeds more than, the same as, or less than the Seed. Non-origin cells on the left or bottom edges indicate gained or lost coverage.

Figure 3: Coverage–cost curves for MiniSWE and Codex on both benchmarks. Points pair

\mathrm{Pass}@k

with

C_{\mathrm{dep}}@k

from Equation 3 for

k=1,2,4,8

Benchmark	Harness	Method	$\Delta\mathrm{Pass}@8$				$\Delta\mathrm{Avg}@8$
			$\xrightarrow{\text{From evolution data to OOD transfer}}$				$\xrightarrow{\text{From evolution data to OOD transfer}}$
			Train	Validation	Test	Transfer	Train	Validation	Test	Transfer
$\tau^{3}$ -Banking	MiniSWE	AHE	$+3.1$	$+4.1$	$+4.1$	$-0.7$	$+3.4$	$+0.9$	$+1.2$	$+2.7$
		HyperAgents	$+2.1$	$+1.0$	$0.0$	$-0.4$	$+3.6$	$+1.7$	$-0.1$	$+0.8$
		A-Evolve	$0.0$	$+1.0$	$0.0$	$-0.3$	$-1.7$	$+0.4$	$+0.1$	$+1.4$
		GEPA	$+1.0$	$+2.1$	$+1.0$	$+1.5$	$+0.4$	$+0.5$	$-0.1$	$+2.6$

Table 3: Seed-relative gains across evolution splits and external transfer in percentage points. Darker cells indicate larger gains on a shared linear scale.

Benchmark	Harness	Original	Random
$\tau^{3}$ -Banking	MiniSWE	$17.5$	25.0
	Codex	$8.8$	25.0
Terminal-Bench 2	MiniSWE	59.5	$25.0$
	Codex	55.7	$25.0$

Table 4: Best Parent selection rates (%). Higher values are bold.

Benchmark	Harness	Method	$\mathrm{Avg}@8$ (%)				$\mathrm{Pass}@8$ (%)
	Harness	Method	Seed	Seed + A	Seed + B	Champion	Seed	Seed + A	Seed + B	Champion
$\tau^{3}$ -Banking	MiniSWE	AHE	$27.4$	$28.2$	$32.1$	32.9	$54.6$	$53.6$	$64.9$	66.0
	MiniSWE	HyperAgents		34.4	$32.0$	$32.6$		59.8	$55.7$	$57.7$
	Codex	AHE	$30.3$	$29.6$	$34.8$	36.2	$56.7$	$56.7$	63.9	$62.9$
	Codex	HyperAgents		37.2	$36.5$	$36.9$		$67.0$	72.2	$66.0$
Terminal-Bench 2	MiniSWE	AHE	$65.9$	$67.7$	$69.1$	70.2	$86.5$	88.8	$87.6$	$87.6$

Table 5: Isolated Components versus complete Champions. Appendix D.5 defines each row’s A and B. Scores are percentages. Best values are bold.

Benchmark	Harness	$\mathrm{Avg}@8$ (%)		$\mathrm{Pass}@8$ (%)
		Seed	Generic	Seed	Generic
			reminder		reminder
$\tau^{3}$ -Banking	MiniSWE	$27.4$	27.8	$54.6$	55.7
	Codex	30.3	$28.6$	56.7	56.7
Terminal-Bench 2	MiniSWE	65.9	65.9	86.5	$84.3$
	Codex	69.9	$65.7$	89.9	$88.8$

Table 9

Integrated Finding	Supporting Evidence	Design Implication
Gains Involve Trade-offs.	New coverage accompanies regressions and higher cost (Table 1 , Figures 2 – 3 ).	Consider gains, regressions, and cost in select and gate .
Gains Are Context-Dependent.	Benefits vary with exposure and transfer target (Table 3 ).	Specify applicability in analyze and mutate .
Generation and Retention Shape Gains.	Current scores, combinations, and admission can mislead search (Tables 4 – 7 ).	Check subsequent gains, trace evidence, compatibility, and retained alternatives.

Table 8: Cross-experiment findings and the design hypotheses they motivate.

Figure 4: EvoAudit diagnoses archived evolution runs and returns validated operator revisions.

Measure	Mean / 8
Direction agreement	$6/8$
Evidence support	$6.25/8$
Problem addressing	$6.5/8$
Proposal endorsement	$6.75/8$

Table 9: Expert review of four revisions.

Benchmark	Harness	Variant	Reliability		Coverage		Task Turnover		Cost
Benchmark	Harness	Variant	$\mathrm{Avg}@8$	$\Delta\mathrm{Avg}$	$\mathrm{Pass}@8$	$\Delta\mathrm{Pass}$	Gained	Lost	Deployment	Evolution
$\tau^{3}$ -Banking	MiniSWE	Seed	$27.4$	$-$	$54.6$	$-$	$-$	$-$	$6.2$	$-$
		AHE	$32.9$	$+5.4$	66.0	$+11.3$	15	4	$7.1$	$1235.0$
		HyperAgents	$32.6$	$+5.2$	$57.7$	$+3.1$	13	10	$6.6$	$867.0$
		A-Evolve	$26.3$	$-1.2$	$55.7$	$+1.0$	7	6	$5.5$	$1057.6$
		GEPA	$28.2$	$+0.8$	$58.8$	$+4.1$	10	6	$7.9$	$2262.8$

Table 10: Comparison of AGE with existing evolution methods. Score changes are relative to the Seed. Costs follow Equation 3 , with

k=8

for deployment and 20 generations for evolution.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Operator	AHE	HyperAgents	A-Evolve	GEPA
select	Newest eligible Parent	Score-weighted sampling with a child-count penalty	Highest-scoring eligible Parent	Per-task Pareto-front sampling
rollout	Reuse Parent evaluation traces	Reuse Parent evaluation traces	Fresh Train trajectories	Fresh Train trajectories
analyze	Per-task debugger reports	Metrics and browsable raw traces	Behavior-only trajectory judge	Focused reflective examples
mutate	One testable target change	Edit the target and allowed operators	Distill experience into target changes	Reflection-guided target revision
validate	$-$	Python syntax checks	$-$	Strict improvement on the Parent’s Train batch
gate	Admit valid completed evaluations	Admit valid completed evaluations	Strict improvement over the matched Parent	Admit valid completed evaluations

Appendix

Table 11: Initial operator policies of the four methods. Dashes indicate an omitted optional stage. HyperAgents can subsequently revise its permitted operators.

Domain	Task and lineage	Count	Observed difference in the reference executions
General	Task 054, MiniSWE with A-Evolve	$1\to 4$	Seed handles replacement and dispute first, blocking the requested credit-limit increase. Champion completes the increase before those dependent actions.
General	Task 037, MiniSWE with A-Evolve and GEPA	$7\to 3/1$	Both Champions successfully file disputes but use the wrong issue-noticed date. Seed’s successful reference uses the date required by the evaluator.
General	Task 056, Codex with AHE	$5\to 0$	Champion terminates after opening checking. Seed continues the interaction and completes the additional savings and funding requests.
General	Task 090, Codex with GEPA	$1\to 5$	Seed repeatedly fails through a generic tool wrapper. Champion switches to the available direct lookup and completes the investigation.
Coding	db-wal-recovery , Codex with GEPA	$0\to 7$	Seed loses the original write-ahead log during inspection and reconstructs incorrect values. Champion preserves a copy before repair and verifies the recovered values.
Coding	polyglot-c-py , MiniSWE with HyperAgents	$1\to 8$	Seed leaves test binaries and a script beside the required single source file. Champion leaves only the required file; the contrast concerns final directory contents.

Appendix

Table 12: Illustrative trace contrasts on

\tau^{3}

-Banking and Terminal-Bench 2. Counts are Seed

\rightarrow

Champion successes out of eight; paired Champion counts follow the listed method order.

Benchmark	Harness	Variant	$\mathrm{Pass}@1$	$\mathrm{Pass}@2$	$\mathrm{Pass}@4$	$\mathrm{Pass}@8$	Mean cost/trial	Recorded cost
$\tau^{3}$ -Banking	MiniSWE	Seed	$27.8$	$39.2$	$46.4$	$54.6$	$\$ 0.779$	$\$ 604.52$
		AHE	$28.9$	$46.4$	$56.7$	$66.0$	$\$ 0.891$	$\$ 691.57$
		HyperAgents	$33.0$	$39.2$	$50.5$	$57.7$	$\$ 0.819$	$\$ 635.56$
		A-Evolve	$26.8$	$36.1$	$48.5$	$55.7$	$\$ 0.690$	$\$ 535.57$
		GEPA	$29.9$	$40.2$	$51.5$	$58.8$	$\$ 0.985$	$\$ 764.57$
	Codex	Seed	$24.7$	$42.3$	$50.5$	$56.7$	$\$ 0.983$	$\$ 762.45$

Appendix

Table 13: Deployment-side coverage and recorded costs on both primary benchmarks. Scores are percentages and costs are in USD.

Figure 5: Task-level Seed-to-Champion transitions over eight passes on the Train split. Each panel contains 50

\tau^{3}

-Banking tasks (blue) or 50 Terminal-Bench 2 tasks (orange).

Figure 6: Task-level Seed-to-Champion transitions over eight passes on the Validation split. Each panel contains 20

\tau^{3}

-Banking tasks (blue) or 19 Terminal-Bench 2 tasks (orange).

Figure 7: Task-level Seed-to-Champion transitions over eight passes on the Test split. Each panel contains 27

\tau^{3}

-Banking tasks (blue) or 20 Terminal-Bench 2 tasks (orange).

Benchmark	Harness	Variant	$\mathrm{Pass}@8$	$\mathrm{Avg}@8$
$\tau^{2}$ -bench	MiniSWE	Seed	96.2	84.0
		AHE	95.6	86.7
		HyperAgents	95.9	84.8
		A-Evolve	96.0	85.4
		GEPA	97.8	86.6
	Codex	Seed	97.5	87.0

Appendix

Table 14: Absolute performance on external transfer benchmarks. Scores are percentages.

\tau^{2}

-bench evaluates agents evolved on

\tau^{3}

-Banking using the official equal-weight aggregation across its three domains; DeepSWE v1.1 evaluates agents evolved on Terminal-Bench 2.

Benchmark	Harness	Method	Empirical best	Original gate	Constrained
Benchmark	Harness	Method		$H/N$	$H/N$
$\tau^{3}$ -Banking	MiniSWE	AHE	Generation 9	$1/4$	$0/1$
		HyperAgents	Generation 3	$1/4$	$1/2$
		A-Evolve	Generation 1	$1/2$	$1/2$
		GEPA	Generation 9	$1/4$	$0/1$
	Codex	AHE	Generation 10	$1/4$	$0/2$

Appendix

Table 15: Per-pool audit of size-normalized best-Candidate retention. Each policy cell gives

H/N

, the binary best-retention indicator divided by set size, including Seed. The empirical best maximizes reporting-panel

\mathrm{Avg}@8

; Generation numbers identify the frozen artifacts.

Figure 8: Human review of select before proposal disclosure. Experts inspect the original rule and evidence, then submit their preferred revision direction.

Figure 9: Human review of select after proposal disclosure. Experts inspect the revision and its reasoning, then assess evidence support, problem addressing, and endorsement for validation.

Figure 10: Evolution trees on

\tau^{3}

-Banking with MiniSWE. Nodes are labeled by generation, with 0 denoting the Seed. Blue marks the Champion used in the paper and its ancestral path.

Figure 11: Evolution trees on

\tau^{3}

-Banking with Codex. Nodes are labeled by generation, with 0 denoting the Seed. Blue marks the Champion used in the paper and its ancestral path.

Figure 12: Evolution trees on Terminal-Bench 2 with MiniSWE. Nodes are labeled by generation, with 0 denoting the Seed. Blue marks the Champion used in the paper and its ancestral path.

Figure 13: Evolution trees on Terminal-Bench 2 with Codex. Nodes are labeled by generation, with 0 denoting the Seed. Blue marks the Champion used in the paper and its ancestral path.

Agent Harness Self-Evolution

See all 50 papers

Sep 3, 2026

EvoHarnessBench: Can Your Agents Keep Pace with an Evolving Harness?

Modern LLM-based agents operate through a harness of tools, reusable skills, and specialist agents that shapes what they observe and what they can do. In practice, this harness continually evolves as new capabilities are added. We introduce EvoHarnessBench, a benchmark for evaluating agents under controlled harness evolution across three axes (tools, skills, and agents). Unlike existing continual-learning benchmarks for agents, which typically place non-stationarity (i.e., what changes over time) in the task stream while keeping the harness fixed, EvoHarnessBench places non-stationarity in the externally supplied harness itself. It contains 17 multi-stage harness streams constructed deterministically from verifier-based benchmarks, comprising 802 tasks, 520 tools, 42 skills, and 62 agents. We evaluate two complementary settings corresponding to the central challenges of harness evolution: deployment evaluation, which isolates retention of previously accessible competence as the harness expands, and self-evolving adaptation evaluation, which tests whether accumulated experience remains useful as new capabilities are introduced. Our results reveal three persistent gaps. First, harness expansion alone can degrade performance on previously solved tasks, producing harness-induced forgetting. Second, gains from self-evolving adaptation remain inconsistent across stages of harness evolution, capability axes, and environments. Third, retention and adaptation can pull in different directions: preserving earlier competence does not necessarily improve adaptation to newly introduced capabilities, and vice versa. These results establish harness evolution as a distinct challenge for building agents that can keep pace with an evolving harness while preserving previously effective behavior.

Zixuan Ke, Vaidehi Patil, Haizhou Shi +9

Salesforce Research · University of North Carolina at Chapel Hill, conducted this work during an internship at Salesforce Research · University of Wisconsin–Madison +1

Continual Learning for LLM Agents LLM Agent Harnesses Self-Evolving Agents Agent Benchmarks Agent Harness Self-Evolution

Ask PDF

Oct 15, 2026

Harness-Delta Attribution: Dissecting Gains From Harness Evolution

Harness evolution automatically optimizes the prompts, skills, tools, and orchestration code around a fixed-weight model, offering a way to improve agentic systems without updating model weights. The agent harness is iteratively optimized to achieve higher performance on the search set. But how much of this gain reflects benchmark-specific overfitting or additional inference-time compute, and how much may generalize beyond the search set? In this work, we introduce Harness-Delta Attribution (HDA), an evaluation protocol that dissects an evolution gain into artifact-driven overfitting, gains explained by test-time scaling, and the remainder, which gives an upper bound on residual generalizable improvement. We apply HDA to 11 executor models and three harness-evolution frameworks across math, coding, creativity, and agentic tasks. In 21 of these settings that show statistically significant search-set gains, we find that a large share of the improvement is explained by overfitting or test-time scaling. Specifically, among these 21 settings, overfitting and test-time scaling account for 54% and 14% of the gain on average; their combined share exceeds 50% in 14 settings. The residual and held-out transfer vary substantially across tasks, models, and frameworks. Importantly, search-set gains often fail to transfer to held-out tasks, and provide limited evidence that they will persist under practical distribution shifts. Moreover, validation-based signals are insufficient to prevent overfitting. These results motivate attribution-aware evaluation in future harness evolution research.

Wenxuan Ding, Qin Lu, Changlong Yu +5

Amazon · New York University

Agent Harness Optimization Agent Harness Self-Evolution Agent Evaluation Benchmark Contamination Test-Time Scaling

Ask PDF

Jul 14, 2026

Rethinking the Evaluation of Harness Evolution for Agents

Harness evolution is an iterative search procedure that repeatedly evaluates and revises candidate harnesses used for LLM agents using task feedback. We revisit the evaluation of such automatic harness evolution procedures and identify two fundamental issues in the protocol. First, prior work does not compare these approaches with simple task-level search baselines under matched feedback and inference budgets. Second, prior work searches for harness configurations using verification signals (e.g., unit test cases) drawn from the same benchmarks on which it reports the final performance of the evolved harnesses, violating the standard separation between training and test data. To address this, we compare automatic harness evolution with simple test-time scaling and discovery baselines under comparable feedback and inference budgets, and evaluate evolved harnesses on held-out tasks to assess generalization. Following prior work, we experiment on Terminal-Bench 2.1 and find that automatic harness evolution fails to outperform simple test-time scaling methods both with and without test cases, and exhibits limited generalization. However, we find that long-horizon games are a promising setting for automatic harness evolution, as they are difficult enough to leave headroom, rely heavily on adaptation to out-of-distribution dynamics, and provide granular feedback by design. In these settings, task-specific harness evolution improves over the search baseline by 80% on ARC-AGI-3 and by 11% on EdgeBench games under matched budgets. Together, these findings highlight the need for matched-budget baselines and held-out evaluation to distinguish genuine harness improvements from benchmark-specific search and overfitting, and point to a more careful characterization of when automatic harness evolution is actually useful. Our code is available at https://github.com/rethinking-harness-evolution.

Yike Wang, Huaisheng Zhu, Zhengyu Hu +8

Allen Institute for AI · University of Washington · Independent

LLM Agent Evaluation Agent Harness Optimization LLM Agent Harnesses Agent Harness Self-Evolution

Ask PDF

Abstract

Agent Harness Self-Evolution

Explore similar work

EvoHarnessBench: Can Your Agents Keep Pace with an Evolving Harness?

Harness-Delta Attribution: Dissecting Gains From Harness Evolution

Rethinking the Evaluation of Harness Evolution for Agents