Forging More Tasks into Better Harnesses: Agentic Harness Evolution via Successive Scale Racing

Jingzhe Liu; Jiliang Tang; Tong Zhao; Neil Shah; Clark Mingxuan Ju

cs.AIOct 20, 2026

Forging More Tasks into Better Harnesses: Agentic Harness Evolution via Successive Scale Racing

Authors: Jingzhe Liu, Jiliang Tang, Tong Zhao, Neil Shah, Clark Mingxuan Ju

Organizations: Michigan State University · Snap Inc.

Abstract

Agentic harness evolution optimizes an agent's instructions, tools, and control flow by using its task execution experiences as evolving evidence. While more evolution tasks should intuitively improve performance, our preliminary study reveals a paradoxical failure mode: the performance of existing methods degrades as the task pool expands. Our further analysis demonstrates that this degradation stems from suboptimal evidence selection during optimization, as different evidence combinations lead to substantially different evolution outcomes. To address this issue, we propose Successive Scale Racing (SSR), a framework that adaptively searches for optimal evidence combinations in each evolution round. SSR constructs diverse evidence combinations based on the corresponding task quality and coverage, and efficiently races the resulting candidate harnesses under a small compute budget. Empirically, across three distinct benchmarks, SSR achieves state-of-the-art results in all six configurations. Notably, as the task pool grows from 40 to 200, SSR steadily improves the pass rate from 64.3% to 68.3%, while the baselines' performances all drop. Furthermore, its racing mechanism identifies near-optimal candidates while reducing validation executions by 60%.

Figures & tables

Figure 1: Many existing methods evolve on large, fixed task pools, which can induce noisy or redundant signals. SSR searches over ordered evidence groups.

Figure 4: Overview of Successive Scale Racing (SSR). (a) RER ranks training trajectories of the parent

H_{t}

by quality and marginal coverage gain. (b) Nested evidence prefixes of different scales independently modify

H_{t}

to generate candidate harnesses. (c) Candidates race against the protected parent

H_{t}

over shared task batches, progressively eliminating weaker harnesses by cumulative score and updating the incumbent to

H_{t+1}

for the next evolution round.

	Codex: GPT 5.5 high			Claude Code: Opus 4.8 high
Method	SWE-Pro-V	DeepSWE	GAIA-2	SWE-Pro-V	DeepSWE	GAIA-2
Initial harness	57.7	59.3	29.0	56.3	56.7	26.0
Meta-Harness	59.7	60.7	31.7	60.0	58.0	29.3
HGM	58.7	62.7	32.3	60.3	58.7	29.3
MGM	60.0	63.3	32.3	61.0	60.7	29.7
AHE	60.7	65.3	36.0	62.0	63.3	33.7

Table 1: Final harness performance (Avg Pass@1, %). SWE-Pro-V denotes SWE-Bench Pro Verified. Each entry is the mean over three independent evolution runs; standard deviations are in Appendix D . Bold marks the best score; underlining marks the strongest baseline.

Figure 5: Scalability of harness evolution on SWE-Bench Pro Verified. (a) Task-pool growth: Evaluation performance when expanding available tasks from 40 to 200 with proportionally scaled execution caps. (b) Search-resource growth: Performance trajectory when increasing total execution caps under a fixed task pool.

Figure 7: Racing quality versus validation cost, averaged over three task-order replays of a fixed pool. Racing uses 40% of the full-evaluation cost to achieve a score within 0.4% of the best full-evaluation score.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Main	Task-pool growth	Resource growth	Fixed strategies	Racing audit	Scale dynamics
Benchmarks	S / D / G	S	S	S / D	S	S
Train / validation / test	S,G: 80/20/100 D: 50/13/50	$0.8T$ / $0.2T$ / 100	80/20/100	S: 80/20/100 D: 50/13/50	Separate 160-task audit set	(a) 160/20/100 (b) 80/20/100
Candidate generation
Children $M$	4	4	4	4	12	(a) 6 (b) 4
Evidence scales	$25/50/$ $75/100\%$	Main	Main	25% or 100% (see text)	Main; three children per scale	(a) $10/20/40/$ $80/120/160$ (b) Main
Raw-trajectory access	On demand, for records selected as candidate-generation evidence.

Appendix

Table 2: Experimental configurations. S, D, and G denote SWE-Pro-V, DeepSWE, and GAIA-2. In the scale-dynamics column, (a) denotes the parent–scale probes and (b) the final-winner scale analysis. Lists in the task-pool and resource columns follow increasing task-pool size and execution budget, respectively.

	Codex: GPT 5.5 high			Claude Code: Opus 4.8 high
Method	SWE-Pro-V	DeepSWE	GAIA-2	SWE-Pro-V	DeepSWE	GAIA-2
Initial harness	57.7 $\pm$ 0.72	59.3 $\pm$ 0.54	29.0 $\pm$ 0.98	56.3 $\pm$ 0.98	56.7 $\pm$ 0.54	26.0 $\pm$ 0.54
Meta-Harness	59.7 $\pm$ 0.82	60.7 $\pm$ 0.54	31.7 $\pm$ 0.94	60.0 $\pm$ 0.47	58.0 $\pm$ 0.94	29.3 $\pm$ 0.82
HGM	58.7 $\pm$ 0.98	62.7 $\pm$ 0.94	32.3 $\pm$ 0.94	60.3 $\pm$ 0.72	58.7 $\pm$ 0.94	29.3 $\pm$ 0.82
MGM	60.0 $\pm$ 0.94	63.3 $\pm$ 0.54	32.3 $\pm$ 0.72	61.0 $\pm$ 0.47	60.7 $\pm$ 0.94	29.7 $\pm$ 0.54
AHE	60.7 $\pm$ 0.54	65.3 $\pm$ 0.54	36.0 $\pm$ 0.82	62.0 $\pm$ 0.98	63.3 $\pm$ 0.54	33.7 $\pm$ 0.47

Appendix

Table 3: Final harness performance as mean

\pm

standard deviation (Avg Pass@1, %) over three independent evolution runs. Means are those of Table 1 ; bold marks the best mean in each setting.

Figure 8: Evidence scales across parents and rounds. (a) Single-candidate gains (one per scale, no intervals) for parents from different independent runs. (b) Final-winner scale shares over 20 runs; each column sums to 100%. These are winner frequencies, not survival probabilities.

Agent Harness Optimization

Agent harness optimization improves the prompts, tools, memory, workflows, and control code that shape how AI agents act.

See all 185 papers

Harness-Delta Attribution: Dissecting Gains From Harness Evolution
Oct 15, 2026Wenxuan Ding, Qin Lu, Changlong Yu +5Agent Harness Optimization Agent Evaluation
Amazon · New York University
Rethinking the Evaluation of Harness Evolution for Agents
Jul 14, 2026Yike Wang, Huaisheng Zhu, Zhengyu Hu +8LLM Agent Evaluation Agent Harness Optimization
Allen Institute for AI · University of Washington · Independent
EvoHarnessBench: Can Your Agents Keep Pace with an Evolving Harness?
Sep 3, 2026Zixuan Ke, Vaidehi Patil, Haizhou Shi +9Continual Learning for LLM Agents LLM Agent Harnesses
Salesforce Research · University of North Carolina at Chapel Hill, conducted this work during an internship at Salesforce Research · University of Wisconsin–Madison +1

Oct 15, 2026J/K move · Enter open · S save

Harness-Delta Attribution: Dissecting Gains From Harness Evolution

Wenxuan Ding, Qin Lu, Changlong Yu, Sha Li+4

Amazon · New York University

Agent Harness Optimization Agent Evaluation Benchmark Contamination Test-Time Scaling

Open paper AskPDF arXiv

Harness evolution automatically optimizes the prompts, skills, tools, and orchestration code around a fixed-weight model, offering a way to improve agentic systems without updating model weights. The agent harness is iteratively optimized to achieve higher performance on the search set. But how much of this gain reflects benchmark-specific overfitting or additional inference-time compute, and how much may generalize beyond the search set? In this work, we introduce Harness-Delta Attribution (HDA), an evaluation protocol that dissects an evolution gain into artifact-driven overfitting, gains explained by test-time scaling, and the remainder, which gives an upper bound on residual generalizable improvement. We apply HDA to 11 executor models and three harness-evolution frameworks across math, coding, creativity, and agentic tasks. In 21 of these settings that show statistically significant search-set gains, we find that a large share of the improvement is explained by overfitting or test-time scaling. Specifically, among these 21 settings, overfitting and test-time scaling account for 54% and 14% of the gain on average; their combined share exceeds 50% in 14 settings. The residual and held-out transfer vary substantially across tasks, models, and frameworks. Importantly, search-set gains often fail to transfer to held-out tasks, and provide limited evidence that they will persist under practical distribution shifts. Moreover, validation-based signals are insufficient to prevent overfitting. These results motivate attribution-aware evaluation in future harness evolution research.

Abstract

Agent Harness Optimization

Explore similar work

Harness-Delta Attribution: Dissecting Gains From Harness Evolution

Rethinking the Evaluation of Harness Evolution for Agents

EvoHarnessBench: Can Your Agents Keep Pace with an Evolving Harness?