Harness-Delta Attribution: Dissecting Gains From Harness Evolution

Wenxuan Ding; Qin Lu; Changlong Yu; Sha Li; Shuowei Jin; Xin Liu; Zhenyu Shi; Greg Durrett

cs.AIOct 15, 2026

Harness-Delta Attribution: Dissecting Gains From Harness Evolution

Authors: Wenxuan Ding, Qin Lu, Changlong Yu, Sha Li, Shuowei Jin, Xin Liu, Zhenyu Shi, Greg Durrett

Organizations: Amazon · New York University

Abstract

Harness evolution automatically optimizes the prompts, skills, tools, and orchestration code around a fixed-weight model, offering a way to improve agentic systems without updating model weights. The agent harness is iteratively optimized to achieve higher performance on the search set. But how much of this gain reflects benchmark-specific overfitting or additional inference-time compute, and how much may generalize beyond the search set? In this work, we introduce Harness-Delta Attribution (HDA), an evaluation protocol that dissects an evolution gain into artifact-driven overfitting, gains explained by test-time scaling, and the remainder, which gives an upper bound on residual generalizable improvement. We apply HDA to 11 executor models and three harness-evolution frameworks across math, coding, creativity, and agentic tasks. In 21 of these settings that show statistically significant search-set gains, we find that a large share of the improvement is explained by overfitting or test-time scaling. Specifically, among these 21 settings, overfitting and test-time scaling account for 54% and 14% of the gain on average; their combined share exceeds 50% in 14 settings. The residual and held-out transfer vary substantially across tasks, models, and frameworks. Importantly, search-set gains often fail to transfer to held-out tasks, and provide limited evidence that they will persist under practical distribution shifts. Moreover, validation-based signals are insufficient to prevent overfitting. These results motivate attribution-aware evaluation in future harness evolution research.

Figures & tables

Figure 1: An evolved harness may improve the performance by overfitting to benchmark artifacts, scaling test-time compute, or making generalizable improvements (left). HDA asks where the observed gain comes from and attributes it to these three sources (right). Overfitting and test-time scaling explain a substantial share across settings.

Figure 2: Illustration of Harness-Delta Attribution (HDA). Comparing

\smash{S(H_{0})}

and

\smash{S(\hat{H})}

gives the observed gain

\Delta S

. Running

\smash{H_{0}}

with

\smash{\hat{H}}

’s compute budget gives the compute-matched baseline

\smash{H_{0}^{\mathrm{scale}}}

; its gain over

\smash{H_{0}}

measures T est-Time Scaling. Disabling detected shortcuts in

\smash{\hat{H}}

gives

\smash{\hat{H}^{\mathrm{neutral}}}

; the resulting drop measures O verfitting. The remaining gap estimates G eneralizable Improvement.

Figure 3: Example of overfitting in harness evolution. As illustrated in this excerpt, the ALFWorld harness evolved by Meta-Harness for Qwen3.5-0.8B overfits to benchmark-specific patterns by hard-coding a deterministic controller that makes 0 LLM calls.

Framework	Benchmark	Executor	$\Delta S$	$O\downarrow$ (%)	$T\downarrow$ (%)	$G\uparrow$ (%)
Meta-Harness	ALFWorld	Qwen3.5-0.8B	$+100.0\%$	100	0	0
		Qwen3.5-2B	$+100.0\%$	100	0	0
		Qwen3.5-4B	$+60.0\%$	93	0	7
		Qwen3-8B	$+64.0\%$	100	0	0
		Qwen3.6-35B-A3B ⋆	$+38.0\%$	95	0	5
	LiveMath	Qwen3.5-0.8B	$+77.1\%$	91	9	0

Table 1: HDA results for all 25 model–benchmark–framework settings.

\Delta S=S(\hat{H})-S(H_{0})

is the search-set gain.

O

T

, and

G

are percentages of

\Delta S

, capped to

[0\%,100\%]

{\color[rgb]{0.418,0.4453,0.5}\dagger}

marks search-set gains that are not statistically significant, for which HDA reports no attribution.

{\color[rgb]{0.707,0.1367,0.0938}\star}

marks settings where the evolved harness performs worse than baseline on the held-out test set. CU denotes Creativity Utility, the metric used for CREATE benchmark.

Figure 4: Search-set gains do not reliably predict held-out transfer. For each setting, the open circle is the search-set gain of the evolved harness (

\Delta S_{search}

) and the filled circle its held-out gain (

\Delta S_{test}

); a red marker flags a negative held-out gain. The length of the line denotes the gap between gain on search set and on held-out set. Diamond means

\hat{H}=H_{0}

;

\times

marks a non-significant gain. Units are pp, except CREATE (CU).

Figure 5: Effect of validation selection across 16 Meta-Harness settings. Open circles show the search-set gain of the search-selected harness

\hat{H}

, gray circles show its held-out test gain, and blue diamonds show the held-out gain of the validation-selected harness

\hat{H}^{\mathrm{val}}

. Gains are reported in percentage points, except for CREATE, which uses Creative Utility (CU).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Overview of the HDA skill. The skill standardizes the attribution procedure while retaining human review at the two task-dependent decisions: compute matching and overfitting neutralization.

Figure 7: Overview of representative evolved-harness mechanisms across the four benchmarks (left four cards, Meta-Harness): hard-coded object-location priors and task recipes on ALFWorld, the meta-option string match on LiveMath, parallel generation frames on CREATE (a test-time-scaling counter-case), and a patch-size prior on SWE-bench. The two right cards show the same LiveMath meta-option artifact independently rediscovered by SkillOpt (as a skill default rule) and AHE (as an explicit prompt rule). Red marks overfitting evidence; yellow marks test-time-scaling evidence; each card’s

O

T

chip is the attribution of the specific Table 1 row whose winner is excerpted — not a benchmark-level average (ALFWorld and LiveMath: Qwen3.5-0.8B; CREATE: Claude Haiku 4.5; SWE-bench: Qwen3-4B-Instruct-2507; SkillOpt: LiveMath Qwen3.5-4B; AHE: LiveMath Claude Opus 4.8). The following sections ( E – G ) present each mechanism in detail with source excerpts.

Panel of Figure 8 — Figure 8: Meta-Harness ALFWorld winner for Qwen3.5-0.8B. The highlighted tables and parser rules encode benchmark-specific object locations and task procedures. The full harness contains no llm_call , so all actions come from the deterministic controller. HDA: $O/T/G=100/0/0$ .

Panel of Figure 9 — Figure 9: SkillOpt ALFWorld skill for Qwen3.5-4B. The highlighted instructions encode the same appliance recipes, simulator rules, and location priors as the Meta-Harness controller, including a rule distilled from a specific search-set failure. HDA: $O/T/G=44/0/56$ .

Panel of Figure 10 — Figure 10: Three representations of the same LiveMath artifact. Meta-Harness directly returns the matching option before invoking the executor and otherwise uses multiple model calls ( $O/T/G=91/9/0$ ). SkillOpt encodes it as a default decision rule in the skill document ( $88/0/12$ ). AHE writes an explicit selection rule into the system prompt ( $100/0/0$ ).

Panel of Figure 11 — Figure 11: CREATE orchestration across the three frameworks. Meta-Harness makes 20 calls per item and obtains $O/T/G=0/49/51$ . AHE increases the mean from 1.04 to 2.46 calls per item through forced expansion turns; using the paper’s $k=3$ compute match gives $0/75/25$ . SkillOpt can edit only the injected skill, while the rollout remains one call; both SkillOpt CREATE gains are statistically insignificant, so HDA assigns no attribution.

Panel of Figure 12 — Figure 12: Examples of mixed and residual-dominated gains on SWE-bench Verified. Red marks the Qwen3-4B harness’s benchmark-derived patch-size heuristic: the $+12/-8$ -line and repetition thresholds encode the search set’s tendency toward small fixes and may reject valid larger patches elsewhere. Disabling only this mechanism accounts for $O=23\%$ . Matching the evolved harness’s higher LLM inference cost raises the baseline from pass@2 to pass@3 and accounts for $T=23\%$ ; the remaining $G=54\%$ comes from mechanisms such as edit-effect feedback. The Qwen3-30B-A3B action-block guardrail uses no benchmark-specific content, yielding $O/T/G=0/6/94$ .

Agent Harness Optimization

Agent harness optimization improves the prompts, tools, memory, workflows, and control code that shape how AI agents act.

See all 183 papers

Rethinking the Evaluation of Harness Evolution for Agents
Jul 14, 2026Yike Wang, Huaisheng Zhu, Zhengyu Hu +8LLM Agent Evaluation Agent Harness Optimization
Allen Institute for AI · University of Washington · Independent
HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundry
Jun 12, 2026Tingyang Chen, Shuo Lu, Kang Zhao +11LLM Agent Orchestration Agent Harness Optimization
Darwin Agent Team
HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
Sep 1, 2026Yuhao Wu, Jingyuan Zhang, Jiajun Shi +16LLM Agent Evaluation Agent Harness Optimization

Jul 14, 2026J/K move · Enter open · S save

Rethinking the Evaluation of Harness Evolution for Agents

Yike Wang, Huaisheng Zhu, Zhengyu Hu, Yige Yuan+7

Allen Institute for AI · University of Washington · Independent

LLM Agent Evaluation Agent Harness Optimization LLM Agent Harnesses

Open paper AskPDF arXiv

Harness evolution is an iterative search procedure that repeatedly evaluates and revises candidate harnesses used for LLM agents using task feedback. We revisit the evaluation of such automatic harness evolution procedures and identify two fundamental issues in the protocol. First, prior work does not compare these approaches with simple task-level search baselines under matched feedback and inference budgets. Second, prior work searches for harness configurations using verification signals (e.g., unit test cases) drawn from the same benchmarks on which it reports the final performance of the evolved harnesses, violating the standard separation between training and test data. To address this, we compare automatic harness evolution with simple test-time scaling and discovery baselines under comparable feedback and inference budgets, and evaluate evolved harnesses on held-out tasks to assess generalization. Following prior work, we experiment on Terminal-Bench 2.1 and find that automatic harness evolution fails to outperform simple test-time scaling methods both with and without test cases, and exhibits limited generalization. However, we find that long-horizon games are a promising setting for automatic harness evolution, as they are difficult enough to leave headroom, rely heavily on adaptation to out-of-distribution dynamics, and provide granular feedback by design. In these settings, task-specific harness evolution improves over the search baseline by 80% on ARC-AGI-3 and by 11% on EdgeBench games under matched budgets. Together, these findings highlight the need for matched-budget baselines and held-out evaluation to distinguish genuine harness improvements from benchmark-specific search and overfitting, and point to a more careful characterization of when automatic harness evolution is actually useful. Our code is available at https://github.com/rethinking-harness-evolution.

Abstract

Agent Harness Optimization

Explore similar work

Rethinking the Evaluation of Harness Evolution for Agents

HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundry

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?