Harness evolution automatically optimizes the prompts, skills, tools, and orchestration code around a fixed-weight model, offering a way to improve agentic systems without updating model weights. The agent harness is iteratively optimized to achieve higher performance on the search set. But how much of this gain reflects benchmark-specific overfitting or additional inference-time compute, and how much may generalize beyond the search set? In this work, we introduce Harness-Delta Attribution (HDA), an evaluation protocol that dissects an evolution gain into artifact-driven overfitting, gains explained by test-time scaling, and the remainder, which gives an upper bound on residual generalizable improvement. We apply HDA to 11 executor models and three harness-evolution frameworks across math, coding, creativity, and agentic tasks. In 21 of these settings that show statistically significant search-set gains, we find that a large share of the improvement is explained by overfitting or test-time scaling. Specifically, among these 21 settings, overfitting and test-time scaling account for 54% and 14% of the gain on average; their combined share exceeds 50% in 14 settings. The residual and held-out transfer vary substantially across tasks, models, and frameworks. Importantly, search-set gains often fail to transfer to held-out tasks, and provide limited evidence that they will persist under practical distribution shifts. Moreover, validation-based signals are insufficient to prevent overfitting. These results motivate attribution-aware evaluation in future harness evolution research.
Figures & tables
Figure 1: An evolved harness may improve the performance by overfitting to benchmark artifacts, scaling test-time compute, or making generalizable improvements (left). HDA asks where the observed gain comes from and attributes it to these three sources (right). Overfitting and test-time scaling explain a substantial share across settings.
Figure 2: Illustration of Harness-Delta Attribution (HDA). Comparing S(H0) and S(H^) gives the observed gain ΔS . Running H0 with H^ ’s compute budget gives the compute-matched baseline H0scale ; its gain over H0 measures T est-Time Scaling. Disabling detected shortcuts in H^ gives H^neutral ; the resulting drop measures O verfitting. The remaining gap estimates G eneralizable Improvement.
Figure 3: Example of overfitting in harness evolution. As illustrated in this excerpt, the ALFWorld harness evolved by Meta-Harness for Qwen3.5-0.8B overfits to benchmark-specific patterns by hard-coding a deterministic controller that makes 0 LLM calls.
Framework
Benchmark
Executor
ΔS
O↓ (%)
T↓ (%)
G↑ (%)
Meta-Harness
ALFWorld
Qwen3.5-0.8B
+100.0%
100
0
0
Qwen3.5-2B
+100.0%
100
0
0
Qwen3.5-4B
+60.0%
93
0
7
Qwen3-8B
+64.0%
100
0
0
Qwen3.6-35B-A3B ⋆
+38.0%
95
0
5
LiveMath
Qwen3.5-0.8B
+77.1%
91
9
0
Table 1: HDA results for all 25 model–benchmark–framework settings. ΔS=S(H^)−S(H0) is the search-set gain. O , T , and G are percentages of ΔS , capped to [0%,100%] . {\color[rgb]{0.418,0.4453,0.5}\dagger} marks search-set gains that are not statistically significant, for which HDA reports no attribution. {\color[rgb]{0.707,0.1367,0.0938}\star} marks settings where the evolved harness performs worse than baseline on the held-out test set. CU denotes Creativity Utility, the metric used for CREATE benchmark.
Figure 4: Search-set gains do not reliably predict held-out transfer. For each setting, the open circle is the search-set gain of the evolved harness ( ΔSsearch ) and the filled circle its held-out gain ( ΔStest ); a red marker flags a negative held-out gain. The length of the line denotes the gap between gain on search set and on held-out set. Diamond means H^=H0 ; × marks a non-significant gain. Units are pp, except CREATE (CU).
Figure 5: Effect of validation selection across 16 Meta-Harness settings. Open circles show the search-set gain of the search-selected harness H^ , gray circles show its held-out test gain, and blue diamonds show the held-out gain of the validation-selected harness H^val . Gains are reported in percentage points, except for CREATE, which uses Creative Utility (CU).
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Overview of the HDA skill. The skill standardizes the attribution procedure while retaining human review at the two task-dependent decisions: compute matching and overfitting neutralization.
Figure 7: Overview of representative evolved-harness mechanisms across the four benchmarks (left four cards, Meta-Harness): hard-coded object-location priors and task recipes on ALFWorld, the meta-option string match on LiveMath, parallel generation frames on CREATE (a test-time-scaling counter-case), and a patch-size prior on SWE-bench. The two right cards show the same LiveMath meta-option artifact independently rediscovered by SkillOpt (as a skill default rule) and AHE (as an explicit prompt rule). Red marks overfitting evidence; yellow marks test-time-scaling evidence; each card’s O / T chip is the attribution of the specific Table 1 row whose winner is excerpted — not a benchmark-level average (ALFWorld and LiveMath: Qwen3.5-0.8B; CREATE: Claude Haiku 4.5; SWE-bench: Qwen3-4B-Instruct-2507; SkillOpt: LiveMath Qwen3.5-4B; AHE: LiveMath Claude Opus 4.8). The following sections ( E – G ) present each mechanism in detail with source excerpts.
Figure 8: Meta-Harness ALFWorld winner for Qwen3.5-0.8B. The highlighted tables and parser rules encode benchmark-specific object locations and task procedures. The full harness contains no llm_call , so all actions come from the deterministic controller. HDA: O/T/G=100/0/0 .
Figure 9: SkillOpt ALFWorld skill for Qwen3.5-4B. The highlighted instructions encode the same appliance recipes, simulator rules, and location priors as the Meta-Harness controller, including a rule distilled from a specific search-set failure. HDA: O/T/G=44/0/56 .
Figure 10: Three representations of the same LiveMath artifact. Meta-Harness directly returns the matching option before invoking the executor and otherwise uses multiple model calls ( O/T/G=91/9/0 ). SkillOpt encodes it as a default decision rule in the skill document ( 88/0/12 ). AHE writes an explicit selection rule into the system prompt ( 100/0/0 ).
Figure 11: CREATE orchestration across the three frameworks. Meta-Harness makes 20 calls per item and obtains O/T/G=0/49/51 . AHE increases the mean from 1.04 to 2.46 calls per item through forced expansion turns; using the paper’s k=3 compute match gives 0/75/25 . SkillOpt can edit only the injected skill, while the rollout remains one call; both SkillOpt CREATE gains are statistically insignificant, so HDA assigns no attribution.
Figure 12: Examples of mixed and residual-dominated gains on SWE-bench Verified. Red marks the Qwen3-4B harness’s benchmark-derived patch-size heuristic: the +12/−8 -line and repetition thresholds encode the search set’s tendency toward small fixes and may reject valid larger patches elsewhere. Disabling only this mechanism accounts for O=23% . Matching the evolved harness’s higher LLM inference cost raises the baseline from pass@2 to pass@3 and accounts for T=23% ; the remaining G=54% comes from mechanisms such as edit-effect feedback. The Qwen3-30B-A3B action-block guardrail uses no benchmark-specific content, yielding O/T/G=0/6/94 .
Harness evolution is an iterative search procedure that repeatedly evaluates and revises candidate harnesses used for LLM agents using task feedback. We revisit the evaluation of such automatic harness evolution procedures and identify two fundamental issues in the protocol. First, prior work does not compare these approaches with simple task-level search baselines under matched feedback and inference budgets. Second, prior work searches for harness configurations using verification signals (e.g., unit test cases) drawn from the same benchmarks on which it reports the final performance of the evolved harnesses, violating the standard separation between training and test data. To address this, we compare automatic harness evolution with simple test-time scaling and discovery baselines under comparable feedback and inference budgets, and evaluate evolved harnesses on held-out tasks to assess generalization. Following prior work, we experiment on Terminal-Bench 2.1 and find that automatic harness evolution fails to outperform simple test-time scaling methods both with and without test cases, and exhibits limited generalization. However, we find that long-horizon games are a promising setting for automatic harness evolution, as they are difficult enough to leave headroom, rely heavily on adaptation to out-of-distribution dynamics, and provide granular feedback by design. In these settings, task-specific harness evolution improves over the search baseline by 80% on ARC-AGI-3 and by 11% on EdgeBench games under matched budgets. Together, these findings highlight the need for matched-budget baselines and held-out evaluation to distinguish genuine harness improvements from benchmark-specific search and overfitting, and point to a more careful characterization of when automatic harness evolution is actually useful. Our code is available at https://github.com/rethinking-harness-evolution.
Yike Wang, Huaisheng Zhu, Zhengyu Hu +8
Allen Institute for AI · University of Washington · Independent
AI agent performance depends critically on the runtime harness, comprising the prompts, tools, memory, and control flow that mediate how a model observes, reasons, and acts. Yet today's harnesses remain largely hand-crafted and static: each new model or task still demands bespoke scaffolding, and the rich traces produced during execution are rarely distilled back into systematic improvement. We introduce HarnessX, a foundry for composable, adaptive, and evolvable agent harnesses. HarnessX assembles typed harness primitives via a substitution algebra, adapts them through AEGIS, a trace-driven multi-agent evolution engine grounded in an operational mirror between symbolic adaptation and reinforcement learning, and closes the harness-model loop by turning trajectories into both harness updates and model training signal. Across five benchmarks (ALFWorld, GAIA, WebShop, tau^3-Bench, and SWE-bench Verified), HarnessX yields an average gain of +14.5% (up to +44.0%), with gains largest where baselines are lowest. These results suggest that agent progress need not come from model scaling alone: composing and evolving runtime interfaces from execution feedback is an actionable and complementary lever. The complete codebase will be open-sourced in a future release.
As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model's ability to develop the harness itself comparatively underexplored. We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages. In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system. In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance. We then evaluate each constructed harness on capability (task success on held-out benchmarks) and efficiency (execution-token cost). The reported Creation results cover six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development. We find that generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost. Evolution produces some performance gains, but they are unstable and transfer only partially to held-out tasks. Experiments with a fixed runtime model further show that the gains depend strongly on the model executing the harness, indicating limited transfer across models.