Harness development plays a crucial role in building capable AI agents. Recent approaches improve harnesses through recursive self-improvement (RSI), but their performance gains often come with higher execution costs. We introduce HarnessPro, a label-free method for evolving harnesses to improve task performance while reducing execution cost. In this setting, candidate harnesses are evaluated using LLM-generated tests without access to ground-truth solutions or official tests. Specialized agents diagnose execution trajectories, plan updates with different performance-cost priorities, and implement them as candidate harnesses. Rather than returning a single harness, HarnessPro uses normalized weighted squared-distance selection to provide cost-focused (lite), balanced (medium), and performance-focused (heavy) harnesses for different deployment budgets. Selected harnesses enter the next iteration, where execution feedback guides further updates. Our experiments cover Terminal-Bench 2, SWE-Bench Pro, and ProgramBench, using GPT-5.5 with Codex and GLM-5.2 with Claude Code. Across these benchmarks using GPT-5.5 with Codex, the heavy mode of HarnessPro achieves an average pass-rate gain of 2.0 percentage points and an average per-task cost reduction of 26.1% relative to RHO. This advantage also holds when using GLM-5.2 with Claude Code. Code is available at https://github.com/pzs19/HarnessPro
Figures & tables
Method
Goal
Scope
Feedback
Iteration
Output
AFlow ( Zhang et al., 2025 )
Pass
Workflow
Ground truth
Recursive
Single
SkillMOO ( Gong et al., 2026a )
Pass + Cost
Skills
Ground truth
Recursive
Pareto set
RHO ( Pan et al., 2026a )
Pass
Skills + tools
Label-free
Single round
Single
AHE ( Lin et al., 2026 )
Pass
Harness
Ground truth
Recursive
Single
HarnessPro
Pass + Cost
Harness
Label-free
Recursive
Pareto set
Table 1: Comparison of different harness self-improvement methods.
Figure 1: Overview of HarnessPro . In each epoch, it analyzes rollout trajectories, generates candidate harnesses, and selects representatives based on performance and cost.
Figure 2: Confusion matrices pooled across training tasks from Terminal-Bench 2 and SWE-Bench Pro.
Figure 3: Schematic of recursive harness update, filtering, and selection. 1 1 1 In practice, filtering and selection may retain fewer than three distinct harnesses, see Figure 4 .
Terminal-Bench 2
SWE-Bench Pro
ProgramBench
Method
P(%) ↑
C();\downarrow$
T(s) ↓
P(%) ↑
C();\downarrow$
T(s) ↓
P(%) ↑
C();\downarrow$
T(s) ↓
GPT-5.5 × Codex
Vanilla
69.5
0.89
260
59.0
1.88
337
60.3
2.65
669
Dynamic Cheatsheet
72.9
1.06
421
62.0
2.19
335
68.5
3.58
742
ReasoningBank
74.6
0.99
369
61.0
2.13
341
62.5
3.06
661
Sleep-time Compute
74.6
1.04
331
62.0
2.26
337
67.7
3.38
637
Table 2: Performance and efficiency across three benchmarks. P denotes pass rate (%), while C and T denote mean cost ($) and execution time (s). We report mean and standard deviation for HarnessPro . On ProgramBench, P is the macro-average of the fraction of tests passed per task.
Figure 4: Evolution dynamics of HarnessPro with GPT-5.5 and Codex. Dashed lines connect the harnesses retained in each epoch, and diamonds denote baselines. Intermediate-epoch test-set metrics are reported for visualization only and were not used for filtering or selection. Final-epoch points show means across repeated test runs.
Terminal-Bench 2
SWE-Bench Pro
Method
P(%) ↑
C();\downarrow$
CoP();\downarrow$
P(%) ↑
C();\downarrow$
CoP();\downarrow$
RHO
76.3
1.09
1.43
80.0
2.68
3.35
RHO + dynamic routing
76.3
0.92
1.21
76.0
2.13
2.80
RHO + budget hooks
76.3
0.92
1.21
75.0
1.75
2.33
RHO + low reasoning effort
62.7
0.45
0.72
59.0
1.08
1.83
HarnessPro-Lite
72.9
0.43
0.59
72.3
0.80
1.11
Table 3: Ablations with GPT-5.5 and Codex. In the RHO variants, + dynamic routing selects reasoning effort from task descriptions, + budget hooks append budget reminders to tool results and + low reasoning effort fixes effort at low.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Model
Uncached input
Cached input
Output
GPT-5.5 ( <272 K context)
5.00 $/M
0.50 $/M
30.00 $/M
GLM-5.2
1.40 $/M
0.26 $/M
4.40 $/M
Appendix
Table 4: Token prices used to compute execution costs.
As LLMs are increasingly deployed within agentic systems, their capabilities depend not only on the model weights but also on the harness: the prompts, tools, control flow, memory, and orchestration code surrounding them. This makes automated harness optimization -- the iterative and evaluation-guided improvement of a harness by an AI system -- both an important route to improving AI systems and a demanding capability for AI systems themselves. Yet the community lacks a common protocol for measuring how well frontier LLMs perform at this task. We introduce HarnessOpt-Bench, a benchmark for end-to-end harness optimization under expensive and stochastic evaluation. An optimizer, an LLM paired with a coding harness, receives a target agent's seed harness, graded evaluation feedback, and a fixed target-evaluation budget. It edits the harness and nominates a final candidate, which is scored by its normalized gain over the seed on a held-out test partition that remains inaccessible throughout search. A trusted execution environment enforces the evaluation boundary, meters target-agent resource use, and preserves candidate versions for audit. We evaluate 5 frontier LLMs as optimizers both under a shared coding harness and under their native harnesses across 4 downstream tasks, over 111 scored runs. Experiment results show that optimizer models separate more than the coding harnesses they act through, native harnesses are not consistently superior, and gains vary substantially across tasks and seed regimes. These results establish harness optimization as a measurable and discriminative capability with large space for improvement.
AI agent performance depends critically on the runtime harness, comprising the prompts, tools, memory, and control flow that mediate how a model observes, reasons, and acts. Yet today's harnesses remain largely hand-crafted and static: each new model or task still demands bespoke scaffolding, and the rich traces produced during execution are rarely distilled back into systematic improvement. We introduce HarnessX, a foundry for composable, adaptive, and evolvable agent harnesses. HarnessX assembles typed harness primitives via a substitution algebra, adapts them through AEGIS, a trace-driven multi-agent evolution engine grounded in an operational mirror between symbolic adaptation and reinforcement learning, and closes the harness-model loop by turning trajectories into both harness updates and model training signal. Across five benchmarks (ALFWorld, GAIA, WebShop, tau^3-Bench, and SWE-bench Verified), HarnessX yields an average gain of +14.5% (up to +44.0%), with gains largest where baselines are lowest. These results suggest that agent progress need not come from model scaling alone: composing and evolving runtime interfaces from execution feedback is an actionable and complementary lever. The complete codebase will be open-sourced in a future release.
As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model's ability to develop the harness itself comparatively underexplored. We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages. In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system. In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance. We then evaluate each constructed harness on capability (task success on held-out benchmarks) and efficiency (execution-token cost). The reported Creation results cover six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development. We find that generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost. Evolution produces some performance gains, but they are unstable and transfer only partially to held-out tasks. Experiments with a fixed runtime model further show that the gains depend strongly on the model executing the harness, indicating limited transfer across models.