HarnessPro: Recursive Harness Self-Improvement with Pareto Optimization for Performance-Cost Trade-offs

Zhuoshi Pan; ZhiYuan Feng; Yukuo Cen; Honglin Lin; Junru Lu; Zhongxia Chen; Shuo Yang; H. Vicky Zhao

cs.LGOct 19, 2026

HarnessPro: Recursive Harness Self-Improvement with Pareto Optimization for Performance-Cost Trade-offs

Authors: Zhuoshi Pan, ZhiYuan Feng, Yukuo Cen, Honglin Lin, Junru Lu, Zhongxia Chen, Shuo Yang, H. Vicky Zhao,

Organizations: Tsinghua University · Z.AI Co., Ltd. · Shanghai Jiao Tong University · University of Warwick ·

Abstract

Harness development plays a crucial role in building capable AI agents. Recent approaches improve harnesses through recursive self-improvement (RSI), but their performance gains often come with higher execution costs. We introduce HarnessPro, a label-free method for evolving harnesses to improve task performance while reducing execution cost. In this setting, candidate harnesses are evaluated using LLM-generated tests without access to ground-truth solutions or official tests. Specialized agents diagnose execution trajectories, plan updates with different performance-cost priorities, and implement them as candidate harnesses. Rather than returning a single harness, HarnessPro uses normalized weighted squared-distance selection to provide cost-focused (lite), balanced (medium), and performance-focused (heavy) harnesses for different deployment budgets. Selected harnesses enter the next iteration, where execution feedback guides further updates. Our experiments cover Terminal-Bench 2, SWE-Bench Pro, and ProgramBench, using GPT-5.5 with Codex and GLM-5.2 with Claude Code. Across these benchmarks using GPT-5.5 with Codex, the heavy mode of HarnessPro achieves an average pass-rate gain of 2.0 percentage points and an average per-task cost reduction of 26.1% relative to RHO. This advantage also holds when using GLM-5.2 with Claude Code. Code is available at https://github.com/pzs19/HarnessPro

Figures & tables

Method	Goal	Scope	Feedback	Iteration	Output
AFlow ( Zhang et al., 2025 )	Pass	Workflow	Ground truth	Recursive	Single
SkillMOO ( Gong et al., 2026a )	Pass + Cost	Skills	Ground truth	Recursive	Pareto set
RHO ( Pan et al., 2026a )	Pass	Skills + tools	Label-free	Single round	Single
AHE ( Lin et al., 2026 )	Pass	Harness	Ground truth	Recursive	Single
HarnessPro	Pass + Cost	Harness	Label-free	Recursive	Pareto set

Table 1: Comparison of different harness self-improvement methods.

Figure 1: Overview of HarnessPro . In each epoch, it analyzes rollout trajectories, generates candidate harnesses, and selects representatives based on performance and cost.

Figure 2: Confusion matrices pooled across training tasks from Terminal-Bench 2 and SWE-Bench Pro.

Figure 3: Schematic of recursive harness update, filtering, and selection. 1 1 1 In practice, filtering and selection may retain fewer than three distinct harnesses, see Figure 4 .

	Terminal-Bench 2			SWE-Bench Pro			ProgramBench
Method	P(%) $\;\uparrow$	C( $)$ ;\downarrow$	T(s) $\;\downarrow$	P(%) $\;\uparrow$	C( $)$ ;\downarrow$	T(s) $\;\downarrow$	P(%) $\;\uparrow$	C( $)$ ;\downarrow$	T(s) $\;\downarrow$
GPT-5.5 $\times$ Codex
Vanilla	69.5	0.89	260	59.0	1.88	337	60.3	2.65	669
Dynamic Cheatsheet	72.9	1.06	421	62.0	2.19	335	68.5	3.58	742
ReasoningBank	74.6	0.99	369	61.0	2.13	341	62.5	3.06	661
Sleep-time Compute	74.6	1.04	331	62.0	2.26	337	67.7	3.38	637

Table 2: Performance and efficiency across three benchmarks. P denotes pass rate (%), while C and T denote mean cost ($) and execution time (s). We report mean and standard deviation for HarnessPro . On ProgramBench, P is the macro-average of the fraction of tests passed per task.

Panel of Figure 4 — Figure 4: Evolution dynamics of HarnessPro with GPT-5.5 and Codex. Dashed lines connect the harnesses retained in each epoch, and diamonds denote baselines. Intermediate-epoch test-set metrics are reported for visualization only and were not used for filtering or selection. Final-epoch points show means across repeated test runs.

	Terminal-Bench 2			SWE-Bench Pro
Method	P(%) $\;\uparrow$	C( $)$ ;\downarrow$	CoP( $)$ ;\downarrow$	P(%) $\;\uparrow$	C( $)$ ;\downarrow$	CoP( $)$ ;\downarrow$
RHO	76.3	1.09	1.43	80.0	2.68	3.35
RHO + dynamic routing	76.3	0.92	1.21	76.0	2.13	2.80
RHO + budget hooks	76.3	0.92	1.21	75.0	1.75	2.33
RHO + low reasoning effort	62.7	0.45	0.72	59.0	1.08	1.83
HarnessPro-Lite	72.9	0.43	0.59	72.3	0.80	1.11

Table 3: Ablations with GPT-5.5 and Codex. In the RHO variants, + dynamic routing selects reasoning effort from task descriptions, + budget hooks append budget reminders to tool results and + low reasoning effort fixes effort at low.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Model	Uncached input	Cached input	Output
GPT-5.5 ( $<272$ K context)	5.00 $/M	0.50 $/M	30.00 $/M
GLM-5.2	1.40 $/M	0.26 $/M	4.40 $/M

Appendix

Table 4: Token prices used to compute execution costs.

Agent Harness Optimization

Agent harness optimization improves the prompts, tools, memory, workflows, and control code that shape how AI agents act.

See all 183 papers

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
Aug 6, 2026Varun Ursekar, Apaar Shanker, Yash Maurya +4Large Language Model-Guided Optimization LLM Agent Evaluation
Scale AI
HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundry
Jun 12, 2026Tingyang Chen, Shuo Lu, Kang Zhao +11LLM Agent Orchestration Agent Harness Optimization
Darwin Agent Team
HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
Sep 1, 2026Yuhao Wu, Jingyuan Zhang, Jiajun Shi +16LLM Agent Evaluation Agent Harness Optimization

Aug 6, 2026J/K move · Enter open · S save

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

Varun Ursekar, Apaar Shanker, Yash Maurya, Shehab Yasser+3

Scale AI

Large Language Model-Guided Optimization LLM Agent Evaluation Agent Harness Optimization LLM Agent Workflow Optimization AI Agent Benchmarks

Open paper AskPDF arXiv

As LLMs are increasingly deployed within agentic systems, their capabilities depend not only on the model weights but also on the harness: the prompts, tools, control flow, memory, and orchestration code surrounding them. This makes automated harness optimization -- the iterative and evaluation-guided improvement of a harness by an AI system -- both an important route to improving AI systems and a demanding capability for AI systems themselves. Yet the community lacks a common protocol for measuring how well frontier LLMs perform at this task. We introduce HarnessOpt-Bench, a benchmark for end-to-end harness optimization under expensive and stochastic evaluation. An optimizer, an LLM paired with a coding harness, receives a target agent's seed harness, graded evaluation feedback, and a fixed target-evaluation budget. It edits the harness and nominates a final candidate, which is scored by its normalized gain over the seed on a held-out test partition that remains inaccessible throughout search. A trusted execution environment enforces the evaluation boundary, meters target-agent resource use, and preserves candidate versions for audit. We evaluate 5 frontier LLMs as optimizers both under a shared coding harness and under their native harnesses across 4 downstream tasks, over 111 scored runs. Experiment results show that optimizer models separate more than the coding harnesses they act through, native harnesses are not consistently superior, and gains vary substantially across tasks and seed regimes. These results establish harness optimization as a measurable and discriminative capability with large space for improvement.

Abstract

Agent Harness Optimization

Explore similar work

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundry

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?