Turbo Harness: Instance-Adaptive Harness Optimization

Tunyu Zhang; Hao Wang; Kai Xu; Dimitris N. Metaxas

cs.AISep 30, 2026

Turbo Harness: Instance-Adaptive Harness Optimization

Authors: Tunyu Zhang, Hao Wang, Kai Xu, Dimitris N. Metaxas

Organizations: Rutgers University · Red Hat AI Innovation · MIT-IBM Watson AI Lab

Abstract

Automating the search for effective harnesses is an important step toward enabling agents to recursively self-improve. Existing harness optimizations typically produce a single global harness that is applied uniformly across task instances. However, a harness that works well on average may not be optimal for every instance. We introduce Turbo Harness, a framework that can adapt a globally optimized harness to each instance by reusing information generated during the original optimization process. Specifically, Turbo Harness recycles artifacts produced during a completed global harness optimization run, and summarizes them into a structured playbook. We train a harness editor to leverage this prior optimization experience to generate instance-specific patches to the global harness. At inference time, the editor uses the instance and the playbook to construct a tailored harness in which the execution model operates. Through numerical experiments, we show that Turbo Harness consistently outperforms existing harness optimization baselines across seven benchmarks spanning interactive agent tasks, software engineering, and long-horizon terminal tasks.

Figures & tables

Figure 1: Turbo Harness adapts a globally optimized harness

\texttt{H}^{\star}

to each instance. (a) A harness editor produces an instance-specific patch to

\texttt{H}^{\star}

, yielding a per-instance harness. (b) On Terminal-Bench-2.1 (Claude Sonnet 4.5,

44

-task test split), Turbo Harness achieves the highest pass rate compared with several widely-used human-designed harnesses.

Figure 2: Turbo Harness overview. Left: an outer loop optimizes a global harness

\texttt{H}^{\star}

, producing “exhaust” (traces, reflections, evaluations) as a byproduct. Right: we recycle this exhaust into a playbook ; a small RL-trained harness editor

\pi_{\theta}

conditions on an instance

x

and the playbook to patch

\texttt{H}^{\star}

into an instance-specific harness

\texttt{H}_{x}

, which the frozen model M then runs.

Outer-loop harness-search archive

\mathcal{A}

; training instances

\mathcal{D}_{\mathrm{train}}

; global harness

\texttt{H}^{\star}

;

frozen execution model M ; initial editor

\pi_{\theta}

; proposals per instance

G

Algorithm 1 Turbo Harness

Method	ALFWorld	ScienceWorld	DBBench	WebShop	Avg.
Default	40.7	25.7	57.5	34.5	39.6
Prompt-based methods
ReAct	60.7	26.7	55.8	33.5	44.2
Self-Refine	40.7	21.3	53.3	18.5	33.5
Reflection	50.0	38.2	60.8	41.0	47.5
Harness optimization

Table 1: Task performance on four agentic benchmarks (frozen Qwen3.5-9B). Higher is better; metrics are success rate (%, ALFWorld), mean score (

/100

, ScienceWorld), accuracy (%, DBBench), and Success@1 (%, WebShop), with Avg. their equal-weight mean. Harness-R1 uses two rollouts (

\dagger

: author-reported). Bold denotes the best in each column.

Method	SWE-smith-MR			SWE-bench Verified
Method	Pass rate	Steps	Cost ($)	Pass rate	Steps	Cost ($)
Execution model: Claude Haiku 4.5
Default	$42.0$ $\pm 4.2$	$29.5$ $\pm 0.8$	$0.325$ $\pm 0.015$	$31.6$ $\pm 0.2$	$35.5$ $\pm 0.2$	$0.426$ $\pm 0.006$
GEPA	$44.7$ $\pm 2.9$	$25.8$ $\pm 0.7$	$0.308$ $\pm 0.014$	$41.3$ $\pm 0.4$	$27.2$ $\pm 1.0$	$0.352$ $\pm 0.013$
ACE	$51.3$ $\pm 1.8$	$26.8$ $\pm 0.4$	$0.314$ $\pm 0.011$	$34.7$ $\pm 1.9$	$35.5$ $\pm 0.4$	$0.473$ $\pm 0.009$
Meta-Harness	$50.7$ $\pm 2.9$	$18.5$ $\pm 0.2$	$0.151$ $\pm 0.004$	$56.7$ $\pm 2.0$	$\mathbf{18.8}$ $\pm 0.5$	$\mathbf{0.153}$ $\pm 0.006$

Table 2: Issue resolution and execution efficiency on two coding benchmarks (frozen Haiku 4.5 and Gemini 3.7 Flash). Per benchmark: pass rate (%, higher is better), average steps, and cost per issue (both lower is better), as mean

\pm

standard error of the mean (SEM) over three runs under a

40

-step,

3 budget. Verified is our held-out

150$ -issue subset. Bold marks the best in each column.

Harness	Pass rate (%)	Turns	Cost ($/task)	Input tokens
OpenHands	$47.3$ $\pm 1.5$	$54.4$	$0.85$	$1.55$ M
Terminus-Kira	$52.3$ $\pm 1.1$	$72.2$	$1.42$	$2.76$ M
Terminus-2	$50.9$ $\pm 0.8$	$75.2$	$1.54$	$3.00$ M
Meta-Harness	$50.5$ $\pm 1.7$	$76.3$	$1.56$	$3.15$ M
Turbo Harness (ours)	$\mathbf{55.5}$ $\pm 1.4$	${71.2}$	${1.38}$	${2.65}$ M

Table 3: Terminal-Bench-2.1: performance and efficiency (frozen Sonnet 4.5, held-out

44

-task test split). Pass rate (%) is mean

\pm

SEM over five runs; turns, cost, and input tokens are per-task means. OpenHands, Terminus-Kira, and Terminus-2 are human-designed harnesses; Meta-Harness is automated. Bold marks the best pass rate.

Figure 3: Ablation on SWE-smith-MR (frozen Haiku 4.5; pass rate %, mean

\pm

SEM over three runs; dashed line = Meta-Harness). (a) For the lightweight Qwen3.5-9B editor, neither the playbook nor RL alone beats Meta-Harness; combined they reach

64.0\%

. (b) Stronger editors exploit the playbook better, yet our small RL-trained editor matches the best.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Epochs	Batch	Group $G$	Max prompt	Max gen.	GPUs
SWE-smith-MR	$5$	$4$	$8$	$16384$	$4096$	$8$
SWE-bench Verified	$5$	$8$	$16$	$16384$	$4096$	$8$
ScienceWorld	$3$	$8$	$8$	$24576$	$8192$	$4$
ALFWorld	$3$	$8$	$8$	$16384$	$8192$	$4$
DBBench	$3$	$8$	$8$	$16384$	$8192$	$4$
WebShop	$8$	$4$	$16$	$16384$	$8192$	$4$

Appendix

Table 4: Harness-editor GRPO training settings per benchmark. All editors are Qwen3.5-9B, full-parameter fine-tuned;

G

is the group size (candidate edits per instance). WebShop uses the tuned recipe (see below): it disables within-group standardization (Dr. GRPO), an improvement bonus, and an abstention penalty.

Figure 4: SWE-smith-MR reliability. The

50

test issues grouped by success count over three runs. Dark orange: all three succeed; dark gray: none.

Explore similar work

Jun 12, 2026cs.AI

HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundry

AI agent performance depends critically on the runtime harness, comprising the prompts, tools, memory, and control flow that mediate how a model observes, reasons, and acts. Yet today's harnesses remain largely hand-crafted and static: each new model or task still demands bespoke scaffolding, and the rich traces produced during execution are rarely distilled back into systematic improvement. We introduce HarnessX, a foundry for composable, adaptive, and evolvable agent harnesses. HarnessX assembles typed harness primitives via a substitution algebra, adapts them through AEGIS, a trace-driven multi-agent evolution engine grounded in an operational mirror between symbolic adaptation and reinforcement learning, and closes the harness-model loop by turning trajectories into both harness updates and model training signal. Across five benchmarks (ALFWorld, GAIA, WebShop, tau^3-Bench, and SWE-bench Verified), HarnessX yields an average gain of +14.5% (up to +44.0%), with gains largest where baselines are lowest. These results suggest that agent progress need not come from model scaling alone: composing and evolving runtime interfaces from execution feedback is an actionable and complementary lever. The complete codebase will be open-sourced in a future release.

Tingyang Chen, Shuo Lu, Kang Zhao +11

Darwin Agent Team

Agent Harness Multi-Agent Evolution

Ask PDF

Sep 29, 2026cs.AI

Self-Evolving Harness on Multiple Tasks with the Agent as Its Own Optimizer

A harness is the code around a language-model agent that organizes prompts, calls tools, manages context, and controls execution. As models grow stronger, recent work has begun to let agents improve their own harnesses, a line of work known as self-evolving harnesses. In most existing methods, a separate proposer running on a human-designed harness modifies the solver's harness, and a separate harness is evolved for each benchmark. Real-world tasks come from many domains, so both the evolution and the evaluation of a harness should cover a diverse range of tasks. We propose a framework close to recursive self-improvement: the same frozen model, on the same version of the harness, first solves tasks as the solver and then, as the proposer, reads the complete run records and directly edits the harness that runs it. Each evolution batch draws tasks from five benchmarks in different domains. To measure generalization, training and held-out tasks are strictly separated, and we additionally evaluate on five out-of-distribution benchmarks never used during evolution. We frame the evolution process as deep-learning training with two stages, multi-task pretraining and continual training. Starting from a 49-line seed harness, the harness obtained at the end of the first stage improves the average score by 4.48 points on the in-distribution benchmarks and by 12.64 points on the out-of-distribution benchmarks, surpassing Codex on the former and matching it on the latter. In the second stage, continued evolution on Claw-Eval, one of the out-of-distribution benchmarks, further raises the score on that benchmark from 66.17 to 68.06, exceeding Codex. We also provide an in-depth analysis of the mechanisms that emerged during evolution, including output truncation, history compaction, and independent review.

Qiankai Xu

Nanjing University

Evolving Harness Language-Model Agents New Tasks

Ask PDF

Aug 3, 2026cs.LG

HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses

Harness design plays a critical role in agent performance by shaping how large language models (LLMs) perceive, reason over, and act within executable environments. Recent work has proposed automatic harness evolution, which iteratively improves the harness from agent--environment interactions. However, existing methods often overfit to the evolution tasks, rely exclusively on trajectory-derived signals, and optimize harness components jointly, causing interference across components. We propose HarnessCompass, a novel automatic harness evolution framework built around constrained evolution, proactive feedback, and component-wise optimization. HarnessCompass first enforces global constraints on evolution, restricting modifications to task-agnostic harness changes that generalize beyond the evolution tasks. It then augments trajectory-derived evidence with proactive first-person feedback from the agent about harness usage, yielding richer signals for evolution. Finally, it decouples the optimization of different harness components before consolidating them into a unified harness, reducing cross-component interference while preserving component synergy. On SWE-bench Verified with GPT-5.4, HarnessCompass improves Pass@1 from 54% to 66% in only 5 evolution iterations, outperforming AHE in both effectiveness and evolution efficiency. In addition, the evolved harness transfers effectively to held-out tasks and other models, demonstrating substantially stronger generalization than prior automatic harness evolution methods.

Luan Zhang, Ruochen Zhou, Dandan Song +9

Beijing Institute of Technology, China · City University of Hong Kong, China · Independent, China

Evolving Harness Agent Harness Evolutionary Optimization Methods

Ask PDF