HarnessFlow: MCTS-Guided Harness Optimization for LLM Agents

Zhi Hong; Qianlong Wang; Hongchen Wei; Aidan Wu; Daohong Ying; Mingze Kong; Bei Liu; Chong Luo

cs.AINov 4, 2026

HarnessFlow: MCTS-Guided Harness Optimization for LLM Agents

Authors: Zhi Hong, Qianlong Wang, Hongchen Wei, Aidan Wu, Daohong Ying, Mingze Kong, Bei Liu, Chong Luo

Organizations: South China University of Technology · Fudan University · Wuhan University · Cornell University ·

Abstract

The performance of large language model (LLM) agents depends on both the underlying model and the harness that governs interaction with tools and the environment. Yet harness design remains largely manual, and automating it requires navigating an open-ended search space through costly end-to-end evaluations. We introduce HarnessFlow, a framework for optimizing complete executable harnesses under a limited candidate-evaluation budget, with the model and task environment held fixed. HarnessFlow organizes candidates and their evaluation histories in a refinement tree, using a Monte Carlo Tree Search (MCTS)-based policy to balance exploitation and exploration. A coding agent refines selected harnesses using archived code, execution feedback, and a library of reusable harness mechanisms. With 15 refinement attempts per benchmark, HarnessFlow achieves the highest mean test scores among the compared methods on six benchmarks spanning agentic tasks, coding, and web interaction. The optimized harnesses achieve an average score of 69.48%, outperforming the initial harnesses and Meta-Harness by 10.89 and 3.95 percentage points, respectively, while incurring lower average per-task execution costs than hand-engineered baselines. These results suggest that structured harness search offers a promising route toward automating the design of capable and efficient LLM agents.

Figures & tables

Panel of Figure 1 — Figure 1: Performance across six benchmarks. (a) Scores on individual benchmarks; labels indicate the best score in each group. (b) Ranking by average score. HarnessFlow outperforms Vanilla and the strongest prior method by $10.89$ and $3.95$ points on average, respectively.

Figure 2: Overview of HarnessFlow . Selection, expansion, validation, and experience backpropagation organize full-harness search. A coding agent uses reusable components and archived code, scores, and traces to refine selected harnesses.

	Agentic		Web		Code
Method	TAU2	AppWorld	WebShop	BC+	SWE-V	BCB-H	Avg.
Vanilla	63.98 $\pm$ 2.15	72.00 $\pm$ 1.86	47.79 $\pm$ 1.14	72.40 $\pm$ 1.85	67.10 $\pm$ 0.89	28.30 $\pm$ 0.76	58.59
Claude Code ( Anthropic, 2026 )	73.12 $\pm$ 1.14	70.86 $\pm$ 2.00	56.98 $\pm$ 1.19	73.20 $\pm$ 1.72	65.30 $\pm$ 1.30	30.80 $\pm$ 2.61	61.71
OpenHands ( Wang et al., 2025 )	71.83 $\pm$ 1.59	66.29 $\pm$ 1.32	65.54 $\pm$ 1.77	74.00 $\pm$ 1.97	67.20 $\pm$ 1.64	30.80 $\pm$ 2.86	62.61
Hermes ( Nous Research, 2026 )	72.90 $\pm$ 2.17	74.19 $\pm$ 1.03	64.50 $\pm$ 1.34	74.30 $\pm$ 1.30	68.10 $\pm$ 2.53	31.70 $\pm$ 1.15	64.28
Codex ( OpenAI, 2026a )	75.91 $\pm$ 2.98	73.24 $\pm$ 1.97	66.15 $\pm$ 1.49	74.70 $\pm$ 2.36	68.70 $\pm$ 2.11	30.20 $\pm$ 1.10	64.82

Table 1: Performance comparison across six widely used benchmarks. We report mean score (%)

\pm

standard deviation over five runs. All methods use the same backbone model, tool access, and per-task limits. Bold marks the highest mean score for each benchmark.

Panel of Figure 3 — Figure 3: Search efficiency and execution cost across six benchmarks. (a) Mean candidate validation scores (solid) and mean per-benchmark best-so-far scores (dashed), from a shared round- $0$ harness. (b) Mean test score versus geometric-mean execution cost per task, excluding search cost; dashed segment: Pareto frontier. Per-benchmark results: Appendices D.1 and D.3 .

Figure 4: Harness evolution on SWE-bench Verified. Code-level refinements along the selected lineage raise the validation score from

67.60\%

79.20\%

. Parent–child labels show actual search edges; scores describe cumulative harness designs, not isolated effects of individual edits.

Table 6

	TAU2		AppWorld		WebShop		Avg.
Method	Score	Cost	Score	Cost	Score	Cost	Score	Cost
Vanilla	83.76 $\pm$ 2.83	0.23	82.48 $\pm$ 1.09	0.38	56.29 $\pm$ 1.20	0.08	74.18	0.23
Codex	85.91 $\pm$ 1.59	0.47	75.52 $\pm$ 2.37	0.57	73.59 $\pm$ 0.89	0.38	78.34	0.47
Hermes	85.81 $\pm$ 1.73	0.70	79.43 $\pm$ 1.36	0.62	72.76 $\pm$ 1.43	0.47	79.33	0.60
OpenHands	83.12 $\pm$ 3.64	0.48	83.71 $\pm$ 1.32	0.52	71.62 $\pm$ 0.59	0.41	79.48	0.47
Claude Code	82.69 $\pm$ 1.76	0.61	87.81 $\pm$ 1.92	0.81	72.06 $\pm$ 1.36	0.59	80.85	0.67

Table 4: Comparison with a stronger executor. All methods use GPT-5.4 as the executor. For HarnessFlow and Meta-Harness, we rerun harness search from the same initial harness using this executor. Cost is USD per test case at the executor’s list price (

2.50 per million input tokens and

15.00 per million output tokens). Rows are ordered by average score.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Functional organization of the Harness Component Library’s

21

indexed modules. The four columns and the shared-support row indicate functional roles. The coding agent copies adopted modules from a read-only snapshot into the candidate.

Benchmark	Reference agent	Revision
TAU2-bench	tau2-bench LLMAgent ( Barres et al., 2025 )	a2c0247
AppWorld	Official ReAct experiment harness ( Trivedi et al., 2024 )	66ad809 (v0.1.3.post1)
WebShop	ReAct WebShop scaffold ( Yao et al., 2022a )	a5c0827
BrowseComp-Plus	search_agent/openai_client.py ( Chen et al., 2026b )	0469490
SWE-bench Verified	mini-SWE-agent DefaultAgent ( Yang et al., 2024b )	04d809c
BigCodeBench-Hard	bigcodebench.generate ( Zhuo et al., 2025 )	v0.1.4

Appendix

Table 5: The initial harness of each benchmark and its upstream source. Revision is the git commit (abbreviated to seven characters) or release of the upstream repository from which the harness was reproduced.

Agent	Release	Source
Claude Code ( Anthropic, 2026 )	2.1.272	npm @anthropic-ai/claude-code
Codex ( OpenAI, 2026a )	0.154.0	npm @openai/codex
OpenHands ( Wang et al., 2025 )	cloud-1.45.0 (SDK 1.34.0)	GitHub
Hermes ( Nous Research, 2026 )	v2026.9.7	GitHub

Appendix

Table 6: Production agent harnesses and the release used throughout.

Benchmark	Episode limit	Deadline
TAU2-bench	$200$ steps	$1{,}800$ s
AppWorld	$50$ interactions	$1{,}800$ s
WebShop	$30$ actions	$900$ s
BrowseComp-Plus	$100$ tool calls	$1{,}800$ s
SWE-bench Verified	$250$ model calls	$2{,}400$ s
BigCodeBench-Hard	$30$ iterations	$1{,}800$ s

Appendix

Table 7: Per-benchmark execution limits, applied identically to every method. Deadlines are wall-clock per task.

Figure 6: Best validation score reached so far at each round, per benchmark, as the dashed lines of Figure 3(a) . Round

0

is the shared initial harness.

Figure 7: Harness evolution on TAU2-bench. Action reviews, structured evidence, and troubleshooting continuation raise the validation score from

59.35\%

81.30\%

Method	TAU2	AppWorld	WebShop	BC+	SWE-V	BCB-H	Avg.
	Agentic		Web		Code
Vanilla	0.067	0.132	0.030	2.292	0.181	0.007	0.451
Claude Code	0.177	0.383	0.191	1.444	0.456	0.024	0.446
OpenHands	0.148	0.286	0.128	1.365	0.445	0.017	0.398
Hermes	0.215	0.350	0.167	1.507	1.050	0.043	0.555
Codex	0.128	0.481	0.134	2.027	0.569	0.025	0.561

Appendix

Table 8: Execution cost of every method in Table 1 , in USD per test task: input and output tokens priced separately at the executor’s list price (GPT-5.4-mini:

0.75 per million input tokens and

4.50 per million output tokens). Bold marks the cheapest method per column.

Figure 8: Test score (Table 1 ) against execution cost per task (Table 8 ) on each benchmark. The dashed line joins the methods on the Pareto frontier; axis limits differ per panel because costs span two orders of magnitude between benchmarks.

Agent Harness Optimization

Agent harness optimization improves the prompts, tools, memory, workflows, and control code that shape how AI agents act.

See all 197 papers

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
Aug 6, 2026Varun Ursekar, Apaar Shanker, Yash Maurya +4Large Language Model-Guided Optimization LLM Agent Evaluation
Scale AI
MESH-Harness: Self-Improving Agent Harnesses via Bandit-Guided Compositional Evolution
Oct 4, 2026Zhiwei Shang, Yu Huo, Mingrong Gong +6Agent Harness Optimization LLM Agent Workflow Optimization
The Chinese University of Hong Kong, Shenzhen · DeepWisdom · The Chinese University of Hong Kong +3
HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
Sep 1, 2026Yuhao Wu, Jingyuan Zhang, Jiajun Shi +16LLM Agent Evaluation Agent Harness Optimization

Aug 6, 2026J/K move · Enter open · S save

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

Varun Ursekar, Apaar Shanker, Yash Maurya, Shehab Yasser+3

Scale AI

Large Language Model-Guided Optimization LLM Agent Evaluation Agent Harness Optimization LLM Agent Workflow Optimization AI Agent Benchmarks

Open paper AskPDF arXiv

As LLMs are increasingly deployed within agentic systems, their capabilities depend not only on the model weights but also on the harness: the prompts, tools, control flow, memory, and orchestration code surrounding them. This makes automated harness optimization -- the iterative and evaluation-guided improvement of a harness by an AI system -- both an important route to improving AI systems and a demanding capability for AI systems themselves. Yet the community lacks a common protocol for measuring how well frontier LLMs perform at this task. We introduce HarnessOpt-Bench, a benchmark for end-to-end harness optimization under expensive and stochastic evaluation. An optimizer, an LLM paired with a coding harness, receives a target agent's seed harness, graded evaluation feedback, and a fixed target-evaluation budget. It edits the harness and nominates a final candidate, which is scored by its normalized gain over the seed on a held-out test partition that remains inaccessible throughout search. A trusted execution environment enforces the evaluation boundary, meters target-agent resource use, and preserves candidate versions for audit. We evaluate 5 frontier LLMs as optimizers both under a shared coding harness and under their native harnesses across 4 downstream tasks, over 111 scored runs. Experiment results show that optimizer models separate more than the coding harnesses they act through, native harnesses are not consistently superior, and gains vary substantially across tasks and seed regimes. These results establish harness optimization as a measurable and discriminative capability with large space for improvement.

Abstract

Agent Harness Optimization

Explore similar work

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

MESH-Harness: Self-Improving Agent Harnesses via Bandit-Guided Compositional Evolution

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?