Elicitation or Expansion? Demystifying Gains from Agent Harness Optimization

Younghyun Kim; Wooseok Seo; Minseo Kim; Seunghyuk Oh; Minjae Lee; Donghoon Kim; Hyung Il Koo; Jaehyung Kim

cs.LGOct 20, 2026

Elicitation or Expansion? Demystifying Gains from Agent Harness Optimization

Authors: Younghyun Kim, Wooseok Seo, Minseo Kim, Seunghyuk Oh, Minjae Lee, Donghoon Kim, Hyung Il Koo, Jaehyung Kim,

Organizations: FuriosaAI · Yonsei University · University of California, Berkeley

Abstract

Recent studies in harness optimization have shown substantial performance gains in language model agents without updating model weights. A well-designed harness more effectively leverages the underlying model's capabilities by shaping context, tool use, and execution workflows. However, naïve success rates alone do not fully characterize whether optimized harness yields more reliable success on tasks already within reach (elicitation) or a broader range of solvable tasks (expansion). This distinction is crucial for recursive self-improvement and evolutionary search, as expansion may provide a broader range of successful experiences to push the capability boundaries. Despite the growing significance, this distinction remains underexplored in harness optimization. We conduct a systematic analysis to study the effect of gains from harness optimization. Building on pass@k analyses in the reinforcement learning (RL) literature that compare models before and after RL training, we compare optimized harnesses with their pre-optimization counterparts. Specifically, we assess changes in reachable task coverage, compare harness optimization with model scaling and increased inference-time compute, and examine how gains vary with task difficulty. Across static prediction and dynamic tasks, we find that pass@1 advantages of harness over stronger interventions can reverse at larger attempt budgets and aggregate gains conceal shortfalls on hard tasks, underscoring the need for a fine-grained diagnosis.

Figures & tables

Panel of Figure 1 — Figure 1: On HotpotQA, the optimized harness enables Qwen3-8B to outperform even 32B under base harness in overall $\mathrm{pass@}1$ . On easy questions, this advantage holds at both $\mathrm{pass@}1$ and $\mathrm{pass@}128$ . On hard questions, however, the optimized harness outperforms the original 8B baseline at $\mathrm{pass@}1$ but falls below it at $\mathrm{pass@}128$ .

Figure 2: Mean accuracy and coverage for harnesses from one optimization run.

Panel of Figure 3 — Figure 3: $\mathrm{pass@}k$ curves for the base solver with baseline ( $H_{0}$ ) and selected optimized ( $H^{\star}$ ) harnesses, compared with a stronger reference using $H_{0}$ . “base” and “reference” denote GPT-OSS-20B and GPT-OSS-120B for text classification (a-c), Qwen3-8B and Qwen3-32B for HotpotQA (d), and GPT-OSS-120B at medium and high reasoning effort for AppWorld (e), respectively. Each red curve represents a harness selected from one optimization run.

Panel of Figure 5 — Figure 5: HotpotQA $\mathrm{pass@}k$ curves for the Qwen3-8B baseline ( $H_{0}$ ) and selected optimized harnesses ( $H^{\star}$ ), with a Qwen3-32B baseline. Each red curve represents one selection from two optimization runs. Neither selected harness exceeds the Qwen3-32B baseline on the hard subset.

Panel of Figure 7 — Figure 7: Task-level $\mathrm{pass@}1$ gains (pp) from harness optimization versus model scaling (Text: GPT-OSS-20B to 120B; HotpotQA: Qwen3-8B to 32B) or reasoning effort (AppWorld: medium to high). Small markers denote tasks; outlined symbols show equal-task group means. The diagonal indicates equal gains.

Panel of Figure 9 — Figure 9: $\mathrm{pass@}k$ curves for Qwen3-8B harnesses optimized with the original and hard-focused task mixtures, alongside the 8B and 32B baselines, overall and by difficulty.

Panel of Figure 10 — Figure 10: Harness transfer to new text-classification datasets. $\mathrm{pass@}k$ curves for GPT-OSS-20B with the baseline ( $H_{0}$ ) and in-distribution selected harness ( $H^{\star}$ ) on out-of-distribution benchmarks; FiNER, GoEmotions, and Banking77.

Panel of Figure 11 — Figure 11: Pseudocode derived from the base harness (a) and the subpart of the selected harness (b).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Prep.	Search/val.	Select.	Test
Symptom2Disease (ID)	200	50	reused	212
LawBench (ID)	200	50	reused	100
USPTO (ID)	50	30	reused	100
FiNER (OOD)	200	—	—	100
GoEmotions (OOD)	200	—	—	150
Banking77 (OOD)	200	—	—	150

Appendix

Table 1: Dataset sizes and split roles. Prep. supplies examples or tasks to the harness; search/validation supplies optimization feedback. Reused selection shares the validation set.

Question	Reference answer	Complete recorded answer
Which band was formed first, Loudspeakers or Marcy Playground?	Marcy Playground	Marcy Playground was formed first.
Which dog is larger, the Cirneco dell’Etna or the Siberian Husky?	Siberian Husky	The Siberian Husky is larger than the Cirneco dell’Etna.
John Alexander is the uncle of the player who plays what position for Liverpool?	right-back	The player is a right-back for Liverpool.
Intimidad de los parques was based on a short story by the novelist and essayist of what nationality?	Argentine	The novelist and essayist was Argentine .
When was the direct-to-video movie that borrows heavily from a 1988 American action film starring Bruce Willis released?	1996	The direct-to-video movie was released in 1996 .

Appendix

Table 2: Exact-match false negatives from the Qwen3-32B HotpotQA baseline. Bold spans mark reference answers in the complete recorded responses.

Panel of Figure 12 — Figure 12: $\mathrm{pass@}k$ across three optimization runs: mean $\pm 1$ sample SD (thick red curves and bands), individual runs (marked curves), and shared baselines (gray).

Panel of Figure 13 — Figure 13: Text-classification $\mathrm{pass@}k$ curves for the GPT-OSS-120B baseline ( $H_{0}$ ) and selected optimized harnesses ( $H^{\star}$ ).

		Tasks			$\mathrm{pass@}1$ gain (pp)
Dataset	$\|S_{0}\|\!\to\!\|S_{\star}\|$	$S_{\star}\setminus S_{0}$	$S_{0}\setminus S_{\star}$	$S_{0}\cap S_{\star}$	Total	$S_{0}\cap S_{\star}$	$S_{\star}\setminus S_{0}$	$S_{0}\setminus S_{\star}$
Symptom2Disease	$203\to 201$	3	5	198	$+2.57$	$+2.63$	$+0.38$	$-0.43$
LawBench	$53\to 56$	7	4	49	$+25.14$	$+21.02$	$+4.64$	$-0.52$
USPTO	$28\to 32$	7	3	25	$+3.39$	$+3.25$	$+0.19$	$-0.05$

Appendix

Table 3: Changes in observed success sets from the GPT-OSS-20B base harness (

H_{0}

) to the selected harness (

H^{\star}

) in Figure 4 (a)-(c), with

n=k=64

S_{0}

and

S_{\star}

are the tasks with at least one success under

H_{0}

and

H^{\star}

	$\mathrm{pass@}1$				$\mathrm{pass@}64$
Dataset	20B $H_{0}$	Rebuilt	Frozen	120B $H_{0}$	20B $H_{0}$	Rebuilt	Frozen	120B $H_{0}$
Symptom2Disease	82.08	84.66	86.34	85.08	95.75	94.81	92.45	95.75
LawBench	18.86	44.00	43.53	27.95	53.00	56.00	53.00	60.00
USPTO	6.25	9.64	10.03	16.94	28.00	32.00	30.00	35.00

Appendix

Table 4:

\mathrm{pass@}1

and

\mathrm{pass@}64

(%) for the text-classification in Figure 4 (a)-(c), with

n=64

. Rebuilt and Frozen denote the selected GPT-OSS-20B harness (

H^{\star}

) with memory rebuilt for each repetition, as in the figure, or prepared once per dataset and reused.

In-distribution
	$\mathrm{pass@}4$			$\mathrm{pass@}8$
Dataset	Guessing	$H_{0}$	$H^{\star}$	Guessing	$H_{0}$	$H^{\star}$
Symptom2Disease	16.98	89.86	90.72	31.08	92.08	92.16
LawBench	—	35.12	50.51	—	42.15	52.59
USPTO	—	12.90	16.10	—	16.08	19.30
Out-of-distribution

Appendix

Table 5: Coverage (%) for uniform-label guessing and the GPT-OSS-20B baseline (

H_{0}

) and selected harness (

H^{\star}

) in Figures 4 and 10 . Guessing values are analytical; harness estimates use

n=64

Panel of Figure 14 — Figure 14: $\mathrm{pass@}k$ curves for the GPT-OSS-20B baseline ( $H_{0}$ ) and selected optimized harnesses ( $H^{\star}$ ), with a GPT-OSS-120B baseline.

Panel of Figure 15 — Figure 15: HotpotQA $\mathrm{pass@}k$ with Judge used for optimization, selection, and evaluation.

Panel of Figure 16 — Figure 16: AppWorld $\mathrm{pass@}k$ with arity-fixed TGC used for optimization, selection, and evaluation.

Caps	$\mathrm{pass@}1$	$\mathrm{pass@}128$
Relaxed	$73.39\pm 1.60$	$87.22\pm 1.54$
Tight	$72.50\pm 1.43$	$87.78\pm 1.39$

Appendix

Table 6: HotpotQA Judge scores (%) on 300 test questions with

n=128

, reported as mean

\pm

sample SD across three runs per cap setting.

Agent Harness Optimization

Agent harness optimization improves the prompts, tools, memory, workflows, and control code that shape how AI agents act.

See all 183 papers

Oct 15, 2026

Harness-Delta Attribution: Dissecting Gains From Harness Evolution

Harness evolution automatically optimizes the prompts, skills, tools, and orchestration code around a fixed-weight model, offering a way to improve agentic systems without updating model weights. The agent harness is iteratively optimized to achieve higher performance on the search set. But how much of this gain reflects benchmark-specific overfitting or additional inference-time compute, and how much may generalize beyond the search set? In this work, we introduce Harness-Delta Attribution (HDA), an evaluation protocol that dissects an evolution gain into artifact-driven overfitting, gains explained by test-time scaling, and the remainder, which gives an upper bound on residual generalizable improvement. We apply HDA to 11 executor models and three harness-evolution frameworks across math, coding, creativity, and agentic tasks. In 21 of these settings that show statistically significant search-set gains, we find that a large share of the improvement is explained by overfitting or test-time scaling. Specifically, among these 21 settings, overfitting and test-time scaling account for 54% and 14% of the gain on average; their combined share exceeds 50% in 14 settings. The residual and held-out transfer vary substantially across tasks, models, and frameworks. Importantly, search-set gains often fail to transfer to held-out tasks, and provide limited evidence that they will persist under practical distribution shifts. Moreover, validation-based signals are insufficient to prevent overfitting. These results motivate attribution-aware evaluation in future harness evolution research.

Wenxuan Ding, Qin Lu, Changlong Yu +5

Amazon · New York University

Agent Harness Optimization Agent Evaluation Benchmark Contamination Test-Time Scaling

Ask PDF

Oct 4, 2026

MESH-Harness: Self-Improving Agent Harnesses via Bandit-Guided Compositional Evolution

An agent harness is the code that organizes context, maintains state, and coordinates tool calls for a language model. We study how to improve the harness under a limited evaluation budget while keeping model weights fixed. Our method, MESH-Harness, organizes each harness into functional modules with explicit role-specific interfaces, allowing alternative implementations of each module to be substituted and recombined. It uses shared module representations and full-covariance LinUCB to score candidate combinations based on predicted performance and exploration value. Mixed-start coordinate ascent selects complete configurations for evaluation without enumerating the combinatorial space. Validation traces then guide local code edits, and the resulting candidates are incorporated into fixed-capacity role-specific pools for subsequent recombination. On text tasks, retrieval-augmented mathematical reasoning, code generation, and interactive scientific tasks, MESH-Harness outperforms Meta-Harness by 5.70, 7.01, 2.00, and 5.00 points, respectively, under matched candidate-evaluation budgets. Iterative harness optimization improves MESH-Harness by 5.63-7.79 points over its first-round configurations. For the reported configurations, aggregate test-time cost is 44.2% lower than that of Meta-Harness, while total cost including search is 14.6% lower. These results show that combining module-level design reuse with feedback-driven compositional search can systematically improve agent harnesses while keeping overall optimization cost under control.

Zhiwei Shang, Yu Huo, Mingrong Gong +6

The Chinese University of Hong Kong, Shenzhen · DeepWisdom · The Chinese University of Hong Kong +3

Agent Harness Optimization LLM Agent Workflow Optimization LLM Agent Self-Improvement LLM Agent Harnesses

Ask PDF

Aug 6, 2026

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

As LLMs are increasingly deployed within agentic systems, their capabilities depend not only on the model weights but also on the harness: the prompts, tools, control flow, memory, and orchestration code surrounding them. This makes automated harness optimization -- the iterative and evaluation-guided improvement of a harness by an AI system -- both an important route to improving AI systems and a demanding capability for AI systems themselves. Yet the community lacks a common protocol for measuring how well frontier LLMs perform at this task. We introduce HarnessOpt-Bench, a benchmark for end-to-end harness optimization under expensive and stochastic evaluation. An optimizer, an LLM paired with a coding harness, receives a target agent's seed harness, graded evaluation feedback, and a fixed target-evaluation budget. It edits the harness and nominates a final candidate, which is scored by its normalized gain over the seed on a held-out test partition that remains inaccessible throughout search. A trusted execution environment enforces the evaluation boundary, meters target-agent resource use, and preserves candidate versions for audit. We evaluate 5 frontier LLMs as optimizers both under a shared coding harness and under their native harnesses across 4 downstream tasks, over 111 scored runs. Experiment results show that optimizer models separate more than the coding harnesses they act through, native harnesses are not consistently superior, and gains vary substantially across tasks and seed regimes. These results establish harness optimization as a measurable and discriminative capability with large space for improvement.

Varun Ursekar, Apaar Shanker, Yash Maurya +4

Scale AI

Large Language Model-Guided Optimization LLM Agent Evaluation Agent Harness Optimization LLM Agent Workflow Optimization AI Agent Benchmarks

Ask PDF

Abstract

Agent Harness Optimization

Explore similar work

Harness-Delta Attribution: Dissecting Gains From Harness Evolution

MESH-Harness: Self-Improving Agent Harnesses via Bandit-Guided Compositional Evolution

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization