Elicitation or Expansion? Demystifying Gains from Agent Harness Optimization

Younghyun Kim; Wooseok Seo; Minseo Kim; Seunghyuk Oh; Minjae Lee; Donghoon Kim; Hyung Il Koo; Jaehyung Kim

cs.LGOct 20, 2026

Elicitation or Expansion? Demystifying Gains from Agent Harness Optimization

Authors: Younghyun Kim, Wooseok Seo, Minseo Kim, Seunghyuk Oh, Minjae Lee, Donghoon Kim, Hyung Il Koo, Jaehyung Kim,

Organizations: FuriosaAI · Yonsei University · University of California, Berkeley

Abstract

Recent studies in harness optimization have shown substantial performance gains in language model agents without updating model weights. A well-designed harness more effectively leverages the underlying model's capabilities by shaping context, tool use, and execution workflows. However, naïve success rates alone do not fully characterize whether optimized harness yields more reliable success on tasks already within reach (elicitation) or a broader range of solvable tasks (expansion). This distinction is crucial for recursive self-improvement and evolutionary search, as expansion may provide a broader range of successful experiences to push the capability boundaries. Despite the growing significance, this distinction remains underexplored in harness optimization. We conduct a systematic analysis to study the effect of gains from harness optimization. Building on pass@k analyses in the reinforcement learning (RL) literature that compare models before and after RL training, we compare optimized harnesses with their pre-optimization counterparts. Specifically, we assess changes in reachable task coverage, compare harness optimization with model scaling and increased inference-time compute, and examine how gains vary with task difficulty. Across static prediction and dynamic tasks, we find that pass@1 advantages of harness over stronger interventions can reverse at larger attempt budgets and aggregate gains conceal shortfalls on hard tasks, underscoring the need for a fine-grained diagnosis.

Figures & tables

Panel of Figure 1 — Figure 1: On HotpotQA, the optimized harness enables Qwen3-8B to outperform even 32B under base harness in overall $\mathrm{pass@}1$ . On easy questions, this advantage holds at both $\mathrm{pass@}1$ and $\mathrm{pass@}128$ . On hard questions, however, the optimized harness outperforms the original 8B baseline at $\mathrm{pass@}1$ but falls below it at $\mathrm{pass@}128$ .

Figure 2: Mean accuracy and coverage for harnesses from one optimization run.

Panel of Figure 3 — Figure 3: $\mathrm{pass@}k$ curves for the base solver with baseline ( $H_{0}$ ) and selected optimized ( $H^{\star}$ ) harnesses, compared with a stronger reference using $H_{0}$ . “base” and “reference” denote GPT-OSS-20B and GPT-OSS-120B for text classification (a-c), Qwen3-8B and Qwen3-32B for HotpotQA (d), and GPT-OSS-120B at medium and high reasoning effort for AppWorld (e), respectively. Each red curve represents a harness selected from one optimization run.

Panel of Figure 5 — Figure 5: HotpotQA $\mathrm{pass@}k$ curves for the Qwen3-8B baseline ( $H_{0}$ ) and selected optimized harnesses ( $H^{\star}$ ), with a Qwen3-32B baseline. Each red curve represents one selection from two optimization runs. Neither selected harness exceeds the Qwen3-32B baseline on the hard subset.

Panel of Figure 7 — Figure 7: Task-level $\mathrm{pass@}1$ gains (pp) from harness optimization versus model scaling (Text: GPT-OSS-20B to 120B; HotpotQA: Qwen3-8B to 32B) or reasoning effort (AppWorld: medium to high). Small markers denote tasks; outlined symbols show equal-task group means. The diagonal indicates equal gains.

Panel of Figure 9 — Figure 9: $\mathrm{pass@}k$ curves for Qwen3-8B harnesses optimized with the original and hard-focused task mixtures, alongside the 8B and 32B baselines, overall and by difficulty.

Panel of Figure 10 — Figure 10: Harness transfer to new text-classification datasets. $\mathrm{pass@}k$ curves for GPT-OSS-20B with the baseline ( $H_{0}$ ) and in-distribution selected harness ( $H^{\star}$ ) on out-of-distribution benchmarks; FiNER, GoEmotions, and Banking77.

Panel of Figure 11 — Figure 11: Pseudocode derived from the base harness (a) and the subpart of the selected harness (b).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Prep.	Search/val.	Select.	Test
Symptom2Disease (ID)	200	50	reused	212
LawBench (ID)	200	50	reused	100
USPTO (ID)	50	30	reused	100
FiNER (OOD)	200	—	—	100
GoEmotions (OOD)	200	—	—	150
Banking77 (OOD)	200	—	—	150

Appendix

Table 1: Dataset sizes and split roles. Prep. supplies examples or tasks to the harness; search/validation supplies optimization feedback. Reused selection shares the validation set.

Question	Reference answer	Complete recorded answer
Which band was formed first, Loudspeakers or Marcy Playground?	Marcy Playground	Marcy Playground was formed first.
Which dog is larger, the Cirneco dell’Etna or the Siberian Husky?	Siberian Husky	The Siberian Husky is larger than the Cirneco dell’Etna.
John Alexander is the uncle of the player who plays what position for Liverpool?	right-back	The player is a right-back for Liverpool.
Intimidad de los parques was based on a short story by the novelist and essayist of what nationality?	Argentine	The novelist and essayist was Argentine .
When was the direct-to-video movie that borrows heavily from a 1988 American action film starring Bruce Willis released?	1996	The direct-to-video movie was released in 1996 .

Appendix

Table 2: Exact-match false negatives from the Qwen3-32B HotpotQA baseline. Bold spans mark reference answers in the complete recorded responses.

Panel of Figure 12 — Figure 12: $\mathrm{pass@}k$ across three optimization runs: mean $\pm 1$ sample SD (thick red curves and bands), individual runs (marked curves), and shared baselines (gray).

Panel of Figure 13 — Figure 13: Text-classification $\mathrm{pass@}k$ curves for the GPT-OSS-120B baseline ( $H_{0}$ ) and selected optimized harnesses ( $H^{\star}$ ).

		Tasks			$\mathrm{pass@}1$ gain (pp)
Dataset	$\|S_{0}\|\!\to\!\|S_{\star}\|$	$S_{\star}\setminus S_{0}$	$S_{0}\setminus S_{\star}$	$S_{0}\cap S_{\star}$	Total	$S_{0}\cap S_{\star}$	$S_{\star}\setminus S_{0}$	$S_{0}\setminus S_{\star}$
Symptom2Disease	$203\to 201$	3	5	198	$+2.57$	$+2.63$	$+0.38$	$-0.43$
LawBench	$53\to 56$	7	4	49	$+25.14$	$+21.02$	$+4.64$	$-0.52$
USPTO	$28\to 32$	7	3	25	$+3.39$	$+3.25$	$+0.19$	$-0.05$

Appendix

Table 3: Changes in observed success sets from the GPT-OSS-20B base harness (

H_{0}

) to the selected harness (

H^{\star}

) in Figure 4 (a)-(c), with

n=k=64

S_{0}

and

S_{\star}

are the tasks with at least one success under

H_{0}

and

H^{\star}

	$\mathrm{pass@}1$				$\mathrm{pass@}64$
Dataset	20B $H_{0}$	Rebuilt	Frozen	120B $H_{0}$	20B $H_{0}$	Rebuilt	Frozen	120B $H_{0}$
Symptom2Disease	82.08	84.66	86.34	85.08	95.75	94.81	92.45	95.75
LawBench	18.86	44.00	43.53	27.95	53.00	56.00	53.00	60.00
USPTO	6.25	9.64	10.03	16.94	28.00	32.00	30.00	35.00

Appendix

Table 4:

\mathrm{pass@}1

and

\mathrm{pass@}64

(%) for the text-classification in Figure 4 (a)-(c), with

n=64

. Rebuilt and Frozen denote the selected GPT-OSS-20B harness (

H^{\star}

) with memory rebuilt for each repetition, as in the figure, or prepared once per dataset and reused.

In-distribution
	$\mathrm{pass@}4$			$\mathrm{pass@}8$
Dataset	Guessing	$H_{0}$	$H^{\star}$	Guessing	$H_{0}$	$H^{\star}$
Symptom2Disease	16.98	89.86	90.72	31.08	92.08	92.16
LawBench	—	35.12	50.51	—	42.15	52.59
USPTO	—	12.90	16.10	—	16.08	19.30
Out-of-distribution

Appendix

Table 5: Coverage (%) for uniform-label guessing and the GPT-OSS-20B baseline (

H_{0}

) and selected harness (

H^{\star}

) in Figures 4 and 10 . Guessing values are analytical; harness estimates use

n=64

Panel of Figure 14 — Figure 14: $\mathrm{pass@}k$ curves for the GPT-OSS-20B baseline ( $H_{0}$ ) and selected optimized harnesses ( $H^{\star}$ ), with a GPT-OSS-120B baseline.

Panel of Figure 15 — Figure 15: HotpotQA $\mathrm{pass@}k$ with Judge used for optimization, selection, and evaluation.

Panel of Figure 16 — Figure 16: AppWorld $\mathrm{pass@}k$ with arity-fixed TGC used for optimization, selection, and evaluation.

Caps	$\mathrm{pass@}1$	$\mathrm{pass@}128$
Relaxed	$73.39\pm 1.60$	$87.22\pm 1.54$
Tight	$72.50\pm 1.43$	$87.78\pm 1.39$

Appendix

Table 6: HotpotQA Judge scores (%) on 300 test questions with

n=128

, reported as mean

\pm

sample SD across three runs per cap setting.

Agent Harness Optimization

Agent harness optimization improves the prompts, tools, memory, workflows, and control code that shape how AI agents act.

See all 183 papers

Harness-Delta Attribution: Dissecting Gains From Harness Evolution
Oct 15, 2026Wenxuan Ding, Qin Lu, Changlong Yu +5Agent Harness Optimization Agent Evaluation
Amazon · New York University
MESH-Harness: Self-Improving Agent Harnesses via Bandit-Guided Compositional Evolution
Oct 4, 2026Zhiwei Shang, Yu Huo, Mingrong Gong +6Agent Harness Optimization LLM Agent Workflow Optimization
The Chinese University of Hong Kong, Shenzhen · DeepWisdom · The Chinese University of Hong Kong +3
HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
Aug 6, 2026Varun Ursekar, Apaar Shanker, Yash Maurya +4Large Language Model-Guided Optimization LLM Agent Evaluation
Scale AI

Oct 15, 2026J/K move · Enter open · S save

Harness-Delta Attribution: Dissecting Gains From Harness Evolution

Wenxuan Ding, Qin Lu, Changlong Yu, Sha Li+4

Amazon · New York University

Agent Harness Optimization Agent Evaluation Benchmark Contamination Test-Time Scaling

Open paper AskPDF arXiv

Harness evolution automatically optimizes the prompts, skills, tools, and orchestration code around a fixed-weight model, offering a way to improve agentic systems without updating model weights. The agent harness is iteratively optimized to achieve higher performance on the search set. But how much of this gain reflects benchmark-specific overfitting or additional inference-time compute, and how much may generalize beyond the search set? In this work, we introduce Harness-Delta Attribution (HDA), an evaluation protocol that dissects an evolution gain into artifact-driven overfitting, gains explained by test-time scaling, and the remainder, which gives an upper bound on residual generalizable improvement. We apply HDA to 11 executor models and three harness-evolution frameworks across math, coding, creativity, and agentic tasks. In 21 of these settings that show statistically significant search-set gains, we find that a large share of the improvement is explained by overfitting or test-time scaling. Specifically, among these 21 settings, overfitting and test-time scaling account for 54% and 14% of the gain on average; their combined share exceeds 50% in 14 settings. The residual and held-out transfer vary substantially across tasks, models, and frameworks. Importantly, search-set gains often fail to transfer to held-out tasks, and provide limited evidence that they will persist under practical distribution shifts. Moreover, validation-based signals are insufficient to prevent overfitting. These results motivate attribution-aware evaluation in future harness evolution research.

Abstract

Agent Harness Optimization

Explore similar work

Harness-Delta Attribution: Dissecting Gains From Harness Evolution

MESH-Harness: Self-Improving Agent Harnesses via Bandit-Guided Compositional Evolution

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization