SHarP: Saliency-based Pruning of Agent Harnesses

Xinyi Gao; Qiucheng Wu; Kaizhi Qian; Handong Zhao; Shiyu Chang; Yang Zhang

cs.AIOct 3, 2026

SHarP: Saliency-based Pruning of Agent Harnesses

Authors: Xinyi Gao, Qiucheng Wu, Kaizhi Qian, Handong Zhao, Shiyu Chang, Yang Zhang

Organizations: University of California, Santa Barbara · MIT-IBM Watson AI Lab · Adobe Research

Abstract

Agent harnesses are systems that coordinate model calls, tool use, and task execution to help large language models complete complex tasks. To meet task requirements and address failures, these systems are often iteratively refined by amending and patching their instructions, tools, and workflows, continuously increasing harness complexity. It is therefore unclear whether some resulting harness modules are redundant, introducing substantial token overhead with little, if any, performance gain. Inspired by neural network pruning, in this paper, we study harness pruning as a means of striking a better balance between task performance and token cost. We propose SHarP (Saliency-based Harness Pruning), a simple yet effective pruning strategy based on the saliency of each harness module with respect to performance and efficiency. Specifically, we first identify tools, instructions, and supporting mechanisms as components that can be individually disabled. We then estimate the saliency of each module by ablating it and assessing its task performance and token cost relative to the full set of single-module ablations. Modules with the smallest contribution to performance or largest computational overhead are subsequently pruned. Our evaluation across various harnesses on held-out validation sets reveals a surprising finding: most harnesses that we studied are highly redundant and can maintain comparable performance and efficiency even after a substantial portion of their modules are pruned. Our pruning approach and empirical findings provide new perspectives on agent harness design and optimization.

Figures & tables

Figure 1: Overview of SHarP ’s efficiency-oriented pruning. Single-module ablations estimate performance and efficiency saliency, which determine the protected module set (green) and pruning order, respectively. Pruned configurations and an all-pruned baseline are evaluated for task performance and token cost.

Panel of Figure 2 — Figure 2: Task performance (left) and token cost (right) as more modules are pruned under the efficiency-oriented strategy. Lines show means over three trials; faint points show individual trials, and shading spans their minimum and maximum. Token cost uses the per-trial median across tasks. When the protected set is nonempty, hollow circles mark the additional all-pruned baseline.

Panel of Figure 3 — Figure 3: Task performance and token cost across pruned configurations. Stars mark full harnesses. Hollow circles mark the additional all-pruned baseline when the protected set is nonempty. Lines trace the Pareto frontier: no other evaluated configuration achieves higher task performance without increasing token cost, or lower token cost without reducing task performance. Panels (b) and (c) show zoomed-in views, omitting PaperQA2’s $\mathcal{S}_{3}$ and JIT agentfold’s $\mathcal{S}_{0}$ , respectively, whose high token costs would compress the remaining configurations into a narrow region. Full-range plots are provided in Appendix Figure 5 .

JIT agentfold / DeepPlanning-Shopping						LIFE-harness / $\tau^{2}$ -bench retail
Performance-oriented			Efficiency-oriented			Performance-oriented			Efficiency-oriented
Config.	$P$	$T$	Config.	$P$	$T$	Config.	$P$	$T$	Config.	$P$	$T$
$\mathcal{S}_{23}$ (full)	75.8	324	$\mathcal{S}_{23}$ (full)	75.8	324	$\mathcal{S}_{48}$ (full)	67.5	121	$\mathcal{S}_{48}$ (full)	67.5	121
$\mathcal{S}^{\prime}_{22}$	77.7	278	$\mathcal{S}_{20}$	76.2	197	$\mathcal{S}^{\prime}_{43}$	80.0	118	$\mathcal{S}_{46}$	72.5	115
$\mathcal{S}^{\prime}_{19}$	82.5	305	$\mathcal{S}_{17}$	67.2	196	$\mathcal{S}^{\prime}_{37}$	72.5	117	$\mathcal{S}_{43}$	87.5	118
$\mathcal{S}^{\prime}_{16}$	76.9	219	$\mathcal{S}_{14}$	71.9	198	$\mathcal{S}^{\prime}_{31}$	75.0	114	$\mathcal{S}_{37}$	80.0	105

Table 1: Comparison of performance-oriented and efficiency-oriented pruning with one trial per validation task.

P

is task performance (%);

T

is mean token cost (

10^{3}

\mathcal{S}^{\prime}_{k}

and

\mathcal{S}_{k}

denote performance-oriented and efficiency-oriented configurations, respectively. Full and all-pruned results are shared between strategies. Bold values mark the best result in each metric column. Retail configurations are selected for brevity; complete results are provided in Appendix A.5 .

Figure 4: Examples from PaperQA2 showing how pruning can increase token cost. Disabling the completion tool prolongs execution after an answer is generated, while disabling evidence retrieval causes all collected passages to be summarized instead of a selected subset.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Module	Function	Disabling notes
Paper search tool	Searches for papers to add to the available document collection.	–
Evidence gathering tool	Collects question-relevant evidence from the available papers.	–
Answer generation tool	Synthesizes an answer using the collected evidence.	–
Reset tool	Clears collected evidence so the agent can gather new evidence.	–
Completion tool	Ends the agent loop using the latest proposed answer.	–
Agent system prompt	Supplies the agent’s system-level instructions.	–

Appendix

Table 2: Candidate modules in PaperQA2. Functions describe enabled behavior; disabling notes clarify alternative behavior where needed.

Module	Function	Disabling notes
Terminal tool	Executes shell commands in the workspace.	–
Finish tool	Submits the final response and ends the agent loop.	–
Think tool	Records a reasoning step without changing the environment.	–
Browser tools	Navigate and interact with web pages, with accompanying usage guidance.	–
Fetch tool	Retrieves web content from a URL.	–
Search tool	Searches the web for relevant information.	–

Appendix

Table 3: Candidate modules in OpenHands. Each row corresponds to one configurable module.

Module	Function	Disabling notes
Initial planning	Generates a plan with goals and candidate solution paths.	Starts without an initial plan; the step budget is unchanged.
Periodic replanning	Reviews progress and updates the plan every eight steps.	–
History folding	Replaces selected ranges of past steps with model-generated summaries.	Keeps the original history instead of applying requested summaries.
Forced final answer	Makes a final answer-generation call when the step limit is reached.	Returns an empty answer if the loop ends without a submitted answer.
Lenient JSON parsing	Repairs malformed action JSON.	Uses strict parsing.
Task restatement	Repeats the original task in each step prompt.	–

Appendix

Table 4: Candidate modules in JIT agentfold. The shopping guidance varies with task difficulty.

Module	Function	Disabling notes
Booking payment limits	Checks the permitted numbers of certificates, credit cards, and gift cards.	–
Booking itinerary check	Checks that flight legs form a connected itinerary with the required endpoints.	–
Cancellation eligibility	Checks cancellation conditions and whether flights have already departed.	–
Basic-economy changes	Blocks flight changes that violate basic-economy restrictions.	–
Compensation eligibility	Checks membership, cabin, or insurance eligibility for compensation.	–
Route preservation	Blocks changes to the reservation’s origin or destination.	–

Appendix

Table 5: Candidate modules in LIFE-harness for the airline domain.

Module	Function	Disabling notes
Cancellation status check	Restricts cancellation to pending orders.	–
Item-modification status check	Restricts item changes to orders still eligible for modification.	–
Exchange status check	Restricts exchanges to eligible delivered orders.	–
Return eligibility check	Checks delivered status and the permitted refund method.	–
Cancellation reason check	Checks that the cancellation reason is an accepted value.	–
Payment-change check	Requires the new payment method to differ from the current one.	–

Appendix

Table 6: Candidate modules in LIFE-harness for the retail domain.

Benchmark	Harness	Development	Validation	$m$
GAIA	OpenHands	50	60	27
LitQA2	PaperQA2	50	38	18
$\tau^{2}$ -bench airline	LIFE-harness	30	20	44
$\tau^{2}$ -bench retail	LIFE-harness	48	40	48
DeepPlanning-Shopping	JIT agentfold	50	30	23

Appendix

Table 7: Dataset and harness statistics. Development and validation denote the numbers of tasks in the respective sets, and

m

is the number of candidate modules.

Model: Qwen3.5-122B-A10B-FP8; inference engine: vLLM; thinking: disabled
Typical deployment: $2$ or $4\times$ NVIDIA H100 GPUs with tensor parallelism
Harness / benchmark	Temperature	Max output tokens	Execution budget
OpenHands / GAIA	0.7	–	30 iterations
PaperQA2 / LitQA2	0	4,096	500 seconds
LIFE-harness / Airline	0	16,384	200 steps
LIFE-harness / Retail	0	16,384	200 steps

Appendix

Table 8: Inference settings and execution budgets. Output limits are per model call. A dash denotes no explicit client-side output-token limit. Step counts follow each harness’s execution loop.

Harness	Included token usage
OpenHands	Model input and output tokens recorded by the SDK during task execution, including recorded delegated-agent calls.
PaperQA2	Model input and output tokens in the question-answering session’s token counts. Offline corpus preprocessing and index construction are excluded.
LIFE-harness	Agent and user-simulator model calls. The separate evaluation judge is excluded.
JIT agentfold	All model calls made by the harness during the task, including planning, action generation, and history compression when enabled.

Appendix

Table 9: Token accounting for each harness.

Module disabled	$P$	$T$	$p^{\mathrm{perf}}$	$p^{\mathrm{eff}}$
Initial planning	73.70	328.7	0.737	0.539
Periodic replanning	67.97	367.0	0.028	0.075
History folding	76.40	535.1	0.923	$\mathbf{<0.001}$
Forced final answer	71.13	359.2	0.314	0.058
Lenient JSON parsing	71.50	325.6	0.344	0.599
Task restatement	67.23	315.8	0.023	0.832

Appendix

Table 10: Single-module ablation results for JIT agentfold on 50 DeepPlanning-Shopping development tasks, with one trial per task.

P

is mean match rate (%);

T

is mean token cost (

10^{3}

) after disabling the named module. The one-sided paired t-test p-values

p^{\mathrm{perf}}

and

p^{\mathrm{eff}}

quantify performance and efficiency saliency relative to the per-task median reference. Bold p-values satisfy

p\leq 0.05

Module disabled	$P$	$T$	$p^{\mathrm{perf}}$	$p^{\mathrm{eff}}$
Paper search tool	0.00	1318.1	$\mathbf{<0.001}$	$\mathbf{<0.001}$
Evidence gathering tool	38.00	1466.1	0.664	$\mathbf{<0.001}$
Answer generation tool	0.00	1232.2	$\mathbf{<0.001}$	$\mathbf{<0.001}$
Reset tool	42.00	184.0	0.887	0.215
Completion tool	80.00	997.9	1.000	$\mathbf{<0.001}$
Agent system prompt	42.00	311.3	0.936	0.072

Appendix

Table 11: Single-module ablation results for PaperQA2 on 50 LitQA2 development tasks, with one trial per task.

P

is success rate (%);

T

is mean token cost (

10^{3}

) after disabling the named module. The one-sided paired t-test p-values

p^{\mathrm{perf}}

and

p^{\mathrm{eff}}

quantify performance and efficiency saliency. Both tests use the per-task median reference. Bold p-values satisfy

p\leq 0.05

Module disabled	$P$	$T$	$p^{\mathrm{perf}}$	$p^{\mathrm{eff}}$
Terminal tool	50.00	576.6	0.084	0.005
Finish tool	58.00	470.1	0.371	0.481
Think tool	60.00	483.0	0.500	0.330
Browser tools	62.00	480.4	0.670	0.347
Fetch tool	54.00	514.8	0.042	0.069
Search tool	44.00	639.9	0.022	0.003

Appendix

Table 12: Single-module ablation results for OpenHands on 50 GAIA development tasks, with one trial per task.

P

is success rate (%);

T

is mean token cost (

10^{3}

) after disabling the named module. The one-sided paired t-test p-values

p^{\mathrm{perf}}

and

p^{\mathrm{eff}}

quantify performance and efficiency saliency. Both tests use the per-task median reference. Bold p-values satisfy

p\leq 0.05

Module disabled	$P$	$T$	$p^{\mathrm{perf}}$	$p^{\mathrm{eff}}$
Booking payment limits	86.67	131.6	0.837	0.024
Booking itinerary check	83.33	112.4	1.000	0.760
Cancellation eligibility	83.33	117.3	0.500	0.501
Basic-economy changes	86.67	120.6	0.837	0.267
Compensation eligibility	83.33	120.2	0.500	0.375
Route preservation	83.33	121.1	1.000	0.319

Appendix

Table 13: Single-module ablation results for LIFE-harness on

\tau^{2}

-bench airline on 30 development tasks, with one trial per task.

P

is success rate (%);

T

is mean token cost (

10^{3}

) after disabling the named module. The one-sided paired t-test p-values

p^{\mathrm{perf}}

and

p^{\mathrm{eff}}

quantify performance and efficiency saliency. Both tests use the per-task median reference. Bold p-values satisfy

p\leq 0.05

Module disabled	$P$	$T$	$p^{\mathrm{perf}}$	$p^{\mathrm{eff}}$
Cancellation status check	81.25	124.0	0.210	0.304
Item-modification status check	79.17	122.0	0.091	0.551
Exchange status check	77.08	123.0	0.052	0.413
Return eligibility check	72.92	123.9	0.016	0.274
Cancellation reason check	79.17	127.0	0.131	0.098
Payment-change check	77.08	129.5	0.022	0.095

Appendix

Table 14: Single-module ablation results for LIFE-harness on the

\tau^{2}

-bench retail development set, with one trial per task.

P

is success rate (%);

T

is mean token cost (

10^{3}

) after disabling the named module. The one-sided paired t-test p-values

p^{\mathrm{perf}}

and

p^{\mathrm{eff}}

quantify performance and efficiency saliency. Both tests use the per-task median reference. Bold p-values satisfy

p\leq 0.05

Config.	Modules removed	$P$	$T$
$\mathcal{S}_{23}$ (full)	None	76.10	329.1
$\mathcal{S}_{20}$	Shopping requirements; Shopping cost strategy; Code execution	74.31	204.3
$\mathcal{S}_{17}$	Lenient JSON parsing; Tool-selection guidance; Final-answer guidance	70.40	203.8
$\mathcal{S}_{14}$	Tool-list summary; Plan-execution guidance; Initial planning	75.96	192.0
$\mathcal{S}_{11}$	Shopping output guidance; Compression guidance; Result-processing guidance	73.38	195.0
$\mathcal{S}_{8}$	Role preamble; Planning-model guidance; General action rules	74.10	215.2

Appendix

Table 15: Efficiency-oriented pruning configurations for JIT agentfold on 30 DeepPlanning-Shopping validation tasks, using three trials per task. Configurations are listed in increasing order of pruning, followed by an additional all-pruned baseline. The module column lists removals relative to the preceding row.

P

is mean match rate (%);

T

is mean token cost (

10^{3}

). Bold values mark the highest performance and lowest token cost within each table.

Config.	Modules removed	$P$	$T$
$\mathcal{S}_{23}$ (full)	None	75.83	323.7
$\mathcal{S}^{\prime}_{22}$	Initial planning	77.66	278.3
$\mathcal{S}^{\prime}_{19}$	Plan-execution guidance; Role preamble; Result-processing guidance	82.49	304.6
$\mathcal{S}^{\prime}_{16}$	Compression guidance; Code execution; Shopping cost strategy	76.88	219.1
$\mathcal{S}^{\prime}_{13}$	Shopping output guidance; Forced final answer; Lenient JSON parsing	72.66	222.8
$\mathcal{S}^{\prime}_{10}$	Planning-model guidance; Shopping requirements; Final-answer guidance	70.66	265.1

Appendix

Table 16: Performance-oriented pruning configurations for JIT agentfold on 30 DeepPlanning-Shopping validation tasks, using one trial per task. Configurations are listed in increasing order of pruning, followed by an additional all-pruned baseline. The module column lists removals relative to the preceding row.

P

is mean match rate (%);

T

is mean token cost (

10^{3}

). Bold values mark the highest performance and lowest token cost within each table.

Config.	Modules removed	$P$	$T$
$\mathcal{S}_{18}$ (full)	None	61.40	1256.6
$\mathcal{S}_{15}$	Multimodal processing; Citation metadata context; Increased evidence depth	56.14	1108.4
$\mathcal{S}_{12}$	Reset tool; Structured evidence output; Answer iteration	50.88	1651.2
$\mathcal{S}_{9}$	Increased concurrency; Larger text chunks; Automatic evidence gathering	42.98	1728.6
$\mathcal{S}_{6}$	Evidence retrieval; Agent system prompt; Document metadata extraction	52.63	2004.0
$\mathcal{S}_{3}$ (protected)	Completion tool; Evidence gathering tool; Workflow guidance	0.00	5915.0

Appendix

Table 17: Efficiency-oriented pruning configurations for PaperQA2 on LitQA2. Configurations are listed in increasing order of pruning, followed by an additional all-pruned baseline. The module column lists removals relative to the preceding row.

P

is success rate (%);

T

is mean token cost (

10^{3}

). Results average three trials per validation task. Bold values mark the highest performance and lowest token cost within the table.

Config.	Modules removed	$P$	$T$
$\mathcal{S}_{27}$ (full)	None	29.44	799.4
$\mathcal{S}_{24}$	Version-control guidance; File editor; Skills (microagents)	32.78	691.8
$\mathcal{S}_{21}$	Risk-assessment guidance; Memory guidance; Documentation guidance	32.78	675.4
$\mathcal{S}_{18}$	Introductory prompt; Finish tool; External-service guidance	33.89	683.3
$\mathcal{S}_{15}$	Browser tools; Pull-request guidance; Code-quality guidance	30.56	654.2
$\mathcal{S}_{12}$	Troubleshooting guidance; Environment-setup guidance; Think tool	29.44	663.0

Appendix

Table 18: Efficiency-oriented pruning configurations for OpenHands on GAIA. Configurations are listed in increasing order of pruning, followed by an additional all-pruned baseline. The module column lists removals relative to the preceding row.

P

is success rate (%);

T

is mean token cost (

10^{3}

). Results average three trials on 60 validation tasks. Token cost uses runs with recorded usage. Bold values mark the highest performance and lowest token cost within the table.

Config.	Modules removed	$P$	$T$
$\mathcal{S}_{44}$ (full)	None	83.33	138.7
$\mathcal{S}_{42}$	Agent instructions; Reservation flight summary	78.33	124.0
$\mathcal{S}_{39}$	Updated baggage allowance; Booking hint; Booking itinerary check	81.67	111.4
$\mathcal{S}_{36}$	Cancellation refund summary; Cancellation eligibility; Other-reservations skill	80.00	113.2
$\mathcal{S}_{33}$	Booking payment total; Compensation amount check; Direct-flight search hint	81.67	114.5
$\mathcal{S}_{30}$	Reservation payment summary; Compensation eligibility; Cancellation-check skill	81.67	107.7

Appendix

Table 19: Efficiency-oriented pruning configurations for LIFE-harness on

\tau^{2}

-bench airline. Configurations are listed from full to all-pruned. Each subsequent row removes the listed modules from the preceding configuration.

P

is success rate (%);

T

is mean token cost (

10^{3}

). Results average three trials per validation task. Bold values mark the highest performance and lowest token cost within the table. The protected set is empty in this setting, so the configuration labeled “protected” is also all-pruned.

Config.	Modules removed	$P$	$T$
$\mathcal{S}_{48}$ (full)	None	75.00	119.1
$\mathcal{S}_{46}$	Order-address hint; Return hint	79.17	116.4
$\mathcal{S}_{43}$	Item availability note; Item-modification hint; Repeated-failure alert	82.50	113.6
$\mathcal{S}_{40}$	Cancellation hint; Pending-replacement skill; Agent instructions	76.67	106.9
$\mathcal{S}_{37}$	Return item check; Cancellation refund summary; Replacement existence check	79.17	106.9
$\mathcal{S}_{34}$	Nonempty replacement list; Payment balance check; Item-modification status check	70.00	110.5

Appendix

Table 20: Efficiency-oriented pruning configurations for LIFE-harness on

\tau^{2}

-bench retail. Configurations are listed in increasing order of pruning, followed by an additional all-pruned baseline. The module column lists removals relative to the preceding row.

P

is success rate (%);

T

is mean token cost (

10^{3}

). Results average three trials per validation task. Bold values mark the highest performance and lowest token cost within the table.

Config.	Modules removed	$P$	$T$
$\mathcal{S}_{48}$ (full)	None	67.50	121.2
$\mathcal{S}^{\prime}_{46}$	Replacement existence check; Payment-change hint	77.50	116.3
$\mathcal{S}^{\prime}_{43}$	Return hint; Nonempty replacement list; Payment balance check	80.00	118.4
$\mathcal{S}^{\prime}_{40}$	Item-list length check; Numeric-option extrema; Item-modification hint	75.00	116.5
$\mathcal{S}^{\prime}_{37}$	Return item check; Pending-replacement skill; Address-history skill	72.50	117.2
$\mathcal{S}^{\prime}_{34}$	Cancellation status check; Modification product-type check; Transfer hint	70.00	116.1

Appendix

Table 21: Performance-oriented pruning configurations for LIFE-harness on

\tau^{2}

-bench retail. Configurations are listed in increasing order of pruning, followed by an additional all-pruned baseline. The module column lists removals relative to the preceding row.

P

is success rate (%);

T

is mean token cost (

10^{3}

). Results use one trial per validation task. Bold values mark the highest performance and lowest token cost within the table.

Panel of Figure 5 — Figure 5: Full-range views of the PaperQA2 and JIT agentfold results in Figure 3 , including $\mathcal{S}_{3}$ and $\mathcal{S}_{0}$ , respectively, which are omitted from the zoomed-in panels. Points average three trials; token cost is the per-trial median across tasks. Stars mark full harnesses, color indicates modules pruned, and lines trace the Pareto frontier.

Agent Harness Optimization

Agent harness optimization improves the prompts, tools, memory, workflows, and control code that shape how AI agents act.

See all 183 papers

Sep 30, 2026

Composing Task-specific Agent Harnesses at Test Time with Reusable Primitives

Agent harnesses govern how large language models (LLMs) gather context, invoke tools, verify results, preserve state, and terminate, largely affecting agent performance. However, the value of each harness mechanism can differ across heterogeneous tasks: a mechanism that improves one task may impose overhead or context distraction on another, leading to the suboptimality of a global harness. We characterize this suboptimality as a mismatch induced by fixed mechanism choices, motivating task-specific harness construction. Nonetheless, generating harness code for each task introduces generation and debugging costs, with execution risks that can compound as more mechanisms are generated. To address those challenges, we introduce Harness Primitives, reusable harness mechanisms with clear application scope and composition contract mined from failed task trajectories. Based on Harness Primitives, we propose STITCH, a framework that Selects suitable primitives given Task Information and compiles them into Task-speCific Harnesses at test time. This separation enables task-specific harnesses without generating or repairing mechanism code at test time. Extensive experiments demonstrate that STITCH not only improves harness adaptability and robustness, but also scales with the primitive library size, boosting task success rates by up to 12 points over fixed harness baselines, surpassing human-designed harnesses like Codex CLI while maintaining a minimal test-time harness composition overhead of only 2.7%, 638 times more efficient than generating task-specific harnesses from scratch. Ultimately, our work demonstrates that building task-adaptive harnesses can be beneficial for completing diverse tasks and that building reusable primitives can be a promising path towards this goal.

Peng Kuang, Haibo Jin, Dehao Wu +4

University of Illinois Urbana-Champaign · University of Michigan, Ann Arbor

Agent Harness Optimization LLM Agent Harnesses

Ask PDF

Oct 4, 2026

MESH-Harness: Self-Improving Agent Harnesses via Bandit-Guided Compositional Evolution

An agent harness is the code that organizes context, maintains state, and coordinates tool calls for a language model. We study how to improve the harness under a limited evaluation budget while keeping model weights fixed. Our method, MESH-Harness, organizes each harness into functional modules with explicit role-specific interfaces, allowing alternative implementations of each module to be substituted and recombined. It uses shared module representations and full-covariance LinUCB to score candidate combinations based on predicted performance and exploration value. Mixed-start coordinate ascent selects complete configurations for evaluation without enumerating the combinatorial space. Validation traces then guide local code edits, and the resulting candidates are incorporated into fixed-capacity role-specific pools for subsequent recombination. On text tasks, retrieval-augmented mathematical reasoning, code generation, and interactive scientific tasks, MESH-Harness outperforms Meta-Harness by 5.70, 7.01, 2.00, and 5.00 points, respectively, under matched candidate-evaluation budgets. Iterative harness optimization improves MESH-Harness by 5.63-7.79 points over its first-round configurations. For the reported configurations, aggregate test-time cost is 44.2% lower than that of Meta-Harness, while total cost including search is 14.6% lower. These results show that combining module-level design reuse with feedback-driven compositional search can systematically improve agent harnesses while keeping overall optimization cost under control.

Zhiwei Shang, Yu Huo, Mingrong Gong +6

The Chinese University of Hong Kong, Shenzhen · DeepWisdom · The Chinese University of Hong Kong +3

Agent Harness Optimization LLM Agent Workflow Optimization LLM Agent Self-Improvement LLM Agent Harnesses

Ask PDF

Oct 20, 2026

Elicitation or Expansion? Demystifying Gains from Agent Harness Optimization

Recent studies in harness optimization have shown substantial performance gains in language model agents without updating model weights. A well-designed harness more effectively leverages the underlying model's capabilities by shaping context, tool use, and execution workflows. However, naïve success rates alone do not fully characterize whether optimized harness yields more reliable success on tasks already within reach (elicitation) or a broader range of solvable tasks (expansion). This distinction is crucial for recursive self-improvement and evolutionary search, as expansion may provide a broader range of successful experiences to push the capability boundaries. Despite the growing significance, this distinction remains underexplored in harness optimization. We conduct a systematic analysis to study the effect of gains from harness optimization. Building on pass@k analyses in the reinforcement learning (RL) literature that compare models before and after RL training, we compare optimized harnesses with their pre-optimization counterparts. Specifically, we assess changes in reachable task coverage, compare harness optimization with model scaling and increased inference-time compute, and examine how gains vary with task difficulty. Across static prediction and dynamic tasks, we find that pass@1 advantages of harness over stronger interventions can reverse at larger attempt budgets and aggregate gains conceal shortfalls on hard tasks, underscoring the need for a fine-grained diagnosis.

Younghyun Kim, Wooseok Seo, Minseo Kim +6

FuriosaAI · Yonsei University · University of California, Berkeley

Agent Harness Optimization LLM Agent Harnesses LLM Agent Evaluation

Ask PDF

Abstract

Agent Harness Optimization

Explore similar work

Composing Task-specific Agent Harnesses at Test Time with Reusable Primitives

MESH-Harness: Self-Improving Agent Harnesses via Bandit-Guided Compositional Evolution

Elicitation or Expansion? Demystifying Gains from Agent Harness Optimization