ParetoSaddler: Pareto-Guided Optimization of Agent Harnesses for Quality and Cost

Xingzhi Qian; Sungho Park; Rongyuan Tan; Jianbo Yu; Jue Zhang; Lorenzo Cavallaro; Menglin Xia; Yanzhi Chen

cs.AINov 1, 2026

ParetoSaddler: Pareto-Guided Optimization of Agent Harnesses for Quality and Cost

Authors: Xingzhi Qian, Sungho Park, Rongyuan Tan, Jianbo Yu, Jue Zhang, Lorenzo Cavallaro, Menglin Xia, Yanzhi Chen,

Organizations: University College London · Pohang University of Science and Technology · Southern University of Science and Technology · Southeast University ·

Abstract

Harness optimization can substantially improve LLM agents without modifying model weights, but existing methods primarily optimize response quality. Extending them to joint quality--cost optimization introduces a challenge: applying the deployment preference during intermediate updates can prematurely reject candidates that are valuable for later improvements. We introduce ParetoSaddler, which addresses this mismatch by separating intermediate candidate promotion from final deployment selection. At each iteration, an optimizer updates a parent harness to produce a candidate, and then compares the two over repeated executions and uses Bayesian bootstrap Pareto support to promote candidates showing credible progress in either quality or cost. Promoted candidates receive broader development evaluation and remain available as parents for later optimization steps. We evaluate ParetoSaddler on ClawMark and GAIA2 using domain-disjoint splits. On ClawMark, ParetoSaddler improves quality from 70.86% to 81.35% while reducing cost from $1.17 to $1.01 per task; on GAIA2, it improves quality from 56.22% to 62.89% while reducing cost from $0.78 to $0.45 per task. The Pareto-selected harnesses also achieve higher quality at lower cost than quality-only harness optimization. Analysis shows that Pareto-aware optimization retains intermediate candidates that would be discarded by deployment preference but can support later improvements, while Bayesian bootstrap screening filters out candidates whose gains are weakly supported across runs.

Figures & tables

Figure 1: Quality–cost optimization on the development sets of ClawMark and GAIA2. Each marker represents a harness evaluated on the development set; better solutions lie toward the upper left. Stars denote the initial harness, and enlarged outlined markers highlight the final candidates retained by each method. Solid green lines trace harness evolution in ParetoSaddler; green markers not connected by solid lines denote candidates not selected for further refinement. Dashed lines connect candidates on the Pareto fronts of ParetoSaddler and Meta-Harness. ParetoSaddler discovers high-quality harnesses in a lower-cost region not reached by the evaluated baseline candidates.

Figure 2: Overview of ParetoSaddler. Repeated execution and statistical Pareto promotion identify candidates for development evaluation. The updated candidate pool and Pareto archive support continued optimization, with final harness selection governed by the deployment preference.

Setting	Metric	Overall		GAIA2 by Universe
Setting	Metric	ClawMark	GAIA2	U21 (107)	U22 (112)	U27 (81)
Default Agent	Qual. $\uparrow$	70.86 $\pm$ 2.29	56.22 $\pm$ 1.07	58.26 $\pm$ 1.08	53.57 $\pm$ 1.55	57.20 $\pm$ 2.57
Default Agent	Cost $\downarrow$	1.17 $\pm$ 0.17	0.78 $\pm$ 0.08	0.82 $\pm$ 0.18	0.71 $\pm$ 0.03	0.82 $\pm$ 0.04
AS-Quality	Qual. $\uparrow$	73.86 $\pm$ 3.54	57.33 $\pm$ 0.67	55.76 $\pm$ 4.70	53.87 $\pm$ 5.67	64.20 $\pm$ 1.23
AS-Quality	Cost $\downarrow$	1.24 $\pm$ 0.04	0.61 $\pm$ 0.03	0.55 $\pm$ 0.02	0.62 $\pm$ 0.04	0.67 $\pm$ 0.05
MH-MinCost	Qual. $\uparrow$	70.02 $\pm$ 2.35	39.78 $\pm$ 0.96	41.12 $\pm$ 2.80	38.69 $\pm$ 2.58	39.51 $\pm$ 2.47

Table 1: Test results on ClawMark and GAIA2. Quality is reported in percentage (%), and cost in USD per task. Values are mean

\pm

sample standard deviation across three test repeats. Bold and underlined means indicate the best and second-best results in each column, respectively. GAIA2 universe-level results are also shown as a detailed breakdown. For Pareto-set optimization, superscripts

Q

and

C

denote the quality- and cost-extreme harnesses selected on the development set.

Figure 3: ClawMark optimization trajectories and final test results of PS-MinCost under Pareto and MinCost local gates. Faint markers denote evaluated candidates not selected for subsequent refinement.

Setting	Qual. $\uparrow$	Cost $\downarrow$
Single-run C	76.95 $\pm$ 5.50	0.95 $\pm$ 0.06
Single-run Q	75.71 $\pm$ 2.10	0.97 $\pm$ 0.05
Three-run C	79.60 $\pm$ 2.43	0.92 $\pm$ 0.08
Three-run Q	81.35 $\pm$ 3.11	1.01 $\pm$ 0.02

Table 2: Overall ClawMark test results for the single- versus three-run PS-Pareto ablation.

Setting	Metric	Mean	Bayesian
PS-Pareto C	Qual. $\uparrow$	76.86 $\pm$ 5.41	79.60 $\pm$ 2.43
PS-Pareto C	Cost $\downarrow$	1.09 $\pm$ 0.09	0.92 $\pm$ 0.08
PS-Pareto Q	Qual. $\uparrow$	76.53 $\pm$ 2.12	81.35 $\pm$ 3.11
PS-Pareto Q	Cost $\downarrow$	1.15 $\pm$ 0.04	1.01 $\pm$ 0.02

Table 3: ClawMark test results with mean-based versus Bayesian screening.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Train	Dev	Test
ClawMark	Clinical Assistant, Content Operation, EDA, Executive Assistant, HR, Research Assistant (50)	Investment Analyst, Project Management, Real Estate (20)	E-commerce, Journalist, Legal Assistant (22)
GAIA2	Universe 29 (75)	Universe 30 (65)	Universes 21, 22, 27 (300)

Appendix

Table 4: Group-disjoint benchmark splits. ClawMark is partitioned by scenario and GAIA2 by Universe. Numbers in parentheses denote the number of tasks.

Benchmark	Model	Reasoning	Fresh input	Cached input	Output
ClawMark	GPT-5.4	High	$2.50	$0.25	$15.00
GAIA2	GPT-5.5	Medium	$5.00	$0.50	$30.00

Appendix

Table 5: Task-model configurations and token prices used for execution-cost accounting. Prices are in USD per one million tokens.

Benchmark	Method	Proposals	Search tasks / proposal	Repeats	Task-execution budget
ClawMark	PS / AS	17	3 + promoted dev. eval.	3	1380
	MH	8	70	3	1890
GAIA2	PS / AS	25	3 + promoted dev. eval.	3	5520
	MH	15	140	3	6720

Appendix

Table 6: Optimization schedules and task-execution budgets. A task execution denotes one complete rollout of the task agent on one benchmark task. For ParetoSaddler ( PS ) and AutoSaddler ( AS ), the reported budget is an upper bound that assumes every locally screened candidate is promoted to full-development evaluation; the actual number of task executions is therefore less than or equal to this value and depends on the number of local promotions. Meta-Harness ( MH ) does not use staged promotion and instead evaluates each proposal on the full search set, so its budget is determined directly by the number of proposals shown. All budgets include the initial default harness evaluation.

Figure 4: Quality–cost outcomes under quality-constrained cost minimization on the development sets of ClawMark and GAIA2. Each marker represents a harness evaluated on the development set; better solutions lie toward the upper left. Stars denote the initial harness, and enlarged outlined markers highlight the final candidates selected by each method. Solid green lines trace harness evolution in PS-MinCost ; green markers not connected by solid lines denote candidates not selected for further refinement. The selected PS-MinCost harnesses achieve substantially lower cost than their MH-MinCost counterparts while also attaining higher quality.

Figure 5: Run-to-run variability on four sampled ClawMark tasks. Markers show six executions per task; the legend gives within-task CV for quality and cost.

Repeated evidence	Diagnostic insight	Patch and repeated verification
Costly successful path. A research audit scores 1.0 in all three runs, but costs $0.72 /$ 1.58 / $0.69. The expensive run makes three subagent-spawn calls; the others make none.	Passing scores conceal different execution strategies. The cheaper local runs provide concrete alternatives to the delegated path; a single passing run would not reveal this contrast.	Prefer local execution for bounded reviews, narrow tool discovery, and disable unavailable web tools. After the patch, all three runs still score 1.0, at $0.71 /$ 0.63 / $0.81. This is consistent with reduced overhead, not an isolated causal estimate of delegation cost.
Medication omission. The medication-discrepancy check yields P/F/F: the first execution passes, but the next two omit the required finding.	One successful trajectory would hide inconsistent preservation of medication-timeline evidence as a distinct discrepancy.	Preserve one structured row per independent evidence issue. Post-patch failures fall from $2/3$ to $0/3$ . A different clinical check fails once after the patch, so targeted repair does not imply complete task correctness.
Customer-risk omission. The customer-champion-loss check yields P/P/F: the missing finding becomes visible only in the third execution.	Combining findings into a generic risk note can lose an independent customer-champion issue, even when other executions retain it.	Keep independent findings as separate evidence rows rather than merging them into one summary. Post-patch failures fall from $1/3$ to $0/3$ .
Persistent claim-handling failure. The taxi-claim amount check yields F/F/F, revealing a recurring problem across all observed executions.	The diagnosis identifies eligible missing-receipt claims being omitted or rejected instead of retained for policy-permitted approval.	Add a policy-conditional path that retains eligible pending claims with supporting evidence and requests approval. Post-patch failures fall from $3/3$ to $2/3$ .

Appendix

Table 7: Representative examples of how repeated executions inform harness optimization on ClawMark. P/F denotes pass/fail of the named check.

Setting	E-commerce (8)		Journalist (8)		Legal Assistant (6)		Overall (22)
Setting	Qual. $\uparrow$	Cost $\downarrow$	Qual. $\uparrow$	Cost $\downarrow$	Qual. $\uparrow$	Cost $\downarrow$	Qual. $\uparrow$	Cost $\downarrow$
Single-run C	72.46 $\pm$ 9.90	0.89 $\pm$ 0.02	82.34 $\pm$ 3.13	1.05 $\pm$ 0.09	75.74 $\pm$ 5.15	0.88 $\pm$ 0.09	76.95 $\pm$ 5.50	0.95 $\pm$ 0.06
Single-run Q	65.87 $\pm$ 4.11	0.83 $\pm$ 0.06	85.70 $\pm$ 1.57	1.19 $\pm$ 0.10	75.51 $\pm$ 7.50	0.87 $\pm$ 0.09	75.71 $\pm$ 2.10	0.97 $\pm$ 0.05
Three-run C	72.99 $\pm$ 4.29	0.81 $\pm$ 0.01	89.88 $\pm$ 3.18	1.12 $\pm$ 0.16	74.72 $\pm$ 2.95	0.83 $\pm$ 0.09	79.60 $\pm$ 2.43	0.92 $\pm$ 0.08
Three-run Q	77.77 $\pm$ 7.51	0.87 $\pm$ 0.07	87.81 $\pm$ 3.55	1.28 $\pm$ 0.03	77.51 $\pm$ 5.88	0.83 $\pm$ 0.08	81.35 $\pm$ 3.11	1.01 $\pm$ 0.02

Appendix

Table 8: Test results for the single-run and three-run PS-Pareto ablation on ClawMark. Quality is reported in percentage (%), and cost in USD per task. Values are mean

\pm

sample standard deviation across three test repeats. Superscripts

Q

and

C

denote the quality- and cost-extreme harnesses selected on the development set, respectively. Bold indicates the best mean result in each column.

Task	Parent		Candidate		Difference
Task	Quality (%)	Cost ($)	Quality (%)	Cost ($)	$\Delta Q$ (pp)	$\Delta C$ ($)
Content operation	$[17,\ 100,\ 100]$	$[2.89,\ 1.26,\ 1.11]$	$[100,\ 100,\ 100]$	$[1.44,\ 2.08,\ 1.94]$	$+27.67$	$+0.067$
Research assistant 4	$[100,\ 100,\ 100]$	$[0.65,\ 1.34,\ 1.69]$	$[13,\ 67,\ 87]$	$[0.76,\ 0.76,\ 0.91]$	$-44.33$	$-0.417$
Research assistant 9	$[100,\ 100,\ 100]$	$[0.95,\ 0.53,\ 0.65]$	$[100,\ 100,\ 100]$	$[1.03,\ 0.63,\ 0.72]$	$0.00$	$+0.083$
Overall mean	90.74	1.230	85.19	1.142	$-5.55$	$-0.088$

Appendix

Table 9: Repeated quality–cost observations for I-12 and its parent. Brackets contain the three training executions for each task. Differences are computed from the corresponding mean observations as candidate minus parent; quality differences are reported in percentage points (pp).

Figure 6: Retrospective sensitivity to the promotion threshold

\gamma

. Each point shows the number of archived proposals that would receive development evaluation under the corresponding threshold. The shaded region marks the setting used in our experiments.

Run	Setting	Metric	Overall	GAIA2 by Universe
Run	Setting	Metric	GAIA2 (300)	U21 (107)	U22 (112)	U27 (81)
–	Default Agent	Qual. $\uparrow$	56.22 $\pm$ 1.07	58.26 $\pm$ 1.08	53.57 $\pm$ 1.55	57.20 $\pm$ 2.57
–	Default Agent	Cost $\downarrow$	0.78 $\pm$ 0.08	0.82 $\pm$ 0.18	0.71 $\pm$ 0.03	0.82 $\pm$ 0.04
Run 1	PS-Pareto C	Qual. $\uparrow$	62.11 $\pm$ 2.46	63.24 $\pm$ 4.80	59.23 $\pm$ 1.03	64.61 $\pm$ 3.77
	PS-Pareto C	Cost $\downarrow$	0.47 $\pm$ 0.00	0.44 $\pm$ 0.01	0.49 $\pm$ 0.02	0.47 $\pm$ 0.00
	PS-Pareto Q	Qual. $\uparrow$	62.89 $\pm$ 2.14	61.99 $\pm$ 2.35	60.71 $\pm$ 2.36	67.08 $\pm$ 1.89

Appendix

Table 10: GAIA2 test results across two PS-Pareto optimization runs. Quality is reported in %, and cost in USD per task. Values are mean

\pm

sample standard deviation across three test repeats of each fixed harness. Superscripts

Q

and

C

denote development-selected quality- and cost-extreme harnesses. The shared default-agent evaluation is shown once.

Method	Generated	Time (s)	Est. cost ($)	LLM calls	Output tokens	Fresh-input tokens	Cache-read tokens
	patches	/patch	/patch	/patch	/patch	/patch	/patch
AS-Quality	25	604	6.01	63.4	47,331	391,870	5,260,513
MH-MinCost	15	610	3.58	67.6	36,055	225,106	2,749,406
MH-Pareto	15	619	5.42	103.3	50,100	328,311	4,544,512
PS-MinCost	25	967	8.02	87.4	63,577	489,543	7,339,602
PS-Pareto	25	982	9.62	86.6	68,526	616,012	8,968,786

Appendix

Table 11: Optimizer-side resource usage on GAIA2. Generated patches are counted per optimization run. All remaining quantities are averaged per generated patch.

Figure 7: Annotated development-set optimization trajectory of ParetoSaddler on ClawMark. Solid arrows trace harness evolution; outlined markers identify the final development Pareto set. Labels indicate proposal iterations, and annotations summarize selected harness modifications.

Figure 8: Annotated development-set optimization trajectory of ParetoSaddler on GAIA2. Solid arrows trace harness evolution; outlined markers identify the final development Pareto set. Labels indicate proposal iterations, and annotations summarize selected harness modifications.

Iter	Diagnosed issue	Implemented change	Final status
1	Private allergy notes and same-name ambiguity were overlooked; schema probing, unavailable Python commands, and large image batches consumed budget.	Added internal-evidence and Python compatibility guidance, Notion raw-API fallback steps, and smaller-batch advice in the image-tool description.	Dominated
2	Live trackers were not updated; budgets omitted actual spending; release conclusions omitted observed log gaps; schema and package probing continued after sufficient evidence.	Added guidance for live updates, complete budget arithmetic, release-evidence preservation, and fallbacks without extra dependencies.	Rejected
3	Required mentor contact was omitted; repeated Notion schema probes continued despite known identifiers and usable raw APIs.	Added responsible-party contact guidance and instructions to use known identifiers or raw APIs after one schema probe.	Dominated
4	Optional presentation and media exploration delayed required artifacts; a higher compensation tier was conflated with a compliance hold.	Added artifact-first guidance and instructions to report provisional tier eligibility separately while preserving compliance holds.	Dominated
5	Missing memory tools and helpers were repeatedly probed; a disputed appeal was prematurely marked resolved.	Added bounded-fallback and unresolved-appeal guidance.	Rejected
6	Contradictions and independent expense flags were omitted; requested financial messages were only drafted; missing helpers triggered retries.	Added row-level discrepancy and communication guidance, and bounded helper retries.	Dominated

Appendix

Table 12: Complete ClawMark multi-objective search trace. Rejected candidates did not receive full-development evaluation during optimization. Dominated and Pareto refer to final development status within this search, not status when first evaluated. Prompt and skill guidance describes intended behavior, not an enforced execution guarantee.

File	Summary of changes
Tool-use skills skills/
gog/SKILL.md	Adds direct Google Sheets REST API fallback when helper tools or packages are unavailable; requires reading back updated ranges to verify live writes.
mcporter/SKILL.md	Discourages full schema dumps; recommends compact tool listings or inspection of one specific function.
notion/SKILL.md	Adds known mcporter calls and raw-API fallback for task-provided database IDs; requires querying records before concluding and retaining relevant private evidence in internal outputs without unauthorized external disclosure.
openai-whisper/SKILL.md	Recommends a small model for short English audio, targeted audio windows for long recordings, one active transcription strategy, and bounded polling before abandoning stalled jobs.
Agent and tool prompts src/agents/

Appendix

Table 13: Cumulative harness-file changes in ClawMark I-16 relative to the initial harness. File paths are relative to the directory shown in each group heading.

File	Summary of changes
Tool-use skills skills/
mcporter/SKILL.md	Discourages repeated full-schema inspection; recommends focused calls and alternative Notion endpoints or raw REST after invalid-URL failures.
notion/SKILL.md	Adds sandbox-specific REST fallback guidance and directs batched operations toward supplied resource IDs rather than repeated searches or ID-format retries.
openai-whisper/SKILL.md	Prioritizes existing transcripts and small transcription models; limits transcription to decision-critical audio and recommends one long wait instead of repeated polling or duplicate jobs.
video-frames/SKILL.md	Recommends an initial contact sheet of four to six frames per video, followed by targeted inspection when needed; preserves visual risks, blockers, ownerless tasks, and delays as structured findings.
Agent tools and execution src/agents/

Appendix

Table 14: Cumulative harness-file changes in ClawMark PS-MinCost I-16 under Pareto local promotion. File paths are relative to the directory shown in each group heading.

Iter	Diagnosed issue	Implemented change	Final status
1	Blocked rescheduling delayed safe deletions; announcements omitted details; contact lookup required paging.	Added partial-completion and message guidance, pagination reminders, and city/country contact search.	Dominated
2	Cancellation messages were vague; punctuation-sensitive product search caused retries.	Added self-contained message guidance and normalized fallback search after empty product results.	Dominated
3	Email replies omitted detail and signatures; bare final text caused invalid-answer retries.	Added reply-completeness guidance and a final-answer tool-use guard.	Rejected
4	Relative dates were misinterpreted; contact IDs were used as phone recipients; separate emails were batched.	Added temporal guidance, Calendar/Email contracts, and phone-recipient validation.	Dominated
5	Literal prefix matching missed required files; missing role search prompted broad contact scans.	Added role-aware contact search and multi-file cleanup guidance.	Rejected
6	Name ambiguity failed to block dependent writes; final answers incurred retries.	Added ambiguity-aware write guards, focused blocker messages, and tool-based final delivery.	Rejected

Appendix

Table 15: Complete GAIA2 search trace. Rejected candidates did not receive full-development evaluation during optimization. Dominated and Pareto refer to final development status within this search, not status when first evaluated.

Source file	Summary of changes
Agent control and prompts agents/default_agent/
base_agent.py	Adds pre-tool-use guards for contact and destination ambiguity, calendar updates and invitations, email arguments, batched queries, product descriptors, conditional cab booking, and notification ordering.
prompts/system_prompt.py	Directs final and clarification messages through the user-interface tool; requires completing clear earlier subtasks before blocked later actions; anchors relative dates to the task or environment time.
termination_methods/are_simulation.py	Continues execution after cab-booking acknowledgements when delay or cancellation handling remains pending; adds termination handling for completed shopping workflows requesting no further notifications.
Application tools apps/
email_client.py	Adds date filters, header-only results, oldest-first listing, scoped whole-word search, and batched contact email activity and recency summaries; refines email-sending instructions.

Appendix

Table 16: Cumulative source-file changes in I-23 relative to the initial harness. File paths are relative to the directory shown in each group heading.

File	Summary of changes
Agent execution and prompts
are/simulation/agents/default_agent/
base_agent.py	Adds heuristic conversion of terminal plain-text responses into user-interface tool calls after prior tool activity; checks intended to exclude refusals and progress messages limit when conversion applies.
prompts/additional_prompts.py	Prioritizes compact search, filtering, and counting over broad listing and repeated lookups; requires tool-based final responses and clarifications; clarifies authorization for simulated app actions.
Application tools are/simulation/apps/
apartment_listing.py	Adds multi-zip-code search, unsaved-property filtering, and sorting by square footage; refines listing and saving guidance.

Appendix

Table 17: Cumulative harness-file changes in GAIA2 PS-MinCost I-22 with local Pareto promotion. File paths are relative to the directory shown in each group heading.

Agent Harness Optimization

Agent harness optimization improves the prompts, tools, memory, workflows, and control code that shape how AI agents act.

See all 197 papers

Aug 6, 2026

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

As LLMs are increasingly deployed within agentic systems, their capabilities depend not only on the model weights but also on the harness: the prompts, tools, control flow, memory, and orchestration code surrounding them. This makes automated harness optimization -- the iterative and evaluation-guided improvement of a harness by an AI system -- both an important route to improving AI systems and a demanding capability for AI systems themselves. Yet the community lacks a common protocol for measuring how well frontier LLMs perform at this task. We introduce HarnessOpt-Bench, a benchmark for end-to-end harness optimization under expensive and stochastic evaluation. An optimizer, an LLM paired with a coding harness, receives a target agent's seed harness, graded evaluation feedback, and a fixed target-evaluation budget. It edits the harness and nominates a final candidate, which is scored by its normalized gain over the seed on a held-out test partition that remains inaccessible throughout search. A trusted execution environment enforces the evaluation boundary, meters target-agent resource use, and preserves candidate versions for audit. We evaluate 5 frontier LLMs as optimizers both under a shared coding harness and under their native harnesses across 4 downstream tasks, over 111 scored runs. Experiment results show that optimizer models separate more than the coding harnesses they act through, native harnesses are not consistently superior, and gains vary substantially across tasks and seed regimes. These results establish harness optimization as a measurable and discriminative capability with large space for improvement.

Varun Ursekar, Apaar Shanker, Yash Maurya +4

Scale AI

Large Language Model-Guided Optimization LLM Agent Evaluation Agent Harness Optimization LLM Agent Workflow Optimization AI Agent Benchmarks

Ask PDF

Aug 24, 2026

AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces

LLM agents remain unreliable on long-horizon tasks, where small local failures can compound over extended interactions and lead to overall task failure. Although external harnesses can substantially improve robustness, harness design remains a manual and expensive process that requires searching over a large space of prompts, tool configurations, and control logic. We propose AutoSaddler, an automatic harness optimization framework that formulates harness improvement as an offline learning problem and iteratively updates the harness using failure signals from mini-batches. AutoSaddler combines failure-trace diagnosis, structured patch generation that treats the harness as code, and validation-based update selection. Experiments on GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0 show that AutoSaddler substantially improves agent performance over the corresponding base harnesses, achieving gains of 9.0, 9.6, and 10.0 percentage points, respectively. Ablation studies further suggest that effective harness optimization benefits from three ingredients: deep debugging rather than shallow reflection, targeted modifications rather than unconstrained editing, and generalization-aware selection rather than trajectory-specific repair. Together, these results suggest that automatic harness optimization is a promising path toward more performant and reliable agent systems.

Sungho Park, Wonjoong Kim, Rongyuan Tan +10

POSTECH · KAIST · Southern University of Science and Technology +1

Agent Harness Optimization LLM Agent Harnesses LLM Agent Reliability Long-Horizon LLM Agents

Ask PDF

Oct 19, 2026

HarnessPro: Recursive Harness Self-Improvement with Pareto Optimization for Performance-Cost Trade-offs

Harness development plays a crucial role in building capable AI agents. Recent approaches improve harnesses through recursive self-improvement (RSI), but their performance gains often come with higher execution costs. We introduce HarnessPro, a label-free method for evolving harnesses to improve task performance while reducing execution cost. In this setting, candidate harnesses are evaluated using LLM-generated tests without access to ground-truth solutions or official tests. Specialized agents diagnose execution trajectories, plan updates with different performance-cost priorities, and implement them as candidate harnesses. Rather than returning a single harness, HarnessPro uses normalized weighted squared-distance selection to provide cost-focused (lite), balanced (medium), and performance-focused (heavy) harnesses for different deployment budgets. Selected harnesses enter the next iteration, where execution feedback guides further updates. Our experiments cover Terminal-Bench 2, SWE-Bench Pro, and ProgramBench, using GPT-5.5 with Codex and GLM-5.2 with Claude Code. Across these benchmarks using GPT-5.5 with Codex, the heavy mode of HarnessPro achieves an average pass-rate gain of 2.0 percentage points and an average per-task cost reduction of 26.1% relative to RHO. This advantage also holds when using GLM-5.2 with Claude Code. Code is available at https://github.com/pzs19/HarnessPro

Zhuoshi Pan, ZhiYuan Feng, Yukuo Cen +6

Tsinghua University · Z.AI Co., Ltd. · Shanghai Jiao Tong University +2

Agent Harness Optimization Agent Harness Self-Evolution LLM Agent Harnesses LLM Agent Self-Improvement

Ask PDF

Abstract

Agent Harness Optimization

Explore similar work

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces

HarnessPro: Recursive Harness Self-Improvement with Pareto Optimization for Performance-Cost Trade-offs