AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces

Sungho Park; Wonjoong Kim; Rongyuan Tan; Jue Zhang; Wook-Shin Han; Pengfei Gao; Chanyoung Park; Yongqiang Yao

cs.AIAug 24, 2026

AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces

Authors: Sungho Park, Wonjoong Kim, Rongyuan Tan, Jue Zhang, Wook-Shin Han, Pengfei Gao, Chanyoung Park, Yongqiang Yao,

Organizations: POSTECH · KAIST · Southern University of Science and Technology · Microsoft

Abstract

LLM agents remain unreliable on long-horizon tasks, where small local failures can compound over extended interactions and lead to overall task failure. Although external harnesses can substantially improve robustness, harness design remains a manual and expensive process that requires searching over a large space of prompts, tool configurations, and control logic. We propose AutoSaddler, an automatic harness optimization framework that formulates harness improvement as an offline learning problem and iteratively updates the harness using failure signals from mini-batches. AutoSaddler combines failure-trace diagnosis, structured patch generation that treats the harness as code, and validation-based update selection. Experiments on GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0 show that AutoSaddler substantially improves agent performance over the corresponding base harnesses, achieving gains of 9.0, 9.6, and 10.0 percentage points, respectively. Ablation studies further suggest that effective harness optimization benefits from three ingredients: deep debugging rather than shallow reflection, targeted modifications rather than unconstrained editing, and generalization-aware selection rather than trajectory-specific repair. Together, these results suggest that automatic harness optimization is a promising path toward more performant and reliable agent systems.

Figures & tables

Panel of Figure 1 — Figure 1: Comparison of optimization performance and efficiency on GAIA2. (a) AutoSaddler reaches 72.3% dev accuracy with ${\sim}$ 1,000 total task executions, whereas GEPA and Meta-Harness saturate at 64.6% and 61.5% , respectively, despite consuming ${\sim}$ 2,800 task executions. (b) When measured by the number of execution rollouts leveraged for optimization, AutoSaddler achieves its best performance after consuming only 147 traces, ${\sim}$ 10 $\times$ fewer than Meta-Harness (1,400 traces).

Figure 2: Overview of AutoSaddler . The iterative optimization loop: the current harness is tested on a mini-batch, diagnosed and patched across harness components, verified for improvement, and then reflected upon to extract lessons into the EvoDAG, which guides the evolution of the next harness.

Category	Subtype	Description
Prompt Patch	Prompt Rule Addition (S)	Add or modify behavioral rules in the system prompt.
Prompt Patch	Prompt Rule Modification (S)	Revise existing prompt rules to resolve conflicts or sharpen guidance.
Tool Patch	New Tool Addition (C)	Add a new tool when no existing tool supports the required action.
	Argument Modification (C)	Add or fix tool parameters to enhance filtering or selection capabilities.
	Implementation Fix (C)	Fix bugs or extend internal tool functionality to produce correct results.
	Tool Description Fix (S)	Modify tool docstrings to avoid tool misuse.

Table 1: Adopted patch categories, with subtypes labeled as Capability (C) or Steering (S) Patch.

Harness (Type)	GAIA2 Universe (Pass@1)			Avg.
Harness (Type)	21 (107)	22 (112)	27 (81)	Avg.
Default Agent (Manual)	$54.8\pm 48$	$51.5\pm 49$	$52.7\pm 43$	$53.0\pm 15$
GEPA (Auto)	$60.1\pm 39$	$47.9\pm 34$	$56.4\pm 7$	$54.6\pm 25$
Meta-Harness (Auto)	$53.0\pm 11$	$51.5\pm 52$	$56.0\pm 7$	$53.2\pm 22$
AutoSaddler (Auto)	61.4(24)	60.7(24)	64.6(31)	62.0(12)
w/o In-depth Diagnosis (Auto)	$56.7\pm 42$	$57.1\pm 39$	$60.1\pm 50$	$57.8\pm 38$

Table 2: Test-set Pass@1 results on GAIA2, reported as mean

\pm

standard deviation over three runs. Values in parentheses indicate the number of tasks; boldface denotes the best result.

SWE-Bench Pro (Pass@1) Harness (Type) Ansible (96) Flipt (85) Element-web (56) Avg. SWE-agent (Manual)

40.6\pm 19

31.0\pm 35

41.1\pm 17

37.3\pm 48

GEPA (Auto)

50.0\pm 13

32.2\pm 14

45.2(06)

42.5\pm 12

Meta-Harness (Auto)

36.9\pm 29

31.3\pm 17

38.7\pm 08

35.3\pm 20

AutoSaddler (Auto) 58.0(18) 36.5(18)

43.5\pm 16

46.9(18)

Terminal-Bench 2.0 (Pass@1) Harness (Type) Test-Split (40) Terminus 2 (Manual)

40.0\pm 0

Terminus KIRA (Manual)

47.5\pm 25

GEPA (Auto)

42.5\pm 25

Meta-Harness (Auto)

43.3\pm 58

AutoSaddler (Iter2) (Auto)

45.0\pm 0

AutoSaddler (Iter34) (Auto) 50.0(0)

Table 3: Test-set Pass@1 results on SWE-Bench Pro and Terminal-Bench 2.0, reported as mean

\pm

standard deviation over three runs. Parentheses indicate the number of tasks; bold denotes the best result per column.

Panel of Figure 3 — Figure 3: Patch type distribution and acceptance: AutoSaddler vs. w/o Structured Intervention. Without structural intervention, patches collapse onto Steering (91.5%), while AutoSaddler produces a balanced mix spanning Prompt, Tool, and Middleware edits.

Panel of Figure 4 — Figure 4: Performance comparison on the dev-set across iterations. AutoSaddler maintains a lower regression rate while achieving comparable fix rates to the w/o Generalization-Aware Selection ablation, yielding a consistently positive net gain that the ablation fails to sustain.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Traditional Mini-batch Training	AutoSaddler Stage	Interpretation
Step 1: Sample mini-batch	1 Sample mini-batch for evaluating current harness	Same role
Step 2: Run forward pass	1 Execute the current harness	The agent rollout is the forward computation. It produces both a final answer and a trajectory.
Step 3: Compute loss	1 Record outcomes and traces	The task outcome provides the sparse metric signal; the trace provides evidence needed to explain the outcome.
Step 4: Backpropagate gradients	2 Diagnose and patch + 3 verify on the same mini-batch	AutoSaddler has no numerical gradient; it constructs a textual update through hypothesis generation on root cause, intervention, and verification.
Step 6: Validate/checkpoint	4 Evaluate on the dev-set	Dev-set evaluation checks whether the candidate improvement generalizes beyond the mini-batch.
Step 5: Apply optimizer update	5 Reflect + 6 store in EvoDAG + 7 evolve the next harness	The committed update is not merely the raw patch. AutoSaddler updates optimizer state and uses it to synthesize the next candidate harness.

Appendix

Table 4: Mapping between conventional mini-batch training and the AutoSaddler loop in Figure 2 . AutoSaddler follows the outer structure of mini-batch learning but replaces differentiable gradient computation with trace-grounded symbolic optimization.

Panel of Figure 9 — Algorithm 1 AutoSaddler as mini-batch learning over harness parameters

Command	Purpose	When to use
Read operations
evo-dag summary	DAG topology, best candidate, all edges	Start of session; quick orientation
evo-dag show history	Full patch history: diffs, reflections, lessons	Deep dive into prior attempts
evo-dag show node <idx>	Node details: scores, intent, verdict, output dirs	Inspecting a specific candidate
evo-dag show edge <parent> <child>	Code diff, per-scenario impacts, files changed	Understanding a specific patch
evo-dag show scenario <id>	Per-scenario history, root causes, attempted fixes	Before diagnosing a failing scenario

Appendix

Table 5: Commands exposed by the evo-dag CLI, grouped into read operations for scoped EvoDAG views and write operations for per-session node or scenario updates.

Benchmark	Split Axis	Split	Task Group	# Tasks
GAIA2	Universe (persona)	Train	Universe 29	75
		Dev	Universe 30	65
		Test	Universe 21	107
			Universe 22	112
			Universe 27	81
SWE-Bench Pro	Repository (language)	Train	qutebrowser (Python)	79

Appendix

Table 6: Data splits across the three benchmarks. To evaluate generalization under distribution shift, we partition each benchmark such that train, development, and test sets contain tasks from disjoint task groups — Universes (personas) for GAIA2 and repositories (programming languages) for SWE-Bench Pro—rather than random task-level splits. Terminal-Bench 2.0 contains only 89 tasks across diverse domains and offers no natural grouping axis, so we adopt a uniform random partition.

Method	GAIA2 Universe 22 (112 scenarios; Pass@1)
Default Agent (Manual)	$51.5\pm 49$
GEPA (Run 1)	$47.9\pm 34$
GEPA (Run 2)	$50.6\pm 05$
Meta-Harness (Run 1)	$51.5\pm 52$
Meta-Harness (Run 2)	$51.2\pm 22$
AutoSaddler (Run 1)	60.7(24)

Appendix

Table 7: Robustness to optimization stochasticity . Pass@1 on GAIA2 Universe 22 for harnesses obtained from two independent optimization runs. Test performance is reported as mean

\pm

standard deviation over three executions.

Method	GAIA2 Universe 22 (112 scenarios; Pass@1)
Default Agent (Manual)	$51.5\pm 49$
AutoSaddler (Run 1; Universe 29)	60.7(24)
AutoSaddler (Run 2; Universe 29)	$58.6\pm 05$
AutoSaddler (Different Train Set; Universe 24)	$57.4\pm 21$

Appendix

Table 8: Robustness to training-distribution shift . Pass@1 on GAIA2 Universe 22 when AutoSaddler is independently optimized using different training universes. Values are mean

\pm

standard deviation over three test executions.

Harness (Type)	GAIA2 Universe (Pass@1)			Avg.
Harness (Type)	21 (107)	22 (112)	27 (81)	Avg.
Default Agent (Manual)	$31.2\pm 59$	$24.4\pm 14$	$36.2\pm 19$	$30.0\pm 18$
GEPA (Auto)	$33.6\pm 19$	$24.4\pm 05$	$38.3\pm 12$	$31.4\pm 04$
Meta-Harness (Auto)	$30.8\pm 00$	$26.8\pm 47$	$34.2\pm 40$	$30.2\pm 27$
AutoSaddler (Auto)	38.3(00)	30.4(18)	39.1(26)	35.6(08)
w/o In-depth Diagnosis (Auto)	$36.1\pm 55$	$19.0\pm 52$	$34.2\pm 07$	$29.2\pm 34$

Appendix

Table 9: Cross-model transferability . Pass@1 success rates on the test-set of GAIA2 using Claude Haiku 4.5 as the agent backbone, with harnesses optimized by Claude Opus 4.6 . Values are mean

\pm

standard deviation over three runs.

Method	GAIA2 Universe 22 (112 scenarios; Pass@1)
Default Agent (Manual)	$51.5\pm 49$
AutoSaddler	60.7(24)
w/o Structured Intervention	$53.3\pm 76$
w/o Phase Scheduling	$54.8\pm 44$
w/o Generalization-Aware Selection	$44.9\pm 67$
w/o Dev-Set Filtering	$50.0\pm 50$

Appendix

Table 10: Fine-grained ablations . Pass@1 on GAIA2 Universe 22. Values are mean

\pm

standard deviation over three runs.

Patch Category	Fix Rate (%)	Regression Rate (%)
Prompt	59	14
Tool	57	19
Middleware	57	14

Appendix

Table 11: Patch durability by harness component . Fix and regression rates are measured by comparing pre- and post-patch outcomes on the same training mini-batch.

Patch Type	Fix Rate (%)	Regression Rate (%)
Steering Patch	58	17
Capability Patch	55	8

Appendix

Table 12: Patch durability by intervention mechanism . Capability and Steering Patches exhibit similar fix rates, but Steering Patches induce substantially more regressions.

Panel of Figure 5 — Figure 5: Comparison of optimization performance and efficiency on Terminal-Bench 2.0. (a) AutoSaddler reaches 73.7% dev accuracy with only 31 task executions, whereas Meta-Harness requires 98 executions to plateau at 63.2% and GEPA reaches 57.9% at 90 executions. (b) When measured by the number of execution traces leveraged for optimization, AutoSaddler achieves its best performance after leveraging only 12 traces, over 8 $\times$ fewer than Meta-Harness (98 traces), while surpassing it by 10.5 percentage points.

Method	Generated Patches	Rejected Patches	Accepted Patches	Wall-Clock Time (s) / Patch	Cost ($) / Patch	LLM Calls / Patch	Output Tokens / Patch	Cache-Creation Input Tokens / Patch	Cache-Read Input Tokens / Patch
GEPA	76	58	18	386	5.50	21.3	11,416	156,788	280,940
Meta-Harness 3 3 3 Meta-Harness does not employ an explicit patch-acceptance gate; therefore, rejected and accepted patch counts are not applicable.	20	–	–	883	12.65	66.9	43,429	250,466	3,528,484
AutoSaddler	39	19	20	533	14.56	76.1	21,464	430,933	3,246,158

Appendix

Table 13: Optimizer-side cost on GAIA2 . Runtime, monetary cost, LLM calls, and token usage are averaged per generated patch.

Metric	Average per Rollout
LLM calls	20.2
Input tokens	550,988
Output tokens	7,380
Wall-clock time	203.9 s

Appendix

Table 14: Average task-agent evaluation cost per GAIA2 rollout .

Figure 6: Evolutionary search trajectory of AutoSaddler as the EvoDAG. Four phases (foundation, rapid improvement, selective merging, and consolidation) lead to a global dev-set peak of 72.3% at Iter27, escaping the Iter20 regression and local optima.

Method	Avg. Tool Calls	Avg. File Accesses
w/o In-Depth Diagnosis	63.5	39.7
AutoSaddler	69.7	45.5
$\Delta$ (Attributable to In-Depth Diagnosis)	+6.2	+5.8

Appendix

Table 15: Average tool calls and file accesses per optimization step for AutoSaddler and its ablated variant without in-depth diagnosis.

Figure 7: Case study on a calendar task with ambiguous cancellations, where AutoSaddler’s in-depth diagnosis accurately identifies the true root cause via deep debugging, unlike shallow diagnosis.

Figure 8: Case study on a file-management task that requires moving PNG files into a sims3 folder under benchmarks . Shallow diagnosis misattributes the failure to a relative-path error, while AutoSaddler verifies the filesystem and locates the true cause: an unverified parent directory.

Figure 9: Case study on a cab-booking task with two candidate rides on the same day. Shallow diagnosis overlooks the second matching ride and proposes an off-target fix, while AutoSaddler inspects the full ride history and identifies the missed candidate.

Figure 10: Cumulative accepted patches. AutoSaddler accumulates 20 accepted patches, whereas the w/o In-depth Diagnosis ablation reaches 15.

Figure 11: Representative patches discovered over the optimization trajectory. During the Exploration phase (Iterations $1$ – $25$ ), the search is dominated by Capability patches, including New Tool Addition, Implementation Fix, and Agent Loop Logic Change, which address structural limitations in the harness (e.g., Iterations $11$ and $13$ ). During the Refinement phase (Iterations $26$ – $50$ ), the search shifts toward Steering patches that refine prompt rules, tool descriptions, and PreToolUse hooks (e.g., Iteration $27$ ). Together, these two phases produce a cumulative $24.6\,pp$ improvement.

Figure 12: Reflection extracts lessons from (a) patches that generalize and (b) patches that overfit despite identical mini-batch outcomes. (c) The lineage of reply_to_email -related failures illustrates how atomic lessons gradually evolve into a generalization principle.


C/S	Component	Base Harness	Final Harness ( $\hat{\theta}_{\mathrm{AS}}$ )
Prompt Patch (modifications to default_agent/prompts/system_prompt.py )
Prompt Rule Modification
S	Rule 1 (Communication)	1. COMMUNICATION: Only message the user when completely done or if the task is impossible.	1. COMMUNICATION: Only message the user when completely done or if the task is impossible. When the user asks a factual question expecting a specific answer (a city name, a number, a person’s name, etc.), respond with ONLY the direct answer — no extra calculations, context, or formatting.
S	Rule 6 (Ambiguity)	6. AMBIGUITY: Execute all clear and unambiguous parts of a request immediately. When you encounter ambiguities, contradictions, or impossible elements, finish unambiguous subtasks and then stop and explicitly ask the user for clarification before proceeding with those specific parts.	6. AMBIGUITY: Execute all clear and unambiguous parts … ask the user for clarification before proceeding with those specific parts. A subtask is CLEAR if all its required tool parameters (recipients, content, IDs, paths) can be fully determined — even if it semantically relates to an ambiguous part. Common ambiguity patterns that REQUIRE clarification: (a) TIES — when the task references ‘the item with the highest/lowest X’ but multiple items are tied at that value; (b) SINGULAR vs MULTIPLE — when the task uses singular language (‘the event’, ‘the property’) but multiple items match the criteria; (c) PER-GROUP vs PER-ITEM — when applying an action ‘for each day/group’ at a time derived from a singular source, but multiple sources exist within that group.
Prompt Rule Addition

Appendix

Table 16: Representative patches discovered by AutoSaddler on GAIA2, producing the final harness

\hat{\theta}_{\mathrm{AS}}

from the base harness. C = Capability, S = Steering. Inserted text is highlighted in teal .


C/S	Component	Base Harness	Final Harness ( $\hat{\theta}_{\mathrm{AS}}$ )
Prompt Patch (modifications to the system prompt template prompt-templates/verify-before-complete.txt )
Prompt Rule Addition
S	Verification before completion	(no verification guidance; the base prompt ends immediately after the format specification)	CRITICAL – Verification before completion: You MUST verify your solution works before setting "task_complete": true . When setting up services, test endpoints. When creating files, verify contents. Verify file permissions for all service users. Perform a dry run of automated hooks. If verification fails, fix and verify again before completing.
S	Different-input testing & build-artifact clean-up	(rule does not exist)	For scripts or programs, test with a DIFFERENT input than the provided example to confirm generalizability. After testing, clean up the output directory: remove any build artifacts (compiled binaries, .o files), temporary files, or intermediate outputs that the task did NOT ask you to create. Verifiers often check that output directories contain ONLY the expected files.
S	Library preference	(rule does not exist)	When writing scripts, check the terminal output for pre-installed packages and prefer established libraries over fragile custom implementations.

Appendix

Table 17: Representative patches discovered by AutoSaddler on Terminal-Bench 2.0, producing the final harness

\hat{\theta}_{\mathrm{AS}}

from the base harness. C = Capability, S = Steering. Inserted text is highlighted in teal .

Agent Harness Optimization

Agent harness optimization improves the prompts, tools, memory, workflows, and control code that shape how AI agents act.

See all 185 papers

ActiveSaddler: Automated Curriculum Learning for Agent Harness Optimization
Oct 1, 2026Sungho Park, Wonjoong Kim, Jue Zhang +8Agent Harness Optimization Curriculum Learning
POSTECH · KAIST · Microsoft
Adaptive Auto-Harness: Sustained Self-Improvement for Agentic System Deployment on Open-Ended Task Streams
Jun 1, 2026Zewen Liu, Zhan Shi, Yisi Sang +7Continual Learning for LLM Agents Agent Harness Optimization
Emory University · Amazon · The Pennsylvania State University +2
Self-Harness: Harnesses That Improve Themselves
Jun 8, 2026Hangfan Zhang, Shao Zhang, Kangcong Li +5Agent Harness Optimization LLM Agent Self-Improvement
Shanghai Artificial Intelligence Laboratory

Oct 1, 2026J/K move · Enter open · S save

ActiveSaddler: Automated Curriculum Learning for Agent Harness Optimization

Sungho Park, Wonjoong Kim, Jue Zhang, Wook-Shin Han+7

POSTECH · KAIST · Microsoft

Agent Harness Optimization Curriculum Learning LLM Agent Harnesses

Open paper AskPDF arXiv

Automated harness optimization can substantially improve LLM agents by iteratively updating their prompts, tool interfaces, and control logic from execution feedback. However, existing methods primarily optimize how the harness is updated while largely fixing which training scenarios generate the feedback that drives those updates. As the harness evolves, the scenarios most useful for further optimization can change, suggesting that the training curriculum itself should adapt alongside the harness. We formulate this missing dimension of harness optimization as an automated curriculum learning problem and introduce ActiveSaddler. ActiveSaddler models the evolving curriculum as a non-stationary bandit with dynamically instantiated optimization targets. It abstracts recurring failures into reusable failure-pattern arms, estimates the potential learning progress from further targeting each pattern, and adaptively balances revisiting known weaknesses with exploring unseen scenarios for new ones. Optimization outcomes continually update both the set of discovered failure patterns and their priorities, allowing the curriculum to co-evolve with the harness. Experiments on GAIA2 and Terminal-Bench 2.0 show that ActiveSaddler consistently discovers stronger harnesses, improving test Pass@1 by 4.4 and 7.5 percentage points over the same harness optimizer using a scenario order fixed before optimization, respectively. Ablations further show that these gains depend on dynamically constructing optimization targets, estimating their evolving utility, and balancing continued optimization with new failure discovery. Together, these results establish automated curriculum learning as a new crucial optimization dimension for harness optimization.

Abstract

Agent Harness Optimization

Explore similar work

ActiveSaddler: Automated Curriculum Learning for Agent Harness Optimization

Adaptive Auto-Harness: Sustained Self-Improvement for Agentic System Deployment on Open-Ended Task Streams

Self-Harness: Harnesses That Improve Themselves