LLM agents remain unreliable on long-horizon tasks, where small local failures can compound over extended interactions and lead to overall task failure. Although external harnesses can substantially improve robustness, harness design remains a manual and expensive process that requires searching over a large space of prompts, tool configurations, and control logic. We propose AutoSaddler, an automatic harness optimization framework that formulates harness improvement as an offline learning problem and iteratively updates the harness using failure signals from mini-batches. AutoSaddler combines failure-trace diagnosis, structured patch generation that treats the harness as code, and validation-based update selection. Experiments on GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0 show that AutoSaddler substantially improves agent performance over the corresponding base harnesses, achieving gains of 9.0, 9.6, and 10.0 percentage points, respectively. Ablation studies further suggest that effective harness optimization benefits from three ingredients: deep debugging rather than shallow reflection, targeted modifications rather than unconstrained editing, and generalization-aware selection rather than trajectory-specific repair. Together, these results suggest that automatic harness optimization is a promising path toward more performant and reliable agent systems.
Figures & tables
Figure 1: Comparison of optimization performance and efficiency on GAIA2. (a) AutoSaddler reaches 72.3% dev accuracy with ∼ 1,000 total task executions, whereas GEPA and Meta-Harness saturate at 64.6% and 61.5% , respectively, despite consuming ∼ 2,800 task executions. (b) When measured by the number of execution rollouts leveraged for optimization, AutoSaddler achieves its best performance after consuming only 147 traces, ∼ 10 × fewer than Meta-Harness (1,400 traces).
Figure 2: Overview of AutoSaddler . The iterative optimization loop: the current harness is tested on a mini-batch, diagnosed and patched across harness components, verified for improvement, and then reflected upon to extract lessons into the EvoDAG, which guides the evolution of the next harness.
Category
Subtype
Description
Prompt Patch
Prompt Rule Addition (S)
Add or modify behavioral rules in the system prompt.
Prompt Rule Modification (S)
Revise existing prompt rules to resolve conflicts or sharpen guidance.
Tool Patch
New Tool Addition (C)
Add a new tool when no existing tool supports the required action.
Argument Modification (C)
Add or fix tool parameters to enhance filtering or selection capabilities.
Implementation Fix (C)
Fix bugs or extend internal tool functionality to produce correct results.
Tool Description Fix (S)
Modify tool docstrings to avoid tool misuse.
Table 1: Adopted patch categories, with subtypes labeled as Capability (C) or Steering (S) Patch.
Harness (Type)
GAIA2 Universe (Pass@1)
Avg.
21 (107)
22 (112)
27 (81)
Default Agent (Manual)
54.8±48
51.5±49
52.7±43
53.0±15
GEPA (Auto)
60.1±39
47.9±34
56.4±7
54.6±25
Meta-Harness (Auto)
53.0±11
51.5±52
56.0±7
53.2±22
AutoSaddler (Auto)
61.4(24)
60.7(24)
64.6(31)
62.0(12)
w/o In-depth Diagnosis (Auto)
56.7±42
57.1±39
60.1±50
57.8±38
Table 2: Test-set Pass@1 results on GAIA2, reported as mean ± standard deviation over three runs. Values in parentheses indicate the number of tasks; boldface denotes the best result.
Table 3: Test-set Pass@1 results on SWE-Bench Pro and Terminal-Bench 2.0, reported as mean ± standard deviation over three runs. Parentheses indicate the number of tasks; bold denotes the best result per column.
Figure 3: Patch type distribution and acceptance: AutoSaddler vs. w/o Structured Intervention. Without structural intervention, patches collapse onto Steering (91.5%), while AutoSaddler produces a balanced mix spanning Prompt, Tool, and Middleware edits.
Figure 4: Performance comparison on the dev-set across iterations. AutoSaddler maintains a lower regression rate while achieving comparable fix rates to the w/o Generalization-Aware Selection ablation, yielding a consistently positive net gain that the ablation fails to sustain.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Traditional Mini-batch Training
AutoSaddler Stage
Interpretation
Step 1: Sample mini-batch
1 Sample mini-batch for evaluating current harness
Same role
Step 2: Run forward pass
1 Execute the current harness
The agent rollout is the forward computation. It produces both a final answer and a trajectory.
Step 3: Compute loss
1 Record outcomes and traces
The task outcome provides the sparse metric signal; the trace provides evidence needed to explain the outcome.
Step 4: Backpropagate gradients
2 Diagnose and patch + 3 verify on the same mini-batch
AutoSaddler has no numerical gradient; it constructs a textual update through hypothesis generation on root cause, intervention, and verification.
Step 6: Validate/checkpoint
4 Evaluate on the dev-set
Dev-set evaluation checks whether the candidate improvement generalizes beyond the mini-batch.
Step 5: Apply optimizer update
5 Reflect + 6 store in EvoDAG + 7 evolve the next harness
The committed update is not merely the raw patch. AutoSaddler updates optimizer state and uses it to synthesize the next candidate harness.
Appendix
Table 4: Mapping between conventional mini-batch training and the AutoSaddler loop in Figure 2 . AutoSaddler follows the outer structure of mini-batch learning but replaces differentiable gradient computation with trace-grounded symbolic optimization.
Algorithm 1 AutoSaddler as mini-batch learning over harness parameters
Table 5: Commands exposed by the evo-dag CLI, grouped into read operations for scoped EvoDAG views and write operations for per-session node or scenario updates.
Benchmark
Split Axis
Split
Task Group
# Tasks
GAIA2
Universe (persona)
Train
Universe 29
75
Dev
Universe 30
65
Test
Universe 21
107
Universe 22
112
Universe 27
81
SWE-Bench Pro
Repository (language)
Train
qutebrowser (Python)
79
Appendix
Table 6: Data splits across the three benchmarks. To evaluate generalization under distribution shift, we partition each benchmark such that train, development, and test sets contain tasks from disjoint task groups — Universes (personas) for GAIA2 and repositories (programming languages) for SWE-Bench Pro—rather than random task-level splits. Terminal-Bench 2.0 contains only 89 tasks across diverse domains and offers no natural grouping axis, so we adopt a uniform random partition.
Method
GAIA2 Universe 22 (112 scenarios; Pass@1)
Default Agent (Manual)
51.5±49
GEPA (Run 1)
47.9±34
GEPA (Run 2)
50.6±05
Meta-Harness (Run 1)
51.5±52
Meta-Harness (Run 2)
51.2±22
AutoSaddler (Run 1)
60.7(24)
Appendix
Table 7: Robustness to optimization stochasticity . Pass@1 on GAIA2 Universe 22 for harnesses obtained from two independent optimization runs. Test performance is reported as mean ± standard deviation over three executions.
Method
GAIA2 Universe 22 (112 scenarios; Pass@1)
Default Agent (Manual)
51.5±49
AutoSaddler (Run 1; Universe 29)
60.7(24)
AutoSaddler (Run 2; Universe 29)
58.6±05
AutoSaddler (Different Train Set; Universe 24)
57.4±21
Appendix
Table 8: Robustness to training-distribution shift . Pass@1 on GAIA2 Universe 22 when AutoSaddler is independently optimized using different training universes. Values are mean ± standard deviation over three test executions.
Harness (Type)
GAIA2 Universe (Pass@1)
Avg.
21 (107)
22 (112)
27 (81)
Default Agent (Manual)
31.2±59
24.4±14
36.2±19
30.0±18
GEPA (Auto)
33.6±19
24.4±05
38.3±12
31.4±04
Meta-Harness (Auto)
30.8±00
26.8±47
34.2±40
30.2±27
AutoSaddler (Auto)
38.3(00)
30.4(18)
39.1(26)
35.6(08)
w/o In-depth Diagnosis (Auto)
36.1±55
19.0±52
34.2±07
29.2±34
Appendix
Table 9: Cross-model transferability . Pass@1 success rates on the test-set of GAIA2 using Claude Haiku 4.5 as the agent backbone, with harnesses optimized by Claude Opus 4.6 . Values are mean ± standard deviation over three runs.
Method
GAIA2 Universe 22 (112 scenarios; Pass@1)
Default Agent (Manual)
51.5±49
AutoSaddler
60.7(24)
w/o Structured Intervention
53.3±76
w/o Phase Scheduling
54.8±44
w/o Generalization-Aware Selection
44.9±67
w/o Dev-Set Filtering
50.0±50
Appendix
Table 10: Fine-grained ablations . Pass@1 on GAIA2 Universe 22. Values are mean ± standard deviation over three runs.
Patch Category
Fix Rate (%)
Regression Rate (%)
Prompt
59
14
Tool
57
19
Middleware
57
14
Appendix
Table 11: Patch durability by harness component . Fix and regression rates are measured by comparing pre- and post-patch outcomes on the same training mini-batch.
Patch Type
Fix Rate (%)
Regression Rate (%)
Steering Patch
58
17
Capability Patch
55
8
Appendix
Table 12: Patch durability by intervention mechanism . Capability and Steering Patches exhibit similar fix rates, but Steering Patches induce substantially more regressions.
Figure 5: Comparison of optimization performance and efficiency on Terminal-Bench 2.0. (a) AutoSaddler reaches 73.7% dev accuracy with only 31 task executions, whereas Meta-Harness requires 98 executions to plateau at 63.2% and GEPA reaches 57.9% at 90 executions. (b) When measured by the number of execution traces leveraged for optimization, AutoSaddler achieves its best performance after leveraging only 12 traces, over 8 × fewer than Meta-Harness (98 traces), while surpassing it by 10.5 percentage points.
Method
Generated Patches
Rejected Patches
Accepted Patches
Wall-Clock Time (s) / Patch
Cost ($) / Patch
LLM Calls / Patch
Output Tokens / Patch
Cache-Creation Input Tokens / Patch
Cache-Read Input Tokens / Patch
GEPA
76
58
18
386
5.50
21.3
11,416
156,788
280,940
Meta-Harness 3 3 3 Meta-Harness does not employ an explicit patch-acceptance gate; therefore, rejected and accepted patch counts are not applicable.
20
–
–
883
12.65
66.9
43,429
250,466
3,528,484
AutoSaddler
39
19
20
533
14.56
76.1
21,464
430,933
3,246,158
Appendix
Table 13: Optimizer-side cost on GAIA2 . Runtime, monetary cost, LLM calls, and token usage are averaged per generated patch.
Metric
Average per Rollout
LLM calls
20.2
Input tokens
550,988
Output tokens
7,380
Wall-clock time
203.9 s
Appendix
Table 14: Average task-agent evaluation cost per GAIA2 rollout .
Figure 6: Evolutionary search trajectory of AutoSaddler as the EvoDAG. Four phases (foundation, rapid improvement, selective merging, and consolidation) lead to a global dev-set peak of 72.3% at Iter27, escaping the Iter20 regression and local optima.
Method
Avg. Tool Calls
Avg. File Accesses
w/o In-Depth Diagnosis
63.5
39.7
AutoSaddler
69.7
45.5
Δ (Attributable to In-Depth Diagnosis)
+6.2
+5.8
Appendix
Table 15: Average tool calls and file accesses per optimization step for AutoSaddler and its ablated variant without in-depth diagnosis.
Figure 7: Case study on a calendar task with ambiguous cancellations, where AutoSaddler’s in-depth diagnosis accurately identifies the true root cause via deep debugging, unlike shallow diagnosis.
Figure 8: Case study on a file-management task that requires moving PNG files into a sims3 folder under benchmarks . Shallow diagnosis misattributes the failure to a relative-path error, while AutoSaddler verifies the filesystem and locates the true cause: an unverified parent directory.
Figure 9: Case study on a cab-booking task with two candidate rides on the same day. Shallow diagnosis overlooks the second matching ride and proposes an off-target fix, while AutoSaddler inspects the full ride history and identifies the missed candidate.
Figure 11: Representative patches discovered over the optimization trajectory. During the Exploration phase (Iterations 1 – 25 ), the search is dominated by Capability patches, including New Tool Addition, Implementation Fix, and Agent Loop Logic Change, which address structural limitations in the harness (e.g., Iterations 11 and 13 ). During the Refinement phase (Iterations 26 – 50 ), the search shifts toward Steering patches that refine prompt rules, tool descriptions, and PreToolUse hooks (e.g., Iteration 27 ). Together, these two phases produce a cumulative 24.6pp improvement.
Figure 12: Reflection extracts lessons from (a) patches that generalize and (b) patches that overfit despite identical mini-batch outcomes. (c) The lineage of reply_to_email -related failures illustrates how atomic lessons gradually evolve into a generalization principle.
C/S
Component
Base Harness
Final Harness ( θ^AS )
Prompt Patch (modifications to default_agent/prompts/system_prompt.py )
Prompt Rule Modification
S
Rule 1 (Communication)
1. COMMUNICATION: Only message the user when completely done or if the task is impossible.
1. COMMUNICATION: Only message the user when completely done or if the task is impossible. When the user asks a factual question expecting a specific answer (a city name, a number, a person’s name, etc.), respond with ONLY the direct answer — no extra calculations, context, or formatting.
S
Rule 6 (Ambiguity)
6. AMBIGUITY: Execute all clear and unambiguous parts of a request immediately. When you encounter ambiguities, contradictions, or impossible elements, finish unambiguous subtasks and then stop and explicitly ask the user for clarification before proceeding with those specific parts.
6. AMBIGUITY: Execute all clear and unambiguous parts … ask the user for clarification before proceeding with those specific parts. A subtask is CLEAR if all its required tool parameters (recipients, content, IDs, paths) can be fully determined — even if it semantically relates to an ambiguous part. Common ambiguity patterns that REQUIRE clarification: (a) TIES — when the task references ‘the item with the highest/lowest X’ but multiple items are tied at that value; (b) SINGULAR vs MULTIPLE — when the task uses singular language (‘the event’, ‘the property’) but multiple items match the criteria; (c) PER-GROUP vs PER-ITEM — when applying an action ‘for each day/group’ at a time derived from a singular source, but multiple sources exist within that group.
Prompt Rule Addition
Appendix
Table 16: Representative patches discovered by AutoSaddler on GAIA2, producing the final harness θ^AS from the base harness. C = Capability, S = Steering. Inserted text is highlighted in teal .
C/S
Component
Base Harness
Final Harness ( θ^AS )
Prompt Patch (modifications to the system prompt template prompt-templates/verify-before-complete.txt )
Prompt Rule Addition
S
Verification before completion
(no verification guidance; the base prompt ends immediately after the format specification)
CRITICAL – Verification before completion: You MUST verify your solution works before setting "task_complete": true . When setting up services, test endpoints. When creating files, verify contents. Verify file permissions for all service users. Perform a dry run of automated hooks. If verification fails, fix and verify again before completing.
S
Different-input testing & build-artifact clean-up
(rule does not exist)
For scripts or programs, test with a DIFFERENT input than the provided example to confirm generalizability. After testing, clean up the output directory: remove any build artifacts (compiled binaries, .o files), temporary files, or intermediate outputs that the task did NOT ask you to create. Verifiers often check that output directories contain ONLY the expected files.
S
Library preference
(rule does not exist)
When writing scripts, check the terminal output for pre-installed packages and prefer established libraries over fragile custom implementations.
Appendix
Table 17: Representative patches discovered by AutoSaddler on Terminal-Bench 2.0, producing the final harness θ^AS from the base harness. C = Capability, S = Steering. Inserted text is highlighted in teal .
Automated harness optimization can substantially improve LLM agents by iteratively updating their prompts, tool interfaces, and control logic from execution feedback. However, existing methods primarily optimize how the harness is updated while largely fixing which training scenarios generate the feedback that drives those updates. As the harness evolves, the scenarios most useful for further optimization can change, suggesting that the training curriculum itself should adapt alongside the harness. We formulate this missing dimension of harness optimization as an automated curriculum learning problem and introduce ActiveSaddler. ActiveSaddler models the evolving curriculum as a non-stationary bandit with dynamically instantiated optimization targets. It abstracts recurring failures into reusable failure-pattern arms, estimates the potential learning progress from further targeting each pattern, and adaptively balances revisiting known weaknesses with exploring unseen scenarios for new ones. Optimization outcomes continually update both the set of discovered failure patterns and their priorities, allowing the curriculum to co-evolve with the harness. Experiments on GAIA2 and Terminal-Bench 2.0 show that ActiveSaddler consistently discovers stronger harnesses, improving test Pass@1 by 4.4 and 7.5 percentage points over the same harness optimizer using a scenario order fixed before optimization, respectively. Ablations further show that these gains depend on dynamically constructing optimization targets, estimating their evolving utility, and balancing continued optimization with new failure discovery. Together, these results establish automated curriculum learning as a new crucial optimization dimension for harness optimization.
Auto-harness systems such as A-Evolve, GEPA, and Meta-Harness improve LLM agents by optimizing prompts, skills, tools, memories, and supporting infrastructure from execution feedback, but they are typically evaluated on fixed offline benchmarks. Real deployments instead present open-ended task streams: histories grow without a fixed endpoint, heterogeneous tasks require different harnesses, and problem distributions shift over time. These challenges make a single repeatedly and densely updated harness brittle, causing performance degradation as accuracy peaks early and then declines. This motivates sustained harness construction with task-wise adaptation. We introduce Adaptive Auto-Harness, a framework and system for such streams. The framework decomposes the gap to an oracle harness into evolution loss and adaptation loss. The system addresses these losses with a stateful multi-agent evolver, a harness tree with solve-time routing, and human-steering hooks for cases where history lacks the needed signal. Across prediction-market, security-competition, and event-forecasting streams, Adaptive Auto-Harness outperforms five existing auto-harness baselines and ablations attribute gains to better construction, routing, or targeted human steering. Code is available in Link.
Zewen Liu, Zhan Shi, Yisi Sang +7
Emory University · Amazon · The Pennsylvania State University +2
The performance of LLM-based agents is jointly shaped by their base models and the harnesses that mediate their interaction with the environment. Because different models exhibit distinct behaviors, effective harness design is inherently model-specific. Yet agent harnesses are still largely engineered by human experts, a paradigm that scales poorly as modern LLMs become increasingly diverse and rapidly evolving. In this paper, we introduce Self-Harness, a new paradigm in which an LLM-based agent improves its own operating harness, without relying on human engineers or stronger external agents. We operationalize Self-Harness as an iterative loop with three stages: Weakness Mining, which identifies model-specific failure patterns from execution traces; Harness Proposal, which generates diverse yet minimal harness modifications tied to these failures; and Proposal Validation, which accepts candidate edits only after regression testing. We instantiate Self-Harness on Terminal-Bench-2.0 using a minimal initial harness and three base models from diverse families: MiniMax M2.5, Qwen3.5-35B-A3B, and GLM-5. Across all three models, Self-Harness consistently improves performance, with held-out pass rates increasing from 40.5% to 61.9%, 23.8% to 38.1%, and 42.9% to 57.1%, respectively. Qualitative analyses further show that Self-Harness does not simply add generic instructions, but effectively turns model-specific weaknesses into concrete, executable harness changes. These results suggest a path toward LLM-based agents that are not merely shaped by their harnesses, but can also participate in reshaping them.