Organizations: Tencent HY LLM Frontier · Indiana University · University of Maryland, College Park · National University of Singapore · University of Georgia
Long-horizon terminal agents may solve a task in one run yet fail across repeated runs. Stochastic errors can propagate through actions and context, creating a gap between capability and reliable execution. We present RiseHarness, a framework for iteratively improving agent harnesses with model weights fixed. It focuses on flipping tasks with both successful and failed runs under the same configuration, using successful runs as references for failure analysis. RiseHarness combines agent trajectories with internal harness traces from a runtime monitor to examine how model actions interact with command execution, feedback, context management, and completion decisions. A Harness Handbook links this evidence to harness behaviors and source code, helping identify recurring problems in mechanisms shared across tasks. RiseHarness then searches other harness frameworks for relevant implementations and adapts them as design references to the current model and execution protocol. Candidate changes undergo behavioral checks and repeated task evaluation; versions with measured gains seed the next iteration. This loop integrates suitable mechanisms from different frameworks into a single harness and uses new execution evidence to guide further changes. We evaluate task success and reliability across models, ablate harness execution evidence and external research, and test transfer to unseen benchmarks with the same model. RiseHarness treats a specific model as the unit of harness adaptation, studying how changes to shared interaction mechanisms can make existing capabilities more reliable across tasks.
Figures & tables
Figure 1: Operations within a discovery round of RiseHarness : record dual traces, identify shared behavioral problems, research and adapt mechanisms, and implement and evaluate changes. The Handbook connects execution evidence to harness behavior and code.
Harness
TBH
TB2.1
Pass@1
Pass 4
Pass@4
Flip ↓
Pass@1
Pass 4
Pass@4
Flip ↓
GPT-6-sol
Terminus-2
50.25
39.00
59.00
20.00
78.65
68.54
86.52
17.98
ModularRSI
50.75
43.00
58.00
15.00
78.65
66.29
91.01
24.72
AHE Harness
54.75
44.00
59.00
15.00
78.09
60.67
89.89
29.21
RiseHarness
55.50 +5.25
46.00 +7
60.00 +1
14.00 -6
81.74 +3.09
69.66 +1.12
88.76 +2.25
19.10 +1.12
Table 1: Performance and reliability ( r=4 ; %). Coral annotations show absolute changes from Terminus-2 for RiseHarness within each model and benchmark. Higher pass metrics and lower Flip are better. Dashes denote unreported results. DeepSeek uses the final four-modification version on TBH and the best observed three-modification version on TB2.1. GPT-6-sol trial counts appear in Appendix D.1 .
Figure 3
Figure 4: Model-specific interventions mapped to Handbook behaviors. GPT-6-sol modifies session recovery, observation construction, and completion context; DeepSeek-V4-Flash adds completion-evidence checks. Details appear in Appendix B .
Figure 5
Figure 7: Model-adapted vs. swapped final harnesses ( r=4 ). GPT: GPT-6-sol; DeepSeek: DeepSeek-V4-Flash, using its final + IR harness. Y-axes start above zero.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Behavior
Primary stage
Supporting stages
Executable completion verification (SE)
Completion confirmation handshake
LLM query and context rescue; Response parsing and command extraction: preserve and parse the verification command and expected exit code.
Designated-invocation verification (DI)
Completion confirmation handshake
Command dispatch and timeout handling: record dispatched commands.
Completion-outcome observation (OO)
Completion confirmation handshake
LLM query and context rescue; Response parsing and command extraction: obtain verification fields for the environment probe.
Independent-reading corroboration (IR)
Completion confirmation handshake
Command dispatch and timeout handling; Observation construction and output truncation: record commands and construct observations containing reading results.
Appendix
Table 2: Behavioral modifications for DeepSeek-V4-Flash and their Handbook mappings.
Table 5: DeepSeek-V4-Flash ablations ( r=4 ; %). These values support Section 4.3 and are kept separate from the main comparison table.
Benchmark
Passed / total
Pass@1
Pass 4
Pass@4
Flip
DeepSeek-V4-Flash harness → GPT-6-sol
TBH
211 / 400
52.75
42.00
62.00
20.00
TB2.1
270 / 356
75.84
60.67
89.89
29.21
GPT-6-sol harness → DeepSeek-V4-Flash
TBH
233 / 400
58.25
45.00
69.00
24.00
TB2.1
260 / 356
73.03
62.92
83.15
20.22
Appendix
Table 6: Cross-model transfer of final evolved harnesses ( r=4 ). Pass metrics and Flip are in %; task counts use 100 tasks on TBH and 89 on TB2.1. Arrows indicate harness source model → execution model.
Harness
TBH
TB2.1
Pass@1
Pass 4
Pass@4
Flip ↓
Pass@1
Pass 4
Pass@4
Flip ↓
Terminus-2
41.25
22.00
58.00
37.00
58.10
34.83
75.30
44.94
RiseHarness
43.50 +2.25
29.00 +7
59.00 +1
30.00 -7
62.92 +4.82
43.82 +8.99
80.90 +5.6
37.08 -7.86
Appendix
Table 7: Historical GPT-5.6-Terra results ( r=4 ; %), separate from GPT-6-sol. Coral annotations show absolute changes from Terminus-2.
Agents built around large language models continually accumulate interaction trajectories during deployment, yet their behavior typically remains fixed. Beyond updating model weights, these trajectories can improve the agent harness that constructs context, mediates tools, validates actions, and recovers execution. We introduce Harness-R1, the first method, to our knowledge, that makes failure-conditioned, lifecycle-wide editing of an existing executable runtime a learned capability. It post-trains a dedicated harness engineer with online reinforcement learning so that its edits are optimized for the realized task success they produce, rather than proposed by a fixed editor. A separate 9B engineer converts batches of target-agent failures into validated executable patches; fresh same-batch reruns of the frozen target provide outcome rewards, so training updates only the engineer. Cold-start supervised fine-tuning initializes this editing policy, which is then trained online with group-relative policy optimization. Across WebShop, ALFWorld, and DBBench, Harness-R1 raises vanilla Qwen3.5-9B success from 44.3% to 53.6% (+9.3 percentage points). After direct target-agent fine-tuning, a target-specific engineer raises the average further from 59.2% to 64.2% (+5.0 points); because these gains hold both before and after fine-tuning the target, Harness-R1 points toward co-evolving the harness engineer and the target agent.
Shuai Shao, Kangning Zhang, Qingyao Li +7
Shanghai Jiao Tong University · Xiaohongshu Inc. · Southeast University
Large language model (LLM) agents increasingly undertake long-horizon tasks that require sustained reasoning, tool use, and revision across many interdependent steps. However, existing agent harnesses maintain task execution, task state, and completion assessment within a growing context, making the state difficult to track and allowing incorrect self-assessments to propagate into later decisions. We reformulate long-horizon execution as a task-state management problem and propose LongHorizon-Harness, which maintains the task state explicitly outside execution and updates it only with facts independently verified from the environment. Its Manage-Execute-Audit(MEA) loop uses a manager to maintain the task state and determine the next subtask, a fresh-context executor to perform it, and a read-only auditor to verify the resulting environment state before the next round. A lightweight AgentAdapter supports interchangeable model and harness backends without modifying their native agent loops. LongHorizon-Harness improves Qwen3.7-Plus from 51.8% to 80.7% on WeaveBench, from 69.7% to 77.2% on Terminal-Bench2.1, and from 2.8% to 8.3% on OSWorld2.0. It also raises Claude Opus4.7 from 20.0% to 34.3% on an OSWorld2.0 subset, demonstrating consistent gains across models, harnesses, and interaction domains.
AI agent performance depends critically on the runtime harness, comprising the prompts, tools, memory, and control flow that mediate how a model observes, reasons, and acts. Yet today's harnesses remain largely hand-crafted and static: each new model or task still demands bespoke scaffolding, and the rich traces produced during execution are rarely distilled back into systematic improvement. We introduce HarnessX, a foundry for composable, adaptive, and evolvable agent harnesses. HarnessX assembles typed harness primitives via a substitution algebra, adapts them through AEGIS, a trace-driven multi-agent evolution engine grounded in an operational mirror between symbolic adaptation and reinforcement learning, and closes the harness-model loop by turning trajectories into both harness updates and model training signal. Across five benchmarks (ALFWorld, GAIA, WebShop, tau^3-Bench, and SWE-bench Verified), HarnessX yields an average gain of +14.5% (up to +44.0%), with gains largest where baselines are lowest. These results suggest that agent progress need not come from model scaling alone: composing and evolving runtime interfaces from execution feedback is an actionable and complementary lever. The complete codebase will be open-sourced in a future release.