Organizations: Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology · Department of Computer Science and Technology, Institute for AI, Tsinghua University
Autonomous agents increasingly depend on harnesses that govern interaction, memory, tools, and execution around large language models (LLMs). While recent work has begun to evolve these harnesses from execution experience, the supervision used to guide evolution often remains relatively static, making it difficult to track changing behavioral weaknesses across rounds and potentially limiting generalization to unseen tasks. To address this issue, we propose RuHarness, a rubric--harness co-evolution framework for continual agent self-improvement. RuHarness continually discovers and reassesses behavioral criteria from execution trajectories, prioritizes the most relevant unresolved weaknesses, and uses them to guide targeted harness updates while keeping the task objective fixed. Extensive experiments across diverse agent tasks and backbone models show that RuHarness outperforms strong harness-evolution baselines, particularly on unseen held-out tasks. Further analyses show that adaptive rubric prioritization tracks shifting weaknesses and supports more sustained improvement and stable generalization across evolution rounds.
Figures & tables
Figure 1: Illustration of rubric–harness co-evolution. Static rubrics may fall behind as behavioral weaknesses shift during harness evolution, while co-evolving rubrics track emerging failures and guide harness improvement.
Figure 2: Overview of the Rubric–Harness Co-Evolution Framework. Rubric evolution adaptively identifies and prioritizes unresolved behavioral weaknesses from trajectory evidence, while harness evolution performs targeted updates accordingly, yielding new behaviors that drive the next round of evolution.
Method
Information Retrieval
Knowledge Work
Software Engineering
Code Implementation
Math Reasoning
Average
Held-in
Held-out
Held-in
Held-out
Held-in
Held-out
Held-in
Held-out
Held-in
Held-out
Held-in
Held-out
Qwen3.5-35B-A3B
Vanilla Agent
28.0
20.0
51.7
40.0
40.0
50.0
83.6
65.1
69.5
46.0
54.6
44.2
ACE ( 2026b )
31.0
21.5
56.7
40.0
46.7
55.6
86.7
69.8
67.9
45.0
57.8
46.4
EvoSkill ( 2026 )
36.0
27.7
53.3
42.0
44.4
47.2
85.9
68.6
70.3
47.0
58.0
46.5
Self-Harness ( 2026a )
34.0
27.7
53.3
44.0
51.1
55.6
90.6
67.4
72.9
47.0
60.4
48.3
Table 1: Performance comparison across five domains on the held-in and held-out sets. Average denotes the macro-average across all five domains. The best result is shown in bold , and the second-best result is underlined .
Method
Information Retrieval
Knowledge Work
Software Engineering
Code Implementation
Math Reasoning
Average
Held-in
Held-out
Held-in
Held-out
Held-in
Held-out
Held-in
Held-out
Held-in
Held-out
Held-in
Held-out
RuHarness
37.0
30.8
60.0
52.0
53.3
63.9
89.8
69.8
72.1
52.0
62.4
53.7
w/o Rubric Module
31.0
23.1
51.7
44.0
48.9
52.8
82.8
65.1
71.1
44.0
57.1
45.8
w/o Rubric Evolution
35.0
29.2
55.0
34.0
44.4
55.6
76.6
57.0
71.3
46.0
56.5
44.4
w/o Rubric Frontier
35.0
30.8
56.7
50.0
46.7
50.0
84.4
67.4
72.6
47.0
59.1
49.0
Table 2: Ablation study of RuHarness on Qwen3.5-35B-A3B across five domains.
Figure 3: Rubric similarity across evolution rounds.
Figure 4: Analysis of adaptive rubric prioritization and subsequent harness improvement.
Figure 5: Evolution trajectories of different methods.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Domain
Evol. (Held-in)
Val.
Test (Held-out)
Information Retrieval
100
54
65
Knowledge Work
60
36
50
Software Engineering
45
26
36
Code Implementation
128
54
86
Math Reasoning
380
98
100
Appendix
Table 3: Dataset statistics for the evolution, validation, and test splits across five evaluation domains.
Coordinate
Functional Scope
Editable Content
Equip
Capability provisioning
Skills, tools, and reusable modules
Instruct
Behavioral guidance
System and task instructions
Act
Action execution
Tool-use and action strategies
Observe
Observation processing
Parsing, filtering, and summarization
Remember
Context management
Memory and context handling
Terminate
Completion control
Stopping and finalization
Appendix
Table 4: Unified harness coordinates, their functional scopes, and editable contents.
Current State
Condition
Next State
Probation
P>0
Active
Probation
Untriggered for 4 rounds
Retired
Active
Trigger rate <5% for 3 rounds
Dormant
Active
N/A drift for 3 rounds
Dormant
Active
P<0.10 for 3 rounds
Retired
Dormant
Trigger rate ≥15%
Active
Appendix
Table 5: Transition rules for the rubric lifecycle. Trigger rate is the violation frequency of a criterion, and N/A drift reflects a sustained rise in its not-applicable rate.
Figure 6: Rubric lifecycle across evolution rounds on the Qwen3.5-35B-A3B model with the knowledge work task. Each column shows criterion states at the end of a round, with flow widths proportional to the number of criteria. “Not yet born” denotes criteria introduced in later rounds; “Retired” is cumulative and includes candidates rejected at birth.
Method
Held-in
Held-out
Qwen3.5-35B-A3B on Knowledge Work
Vanilla Agent
51.1±1.0
38.0±2.0
ACE
55.6±1.0
42.0±2.0
EvoSkill
54.4±1.0
41.3±1.2
Self-Harness
52.2±1.9
41.3±3.1
AHE
55.0±1.7
47.3±3.1
Appendix
Table 6: Robustness to random seeds on two representative settings. Results are reported as mean ± standard deviation over three independent runs.
Figure 7: Effectiveness of dynamic rubric evolution on the Qwen3.5-35B-A3B model with the knowledge work task. Compared with a static rubric, the dynamic rubric maintains a smaller held-out–held-in generalization gap and substantially higher frontier coverage over the course of evolution.
Figure 8: Effectiveness of rubric frontier prioritization. Compared with the complete eligible rubric pool, the frontier concentrates criteria with substantially higher core diagnostic value and contains far fewer criteria with non-positive diagnostic value.
Round
Observed Pattern
Rubric Diagnosis
Target Coordinate
Harness Response
Case I: Knowledge Work — From Generic Checking to Executable Verification
R1
Execution
Generated scripts should not be treated as valid merely because they were written; successful execution must be verified .
Observe
Require every generated script to be executed and explicitly checked for errors before subsequent actions.
R3
Error Analysis
Detecting a tool failure is insufficient if the underlying cause is not examined .
Observe
Require the agent to inspect the error output and identify its cause before proceeding with unrelated commands.
R4
Deliverable Check
Successful execution does not guarantee that the expected deliverable has actually been produced .
Terminate + Equip
Add a completion-stage check that confirms the output file exists and matches the required filename .
R5
Programmatic Check
Declarative confirmation remains underspecified for reliable verification .
Terminate
Strengthen the check by requiring programmatic verification of the output filename, e.g., through os.path.basename .
R8
Pre-access Check
File validity should be established before the file is consumed downstream .
Remember
Move verification earlier by requiring file existence to be checked before reading reference files.
Appendix
Table 7: Case studies of rubric–harness co-evolution with Qwen3.5-35B-A3B. Across evolution rounds, diagnostic focus progressively shifts from coarse behavioral checks to more specific and operational requirements, which are translated into targeted modifications of the corresponding harness coordinates.
A harness is the code around a language-model agent that organizes prompts, calls tools, manages context, and controls execution. As models grow stronger, recent work has begun to let agents improve their own harnesses, a line of work known as self-evolving harnesses. In most existing methods, a separate proposer running on a human-designed harness modifies the solver's harness, and a separate harness is evolved for each benchmark. Real-world tasks come from many domains, so both the evolution and the evaluation of a harness should cover a diverse range of tasks. We propose a framework close to recursive self-improvement: the same frozen model, on the same version of the harness, first solves tasks as the solver and then, as the proposer, reads the complete run records and directly edits the harness that runs it. Each evolution batch draws tasks from five benchmarks in different domains. To measure generalization, training and held-out tasks are strictly separated, and we additionally evaluate on five out-of-distribution benchmarks never used during evolution. We frame the evolution process as deep-learning training with two stages, multi-task pretraining and continual training. Starting from a 49-line seed harness, the harness obtained at the end of the first stage improves the average score by 4.48 points on the in-distribution benchmarks and by 12.64 points on the out-of-distribution benchmarks, surpassing Codex on the former and matching it on the latter. In the second stage, continued evolution on Claw-Eval, one of the out-of-distribution benchmarks, further raises the score on that benchmark from 66.17 to 68.06, exceeding Codex. We also provide an in-depth analysis of the mechanisms that emerged during evolution, including output truncation, history compaction, and independent review.
Harness design plays a critical role in agent performance by shaping how large language models (LLMs) perceive, reason over, and act within executable environments. Recent work has proposed automatic harness evolution, which iteratively improves the harness from agent--environment interactions. However, existing methods often overfit to the evolution tasks, rely exclusively on trajectory-derived signals, and optimize harness components jointly, causing interference across components. We propose HarnessCompass, a novel automatic harness evolution framework built around constrained evolution, proactive feedback, and component-wise optimization. HarnessCompass first enforces global constraints on evolution, restricting modifications to task-agnostic harness changes that generalize beyond the evolution tasks. It then augments trajectory-derived evidence with proactive first-person feedback from the agent about harness usage, yielding richer signals for evolution. Finally, it decouples the optimization of different harness components before consolidating them into a unified harness, reducing cross-component interference while preserving component synergy. On SWE-bench Verified with GPT-5.4, HarnessCompass improves Pass@1 from 54% to 66% in only 5 evolution iterations, outperforming AHE in both effectiveness and evolution efficiency. In addition, the evolved harness transfers effectively to held-out tasks and other models, demonstrating substantially stronger generalization than prior automatic harness evolution methods.
Luan Zhang, Ruochen Zhou, Dandan Song +9
Beijing Institute of Technology, China · City University of Hong Kong, China · Independent, China
LLM agents are increasingly deployed as systems built around editable external harnesses, including prompts, skills, memories and tools, that shape task execution without changing model parameters. Harness self-evolution adapts such agents by updating these harnesses from execution evidence. Yet it remains unclear whether a model's base capability in task-solving predicts its capabilities in harness self-evolution: which models produce useful harness updates, and which actually benefit from them? We analyze two harness self-evolution capabilities: (i) harness-updating, the capability to produce useful persistent harness updates from execution evidence; (ii) harness-benefit, the capability to benefit from updated harnesses during task solving. Our analysis reveals two findings. First, harness-updating is flat in base capability: models from different capability tiers produce harness updates that lead to surprisingly similar gains; even Qwen3.5-9B's updates yield gains comparable to those of Claude Opus~4.6. Second, harness-benefit is non-monotonic in base capability: weak-tier models benefit little from updated harnesses, mid-tier models benefit most, and strong-tier models benefit less than mid-tier. We trace low gains at the weak tier to two failure modes: weak-tier models may fail to activate relevant harness artifacts, or activate them but fail to follow them faithfully. These findings suggest investing capability budget in the task-solving agent rather than the evolver, and targeting harness invocation and long-horizon instruction following in agent training. Our source code is publicly available at https://github.com/A-EVO-Lab/a-evolve/tree/release/harness-evolution.
Minhua Lin, Juncheng Wu, Zijun Wang +14
1The Pennsylvania State University · 2UC Santa Cruz · 3Amazon +3