Organizations: Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology · Department of Computer Science and Technology, Institute for AI, Tsinghua University
Autonomous agents increasingly depend on harnesses that govern interaction, memory, tools, and execution around large language models (LLMs). While recent work has begun to evolve these harnesses from execution experience, the supervision used to guide evolution often remains relatively static, making it difficult to track changing behavioral weaknesses across rounds and potentially limiting generalization to unseen tasks. To address this issue, we propose RuHarness, a rubric--harness co-evolution framework for continual agent self-improvement. RuHarness continually discovers and reassesses behavioral criteria from execution trajectories, prioritizes the most relevant unresolved weaknesses, and uses them to guide targeted harness updates while keeping the task objective fixed. Extensive experiments across diverse agent tasks and backbone models show that RuHarness outperforms strong harness-evolution baselines, particularly on unseen held-out tasks. Further analyses show that adaptive rubric prioritization tracks shifting weaknesses and supports more sustained improvement and stable generalization across evolution rounds.
Figures & tables
Figure 1: Illustration of rubric–harness co-evolution. Static rubrics may fall behind as behavioral weaknesses shift during harness evolution, while co-evolving rubrics track emerging failures and guide harness improvement.
Figure 2: Overview of the Rubric–Harness Co-Evolution Framework. Rubric evolution adaptively identifies and prioritizes unresolved behavioral weaknesses from trajectory evidence, while harness evolution performs targeted updates accordingly, yielding new behaviors that drive the next round of evolution.
Method
Information Retrieval
Knowledge Work
Software Engineering
Code Implementation
Math Reasoning
Average
Held-in
Held-out
Held-in
Held-out
Held-in
Held-out
Held-in
Held-out
Held-in
Held-out
Held-in
Held-out
Qwen3.5-35B-A3B
Vanilla Agent
28.0
20.0
51.7
40.0
40.0
50.0
83.6
65.1
69.5
46.0
54.6
44.2
ACE ( 2026b )
31.0
21.5
56.7
40.0
46.7
55.6
86.7
69.8
67.9
45.0
57.8
46.4
EvoSkill ( 2026 )
36.0
27.7
53.3
42.0
44.4
47.2
85.9
68.6
70.3
47.0
58.0
46.5
Self-Harness ( 2026a )
34.0
27.7
53.3
44.0
51.1
55.6
90.6
67.4
72.9
47.0
60.4
48.3
Table 1: Performance comparison across five domains on the held-in and held-out sets. Average denotes the macro-average across all five domains. The best result is shown in bold , and the second-best result is underlined .
Method
Information Retrieval
Knowledge Work
Software Engineering
Code Implementation
Math Reasoning
Average
Held-in
Held-out
Held-in
Held-out
Held-in
Held-out
Held-in
Held-out
Held-in
Held-out
Held-in
Held-out
RuHarness
37.0
30.8
60.0
52.0
53.3
63.9
89.8
69.8
72.1
52.0
62.4
53.7
w/o Rubric Module
31.0
23.1
51.7
44.0
48.9
52.8
82.8
65.1
71.1
44.0
57.1
45.8
w/o Rubric Evolution
35.0
29.2
55.0
34.0
44.4
55.6
76.6
57.0
71.3
46.0
56.5
44.4
w/o Rubric Frontier
35.0
30.8
56.7
50.0
46.7
50.0
84.4
67.4
72.6
47.0
59.1
49.0
Table 2: Ablation study of RuHarness on Qwen3.5-35B-A3B across five domains.
Figure 3: Rubric similarity across evolution rounds.
Figure 4: Analysis of adaptive rubric prioritization and subsequent harness improvement.
Figure 5: Evolution trajectories of different methods.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Domain
Evol. (Held-in)
Val.
Test (Held-out)
Information Retrieval
100
54
65
Knowledge Work
60
36
50
Software Engineering
45
26
36
Code Implementation
128
54
86
Math Reasoning
380
98
100
Appendix
Table 3: Dataset statistics for the evolution, validation, and test splits across five evaluation domains.
Coordinate
Functional Scope
Editable Content
Equip
Capability provisioning
Skills, tools, and reusable modules
Instruct
Behavioral guidance
System and task instructions
Act
Action execution
Tool-use and action strategies
Observe
Observation processing
Parsing, filtering, and summarization
Remember
Context management
Memory and context handling
Terminate
Completion control
Stopping and finalization
Appendix
Table 4: Unified harness coordinates, their functional scopes, and editable contents.
Current State
Condition
Next State
Probation
P>0
Active
Probation
Untriggered for 4 rounds
Retired
Active
Trigger rate <5% for 3 rounds
Dormant
Active
N/A drift for 3 rounds
Dormant
Active
P<0.10 for 3 rounds
Retired
Dormant
Trigger rate ≥15%
Active
Appendix
Table 5: Transition rules for the rubric lifecycle. Trigger rate is the violation frequency of a criterion, and N/A drift reflects a sustained rise in its not-applicable rate.
Figure 6: Rubric lifecycle across evolution rounds on the Qwen3.5-35B-A3B model with the knowledge work task. Each column shows criterion states at the end of a round, with flow widths proportional to the number of criteria. “Not yet born” denotes criteria introduced in later rounds; “Retired” is cumulative and includes candidates rejected at birth.
Method
Held-in
Held-out
Qwen3.5-35B-A3B on Knowledge Work
Vanilla Agent
51.1±1.0
38.0±2.0
ACE
55.6±1.0
42.0±2.0
EvoSkill
54.4±1.0
41.3±1.2
Self-Harness
52.2±1.9
41.3±3.1
AHE
55.0±1.7
47.3±3.1
Appendix
Table 6: Robustness to random seeds on two representative settings. Results are reported as mean ± standard deviation over three independent runs.
Figure 7: Effectiveness of dynamic rubric evolution on the Qwen3.5-35B-A3B model with the knowledge work task. Compared with a static rubric, the dynamic rubric maintains a smaller held-out–held-in generalization gap and substantially higher frontier coverage over the course of evolution.
Figure 8: Effectiveness of rubric frontier prioritization. Compared with the complete eligible rubric pool, the frontier concentrates criteria with substantially higher core diagnostic value and contains far fewer criteria with non-positive diagnostic value.
Round
Observed Pattern
Rubric Diagnosis
Target Coordinate
Harness Response
Case I: Knowledge Work — From Generic Checking to Executable Verification
R1
Execution
Generated scripts should not be treated as valid merely because they were written; successful execution must be verified .
Observe
Require every generated script to be executed and explicitly checked for errors before subsequent actions.
R3
Error Analysis
Detecting a tool failure is insufficient if the underlying cause is not examined .
Observe
Require the agent to inspect the error output and identify its cause before proceeding with unrelated commands.
R4
Deliverable Check
Successful execution does not guarantee that the expected deliverable has actually been produced .
Terminate + Equip
Add a completion-stage check that confirms the output file exists and matches the required filename .
R5
Programmatic Check
Declarative confirmation remains underspecified for reliable verification .
Terminate
Strengthen the check by requiring programmatic verification of the output filename, e.g., through os.path.basename .
R8
Pre-access Check
File validity should be established before the file is consumed downstream .
Remember
Move verification earlier by requiring file existence to be checked before reading reference files.
Appendix
Table 7: Case studies of rubric–harness co-evolution with Qwen3.5-35B-A3B. Across evolution rounds, diagnostic focus progressively shifts from coarse behavioral checks to more specific and operational requirements, which are translated into targeted modifications of the corresponding harness coordinates.