Agentic Harness Engineering Beta: Requirement-Centered Evolution of Agent Harnesses
Organizations: Fudan University · Peking University · Renmin University of China · Shanghai Qiji Zhifeng Co., Ltd
Abstract
Real-world harness engineering is driven by explicit and implicit user requirements, which guide development and validation. However, existing harness evolution methods often overlook implicit requirements and interaction needs, while using a single evolution agent to modify the harness and benchmark scores to select changes. This process often produces narrow patches and leads to benchmark-specific local optima. We introduce Agentic Harness Engineering Beta (AHE-Beta), a requirement-centered loop that brings real-world engineering practices into harness evolution. The loop consists of three steps: (1) User-driven evaluation lets simulated users raise their requirements as issues while they work with the harness over multiple turns. (2) Team-based modification implements those requirements as generalizable changes rather than task-specific patches. (3) Requirement-driven selection merges a change only when it resolves a verified requirement and passes review. Empirically, on held-out tasks, AHE-Beta raises pass@1 from 21.9% to 33.3% while using 40.8% fewer tokens than Base Harness, and achieves higher pass@1 than Claude Code, Meta-Harness, and AHE. In a blind human A/B test, real users prefer AHE-Beta in 31 of 48 valid pairs and rate it higher on all five dimensions, matching the directional preference of the user simulator. The human A/B results show that AHE-Beta achieves its goal of aligning harness evolution with real-world user requirements. Without further evolution, the gains transfer to GLM-5.3 and DeepSeek-V4.1-Flash with pass@1 improvements of 15.8% and 5.3% over Base Harness. Together, these results show that requirement-centered harness evolution is a more general method for improving agent capabilities.
Figures & tables
| Mean score | AHE-Beta against Base Harness | ||||
| Measure | Base Harness | AHE-Beta | Higher | Equal | Lower |
| Overall | 5.39 | 6.01 | 31 | 10 | 7 |
| Process understandability | 4.13 | 5.33 | 29 | 15 | 4 |
| Usefulness | 5.40 | 6.15 | 16 | 26 | 5 |
| Control | 5.23 | 5.77 | 18 | 25 | 4 |
| User burden | 5.98 | 6.27 | 17 | 25 | 6 |
| Setting | Pass@1 | Tokens |
|---|---|---|
| Base Harness | 21.9% | 2.40 |
| AHE-Beta ( , w/o interaction) | 23.7% | 3.15 |
| AHE-Beta ( ) | 28.9% | 1.71 |
| Harness | Interaction | Pass@1 | Tokens |
|---|---|---|---|
| Claude Code | with | 27.2% | 2.89 |
| Meta-Harness | with | 21.1% | 2.26 |
| AHE-Beta | with | 33.3% | 1.42 |
| Claude Code | w/o | 23.7% | 3.24 |
| Meta-Harness | w/o | 13.2% | 2.53 |
| AHE-Beta | w/o | 26.3% | 1.55 |
| Pass@1 | User rating | Tokens | ||||
|---|---|---|---|---|---|---|
| Execution model | Base | AHE-Beta | Base | AHE-Beta | Base | AHE-Beta |
| GLM-5.3-Flash | 21.9% | 33.3% | 2.59 | 3.04 | 2.40 | 1.42 |
| GLM-5.3 | 23.7% | 39.5% | 1.70 | 3.03 | 1.91 | 1.99 |
| DeepSeek-V4.1-Flash | 36.8% | 42.1% | 3.33 | 3.39 | 3.80 | 2.81 |
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Column | Meaning | Values |
|---|---|---|
| Evaluation | Evaluation capability: how well the user checks the output of the harness. | blind : skims the output and trusts it. competent : runs tests, reads diffs, and spots plausible but wrong APIs. expert : verifies systematically and catches missing edge cases and boundary bugs. |
| Acceptance | Acceptance bar: the lowest quality the user accepts. | runs : the result runs without error. functional : it works on the normal case the user tried. idiomatic : it follows modern idiomatic practice. rigorous : it handles types and boundary cases. |
| Delegation | Interaction style: how much context the user writes into a request. | one-liner : one line with little context. half context : background but no acceptance criteria. spec level : a full specification with the information the harness needs. |
| Supervision | Interaction style: how much of the ongoing work the user needs to see. | every step : grows uneasy during long opaque stretches and asks for visibility. phase check : wants a signal at key moments that the work is on track. result only : cares about the final result alone. |
| Response | Interaction style: what the user does when a result is wrong. | feedback : describes the problem and asks for a fix. handover : takes over the remaining work. resend : sends the request again, possibly rephrased. give up : abandons the task quickly. file issue : stops and files an issue after repeated failures. |
| Trust | Trust: how far the user believes claims of the harness without proof. | blind : trusts by default. balanced : starts neutral and grows skeptical after repeated failures. skeptical : doubts by default and expects proof. |
| Card | Evaluation | Acceptance | Delegation | Supervision | Response | Trust | Domain | Autonomy | Time |
|---|---|---|---|---|---|---|---|---|---|
| sr-01 | expert | idiomatic | spec level | phase check | handover | skeptical | dev tooling | collaborative | patient |
| sr-02 | expert | idiomatic | spec level | phase check | feedback | balanced | dev tooling | collaborative | steady |
| sr-08 | expert | idiomatic | spec level | phase check | feedback | balanced | dev tooling | collaborative | steady |
| sr-15 | competent | functional | spec level | phase check | feedback | balanced | dev tooling | collaborative | steady |
| sr-16 | competent | runs | half context | phase check | resend | balanced | web frontend | collaborative | impatient |
| sr-19 | competent | functional | half context | phase check | feedback | balanced | infra and DevOps | collaborative | steady |
| # | Source | Task | Interaction | Persona |
|---|---|---|---|---|
| 1 | SWE-bench | pvlib__pvlib-python-1478 | with | sr-15 |
| 2 | Terminal-Bench | production-planning | w/o | sr-15 |
| 3 | TUA-Bench | 010-pivot-product-revenue | with | sr-20 |
| 4 | mechanism | superset-goal-discovery | with | sr-27 |
| 5 | SWE-bench | pylint-dev__astroid-2240 | with | sr-15 |
| 6 | Terminal-Bench | kv-live-surgery | with | sr-19 |
| Task | Source |
|---|---|
| Coding (14) | |
| build-word2vec-pipeline | BigCodeBench |
| featurebench-add-feature-mlflow-unity-catalog | FeatureBench |
| featurebench-add-feature-xarray-backend-chunks | FeatureBench |
| swebenchpro-fix-file-suffix-chooser | SWE-bench Pro |
| swebenchpro-improve-process-signal-messages | SWE-bench Pro |
| Label | Meaning |
|---|---|
| bug , feature | kind of a new issue, guessed from its wording and corrected by Triage |
| chore , document | further kinds that only Triage assigns |
| needs-triage | a new issue waiting for Triage |
| direct | an issue from a session without interaction |
| round- | the round in which the issue or item was written |
| proposal | a requirement, consolidated requirement, or PRD waiting for admission |
| Overall | Coding | Optimization | Analysis | Tool Use | |||||||||||
| 38 tasks | 14 tasks | 12 tasks | 7 tasks | 5 tasks | |||||||||||
| Harness | Pass@1 | User rating | Tokens | Pass@1 | User rating | Tokens | Pass@1 | User rating | Tokens | Pass@1 | User rating | Tokens | Pass@1 | User rating | Tokens |
| External harnesses | |||||||||||||||
| Claude Code | 27.2% | 2.46 | 2.89 | 38.1% | 3.31 | 3.53 | 22.2% | 1.53 | 3.37 | 19.0% | 2.05 | 1.32 | 20.0% | 2.87 | 2.12 |
| 1.5 | 0.18 | 0.20 | 4.1 | 0.29 | 0.46 | 9.6 | 0.65 | 0.21 | 8.2 | 0.30 | 0.38 | 20.0 | 0.31 | 0.43 | |
| Meta-Harness | 21.1% | 1.69 | 2.26 | 35.7% | 2.79 | 2.86 | 19.4% | 1.06 | 1.94 | 9.5% | 1.20 | 2.00 | 0.0% | 1.00 | 1.72 |
| Round | Issues | Declined | Requirements | Consolidated | PRDs | RFCs |
|---|---|---|---|---|---|---|
| 1 | 28 | 3 | 36 | 5 | 3 | 18 |
| 2 | 26 | 6 | 23 | 5 | 3 | 18 |
| 3 | 24 | 9 | 19 | 4 | 3 | 16 |
| 4 | 23 | 6 | 18 | 5 | 2 | 10 |
| 5 | 30 | 12 | 19 | 3 | 3 | 9 |
| Total | 131 | 36 | 115 | 22 | 14 | 71 |
| Variant | Harness | User interaction and review | Requirement roles | Developers | Component sum |
|---|---|---|---|---|---|
| AHE-Beta ( , w/o interaction) | 232.9 | 3.6 | 25.7 | 619.1 | 881.4 |
| AHE-Beta ( ) | 113.3 | 15.8 | 36.4 | 837.2 | 1,002.7 |
| AHE-Beta ( ) | 373.6 | 46.1 | 158.3 | 1,022.1 | 1,600.1 |
| AHE-Beta ( ) | 473.1 | 41.2 | 108.7 | 1,301.7 | 1,924.8 |
| Round | Harness | User interaction and review | Requirement roles | Developers | Development coverage |
|---|---|---|---|---|---|
| 1 | 114.5 | 6.9 | 19.6 | 310.4 | 153/153 |
| 2 | 119.4 | 6.2 | 21.7 | 322.9 | 137/137 |
| 3 | 57.8 | 8.6 | 22.7 | 291.3 | 121/121 |
| 4 | 79.7 | 8.8 | 23.2 | 180.3 | 80/80 |
| 5 | 101.7 | 10.7 | 21.5 | 196.8 | 70/70 |
| Total | 473.1 | 41.2 | 108.7 | 1,301.7 | 561/561 |
| Task | Codebase | Pairs | AHE-Beta | Base Harness | Tie | Neither |
| T1 Annotation | Matplotlib | 9 | 6 | 2 | 1 | 0 |
| T2 Staged report | Django | 2 | 2 | 0 | 0 | 0 |
| T3 Oversell | Shop service | 13 | 5 | 1 | 6 | 1 |
| T4 Game feature | Browser game | 10 | 8 | 2 | 0 | 0 |
| T5 Span selector | Matplotlib | 7 | 5 | 1 | 1 | 0 |
| T6 Literal annotation | Sphinx | 7 | 5 | 1 | 1 | 0 |
Agent Harness Optimization
Agent harness optimization improves the prompts, tools, memory, workflows, and control code that shape how AI agents act.
See all 206 papers