Real-world harness engineering is driven by explicit and implicit user requirements, which guide development and validation. However, existing harness evolution methods often overlook implicit requirements and interaction needs, while using a single evolution agent to modify the harness and benchmark scores to select changes. This process often produces narrow patches and leads to benchmark-specific local optima. We introduce Agentic Harness Engineering Beta (AHE-Beta), a requirement-centered loop that brings real-world engineering practices into harness evolution. The loop consists of three steps: (1) User-driven evaluation lets simulated users raise their requirements as issues while they work with the harness over multiple turns. (2) Team-based modification implements those requirements as generalizable changes rather than task-specific patches. (3) Requirement-driven selection merges a change only when it resolves a verified requirement and passes review. Empirically, on held-out tasks, AHE-Beta raises pass@1 from 21.9% to 33.3% while using 40.8% fewer tokens than Base Harness, and achieves higher pass@1 than Claude Code, Meta-Harness, and AHE. In a blind human A/B test, real users prefer AHE-Beta in 31 of 48 valid pairs and rate it higher on all five dimensions, matching the directional preference of the user simulator. The human A/B results show that AHE-Beta achieves its goal of aligning harness evolution with real-world user requirements. Without further evolution, the gains transfer to GLM-5.3 and DeepSeek-V4.1-Flash with pass@1 improvements of 15.8% and 5.3% over Base Harness. Together, these results show that requirement-centered harness evolution is a more general method for improving agent capabilities.
Figures & tables
Figure 1: Overview of AHE-Beta in round t . ❶ The user simulator works with Ht on evolution tasks, files issues, and rates each session. ❷ Triage, the Architect, the PM, and the Developers distill requirements from the issues, list where the harness should change, and implement them as PRs. ❸ PRs with verified requirements are merged into main, which becomes Ht+1 .
Figure 2: Main comparison on 38 held-out tasks with GLM-5.3-Flash under evaluation with interaction, by domain, with task counts in parentheses. Bars show the mean over three runs. Base Harness is the starting point of AHE-Beta; Meta-Harness and AHE are released evolved harnesses. Hatched bars and bold labels mark AHE-Beta. The user rating panel starts at the scale floor of 1, so a bar of zero height denotes a mean user rating of 1.00. Table 10 in Appendix E lists every value.
Figure 3: AHE-Beta over recorded rounds, with R1 denoting Base Harness. Panels show (a) evolution user rating, (b) held-out user rating, and (c) held-out pass@1. Held-out observations exist only at R1, R3, and R6, and each vertical axis is truncated to its own range.
Mean score
AHE-Beta against Base Harness
Measure
Base Harness
AHE-Beta
Higher
Equal
Lower
Overall
5.39
6.01
31
10
7
Process understandability
4.13
5.33
29
15
4
Usefulness
5.40
6.15
16
26
5
Control
5.23
5.77
18
25
4
User burden
5.98
6.27
17
25
6
Table 1: Blind human A/B study on 48 valid pairs. Mean scores use a 1–7 scale, with higher scores indicating better experience. Overall averages the five displayed dimension means, while its Higher, Equal, and Lower counts report overall preference, including nine ties and one neither response under Equal. Other counts compare the paired dimension scores. Appendix F defines the five dimensions, and usefulness and control each lack one rating.
Setting
Pass@1 ↑
Tokens ↓
Base Harness
21.9%
2.40
AHE-Beta ( N=8 , w/o interaction)
23.7%
3.15
AHE-Beta ( N=8 )
28.9%
1.71
Table 2: Evolution with and without interaction, evaluated on the 38 held-out Flash tasks. AHE-Beta ( N=8 ) runs six of its eight evolution tasks with interaction, and AHE-Beta ( N=8 , w/o interaction) runs none. All rows use evaluation with interaction, and units and bold follow Table 10 .
Harness
Interaction
Pass@1 ↑
Tokens ↓
Claude Code
with
27.2%
2.89
Meta-Harness
with
21.1%
2.26
AHE-Beta
with
33.3%
1.42
Claude Code
w/o
23.7%
3.24
Meta-Harness
w/o
13.2%
2.53
AHE-Beta
w/o
26.3%
1.55
Table 3: User simulation during evaluation on 38 held-out tasks with GLM-5.3-Flash. Evaluation without interaction sends the task once and provides no simulated-user follow-up, so it produces no user rating. Tokens are execution tokens in millions per task, excluding user simulation. Bold marks the best pass@1 and lowest token usage within each protocol.
Pass@1 ↑
User rating ↑
Tokens ↓
Execution model
Base
AHE-Beta
Base
AHE-Beta
Base
AHE-Beta
GLM-5.3-Flash
21.9%
33.3%
2.59
3.04
2.40
1.42
GLM-5.3
23.7%
39.5%
1.70
3.03
1.91
1.99
DeepSeek-V4.1-Flash
36.8%
42.1%
3.33
3.39
3.80
2.81
Table 4: Cross-model transfer of the Flash-evolved harness, paired against Base Harness on the same execution model. GLM-5.3-Flash is the evolution model, and the other two bases receive the harness without in-domain re-evolution. Pass@1 and tokens use all 38 tasks, and user rating uses paired rated tasks (38, 37, and 36 from top to bottom). Bold marks the better value in each pair.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Column
Meaning
Values
Evaluation
Evaluation capability: how well the user checks the output of the harness.
blind : skims the output and trusts it. competent : runs tests, reads diffs, and spots plausible but wrong APIs. expert : verifies systematically and catches missing edge cases and boundary bugs.
Acceptance
Acceptance bar: the lowest quality the user accepts.
runs : the result runs without error. functional : it works on the normal case the user tried. idiomatic : it follows modern idiomatic practice. rigorous : it handles types and boundary cases.
Delegation
Interaction style: how much context the user writes into a request.
one-liner : one line with little context. half context : background but no acceptance criteria. spec level : a full specification with the information the harness needs.
Supervision
Interaction style: how much of the ongoing work the user needs to see.
every step : grows uneasy during long opaque stretches and asks for visibility. phase check : wants a signal at key moments that the work is on track. result only : cares about the final result alone.
Response
Interaction style: what the user does when a result is wrong.
feedback : describes the problem and asks for a fix. handover : takes over the remaining work. resend : sends the request again, possibly rephrased. give up : abandons the task quickly. file issue : stops and files an issue after repeated failures.
Trust
Trust: how far the user believes claims of the harness without proof.
blind : trusts by default. balanced : starts neutral and grows skeptical after repeated failures. skeptical : doubts by default and expects proof.
Appendix
Table 5: Persona dimensions, one row per column of Table 6 . Value meanings paraphrase the fixed sentence that the user simulator receives for each value, and Appendix D.1 gives these sentences verbatim.
Card
Evaluation
Acceptance
Delegation
Supervision
Response
Trust
Domain
Autonomy
Time
sr-01
expert
idiomatic
spec level
phase check
handover
skeptical
dev tooling
collaborative
patient
sr-02
expert
idiomatic
spec level
phase check
feedback
balanced
dev tooling
collaborative
steady
sr-08
expert
idiomatic
spec level
phase check
feedback
balanced
dev tooling
collaborative
steady
sr-15
competent
functional
spec level
phase check
feedback
balanced
dev tooling
collaborative
steady
sr-16
competent
runs
half context
phase check
resend
balanced
web frontend
collaborative
impatient
sr-19
competent
functional
half context
phase check
feedback
balanced
infra and DevOps
collaborative
steady
Appendix
Table 6: Personas used in evolution and evaluation, with columns defined in Table 5 . sr marks first-person cards and gh marks GitHub cards. Italic values are sampled or derived, and all other card values are grounded in the source. The last row is the hand-set evaluation persona.
#
Source
Task
Interaction
Persona
1
SWE-bench
pvlib__pvlib-python-1478
with
sr-15
2
Terminal-Bench
production-planning
w/o
sr-15
3
TUA-Bench
010-pivot-product-revenue
with
sr-20
4
mechanism
superset-goal-discovery
with
sr-27
5
SWE-bench
pylint-dev__astroid-2240
with
sr-15
6
Terminal-Bench
kv-live-surgery
with
sr-19
Appendix
Table 7: The 32 seeds in run order. Rules after seeds 8 and 16 close the seed sets of AHE-Beta ( N=8 ) and AHE-Beta ( N=16 ) , and AHE-Beta ( N=8 , w/o interaction) reuses seeds 1 to 8. Interaction marks whether a seed runs with interaction or w/o interaction (Section 3.1 ), and persona IDs refer to Table 6 .
Task
Source
Coding (14)
build-word2vec-pipeline
BigCodeBench
featurebench-add-feature-mlflow-unity-catalog
FeatureBench
featurebench-add-feature-xarray-backend-chunks
FeatureBench
swebenchpro-fix-file-suffix-chooser
SWE-bench Pro
swebenchpro-improve-process-signal-messages
SWE-bench Pro
Appendix
Table 8: The 38 held-out Harbor-index tasks by domain, with task counts in parentheses.
Label
Meaning
bug , feature
kind of a new issue, guessed from its wording and corrected by Triage
chore , document
further kinds that only Triage assigns
needs-triage
a new issue waiting for Triage
direct
an issue from a session without interaction
round- N
the round in which the issue or item was written
proposal
a requirement, consolidated requirement, or PRD waiting for admission
Appendix
Table 9: Gitea labels in the order an item meets them.
Overall
Coding
Optimization
Analysis
Tool Use
38 tasks
14 tasks
12 tasks
7 tasks
5 tasks
Harness
Pass@1 ↑
User rating ↑
Tokens ↓
Pass@1 ↑
User rating ↑
Tokens ↓
Pass@1 ↑
User rating ↑
Tokens ↓
Pass@1 ↑
User rating ↑
Tokens ↓
Pass@1 ↑
User rating ↑
Tokens ↓
External harnesses
Claude Code
27.2%
2.46
2.89
38.1%
3.31
3.53
22.2%
1.53
3.37
19.0%
2.05
1.32
20.0%
2.87
2.12
± 1.5
± 0.18
± 0.20
± 4.1
± 0.29
± 0.46
± 9.6
± 0.65
± 0.21
± 8.2
± 0.30
± 0.38
± 20.0
± 0.31
± 0.43
Meta-Harness
21.1%
1.69
2.26
35.7%
2.79
2.86
19.4%
1.06
1.94
9.5%
1.20
2.00
0.0%
1.00
1.72
Appendix
Table 10: Full values behind Figure 2 . Main comparison on 38 held-out tasks with GLM-5.3-Flash under evaluation with interaction. Each cell gives the mean, with the standard deviation in gray below it. Base Harness is the starting point of AHE-Beta; Meta-Harness and AHE are released evolved harnesses. Pass@1: audited fraction of tasks passed with one attempt. User rating: mean rating on a 1–4 scale. Tokens: mean execution tokens in millions per task, excluding user simulation. Bold marks the best mean in each column, and ties are all bold.
Round
Issues
Declined
Requirements
Consolidated
PRDs
RFCs
1
28
3
36
5
3
18
2
26
6
23
5
3
18
3
24
9
19
4
3
16
4
23
6
18
5
2
10
5
30
12
19
3
3
9
Total
131
36
115
22
14
71
Appendix
Table 11: Issue flow of AHE-Beta evolution by round, read from the frozen Gitea records of the run. Issues counts the issues that the 32 sessions of a round file, and Declined counts those that Triage declines, with four issues of round 5 still open. Requirements counts the requirements that Triage writes from the remaining issues together with those that ask the harness to record facts its logs miss. The Architect merges part of the requirements into consolidated requirements, and RFCs counts the changes merged into the harness, one for each consolidated requirement, PRD, and remaining requirement. Round 6 runs sessions only and is omitted.
Variant
Harness
User interaction and review
Requirement roles
Developers
Component sum
AHE-Beta ( N=8 , w/o interaction)
232.9
3.6
25.7
619.1
881.4
AHE-Beta ( N=8 )
113.3
15.8
36.4
837.2
1,002.7
AHE-Beta ( N=16 )
373.6
46.1
158.3
1,022.1
1,600.1
AHE-Beta ( N=32 )
473.1
41.2
108.7
1,301.7
1,924.8
Appendix
Table 12: Recorded evolution token components in millions over rounds 1 to 5. Harness covers selected evolution sessions. User interaction and review covers simulated interaction and user ratings in selected sessions plus one-time reviews, with only reviews for AHE-Beta ( N=8 , w/o interaction). Requirement roles sums recorded cached input and output for Triage, the Architect, and the PM, including recorded retries. Developers covers selected completed development calls. The component sum is a lower bound within these scopes, and missing usage remains unknown. Totals use unrounded values.
Round
Harness
User interaction and review
Requirement roles
Developers
Development coverage
1
114.5
6.9
19.6
310.4
153/153
2
119.4
6.2
21.7
322.9
137/137
3
57.8
8.6
22.7
291.3
121/121
4
79.7
8.8
23.2
180.3
80/80
5
101.7
10.7
21.5
196.8
70/70
Total
473.1
41.2
108.7
1,301.7
561/561
Appendix
Table 13: Recorded token components of AHE-Beta by round, in millions, using the scopes of Table 12 . Development coverage counts selected completed calls with usage against all selected completed calls. Calls are assigned to rounds by completion time between consecutive frozen-round boundaries. Round 6 runs sessions only and is outside this accounting window.
Figure 4: Scenario coverage on the 38 held-out Flash tasks. N is the number of evolution seeds. Panels show (a) audited pass@1, (b) user rating from the scale floor of 1, and (c) execution tokens per task. Hatched bars mark AHE-Beta.
Figure 5: From user feedback to runtime changes. AHE-Beta’s pydicom DS evolution case. Each column shows the problem a simulated user met in one round, the change merged after it, and the effect measured in later rounds. Bars share one time scale from the arrival of a correction. Bottom: the first and final rounds.
Task
Codebase
Pairs
AHE-Beta
Base Harness
Tie
Neither
T1 Annotation
Matplotlib
9
6
2
1
0
T2 Staged report
Django
2
2
0
0
0
T3 Oversell
Shop service
13
5
1
6
1
T4 Game feature
Browser game
10
8
2
0
0
T5 Span selector
Matplotlib
7
5
1
1
0
T6 Literal annotation
Sphinx
7
5
1
1
0
Appendix
Table 14: Overall preference in the human A/B test by task. Pairs counts valid pairs. Without exposure removes the six valid pairs in which the Base Harness side read the upstream fix, and all submitted pairs adds back the two excluded pairs.
Figure 6: Change cards for the three changes aimed at pass@1. Each title gives the change, the RFCs merged for it as numbered in the harness repository, and the layer it edits. Files lists the main harness files those RFCs touch. What changed describes the new behavior. User problem gives the evolution sessions that motivated the change. Held-out evidence comes from the first main run on the 38 held-out tasks with GLM-5.3-Flash.
Figure 7: Change cards for the four changes aimed at user experience. Held-out evaluation lets the user simulator speak only after complete replies, so the evidence for changes that act during a turn also comes from evolution rounds and the human A/B test (Appendix F ).
Salesforce Research · University of North Carolina at Chapel Hill, conducted this work during an internship at Salesforce Research · University of Wisconsin–Madison +1