Agents driven by large language models are increasingly deployed on long-horizon tasks, where they use tools and modify real workspaces. How well they perform depends as much on the harness around the model as on the model itself. Existing benchmarks measure downstream task success under a fixed harness, leaving harness engineering itself unexplored. We propose Doc2Harness, a two-stage benchmark that evaluates long-horizon harness engineering directly. In Stage 1, a build model operating inside a fixed builder harness receives an interface-only contract skeleton and a development document that consolidates behavior across twelve frontier harnesses, and then constructs a complete candidate harness in an isolated container. In Stage 2, the frozen candidate harness runs in containers separate from the construction environment, one per test case, and is driven end to end by a scripted mock model. We score each test case using four observable channels, namely the requests sent to the model, the files on disk, the process exit code, and the command output; positive controls rule out vacuous passes. Across nineteen frontier models, each evaluated with three independent builds on 328 test cases, GPT-6 Astra achieves the highest pass rate at 53.05%, while the weakest model reaches 27.44%, a gap of roughly twenty-six percentage points. Even the strongest model fails nearly half of the test cases, and failures cluster around how cross-cutting functionality is split across modules. To our knowledge, Doc2Harness provides the first deterministic benchmark for measuring long-horizon harness engineering.
Figures & tables
Figure 2: Overview of Doc2Harness. Top (Stage 1): the builder, formed from the build model under evaluation and a fixed builder harness (OpenCode), takes the development document and the contract skeleton as input and emits a submission; the contract check verifies form rather than behavior, freezing the deliverable as the candidate harness on success and returning it to the builder on failure. Bottom (Stage 2): the frozen candidate harness and a mock model form the driven harness, which runs the scenario of each test case; effects are read from four channels, namely the requests sent to the model, the files on disk, the process exit code, and the command output, and the positive control gates the status before the assertion decides PASS or FAIL.
Content
Principal clause types
Deliverable and runtime interface
File structure, entry script
Built-in tools
Invocation form and boundary behavior of tools
Top-level commands and subcommands
Command syntax and exit codes
Permission modes, role modes, sandbox and lists
Five permission modes, five role modes, deny rules
Global flags
Semantics of flags, and their effect on requests
In-session commands and shortcuts
Command dispatch and terminal interaction
Table 1: Chapter structure of the development document.
Figure 3: The three test case synthesis methods.
Figure 4: File operations and permissions have the highest positive-control coverage, whereas the interactive terminal has the lowest. Artifact checks is marked NA, because the contract check already settles in full what its test cases examine, namely that the deliverable is complete.
Model
Perm.
File ops.
Session
Direct
Headless
Interactive
Other
All
61
75
84
33
40
20
15
328
GPT-6 Astra
71.6
46.7
44.0
39.4
45.0
91.7
60.0
53.05 ± 0.25
Opus 5
63.4
39.6
38.1
17.2
43.3
91.7
75.6
46.65 ± 1.80
Qwen3.8 Max
61.2
38.2
43.3
15.2
32.5
93.3
75.6
45.83 ± 0.14
GPT-5.6-sol
47.5
48.9
29.4
32.3
39.2
83.3
68.9
43.80 ± 2.44
Grok 4.7
54.6
50.7
28.2
22.2
33.3
78.3
75.6
43.50 ± 2.49
Table 2: Pass rate of the nineteen models (%), decomposed by evaluation category. The rightmost column is the primary measure: the pass rate over all 328 test cases, reported as the mean ± the standard deviation over the three scored candidate harnesses. “Other” merges artifact checks, startup, and extension discovery, which hold 7, 7, and 1 test cases respectively. Positive controls are excluded from every denominator. Abbreviations in the header are Perm. for permissions, File ops. for file operations, Session for session state, Direct for direct drive, and Interactive for interactive terminal.
Category
Cases
Avg. pass
Strong-weak
Floor
Ceiling
Discriminating
Session state
84
29.9
28.6
14
0
70
File operations
75
39.6
20.9
3
0
72
Permissions
61
43.7
50.8
10
0
51
Headless
40
26.4
41.7
9
0
31
Direct drive
33
18.8
29.3
17
0
16
Interactive terminal
20
79.9
33.3
0
7
13
Table 3: Strong-weak gap and test-case composition by evaluation category. The last three columns split each category’s test cases into floor test cases, which no candidate harness passes, ceiling test cases, which every candidate harness passes, and the remaining discriminating test cases that determine the ranking.
Model
Pass rate
Code lines
Modules
Lines/module
Largest module
GPT-6 Astra
53.05
7964
12.3
646
21%
Opus 5
46.65
13845
33.3
415
13%
Qwen3.8 Max
45.83
16681
40.0
417
10%
GPT-5.6-sol
43.80
2745
14.7
187
23%
Grok 4.7
43.50
5625
13.3
422
25%
GLM 5.3
42.07
8186
21.3
384
16%
Table 4: Architecture of the candidate harnesses. Every column is a mean over each model’s three scored builds, except Lines/module , which is mean code lines divided by mean module count, computed before rounding; Largest module is the share of all code lines held by the single largest module.
Figure 5: Code lines against pass rate. Left, one point per model at its mean over the three scored builds, with a least-squares fit and its 95% confidence band. Right, each model’s three scored builds, spanned by a segment with the mean marked, one row per model, ordered by pass rate.