A hosted score in the Gemma 4 Developer Agent competition is one noisy number, so development needs a local evaluator that is fast, safe for agent-written code, and matching the hosted scorer's prompts. drydock-eval evaluates a small coding agent (Gemma 4 E4B) at about 250 runs per hour on one RTX 5090. Contain runs each job in a fresh container without network and passed 200 scripted escape tests; conform provides eight checks that compare a local stack's requests with the scorer's configuration, down to rendered tokens; compare provides paired statistics that respect repository clusters. AI agents built and attacked the kit; a human owner approved machine use and release. This extended version repeats the short paper and adds the full record of the experiments made with and around the kit: 9,660 rows of agent runs, 2,494 grading controls and 72 serving measurements, all released. On four L4 GPUs the 31B model decodes 38.1 tokens per second in one stream and E4B 105.8. Six configurations of a 31B agent solved between 37 and 44 of 70 development tasks, and no pairwise difference was detected. A pre-registered five-arm experiment (231 tasks, 4,620 jobs) measured four prompt divergences: no E4B solve-rate change was detected (largest estimate -2.1 points, 95% interval -4.4 to +0.4). Three changes of the E4B agent led their screens by +1.0 to +6.2 points; each was then tested once on new tasks under a frozen pass rule, and none was confirmed (gains -0.35, -0.39 and +4.10 points). The unchanged baseline, run again on the same 128 tasks, moved by 1.17 points, and 65% of these tasks were solved in none of eight runs. Prompts and agent bundles are withheld; every run-level result is public.
Figures & tables
Figure 1: The kit: where each module sits.
Model
Context
Streams
Prefill tok/s
Decode tok/s per stream
Decode tok/s, all streams
31B
2,048
1
1,539
38.1
38
31B
2,048
2
1,545
34.4
66
31B
2,048
4
1,546
31.2
122
31B
2,048
8
1,550
17.9
78
31B
2,048
16
1,548
18.2
279
31B
8,192
1
1,479
36.3
36
Table 1: Serving throughput on four L4 GPUs (vLLM, one server per model). Context: prompt tokens already in the request. Streams: concurrent requests. Each request generates 160 tokens.
Figure 2: Decode throughput on four L4 GPUs with a context of 2,048 tokens. Left: tokens per second of one stream as the number of concurrent streams grows. Right: tokens per second summed over the streams.
Server
KV cache, tokens
2k context, 1 stream
8k, 1 stream
8k, 4 streams
no adapter (control)
46,352
38.3
36.4
29.4
rank 16, attention projections
37,328
33.9 ( − 12%)
32.4 ( − 11%)
26.9 ( − 9%)
rank 16, all linear layers
37,328
33.8 ( − 12%)
32.5 ( − 11%)
26.9 ( − 9%)
rank 8, two module types
37,568
33.3 ( − 13%)
32.1 ( − 12%)
26.7 ( − 9%)
Table 2: Cost of serving a LoRA adapter with the 31B model (same server settings; requests addressed to the adapter). Decode speed in tokens per second per stream; change against the control in brackets.
Runner version
Canaries passed
Security findings of independent verifiers
1. Container per command
52/52
4 high: host-side install ran the agent’s setup.py (builder-found); symlink leaked a host file; 10 GiB of host disk reserved in 0.005 s; host copied a sparse file in full
2. Hardened, per command
60/60 (57/60 before canary fixes)
4 high, 1 untested: host overwrote a file through a planted link; cleanup followed links; 6 GiB reserved in 0.012 s; 200,000 files in 2.4 s
3. Container per job
98/98
1 high: the model server could fetch URLs named in requests; 2 medium
4–6. Request filter added, tightened
134/134, 159/159, 191/191
medium at most
7. Final
200/200
no high or medium
Table 3: Runner versions.
Figure 3: Canaries passed and high-severity vectors found, per runner version.
Check
Compares
Mutation runs caught
role
role of tool results
53/53
sampling
temperature, top-p, token and thinking budgets
52/52
tools
declared tool schemas
20/20
shape
message skeleton, system text, field encoding
72/72
template
rendered prompt against the Kaggle template
13/21 by token count, 24/24 by token id
thoughts
history thoughts, under the key the server reads
52/52
Table 4: The eight checks.
Simulated scenario
Small panel
Large panel
No effect, exchangeable tasks
0.23–1.90%
0.00–0.50%
No mean effect, candidate breaks 20% of repositories
7.50%
8.43%
No effect, worst tasks dropped after seeing outcomes
31.75%
40.75%
Even gain of 3/58 (power)
≈ 60%
≈ 90%
Gain confined to four repositories (power)
15.9%
24.9%
Table 5: Simulated pass rates (3,000 simulations per cell; 400 for the dropping row).
Block
Arm
Tasks
Solved
%
95% interval
Wall
Out
Calls
T/O
Empty
Sum.
pub31-1
BF5
30
17
57
39–73
303
6,893
24.2
5
1
18
pub31-1
BF5-p
30
18
60
42–75
296
6,921
23.1
4
4
19
pub31-1
W1
30
20
67
49–81
254
5,942
22.4
3
4
0
pub31-2
W1-b
30
17
57
39–73
290
6,394
23.4
7
7
0
pub31-3
W1-p
30
15
50
33–67
304
6,412
25.2
11
12
0
pub31-4
W3
30
16
53
36–70
272
7,448
30.6
4
6
0
Table 6: The 31B arms on the development tasks, one run per task, with emulated L4 latency. Top: the first 30 tasks. Middle: 40 further tasks. Bottom: all 70. Wall: mean seconds per job. Out: mean generated tokens. T/O: runs stopped by the session time limit. Empty: runs that left no patch. Sum.: runs with at least one history summary.
Configuration
First 30
Next 40
All 70
%
95% interval
Only it
Only BF5
p
BF5
17
20
37
53
41–64
—
—
—
W1/W2
20
19
39
56
44–67
6
4
0.75
W3
16
22
38
54
43–65
6
5
1.00
SH1/SH3
20
21
41
59
47–69
8
4
0.39
SH5
17
24
41
59
47–69
7
3
0.34
SH6
21
23
44
63
51–73
10
3
0.09
Table 7: Six configurations on all 70 development tasks (one run per task). The last three columns compare each configuration with BF5 on the same tasks: tasks only it solved, tasks only BF5 solved, and the two-sided exact sign test.
Figure 4: Six configurations of the 31B agent on the same 70 development tasks, one run per task.
Figure 5: Which configuration solved which task. One column per development task, one row per configuration; a filled cell is a solved run. 25 tasks were solved by all six configurations and 20 by none.
Arm
Standard
Masked
Only masked
Only standard
BF5
17
19
2
0
SH5
17
17
2
2
Table 8: Runs with the test runners masked inside the sandbox, on the first 30 development tasks (solved of 30).
Arm
Tasks
Solved, L4
Solved, emul.
Both
Only L4
Only emul.
Wall, L4
Wall, emul.
Diff.
SH5
10
6
3
3
3
0
280
292
− 4%
SH6
24
14
17
14
0
3
289
314
− 8%
Table 9: The same arm and tasks on real 4 × L4 hardware (serial jobs) and under latency emulation (eight workers). Wall: mean seconds per job.
Arm
Solved
Agent
Setup
Verify
Tokens out
Tool calls
BF5
4/10
191
2.9
10.5
6,454
18.5
BF5-s
4/10
233
4.8
10.1
7,710
19.8
ID24
4/10
193
3.8
10.7
6,551
16.7
ID24-r
4/10
179
4.0
7.8
5,792
17.1
Table 10: Timing block: four 31B arms on ten Berth tasks, jobs run one after another on 4 × L4 with the software stack of the scorer. Times are mean seconds per job.
Date (UTC)
Arm
Model
Public score
Local result of the same arm
2026-11-05
W3
31B
0.12
38 of 70 development tasks (emulated latency)
2026-11-08
ID24
31B
0.17
4 of 10 Berth tasks (4 × L4)
2026-11-09
base
E4B
0.06
20.2% of 1,616 Berth runs (workstation)
2026-11-10
BF5
31B
0.18
37 of 70 development tasks (emulated latency)
2026-11-11
v-a3
E4B
0.05
118 of 512 Berth runs (panel H)
Table 11: Our five hosted submissions (one per day) and the local result of the same configuration. The public score is the solved share on the public part of the hidden test set, as displayed.
Figure 6: Our hosted scores, one run each, against the range of ten public draws of the unchanged BF5 files [ 7 ] .
Block
Arm
Tasks
Runs
Solved
%
95% interval
Time
Tool calls
pubE4-1
e-single
24
24
5
21
9–40
124
8.7
pubE4-1
e-wrap
24
24
5
21
9–40
108
10.3
pubE4-1
e-wrap-ad
24
24
5
21
9–40
118
10.5
berthE4-k1
e-BF
24
48
4
8
3–20
70
9.3
berthE4-k1
base
24
48
8
17
9–30
79
11.9
berthE4-k2
base
8
16
7
44
23–67
82
12.4
Table 12: First E4B experiments on hosted 4 × L4 notebooks. Time: mean seconds per job (whole job in the first block, agent phase in the others).
Tasks and machine
Reference patch passes
Empty patch passes
whole pool, workstation, no container
580/580
1/580
panels C and H, final runner
200/200
0/200
screen set S, final runner
48/48
0/48
case-study panel and two tasks of S, final runner
233/233
0/233
hosted notebook, first batch
38/38
0/38
hosted notebook, second batch
87/87
0/209
Table 13: Control runs of the grading path: every task graded once with its reference patch (must pass) and once with no patch (must fail).
Run
Solved
95% interval, %
Agent
hosted 4 × L4, repeat 1
4/24
7–36
79
hosted 4 × L4, repeat 2
4/24
7–36
80
workstation, no sandbox, 8 workers
3/24
4–31
65
workstation, no sandbox, 1 worker
5/24
9–40
63
workstation, container runner, version 1
3/24
4–31
69
workstation, container runner, version 3
2/24
2–26
58
Table 14: The baseline on the same 24 Berth tasks, one run per task, on the hosted notebook and on the workstation under successive runners. Agent: mean seconds of the agent phase.
Task set
Sessions
Solved runs
%
Range
Same
Never
Always
r
S (48 tasks, 2 repeats)
3
19, 21, 20
19.8, 21.9, 20.8
2.1
39
34
5
0.91
H (128 tasks, 4 repeats)
2
103, 97
20.1, 18.9
1.2
99
83
10
0.92
Table 15: The unchanged baseline run again on the same tasks in separate sessions on the workstation. Range: largest minus smallest session total in percentage points. Same: tasks with the same solved count in every session. Never, always: tasks solved in no run, in every run. r : correlation of the per-task counts of the first and last session.
Figure 7: The unchanged E4B baseline on the workstation. Left: solved runs of each session on the same tasks, with 95% intervals. Right: the 128 tasks of panel H by the number of baseline runs that solved them, of eight runs in two sessions.
Figure 8: Solved share of the E4B baseline by repository, over all its runs in the screen and confirmation blocks, with 95% intervals that treat runs as independent. Numbers of tasks and runs are in Table 16 .
Repository
Tasks
Runs
Solved
%
95% interval
executablebooks/markdown-it-py
6
36
19
52.8
37–68
Kludex/starlette
1
6
3
50.0
19–81
jd/tenacity
15
98
33
33.7
25–43
mahmoud/boltons
17
118
38
32.2
24–41
pypa/packaging
24
128
36
28.1
21–36
marshmallow-code/marshmallow
8
56
15
26.8
17–40
Table 16: Solved share of the E4B baseline by repository, over every baseline run of the screen and confirmation blocks on the workstation. The interval treats runs as independent and is too narrow where few tasks carry many runs.
Setting
Runs
Solved %
Mean
Median
Solved
Unsolved
Out
In (k)
Tools
T/O
Budget
Empty
31B, development tasks, emulated latency
570
57.2
278
255
231
341
7,089
240
20.6
12.6
0.4
13.5
31B, development tasks, 4 × L4
36
58.3
286
256
234
358
8,515
264
23.8
11.1
2.8
11.1
31B, Berth, 4 × L4
104
38.5
226
212
165
265
7,495
196
19.2
8.7
0.0
4.8
E4B, development tasks, 4 × L4
72
20.8
117
106
85
125
4,665
102
9.8
1.4
0.0
4.2
E4B, Berth, 4 × L4
160
20.0
73
61
43
80
6,261
110
10.6
0.0
1.9
11.9
E4B, Berth, workstation
4,082
21.3
75
60
45
84
8,146
156
14.2
0.3
1.2
5.5
Table 17: Time, tokens and endings of the agent runs by model and setting. Time: seconds per run (whole job for the development tasks, agent phase for Berth). In: prompt tokens summed over the requests of a run, in thousands. T/O, Budget, Empty: percentage of runs stopped by the time limit, stopped by the tool-call limit, and left without a patch.
Figure 9: Time of solved and unsolved runs for the two models. E4B: agent phase of the Berth runs on the workstation. 31B: whole job of the development-task runs under emulated latency.
Arm
Difference from the scorer path
Requests changed by the divergence
Change against F [95% interval]; Holm p
F
none: role tool , Kaggle template, compaction at 14,336 tokens, thoughts under both keys
—
solve rate 23.8%
R
model name contains “gemma4”: tool results get role tool_responses
89%
− 0.3 [ − 2.8, +2.2]; p = 1
C
compaction unset: prompts grow until rejected above 24,576 tokens (the run ends; edits are still graded)
21%
+0.2 [ − 2.0, +2.5]; p = 1
H
history thoughts removed from every request, as when a client omits the key the server reads
90%
− 2.1 [ − 4.4, +0.4]; p = 0.44
T
the Hugging Face E4B template: a thought opener is forced after each tool result
85%
+0.8 [ − 1.4, +2.9]; p = 1
Table 18: Arms; changes in percentage points.
Figure 10: (A) Solve-rate difference, local minus Kaggle (pooled 8/48), for eight local runs of one 24-task list (95% Newcombe intervals, runs treated as independent); shaded: the pass rule’s ± 3.45 points. (B) Change in solve rate against F on 231 tasks: estimates, 95% intervals and Holm-adjusted p.
Batch
Jobs started
Analysed runs
Interrupted, row written
Interrupted, no row
How the batch ended
1
967
959
0
8
runner reached its open-file limit
2
378
370
8
0
external termination signal
3
3,074
3,066
8
0
model server ended (out of memory)
4
201
201
0
0
job list completed
5
24
24
0
0
re-runs of the interrupted jobs completed
Table 19: Execution record: the block ran as four batches and one batch of re-runs. An interrupted job is an infrastructure row, with or without a written result row; its slot was re-run once.
Arm
Solved
Unsolved, run completed
Five-minute limit
Over-long prompt rejected
Tool-call budget (28)
No submission
Runs
F
220
676
17
0
6
5
924
R
217
677
20
0
9
1
924
C
222
568
17
114
1
2
924
H
201
597
106
1
18
1
924
T
227
668
19
0
4
6
924
Table 20: Outcome of every analysed run, by arm (one row per task, arm and repeat).
Arm
Solve rate (%)
95% interval
Solved runs
Repeat 1
2
3
4
Runs
F
23.81
[16.7, 30.9]
220
50
59
57
54
924
R
23.48
[17.7, 29.3]
217
53
53
54
57
924
C
24.03
[16.6, 31.5]
222
58
56
55
53
924
H
21.75
[15.9, 27.6]
201
50
49
54
48
924
T
24.57
[18.6, 30.5]
227
59
56
51
61
924
Table 21: Solve rate per arm: task-weighted mean with a 95% interval from a repository-clustered (CR1) standard error; solved runs per repeat.
Arm
0 solved
1 solved
2 solved
3 solved
4 solved
Tasks
F
149
16
21
18
27
231
R
152
14
21
15
29
231
C
151
15
18
17
30
231
H
155
21
13
14
28
231
T
151
14
13
25
28
231
Table 22: Tasks by the number of solved repeats (of four), per arm.
Arm
d
Exact d
Tasks lower; higher
Exact p
Holm p
Monte Carlo p
95% interval
98.75% simultaneous
R
−0.32
−1/308
26; 33
0.8652
1.000
0.8650
[ −2.82 , +2.17 ]
[ −3.51 , +2.85 ]
C
+0.22
1/462
26; 22
0.9253
1.000
0.9243
[ −2.03 , +2.48 ]
[ −2.65 , +3.09 ]
H
−2.06
−19/924
34; 20
0.1102
0.441
0.1106
[ −4.45 , +0.38 ]
[ −5.09 , +0.95 ]
T
+0.76
1/132
25; 30
0.5604
1.000
0.5599
[ −1.42 , +2.94 ]
[ −2.02 , +3.54 ]
Table 23: Primary analysis: change in solve rate against F in percentage points, tasks with fewer and with more solved repeats than in F, exact two-sided task-level sign-flip test, Holm adjustment over the four arms, and intervals from inverting the test.
Figure 11: Tasks whose number of solved repeats differs between an arm and F. Bars to the left of the centre line are tasks with fewer solved repeats in the arm.
Arm
Repositories that differ
Exact p
Webb p
95% interval
CR1 interval
Repository-equal
Leave-one-out range
R
12
0.8188
0.7780
[ −3.81 , +2.38 ]
[ −2.70 , +2.05 ]
+1.62
[ −1.09 , +0.36 ]
C
8
0.8438
0.7788
[ −0.88 , +2.81 ]
[ −1.22 , +1.65 ]
+4.02
[ −0.22 , +0.60 ]
H
12
0.1831
0.1736
[ −4.94 , +1.56 ]
[ −4.65 , +0.53 ]
−2.98
[ −3.01 , −1.44 ]
T
9
0.8047
0.7374
[ −1.44 , +4.62 ]
[ −2.09 , +3.61 ]
+2.65
[ −0.55 , +1.66 ]
Table 24: Secondary analysis at the repository level (15 repositories): exact wild cluster bootstrap- t with Rademacher weights, its inverted 95% interval, the six-point (Webb) variant, a CR1 interval, the repository-equal estimate and the range of leave-one-repository-out estimates. Estimates in percentage points.
Repository
Tasks
F solved runs
R
C
H
T
pypa/packaging
65
65 of 260
+0.50
−0.50
−2.00
−1.00
pydantic/pydantic-settings
48
27 of 192
+1.25
0
+0.75
+2.75
python-poetry/tomlkit
35
31 of 140
−1.00
−0.50
−1.50
0
more-itertools/more-itertools
23
39 of 92
−1.50
+0.25
−1.75
0
pallets/click
22
13 of 88
−0.50
−0.25
−0.50
−0.25
pallets/werkzeug
9
3 of 36
+0.25
0
−0.25
+0.25
Table 25: The 15 repositories of the panel: tasks, runs solved in arm F (of four per task) and, per arm, the summed change against F in task-equivalents (solved runs divided by four).
Figure 12: Change against F in each repository, per arm. Repositories are ordered by their number of tasks.
Arm
Analysis
Tasks
Runs affected
d
Exact p
R
five-minute-limit runs of both arms treated as missing
229
37
−0.29
0.8652
slots of the arm with an infrastructure row scored unsolved
231
5
−0.43
0.8011
slots of F with an infrastructure row scored unsolved
231
3
−0.22
0.9321
C
five-minute-limit runs of both arms treated as missing
229
34
+0.22
0.9253
slots of the arm with an infrastructure row scored unsolved
231
5
+0.11
1.0000
slots of F with an infrastructure row scored unsolved
231
3
+0.32
0.8506
Table 26: Pre-registered sensitivity analyses of the primary estimate (percentage points; exact two-sided sign-flip p , unadjusted).
Arm
Runs
Requests checked
Changed by the divergence
Share (%)
Failed calls recorded
F
924
15,043
0
0.0
17
R
924
15,290
13,616
89.1
20
C
924
13,036
2,709
20.8
131
H
924
17,450
15,684
89.9
112
T
924
14,787
12,629
85.4
19
Table 27: Exposure: requests re-rendered by the frozen checker, and those whose rendered prompt the arm’s divergence changed.
Measure
F mean
R − F
C − F
H − F
T − F
history summaries per run
0.69
+0.13
−0.69
−0.30
−0.02
[ +0.04 , +0.22 ]
[ −0.86 , −0.53 ]
[ −0.47 , −0.12 ]
[ −0.10 , +0.06 ]
runs with a summary (share)
0.482
+0.038
−0.482
−0.161
−0.014
[ −0.024 , +0.099 ]
[ −0.572 , −0.391 ]
[ −0.249 , −0.073 ]
[ −0.059 , +0.030 ]
largest prompt (tokens)
12,721
+220
+2,160
−1,499
−95
[ +111 , +329 ]
[ +1,666 , +2,655 ]
[ −1,903 , −1,095 ]
[ −284 , +95 ]
Table 28: Exploratory run measures: mean in arm F, and the paired difference of each arm against F; below it, its 95% interval (CR1, repository clusters). No multiplicity adjustment.
Figure 13: Agent time per run: each arm (blue) against F (grey).
Block
Arm
Tasks
Runs
Solved
%
Lead
Agent
pilot-2a
v-t1
15
15
4
26.7
—
98
pilot-2a
v-t2
15
15
5
33.3
—
71
pilot-2b
v-t3
15
15
3
20.0
—
96
pilot-2b
v-t4
15
15
7
46.7
—
89
pilot-2b
v-t5
15
15
6
40.0
—
77
pilot-2b
v-t6
15
15
6
40.0
—
66
Table 29: Pilot, development and screen blocks of the E4B candidates on the workstation. Lead: difference to the baseline of the same block in percentage points. Agent: mean seconds of the agent phase.
Figure 14: Gain of each repository in the three applications of the pass rule (candidate minus baseline). Repositories are sorted by their total within each application.
Application 1
Application 2
Application 3
Task set
small panel
large panel minus small
as application 2
Tasks; repositories; repeats
72; 18; 4
128; 19; 4
128; 19; 4
Runs per arm (all with a result row)
288
512
512
Solved runs: baseline, candidate
63, 62
103, 101
97, 118
Mean gain (points; exact fraction)
−0.35 ( −1/288 )
−0.39 ( −1/256 )
+4.10 ( 21/512 )
(1) gain at least 2/58
not met
not met
met
Table 30: Every application of the pass rule of Section 6 to a real candidate, from the frozen gate reports. Baseline and candidate are two versions of the withheld agent bundle.
Figure 15: Lead of each candidate in its screen on S against its gain in the confirmation run on other tasks, both in percentage points against the baseline of the same block. The horizontal line is the gain that the pass rule asks for.
Comparison of two arms
Runs per arm
SD, points
Time as run
Screen on S (48 tasks, 2 repeats)
96
2.9
1–2 hours (3–6 arms)
Confirmation on H (128 tasks, 4 repeats)
512
1.44
about 3 hours (2 arms)
Hosted, one draw per file (58 public tasks)
58
3.8
2 days
Table 31: Standard deviation of the difference of two arms that run on the same tasks, from run-to-run variation alone. The first two rows are estimated from the repeated baseline sessions ( Section 8 ); the third row uses the repeated public draws of unchanged 31B files [ 7 ] .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Arm
Model
Runs
What it is
BF5
31B
142
Single agent: the public Budget-Fit V5 configuration, run unchanged ( reference )
BF5-p
31B
30
BF5 with one changed section of the task prompt ( prompt )
BF5-c
31B
32
BF5 with a prompt-only checking step before the patch is submitted ( prompt )
BF5-s
31B
10
BF5 with changed sampling settings ( sampling )
W1
31B
30
Wrapped: a root agent calls a coder sub-agent as a tool; the coder has its own context ( architecture )
W2
31B
40
W1 with a bounded restart of the coder after a malformed tool call ( architecture )
Appendix
Table 32: Arms of the experiments. Labels are neutral; the category names what was changed against the arm it was built from. Prompt text and bundle files are not released.
Block
Model
Machine
Source
Tasks
Arms
Rep.
Runs
Note
pub31-1
31B
emulated
dev.
30
3
1
90
first 30 development tasks
pub31-2
31B
emulated
dev.
30
1
1
30
same 30 tasks
pub31-3
31B
emulated
dev.
30
1
1
30
same 30 tasks
pub31-4
31B
emulated
dev.
30
2
1
60
same 30 tasks, both arms in one queue
pub31-5
31B
emulated
dev.
40
2
1
80
40 further development tasks
pub31-6
31B
emulated
dev.
40
2
1
80
same 40 tasks, both arms in one queue
Appendix
Table 33: Every block of agent runs in the released file runs.csv . A block is one session on one machine; its arms ran in one job queue. Machine: one GPU with emulated L4 latency (emulated), a hosted notebook with four L4 GPUs (4 × L4), or the single-GPU workstation. Source: the development tasks of the competition (dev.) or Berth. Rep.: repeats per task and arm.
95% interval
Monte Carlo p
Arm
Registered
Kit, panel order
Kit command
Registered
Kit, panel order
Kit command
R
[ −2.82 , +2.17 ]
[ −2.82 , +2.17 ]
[ −2.80 , +2.16 ]
0.8650
0.8650
0.8642
C
[ −2.03 , +2.48 ]
[ −2.03 , +2.48 ]
[ −2.04 , +2.48 ]
0.9243
0.9243
0.9268
H
[ −4.45 , +0.38 ]
[ −4.45 , +0.38 ]
[ −4.45 , +0.25 ]
0.1106
0.1106
0.1095
T
[ −1.42 , +2.94 ]
[ −1.42 , +2.94 ]
[ −1.42 , +2.94 ]
0.5599
0.5599
0.5624
Appendix
Table 34: Reproduction with the released kit: the two quantities that depend on Monte Carlo draws. 95% intervals of the change against F in percentage points, and the Monte Carlo p that accompanies the exact p .
No.
Candidate
Tasks
Repos
Runs/arm
Base
Cand.
Gain
p
+/−
LOO
Time
Verdict
1
v-p2
72
18
288
63
62
− 0.35
0.6034
3/6
− 1.47
1.03
not confirmed
2
v-t6
128
19
512
103
101
− 0.39
0.6034
7/6
− 0.85
1.18
not confirmed
3
v-a3
128
19
512
97
118
+4.10
0.0405
9/6
+2.94
1.10
not confirmed
Appendix
Table 35: The three applications of the pass rule (four repeats per task and arm). Gain: candidate minus baseline in percentage points. p : one-sided, repository-clustered. +/− : repositories with a positive or negative summed gain. LOO: lowest gain when one repository is left out. Time: ratio of mean agent time, candidate to baseline.
File
Records
Content
runs.csv
5,024
one row per agent run of Sections 7 , 8 and 10
controls.csv
2,494
one row per control grading (reference patch or empty patch)
serving_bench.csv
72
one row per cell of the serving benchmark of Section 3
serving_servers.csv
7
start-up time and cache size of each model server of the benchmark
experiments.csv
31
one row per block ( Table 33 )
arms.csv
37
one row per arm ( Table 32 )
Appendix
Table 36: Data files released with this paper, among the ancillary files of its arXiv submission, in the directory data/ .
Column
Content
batch
1 to 5: the batch of the block in which the job ran; batch 5 holds the re-runs
stage
1, 2 or 3: the nested stage of the job list that holds the task (118, 37 and 76 tasks)
arm
F, R, C, H or T
task
task identifier in the public task collection the panel was drawn from
repository
repository of the task (owner/name); the unit of clustering
repeat
1 to 4
Appendix
Table 37: Columns of e8_runs.csv . The rows of interrupted jobs hold the first ten columns and the two times of day only.
Coding agents are beginning to develop the production-grade AI infrastructure that trains and serves large models, yet existing benchmarks cover only single kernels or single-GPU tasks, and grade with tests and prompts that misjudge both correct and incorrect patches. We introduce InfraDev, a benchmark of 36 tasks built from real-world pull requests to Megatron-LM, vLLM, and verl, covering infrastructure development for training, inference, and reinforcement learning. Each task asks an agent to reimplement a real change that spans many files and must run correctly across many GPUs. To grade such changes, we jointly synthesize hidden probes, which enforce the requirements the original tests leave unchecked, and a prompt that states every interface the probes call without revealing the solution. We also design a shared execution environment that reuses dependencies and artifacts across tasks, cutting storage by 52-57% and build time by 13-14% on Megatron-LM and vLLM. Across six frontier models, the best resolves only 38.9% of the tasks; the probes reject 32% of the submissions that pass every official test, and with the original pull request descriptions as prompts, no model resolves a single task. Failure analysis shows that structural and state errors dominate and that agents' own tests rarely check memory, time, or traffic. Turning these findings into an agent skill raises the resolved rate of a frontier model on the Megatron-LM tasks by 10.5%.
Large language model (LLM) agents are rapidly reshaping software engineering, accompanied by an explosion of new code benchmarks. Yet nearly all existing benchmarks still rely on the same decades-old criterion: a solution is correct if it passes a fixed set of unit tests. Such tests are often insufficient: they check only part of what the task requires, so agents can reward hack them or silently miss required behavior while still passing every test. As a result, higher benchmark scores may partly reflect better adaptation to the evaluator rather than better problem solving. Existing works focus on static test augmentation: they strengthen each task's tests once, before any trial is seen, and thus overlook how real trials actually fail. We introduce TestJack, a scalable framework for evaluating patches beyond fixed tests. For each trial, TestJack generates tests targeting prompt requirements the patch may violate, retains only tests passed by the ground-truth patch, and re-examines any trial failures. Each confirmed failure is thus supported by a replayable test. To reduce evaluation cost, we also introduce a lightweight variant which audits a random sample of trials in depth and reuses the resulting tests across all trials for the same task. Across 6 frontier model backends and 5 benchmarks such as DeepSWE and SWE Marathon, we find that about 34.4% of the model trials currently judged correct violate the task requirements, lowering the overall resolution rate from 50.6% to 33.2%. Our results reveal a fundamental limitation of current coding-agent evaluation: as LLMs become better at optimizing against fixed evaluators, those evaluators themselves must become more adaptive.
Shuangjie Yao, Hao Wang, Koushik Sen +3
Columbia University · University of California, Berkeley · George Mason University
A coding agent combines a model with a harness, which decides what the model sees, which tools it can use, and how the work continues. We ask whether changing the harness changes the result when the model and task stay fixed. We compare two configurations of the same harness on three coding benchmarks. The control supplies the full conversation in time order, while the treatment keeps the same record but mechanically shortens older tool results as the context fills and responds to repeated or stalled work. Under tight context, the treatment raises mean per-task fail-to-pass fraction (F2PF) in all three pressure comparisons and increases complete solutions on SWE-bench Verified and SWE-bench Pro. The tight-window Verified comparison uses 169 tasks, a 20,480-token window, and a fixed 480-second attempt endpoint; on this cohort, treatment raises mean per-task F2PF from 28 percent to 49 percent and complete solutions from 43 to 72. Without model-specific retuning, the same frozen treatment also raises both endpoints on the same cohort for three additional models with different designs. In the wide-window Qwen3.6 comparisons, observed arm outcomes are close on Verified and Pro, while FeatureBench retains a higher mean per-task F2PF under treatment. On the wide-window Verified cohort, treatment also serves fewer prompt tokens per turn. Because changing the harness changed what unchanged model weights could accomplish, coding-agent evaluations should treat the model and harness together as the tested solver.