Coding agents are beginning to develop the production-grade AI infrastructure that trains and serves large models, yet existing benchmarks cover only single kernels or single-GPU tasks, and grade with tests and prompts that misjudge both correct and incorrect patches. We introduce InfraDev, a benchmark of 36 tasks built from real-world pull requests to Megatron-LM, vLLM, and verl, covering infrastructure development for training, inference, and reinforcement learning. Each task asks an agent to reimplement a real change that spans many files and must run correctly across many GPUs. To grade such changes, we jointly synthesize hidden probes, which enforce the requirements the original tests leave unchecked, and a prompt that states every interface the probes call without revealing the solution. We also design a shared execution environment that reuses dependencies and artifacts across tasks, cutting storage by 52-57% and build time by 13-14% on Megatron-LM and vLLM. Across six frontier models, the best resolves only 38.9% of the tasks; the probes reject 32% of the submissions that pass every official test, and with the original pull request descriptions as prompts, no model resolves a single task. Failure analysis shows that structural and state errors dominate and that agents' own tests rarely check memory, time, or traffic. Turning these findings into an agent skill raises the resolved rate of a frontier model on the Megatron-LM tasks by 10.5%.
Figures & tables
Figure 1: Overview of InfraDev.
Repository
New feature
Performance
Refactor
Bug fix
Total
Megatron-LM (training)
9
8
1
1
19
verl (post-training)
5
2
0
0
7
vLLM (inference)
5
5
0
0
10
Total
19
15
1
1
36
Table 1: Task types of the 36 selected pull requests. New feature : adds functionality that the pre-PR snapshot lacks; Performance : keeps the functional result but reduces time, memory, or communication; Refactor : restructures code without adding functionality; Bug fix : corrects wrong behavior.
Model
% Resolved
% Passed
Time (min)
Cost ($)
Tokens (M)
GPT-6 Sol
38.9
78.8
43.9
5.37
20.86
GPT-5.6 Sol 2 2 2 GPT-5.6 Sol was measured through the Codex coding plan between September 21 and 24, 2026; we observed possible degradation of the served model around these dates.
30.6
78.6
73.9
17.98
24.71
GLM 5.3
25.0
76.7
136.9
13.89
48.84
HY4 preview
36.1
83.3
118.1
2.09
31.75
DeepSeek V4 Pro
30.6
74.3
94.1
1.89
41.80
Kimi K3
30.6
75.8
136.4
13.48
33.58
Table 2: Results of 6 models on InfraDev. % Resolved is the share of tasks on which every test passes; % Passed is the share of tests passed on a task, averaged over tasks. Time, Cost, and Tokens are per-task means of the agent’s run time, API cost estimated at list prices, and tokens consumed.
Model
Training (19)
Inference (10)
RL (7)
GPT-6 Sol
52.6
10.0
42.9
GPT-5.6 Sol
42.1
10.0
28.6
GLM 5.3
26.3
10.0
42.9
HY4 preview
42.1
10.0
57.1
DeepSeek V4 Pro
42.1
10.0
28.6
Kimi K3
31.6
10.0
57.1
Table 3: % Resolved per task category. The number of tasks is in parentheses.
Figure 5
Figure 4: (a) Share of the graded submissions with at least one error of each class. (b)–(d) Error types within each class; counts are submissions with at least one failing case of the type, and a submission can count under several types.
Figure 7
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Official tests
Official + probes
Megatron-LM: 19 tasks
GPT-6 Sol
68.4
52.6
GPT-5.6 Sol
52.6
42.1
GLM 5.3
36.8
26.3
HY4 preview
52.6
42.1
DeepSeek V4 Pro
52.6
42.1
Appendix
Table 4: Resolved rate (%) under the official tests alone and under the official tests plus the supplementary probes. A task is resolved when every test in the suite passes.
Model
% Resolved
% Passed
Time (min)
Steps
Cost ($)
Tokens (M)
Megatron-LM: 19 tasks
GPT-6 Sol
52.6
80.6
50.4
227.6
6.66
27.00
GPT-5.6 Sol
42.1
80.1
71.6
163.4
16.90
22.67
GLM 5.3
26.3
84.4
149.5
203.2
12.79
45.44
HY4 preview
42.1
91.0
137.8
199.5
2.14
32.35
DeepSeek V4 Pro
42.1
78.0
103.2
186.6
2.23
42.66
Appendix
Table 5: Table 2 per repository. Steps is the mean number of completed model turns per task.
Model
Input (M)
Cache read (M)
Output (M)
Reasoning (M)
Megatron-LM
GPT-6 Sol
0.336
26.600
0.037
0.030
GPT-5.6 Sol
0.768
21.832
0.042
0.029
GLM 5.3
0.485
44.813
0.055
0.086
HY4 preview
0.622
31.608
0.046
0.072
DeepSeek V4 Pro
1.580
40.930
0.061
0.087
Appendix
Table 6: Mean parent-session tokens in millions. Input excludes cached tokens. The four components sum to the token totals in Table 5 before rounding and average the same runs.
Class
Error type
Megatron-LM
verl
vLLM
All
Structural
Execution / API
19
10
19
48
Structural
Shape / layout / dtype
8
4
10
22
Structural
State / lifecycle
10
8
17
35
Structural
Any structural error
30
18
37
85
Behavioral
Resource (memory, speed, traffic)
14
1
2
17
Behavioral
Condition-specific behavior
9
0
26
35
Appendix
Table 7: Submissions exhibiting each error type, counted once per submission. A submission may exhibit several types. An exception or collection failure is structural; otherwise a failure is labelled by the property its assertion checks.