Organizations: University of Washington · Allen Institute for AI · University of Pennsylvania · Pyromind Dynamics · Stanford University · RadixArk · The University of Texas at Austin · Northwestern University
In this work, we study whether frontier agents can continually improve real-world AI infrastructure over long horizons. We introduce Infrathon, a benchmark of diverse, long-horizon AI infrastructure optimization tasks across different levels. We evaluate six models across 150 runs with 24-hour budgets, totaling over 10,000 GPU-hours. Our results show that model rankings vary with the evaluation horizon, while 135 of the 150 runs achieve further improvements during the second half of their budgets. These findings indicate that shorter evaluations can overlook both optimization gains and changes in relative model performance. We also find agents can discover solutions that outperform human-designed baselines: Claude Opus 5.5 increases the mean KV-cache compression ratio from 6.82× with the human-designed Quant-VideoGen codec to 19.86×. Analysis of agent trajectories identifies task violations, unreliable testing, and premature completion as barriers to sustained improvement. Together, our Infrathon framework provides resources for constructing long-horizon tasks, evaluating frontier agents, and discovering good infrastructure solutions. Code is available at https://github.com/InfrathonX/Infrathon
Figures & tables
Figure 1: An optimization example with Infrathon. On Quant-VideoGen KV Codec, Claude Opus 5.5 develops a codec that reduces memory use during video generation, with higher KV-cache compression ratios and better video quality than the human codec. Top: two video examples; compression ratios are red. Bottom: Claude’s scores over time combine compression ratio and video quality; dashed line: Human Solution. Numbered points mark five selected submissions; boxes summarize their code changes (Appendix E.3 ).
Benchmark
Infra only
Long horizon
GPU tasks
Operator level
Module level
System level
KernelBench [ 6 ]
KernelBenchX [ 17 ]
FlashInfer-Bench [ 18 ]
RE-Bench [ 19 ]
InferenceBench [ 7 ]
Φ -Bench [ 8 ]
Table 1: Coverage of related benchmarks. Infra only : exclusively AI infrastructure; Long horizon : support for continuous execution of every task for at least 12 hours; GPU tasks : at least one GPU task in the open-source release. Operator , Module , and System denote computational primitives, components, and complete workflows. and indicate whether coverage is documented (Appendices A.2 and D ).
Figure 2: Framework of Infrathon . (a) Task Definition specifies task requirements. (b) Task Execution coordinates iterative development and evaluation through the Work, Controller, and Judge environments.
Table 2: Task example: KVzip KV Scoring. The specification includes the workspace, public and private tests, GPU resources, and time budget. Private tests use disjoint samples and include additional task types.
Figure 4: Performance on 12 selected tasks.
Figure 5: Rank-based final performance over all 25 tasks. Markers show mean rank scores (higher is better). Error bars show ±1 standard error of the mean across tasks in each group.
Video model
Dataset
Compression ( × ) ↑
PSNR (dB) ↑
SSIM ↑
LPIPS ↓
HY-WorldPlay
WorldScore
6.75→20.89
31.15→36.33
0.923→0.975
0.059→0.023
LongCat-Video
MovieGen
6.88→18.83
23.67→26.95
0.836→0.884
0.126→0.085
Table 3: Human versus Claude Opus 5.5. Entries report Human → Claude; bold marks the better mean. PSNR, SSIM, and LPIPS are measured on the first segment generated with compressed KV (Appendix E ).
Figure 6: Iterative optimization of Quant-VideoGen KV Codec. The figure compares Claude Opus 5.5’s 6th, 7th, 16th, and 22nd submissions with the BF16 reference and the human codec on two LongCat-Video samples. Parentheses show KV-cache compression ratios.
Figure 7: Model rankings, second-half gains, and token and action counts. (a) Model rankings as the time budget increases. (b) Agents’ scores at half of the budget relative to their final scores. (c) Budget fraction agents need to reach 90% of their final scores. (d) Each model’s visible output tokens (top) and actions (bottom).
Figure 8: Long horizon vs. short horizon at equal GPU-hours. Curves take the highest private test score across each setting’s one, two, or four runs up to the same elapsed time. The first half of the budget occupies 75% of each horizontal axis and the shaded second half 25%; ticks show actual hours.
Figure 9: Public-test errors on Attn-QAT NVFP4 Attention. Points show the maximum error-to-tolerance ratio across outputs and gradients for each case.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Complete optimization trajectories for training tasks and inference operators. Panels (a)–(p) cover the first 16 tasks in Appendix A.3 : all 10 training tasks and the first six inference operators. Curves show performance over time.
Figure 11: Complete optimization trajectories for the remaining inference tasks. Panels (q)–(y) continue the task order in Appendix A.3 with two operators, three modules, and four systems. Curves show performance over time.
Figure 12: Visible output tokens and actions for every task–model run. Rows follow the benchmark task order. Cell values show (a) visible output tokens in thousands and (b) agent actions. Color intensity is logarithmic within each panel.
Figure 13: Individual operator trajectories with GPT-6 Astra. Flux/Comet MoE Primitive and LMCache KV Block Transfer each use one 12-hour run, two 6-hour runs, and four 3-hour runs. Curves show performance over time. The 6–12-hour interval is compressed threefold; ticks show actual hours.
Figure 14: Individual module trajectories with GPT-6 Astra. Qwen3 ORPO and KVzip KV Scoring each use one 24-hour run, two 12-hour runs, and four 6-hour runs. Curves show performance over time. The 12–24-hour interval is compressed threefold; ticks show actual hours.
Figure 15: Individual system trajectories with GPT-6 Astra. Miles SD3.5-M Multi-teacher OPD and LLaDA Decoding each use one 24-hour run, two 12-hour runs, and four 6-hour runs. Curves show performance over time. The 12–24-hour interval is compressed threefold; ticks show actual hours.
Figure 16: Individual operator trajectories with Claude Opus 5.5. Flux/Comet MoE Primitive and LMCache KV Block Transfer each use one 12-hour run, two 6-hour runs, and four 3-hour runs. Curves show performance over time. The 6–12-hour interval is compressed threefold; ticks show actual hours.
Figure 17: Individual module trajectories with Claude Opus 5.5. Qwen3 ORPO and KVzip KV Scoring each use one 24-hour run, two 12-hour runs, and four 6-hour runs. Curves show performance over time. The 12–24-hour interval is compressed threefold; ticks show actual hours.
Figure 18: Individual system trajectories with Claude Opus 5.5. Miles SD3.5-M Multi-teacher OPD and LLaDA Decoding each use one 24-hour run, two 12-hour runs, and four 6-hour runs. Curves show performance over time. The 12–24-hour interval is compressed threefold; ticks show actual hours.
Setting
HY-WorldPlay
LongCat-Video
Dataset
WorldScore [ 52 ]
MovieGen [ 54 ]
Input
Image, prompt, and camera actions
Prompt and fixed 73-frame prefix
Resolution; seed
832×480 ; 0
832×480 ; 0
Output frames
285
273: 73 fixed and ten 20-frame segments
Frame rate
24 fps
15 fps
Denoising
Four steps
50 steps; guidance scale 4.0
Appendix
Table 4: Video generation and evaluation settings. Frame indices start at zero.
Component
Human
Claude Opus 5.5
K/V clustering
Separate assignments
Shared assignments; separate when 2D>256
Cluster count
256
Adaptive (Equation 12 )
Lloyd iterations
Two for HY-WorldPlay; up to 100 for LongCat-Video
Ten
Residual transform
None
PCA, retaining all dimensions
Quantization
Ternary values; one FP8 scale per 64 elements
Eight-state trellis quantization
Residual storage
Two bits per symbol
rANS entropy coding
Appendix
Table 5: Codec designs for Human and Claude Opus 5.5.
Point
Submission
Optimization
Score
1
6th
4-bit quantization. Claude tests a simple 4-bit codec and obtains its first valid private test score.
5.80
2
7th
PCA + entropy coding. Claude returns to clustering and PCA residual quantization with rANS entropy coding. Video tests in the Work environment guide changes to the quantization step and encoded symbol set.
14.07
3
8th
Quantization-step tuning. Claude uses video tests in the Work environment to select larger quantization steps for K and V, increasing compression while passing private tests.
17.66
4
16th
Trellis quantization. Claude jointly quantizes PCA coefficients and uses finer steps for high-variance key directions.
21.90
5
22nd
Metadata compression + per-head quantization. Claude losslessly compresses metadata and shares K/V cluster assignments. It also refines clustering and uses finer quantization steps for attention heads with larger residual magnitudes.
28.45
Appendix
Table 6: Claude’s optimization steps on Quant-VideoGen KV Codec. Point numbers match Figure 1 .
Metric
What it measures
Computation or model
Compression ↑
KV storage savings
Ratio of original to stored bytes (Equation 14 )
PSNR ↑
Pixel reconstruction accuracy
Mean squared error, expressed in dB (Equation 15 )
SSIM ↑
Local luminance, contrast, and structural similarity
Uniform 11×11 windows on each RGB channel (Equation 16 )
LPIPS ↓
Perceptual difference from the BF16 reference
Weighted VGG feature distances; LPIPS version 0.1
Appendix
Table 7: Metrics for the codec comparison. Arrows indicate the preferred direction.
Quantity
LongCat-Video
HY-WorldPlay
PSNRHuman (dB)
23.6403
31.0633
SSIMHuman
0.8704
0.9483
LPIPSHuman
0.0964
0.0435
GHuman
0.4060
0.3939
Δ
(1.873, 2.685, 6.174, 4.196)
(2.055, 1.511, 2.415, 1.984)
Appendix
Table 8: Human codec values and quality thresholds. VBench tuples follow aesthetic quality, background consistency, imaging quality, and subject consistency.
Efficiency
Frame similarity
Full-video VBench ↑
Video model
Dataset
Codec
Comp. ↑
Time ↓
PSNR ↑
SSIM ↑
LPIPS ↓
Aes.
Bg.
Img.
Subj.
HY-WorldPlay
WorldScore
Human
6.75
305
31.15
0.923
0.059
59.8
93.2
69.9
88.2
Claude
20.89
262
36.33
0.975
0.023
61.8
94.0
70.9
89.6
LongCat-Video
MovieGen
Human
6.88
1734
23.67
0.836
0.126
63.8
93.0
65.8
89.1
Claude
18.83
1760
26.95
0.884
0.085
66.0
94.2
68.4
91.2
Appendix
Table 9: Human and Claude on WorldScore and MovieGen. Entries are dataset means. Time is in seconds; MovieGen timing uses 188 matched pairs. VBench [ 85 ] columns report aesthetic quality, background consistency, imaging quality, and subject consistency.
Measurement
WorldScore
MovieGen
Higher compression
200/200
200/200
Higher PSNR
200/200
170/200
Higher SSIM
200/200
169/200
Lower LPIPS
200/200
176/200
Higher VBench scores (four dimensions)
615/800
659/800
Mean total encoding time (s)
67.3→41.4
103.2→49.5
Appendix
Table 10: Per-sample improvements and codec timing. Counts compare Claude with the human codec. Times show Human → Claude in seconds per video.
Managing modern computing infrastructure has become a steadily harder problem due to the ever-increasing complexity. Recent advances in AI agents create a timely opportunity to automate infrastructure management tasks, but it remains unclear how well such agents can handle real-world infrastructure complexity. We present InfraBench, a benchmark for infrastructure agents organized around system layers, the operational lifecycle, and risk. The current prototype couples container, VM, and bare-metal backends with task-specific verifiers that distinguish functional repair from durability and cleanup, and analyzes retained action-trace findings for unsafe shortcuts. We evaluate 18 agent-model configurations on 12 seed tasks, with three attempts per task. Mean effective scores range from 45.5% to 89.2%; the strongest configuration completes 77.8% of attempts perfectly. In the available lifecycle checks, pass rates fall from 91.5% for functional repair to 80.9% for durability and 38.9% for cleanup. These results show why immediate recovery is an incomplete measure of operational capability. We use this evidence to motivate a planned expansion toward 40-60 tasks centered on managed cloud infrastructure and redesign after incidents, including provider-native Kubernetes networking. This extension is a research direction; the reported results remain those of the 12-task prototype. InfraBench, its tasks, evaluation harness, and live leaderboard are available at infraben.ch.
Yuan Gao, Zeren Yang, Junnan Li +6
Wanxiang · University of Wisconsin–Madison · Iowa State University
Improving the efficiency of training and inference infrastructure is a practical component of recursive self-improvement. Existing benchmarks for agent-driven optimization largely focus on individual software layers such as kernel, leaving unclear whether agents can deliver end-to-end performance gains under production workloads. We introduce RSIBench-Infra, a benchmark of 20 training and inference tasks spanning diverse model families and modalities, with deployments of up to 32 accelerators on NVIDIA GPUs and Ascend NPUs. Agents start from runnable baselines derived from official recipes and can modify the software stack across engine, kernel, and instruction levels. An OCI image-layer submission protocol captures these changes for reproducible deployment, while independent evaluation measures end-to-end performance under task-specific quality constraints. Across all task results, 66.2% exceed 1.10× validated speedup. These results demonstrate that current agents can improve production-scale training and inference performance, but identifying effective optimization strategies and consistently translating local improvements into end-to-end gains remain challenging.
Hanlin Wu, Yifei Wang, Zhicheng Cai +5
ByteDance Seed · Institute for AI Industry Research (AIR), Tsinghua University · SIA-Lab of Tsinghua AIR and ByteDance Seed
Coding agents are beginning to develop the production-grade AI infrastructure that trains and serves large models, yet existing benchmarks cover only single kernels or single-GPU tasks, and grade with tests and prompts that misjudge both correct and incorrect patches. We introduce InfraDev, a benchmark of 36 tasks built from real-world pull requests to Megatron-LM, vLLM, and verl, covering infrastructure development for training, inference, and reinforcement learning. Each task asks an agent to reimplement a real change that spans many files and must run correctly across many GPUs. To grade such changes, we jointly synthesize hidden probes, which enforce the requirements the original tests leave unchecked, and a prompt that states every interface the probes call without revealing the solution. We also design a shared execution environment that reuses dependencies and artifacts across tasks, cutting storage by 52-57% and build time by 13-14% on Megatron-LM and vLLM. Across six frontier models, the best resolves only 38.9% of the tasks; the probes reject 32% of the submissions that pass every official test, and with the original pull request descriptions as prompts, no model resolves a single task. Failure analysis shows that structural and state errors dominate and that agents' own tests rarely check memory, time, or traffic. Turning these findings into an agent skill raises the resolved rate of a frontier model on the Megatron-LM tasks by 10.5%.