cs.AINov 9, 2026

Infrathon: Benchmarking Language Agents for Real-World, Long-Horizon AI Infra Optimization

Authors: Zhengyu Hu, Zheyuan Xiao, Yike Wang, Hamish Ivison, Haoyu Wang, Yige Yuan, Rulin Shao, Yuetai Li, 

Organizations: University of Washington · Allen Institute for AI · University of Pennsylvania · Pyromind Dynamics · 

Abstract

In this work, we study whether frontier agents can continually improve real-world AI infrastructure over long horizons. We introduce Infrathon, a benchmark of diverse, long-horizon AI infrastructure optimization tasks across different levels. We evaluate six models across 150 runs with 24-hour budgets, totaling over 10,000 GPU-hours. Our results show that model rankings vary with the evaluation horizon, while 135 of the 150 runs achieve further improvements during the second half of their budgets. These findings indicate that shorter evaluations can overlook both optimization gains and changes in relative model performance. We also find agents can discover solutions that outperform human-designed baselines: Claude Opus 5.5 increases the mean KV-cache compression ratio from 6.82×6.82\times with the human-designed Quant-VideoGen codec to 19.86×19.86\times. Analysis of agent trajectories identifies task violations, unreliable testing, and premature completion as barriers to sustained improvement. Together, our Infrathon framework provides resources for constructing long-horizon tasks, evaluating frontier agents, and discovering good infrastructure solutions. Code is available at https://github.com/InfrathonX/Infrathon

Figures & tables

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. InfraBench: Evaluating Infrastructure Agents Across Layers, Lifecycle, and Risk

    Jul 31, 2026Yuan Gao, Zeren Yang, Junnan Li +6AI Agent ReliabilityAI Agent Evaluation

  2. RSIBench-Infra: Can Agents Optimize Their Own Training and Inference Infrastructure at Production Scale?

    Nov 18, 2026Hanlin Wu, Yifei Wang, Zhicheng Cai +5AI Agent BenchmarksEfficient Neural Network Inference

  3. InfraDev: Can Coding Agents Develop AI Infrastructure, and How Can We Tell?

    Oct 27, 2026Yangshen Deng, Jiaqi Ruan, Yeqi Huang +3AI Coding AgentsCoding Agent Benchmarks