Auto-Research Arena: Benchmarking and Advancing Auto-Research in Language Modeling

Xi Chen; Chongjian Ge; Yiran Xu; Yiqun Mei; Henrik I. Christensen; Hao Tan; Ziwen Chen; Hanwen Jiang

cs.LGOct 26, 2026

Auto-Research Arena: Benchmarking and Advancing Auto-Research in Language Modeling

Authors: Xi Chen, Chongjian Ge, Yiran Xu, Yiqun Mei, Henrik I. Christensen, Hao Tan, Ziwen Chen, Hanwen Jiang

Organizations: University of California San Diego · Adobe Research

Abstract

As autonomous research systems become more capable, benchmarks must both evaluate their performance and support meaningful research progress. We introduce Auto-Research Arena, a multi-target benchmark for evaluating autonomous research systems and discovering ideas in language modeling. Arena supports diverse research objectives within a shared experimental environment by separating admissibility from improvement: model quality and resource constraints determine whether a result qualifies, while the selected target measures progress among admissible results. We instantiate Arena in language model pretraining with nine targets spanning model, data, training, and inference efficiency, and evaluate seven research harnesses using two foundation models. Arena clearly differentiates existing harnesses across research objectives, with several sophisticated methods outperformed by our simple sequential search baseline, indicating substantial headroom in current auto-research systems. Beyond evaluation, Arena reveals a substantially broader frontier of model and system ideas aligned with active research directions. Reusing these discoveries in a new quality-driven search yields lower training loss under matched data and FLOPs and lower held-out BPB at the same model size. Together, Arena provides a common framework for benchmarking autonomous research and expanding its scope in language modeling. The code is available at https://github.com/Auto-Research-Arena/auto-research-arena

Figures & tables

Figure 1: Auto-Research Arena evaluates research systems and expands the idea frontier. Left: Arena differentiates research harnesses across nine complementary objectives. Our simple Sequential Search baseline remains competitive, whereas the TPE method tailored to validation-loss minimization underperforms in this broader evaluation. Right: Multi-target research reveals a broad frontier of ideas. A program combining these discoveries achieves lower training loss than the reference program under matched data and FLOP budgets, as well as lower final held-out BPB at the same model size.

Figure 2: Overview of Auto-Research Arena Pipeline. Each search run optimizes one of nine efficiency targets. A research harness proposes and implements changes to a reference program. Arena evaluates the resulting candidates, records outcomes, and provides feedback. The best admissible candidate determines the benchmark result.

	$\displaystyle\underset{p}{\operatorname{minimize}}$	$\displaystyle N_{\mathrm{param}}(p)$
	$\displaystyle\text{subject to}$	$\displaystyle\operatorname{BPB}(p)<1.05,$
		$\displaystyle\operatorname{FLOPs}(p)\leq 239.1\,\text{M/token},$
		$\displaystyle\operatorname{Mem}_{\mathrm{train}}(p)\leq 44.0\,\text{GiB}.$

Table 3

Target	Measured quantity	Why it matters
Model
Total parameters	Number of model parameters	Model storage and weight-memory footprint ( Lan et al., 2019 ) .
Training
Data consumption	Tokens used to reach target quality	Learning efficiency under a token budget ( Hoffmann et al., 2022 ) .
Optimizer updates	Updates to reach the quality threshold	Convergence in fewer updates ( Gupta et al., 2018 ) .
Training FLOPs	FLOPs per training token	Compute demand per token ( Hoffmann et al., 2022 ) .

Table 1: Efficiency targets in Auto-Research Arena. Each track minimizes one target subject to a shared model-quality threshold and target-specific resource constraints.

Figure 3: Research trajectories behind the Opus 5 benchmark results. Curves track the best admissible result found as each search proceeds, ordered by charged experiments; lower is better. Counts include the reference evaluation and charged failures, and dots mark the end of each recorded search. Training time is measured at 30-second validation checkpoints, and KV-cache allocation is shown on a logarithmic scale.

Discovered idea: Loop Transformer with depth-specific residual gain Optimization target: Parameter efficiency

BEFORE: INDEPENDENT BLOCKS

\begin{aligned} &\mathbf{for}\ \ell=0,\ldots,7:\\ &\quad u\leftarrow h+\operatorname{Attn}_{\ell}(h)\\ &\quad h\leftarrow u+\operatorname{MLP}_{\ell}(u)\end{aligned}

[0pt][0pt] AFTER: SHARED BLOCKS, DEPTH-AWARE GAINS

[0pt][0pt]

\begin{aligned} &\mathbf{for}\ \ell=0,\ldots,7:\\ &\quad u\leftarrow h+{\color[rgb]{0.0977,0.3672,0.5313}\alpha_{\ell}\operatorname{Attn}_{\ell\bmod B}(h)}\\ &\quad h\leftarrow u+{\color[rgb]{0.0977,0.3672,0.5313}\beta_{\ell}\operatorname{MLP}_{\ell\bmod B}(u)}\end{aligned}

B=1

: one stored block, eight applications. The learned

\alpha_{\ell},\beta_{\ell}

control attention and MLP residual updates separately at each depth.

RELATION TO PRIOR WORK Recent work scales residual updates with loop count ( Wang et al., 2026 ) ; here the gains are learned separately for each depth.

Table 3: A representative searched idea by improving parameter efficiency. Blue highlights searched changes; more discoveries are shown in Appendix C .

Figure 4: Discovered programs can transfer beyond the search horizon. Programs discovered with 10-minute candidate training are retrained from random initialization for 60 minutes alongside the reference. Several retain their advantages for substantially longer than the original search horizon, although these gains can diminish later as horizon-specific training policies take effect. Curves show held-out BPB against training time. Annotated comparisons use checkpoints after the learning-rate schedule reaches its floor, except in figure (c), where AutoScientists’ final BPB defines the shared target.

Table 8

Automated Machine Learning

Automated machine learning (AutoML) builds and tunes data-to-model pipelines with minimal manual effort, balancing predictive quality, compute, and deployment constraints.

See all 96 papers

AutoTrainess: Teaching Language Models to Improve Language Models Autonomously
Jun 30, 2026Zhaojian Yu, Penghao Yin, Shuzheng Gao +3Tool-Augmented Language Model Agents LLM Tool Use
1Tsinghua University · 2The Chinese University of Hong Kong · 3Simple Agent Lab
MMPostTrainBench: Benchmarking Autonomous Research for Multimodal Post-Training
Oct 4, 2026Yuxin Liu, Yuxuan Wang, Zhenxin Lei +9Multimodal Model Evaluation Agent Benchmarks
University of Science and Technology of China · Alibaba Token Hub, Alibaba Group · University of the Chinese Academy of Sciences +2
Language Models as AI Research World Models
Oct 8, 2026Zijun Wang, Zewen Liu, Minhua Lin +10LLM World Models Autonomous Research Agents
Amazon · UC Santa Cruz · Emory University +2

Jun 30, 2026J/K move · Enter open · S save

AutoTrainess: Teaching Language Models to Improve Language Models Autonomously

Zhaojian Yu, Penghao Yin, Shuzheng Gao, Shilin He+2

1Tsinghua University · 2The Chinese University of Hong Kong · 3Simple Agent Lab

Tool-Augmented Language Model Agents LLM Tool Use LLM Agent Training LLM Post-Training Language Model Self-Improvement

Open paper AskPDF arXiv

Training language models (LMs) remains a highly human-intensive process, even as frontier language model agents become increasingly capable at software engineering and other long-horizon tasks. A central challenge is that autonomous post-training is not just a coding problem: it requires the agent to repeatedly plan iterations, construct benchmark-aligned data, run stable training jobs, evaluate checkpoints, and preserve experiment state across many hours of interaction. We present AutoTrainess, a LM agent that exposes these operations as a repository of agent-computer interfaces for planning, data preparation, training, evaluation, and logging. Rather than leaving the agent to operate in a raw CLI environment with an underspecified action space, AutoTrainess externalizes prior human experience as explicit workflows, rules, and execution constraints that guide the agent toward effective and reliable training behavior. On PostTrainBench, AutoTrainess consistently outperforms CLI-only baselines, achieving 26.94 average score with GPT-5.4 (Codex) versus 23.21 for CLI-only. It also generalizes across models and harnesses, improving DeepSeek-V4-Flash (OpenCode) from 12.13 to 19.58.

Abstract

Automated Machine Learning

Explore similar work

AutoTrainess: Teaching Language Models to Improve Language Models Autonomously

MMPostTrainBench: Benchmarking Autonomous Research for Multimodal Post-Training

Language Models as AI Research World Models