Language Models as AI Research World Models

Zijun Wang; Zewen Liu; Minhua Lin; Zhaotian Weng; Zhan Shi; Bing He; Yisi Sang; Dakuo Wang

cs.CLOct 8, 2026

Language Models as AI Research World Models

Authors: Zijun Wang, Zewen Liu, Minhua Lin, Zhaotian Weng, Zhan Shi, Bing He, Yisi Sang, Dakuo Wang, +5 more

Organizations: Amazon · UC Santa Cruz · Emory University · Pennsylvania State University · UC Santa Barbara

Abstract

AI research agents automate the cycle of proposing, implementing, and evaluating experiments, opening a path toward recursive self-improvement. Yet their ability to propose experiments outpaces their capacity to execute them in real environments, making outcome prediction a key capability for sustained self-improvement under limited experimental budgets. We investigate language models as Research World Models (RWMs), which predict the outcomes of candidate interventions across research environments. Our evaluation draws on over 2,600 experimental records from nine research environments spanning pretraining, post-training, and inference, representing more than 171,000 H100 GPU-hours of experimentation. Research knowledge acquired from real experimental experience improves RWM predictions of unseen interventions within the same environment (Spearman +0.27), and can be reused across environments. For example, using only pretraining experience from OLMo3, Marin, and Nanochat, an RWM reduces selection regret in the Qwen3 environment by 78% compared with zero-experience setting. These benefits extend to multi-round Autoresearch under a fixed selection budget: RWMs with in-env and cross-env research knowledge increase the best gain achieved by 15.8% and 11.6%, respectively. Ablations across 13 language models used as RWMs show that adding research knowledge can improve intervention ranking more than changing models or increasing reasoning effort alone. These findings support language models as RWMs and motivate accumulating experimental data for future RWM training.

Figures & tables

Figure 1: Large language models as Research World Models. Given a research environment $E_{\tau}$ and candidate interventions $a$ , an RWM predicts gains $\hat{g}$ before execution, so that only the most promising interventions consume experimental budget. Executed experiments become records $(E_{\tau_{i}},a_{i},g_{i})$ that form history $H$ for subsequent predictions. Records may come from the same environment or from other environments, allowing the same RWM to serve a target environment that has no records of its own.

Research Env.	Model / System	Eval metric	Records	H100 GPU-hours	In-env	Cross-env
Pretraining
OLMo3-100M	OLMo3-100M	mean BPB $\downarrow$	480	8,572	$\checkmark$	$\checkmark$
Marin	Marin-153M	mean BPB $\downarrow$	132	12,101	—	$\checkmark$
Nanochat	Nanochat-149M	mean BPB $\downarrow$	132	2,584	—	$\checkmark$
Qwen3	Qwen3-153M	mean BPB $\downarrow$	132	15,447	—	$\checkmark$
OLMo3-190M	OLMo3-190M	mean BPB $\downarrow$	132	5,582	—	$\checkmark$

Table 1: Research environments and experimental scale. In-env and Cross-env indicate prediction-evaluation coverage.

			MAE $\downarrow$				Spearman $\uparrow$
Environment (unit)	History	Test	$W_{0}$	kNN	ridge	$W_{\mathrm{in-env}}$	$W_{0}$	kNN	ridge	$W_{\mathrm{in-env}}$
OLMo3-100M (BPB)	320	160	2.93e-2	2.58e-2	3.13e-2	2.03e-2	0.62	0.78	0.67	0.90
Diffusion (BPB)	536	125	1.95e-2	1.58e-2	1.99e-2	8.77e-3	0.46	0.49	0.50	0.78
Math distillation (pp)	180	100	1.58	1.15	1.11	0.91	0.43	0.61	0.62	0.70
Code RL (pp)	34	115	5.98	5.45	5.02	4.17	0.48	0.29	0.48	0.62
Inference optimization (tokens/s)	457	98	13.19	13.18	12.73	1.34	0.54	0.62	0.72	0.88

Table 2: Within-environment prediction performance. Each environment uses a single fixed split. History denotes context records supplied to

W_{\mathrm{in-env}}

; Test denotes the number of held-out candidates.

	Spearman $\uparrow$				MAE (BPB) $\downarrow$
Target Environment	$W_{0}$	kNN	ridge	$W_{\mathrm{cross-env}}$	$W_{0}$	kNN	ridge	$W_{\mathrm{cross-env}}$
OLMo3-100M	0.69	0.56	0.26	0.83	1.11e-2	1.18e-2	1.65e-2	9.28e-3
OLMo3-190M	0.72	0.43	0.26	0.78	7.94e-3	1.08e-2	1.19e-2	8.56e-3
Qwen3	0.74	0.71	0.40	0.82	9.20e-3	1.13e-2	1.69e-2	7.80e-3
Marin	0.42	0.28	0.43	0.50	9.35e-3	1.13e-2	1.03e-2	8.18e-3
Nanochat	0.57	$-0.17$	$-0.12$	0.71	2.79e-2	3.61e-2	3.57e-2	2.70e-2

Table 3: Cross-environment prediction performance. Macro is the equal-weight mean over target environments.

	Regret@3 $\downarrow$					NDCG@3 $\uparrow$
Environment (unit)	Random	$W_{0}$	kNN	ridge	$W_{\mathrm{in-env}}$	Random	$W_{0}$	kNN	ridge	$W_{\mathrm{in-env}}$
OLMo3-100M (BPB)	1.32e-2	2.80e-3	1.81e-3	2.14e-3	7.00e-4	0.28	0.70	0.78	0.72	0.84
Diffusion (BPB)	1.09e-2	3.14e-3	1.79e-3	5.47e-3	4.70e-4	0.16	0.51	0.62	0.32	0.87
Math distillation (pp)	0.87	0.48	0.41	0.45	0.29	0.20	0.33	0.38	0.34	0.47
Code RL (pp)	4.16	2.20	2.12	1.54	0.83	0.42	0.59	0.64	0.73	0.81
Inference optimization (tokens/s)	4.31	3.60	1.62	2.09	2.53	0.69	0.86	0.93	0.92	0.93

Table 4: Mean within-environment experiment-selection performance. Metrics are averaged over 200 shared random partitions of each environment’s fixed test panel, selecting three candidates from each pool of 16.

	Regret@3 (BPB) $\downarrow$					NDCG@3 $\uparrow$
Target	Random	$W_{0}$	kNN	ridge	$W_{\mathrm{cross-env}}$	Random	$W_{0}$	kNN	ridge	$W_{\mathrm{cross-env}}$
OLMo3-100M	1.31e-2	4.27e-3	4.74e-3	4.84e-3	2.24e-3	0.26	0.65	0.66	0.66	0.84
OLMo3-190M	8.42e-3	1.64e-3	1.60e-3	1.96e-3	5.50e-4	0.24	0.68	0.66	0.64	0.81
Qwen3	1.51e-2	9.40e-4	3.50e-4	1.72e-3	2.10e-4	0.27	0.77	0.85	0.73	0.91
Marin	1.07e-2	7.40e-4	1.10e-3	8.40e-4	7.90e-4	0.18	0.74	0.61	0.67	0.70
Nanochat	8.27e-3	4.94e-3	1.53e-2	1.51e-2	2.22e-3	0.14	0.32	0.12	0.06	0.43

Table 5: Mean cross-environment experiment-selection performance. Metrics are averaged over 200 shared random partitions of each target’s fixed test panel, selecting three candidates from each pool of 16. Macro is the equal-weight mean over target environments.

Figure 2: Multi-round Autoresearch on OLMo3-100M. Means over five Autoresearch evaluations, each with eight rounds. (a) Best-so-far gain (BPB). (b) nAULC (%), computed by normalizing each evaluation’s curve area by that of its matched Arrival Oracle and then averaging across evaluations.

Figure 3: RWM rankings on OLMo3-100M across (a) backbones and (b) Claude Opus 5 reasoning-effort settings.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Block	Contents
Environment	Model and data configuration, resource budget, reference recipe and measurements, evaluation suite, and gain definition.
Intervention	Materialized code diff or configuration override and intervention direction.
Outcome	Raw evaluation measurements and reference-relative gain when defined by the scoring protocol.
Metadata	Execution status and random seed.

Appendix

Table 6: Fields retained in experimental records.

Target	vs. $W_{0}$	vs. kNN	vs. ridge
OLMo3-100M	$+0.14$ $[+0.02,\,+0.28]$	$+0.27$ $[+0.04,\,+0.58]$	$+0.57$ $[+0.23,\,+0.93]$
OLMo3-190M	$+0.06$ $[-0.07,\,+0.18]$	$+0.35$ $[+0.13,\,+0.61]$	$+0.52$ $[+0.22,\,+0.83]$
Qwen3	$+0.08$ $[\text{+2.00e-3},\,+0.17]$	$+0.11$ $[\text{+7.00e-3},\,+0.23]$	$+0.43$ $[+0.20,\,+0.66]$
Marin	$+0.08$ $[-0.05,\,+0.24]$	$+0.22$ $[+0.07,\,+0.40]$	$+0.07$ $[-0.25,\,+0.37]$
Nanochat	$+0.15$ $[+0.04,\,+0.27]$	$+0.89$ $[+0.62,\,+1.13]$	$+0.84$ $[+0.57,\,+1.07]$

Appendix

Table 7: Cross-environment Spearman differences:

W_{\mathrm{cross-env}}

minus each comparator. Brackets contain pointwise 95% paired bootstrap confidence intervals.

Explore similar work

Scaling Automatic Research Agents via World Models
Aug 12, 2026Xiyuan Yang, Sheikh Sarwar, Jingru Cheng +8World Model Learning RL Post-Training
University of Illinois Urbana-Champaign · Amazon
Qwen-AgentWorld: Language World Models for General Agents
Jun 23, 2026Yuxin Zuo, Zikai Xiao, Li Sheng +30LLM World Models Agentic RL
Qwen Team
AutoWorldModel-Bench: A State-Centric Benchmark for Automated World-Model Research
Jul 20, 2026Marjan Moodi, Xuankang Zhu, Fernando De Mesentier Silva +2World Model Learning AI Agent Benchmarks
Electronic Arts · Simon Fraser University

Aug 12, 2026, cs.LGJ/K move · Enter open · S save

Scaling Automatic Research Agents via World Models

Xiyuan Yang, Sheikh Sarwar, Jingru Cheng, Zhan Shi+7

University of Illinois Urbana-Champaign · Amazon

World Model Learning RL Post-Training RL for LLM Agents Autonomous Research Agents Model-Based RL

Open paper AskPDF arXiv

Automating empirical research is a long-standing direction of AI. Recent automatic research (AutoResearch) agents bring this goal within reach, as modern LLMs show the capability to independently implement solutions and learn from the execution outcomes. Behind these gains, post-training (especially RL) plays a central role. In this paper, we identify a fundamental tension when scaling RL for these agents: the two components of every AutoResearch trajectory (agent generation and environment execution) scale in very different manners, since all generation shares compute through batching, while each execution occupies its exclusive sandbox and real machine time. As a result, the environment execution dominates the training cost and becomes the bottleneck as trajectories grow. To resolve this tension, we propose World Model RL (WMRL), which replaces environment execution with a world model to remove this bottleneck. Additionally, the world model can be imperfect, as its rewards are corrupted by bias and noise. Therefore, we further equip WMRL with two mitigations, Online Debiasing and Inverse-Variance Denoising, which offset the bias and suppress the noise respectively. Theoretically, we prove that both mitigations of WMRL strictly improve the convergence guarantee. Empirically, WMRL accelerates training by 3-4x on various tasks at different agent scales, while exceeding the performance of standard RL baselines. Moreover, our post-trained 4B and 9B agents outperform much larger open-weight agents of 48B and 120B on held-out benchmarks. Beyond AutoResearch, WMRL also transfers to post-training embodied VLA policies, which demonstrates the generalizability of our method.

Abstract

Explore similar work

Scaling Automatic Research Agents via World Models

Qwen-AgentWorld: Language World Models for General Agents

AutoWorldModel-Bench: A State-Centric Benchmark for Automated World-Model Research