SciExam for ENSO: Can AI Agents Build Climate Models?

Yinling Zhang; Langchen Liu; Dongbin Xiu; Xueyan Zou; Xu Kuang; Mengdi Wang; Shilong Liu

cs.AIOct 7, 2026

SciExam for ENSO: Can AI Agents Build Climate Models?

Authors: Yinling Zhang, Langchen Liu, Dongbin Xiu, Xueyan Zou, Xu Kuang, Mengdi Wang, Shilong Liu

Organizations: The Ohio State University · Yale University · University of California, San Diego · Stanford University · Princeton University

Abstract

Language-model agents are increasingly asked to carry out open-ended scientific research, yet their results are usually graded against a known answer, a rubric, or a language-model reviewer, none of which can tell whether a new scientific model is valid. The AI Science Exam for El Nino-Southern Oscillation (SciExam for ENSO) is a benchmark in which agents build low-order stochastic models of ENSO, the dominant mode of interannual climate variability, from real observations. Within a six-hour budget, agents process the observations, write their own diagnostics, which are then frozen, and develop a model using only these diagnostics as feedback. Hidden graders then test whether the model reproduces ENSO's statistics, recovers unobserved variables, and forecasts held-out years, and score a published model in the same way. Across twelve agent systems, six produce models that score higher than the published model, mainly through better reconstruction and forecasting. The simplified forms of the stronger models are each compatible with one of the two competing explanations of ENSO's warm-cold asymmetry, an open debate that the task never mentions. Controlled runs of the top system under varied information suggest that its scores do not come from recalling the dated observational record and that the information it receives shapes how it builds its model. SciExam for ENSO can thus evaluate agent research where no answer is known, and the results suggest that agents can already build competitive models whose structures bear on questions that scientists still debate.

Figures & tables

Figure 1 : ENSO conceptual modeling and benchmark performance. (a) Agents build a stochastic model of five monthly ENSO indices from observations, with deterministic dynamics $\mathbf{f}$ and stochastic forcing $\boldsymbol{\Sigma}$ . (b) Final composite scores of twelve agent systems. The dashed line marks the published reference model (0.378), which six agent-built models exceed.

Benchmark	Long horizon	Physical system	No known answer	Hidden scores	Scientific evaluation
Long-horizon agent tasks
Agents’ Last Exam [ 27 ]	✓	✗	✗	✓	✗
MLE-bench [ 6 ]	✓	✗	✗	✓	✗
RE-Bench [ 32 ]	✓	✗	✓	$\sim$	✗
ALE-Bench [ 11 ]	✓	✗	✓	✗	✗
EdgeBench [ 36 ]	✓	$\sim$	$\sim$	✗	–

Table 1 : Comparison with related benchmarks. Long horizon means a budget of an hour or more, physical system means that the task concerns a real physical system, no known answer means grading without reference tests, labels, a paper, an equation, or a fixed solution, hidden scores means that the agent does not see its scores during the run, and scientific evaluation means evaluation beyond fit, such as statistics, coupling, and forecasts.

\sim

marks partial and a dash unspecified properties.

Figure 2 : Overview of SciExam for ENSO . Given background material, raw observations, and a reference paper, the agent works through three phases within a six-hour budget: (A) processing the observations into indices that must pass a data check, (B) writing its own diagnostics from the task’s scientific guidance, which are then frozen, and (C) developing the model while rerunning these diagnostics. Hidden graders evaluate the final model on statistical properties, dynamical consistency, and predictive skill without returning scores to the agent. Dialogue is illustrative.

Dimension	What it tests	Measure	Panels
Statistical properties	Whether a long free run of the model behaves like the observed ENSO	Probability density functions (PDFs)	a, e
		Seasonal variance	b, f
		Autocorrelation functions (ACFs)	c, g
		Frequency of different event types	d
		Event strength against location	h
Dynamical consistency	Whether the variables are coupled as observed	$h_{W}$ , $u$ , $\tau$ reconstructed from $T_{C}$ , $T_{E}$	i–k

Table 2 : The three evaluation dimensions. Each is scored from

0

1

by running the submitted model, and the composite score is their weighted sum. The last column gives the panels of Figure 4 that show each measure for the top-scoring model. Persistence, the forecast baseline, keeps the initial anomaly unchanged.

Figure 3 : Model development and final scores. Panel (a): Composite scores of the checkpoints of each system against eligible time from the start of the run, including data processing and diagnostic development. Filled circles mark final checkpoints, and colors and line styles follow the system names in panel (b). Panel (b): Final scores of the twelve systems and the reference model, ranked by composite score. Stacked segments show the weighted contributions of statistical properties, dynamical consistency, and predictive skill, with weights of

0.3

0.3

, and

0.4

, and numbers at the ends give the composite scores.

Figure 4 : Scientific diagnostics of the final ENSO model produced by Claude Fable 5.1. Panels (a–c, e–g): Simulated and observed probability density functions (PDFs), seasonal variance, and autocorrelation functions (ACFs) for $T_{C}$ and $T_{E}$ . Panel (d): Event counts per seventy-one years, with model means and 2.5th–97.5th percentile ranges compared against observed counts. EN and LN denote El Niño and La Niña. EP and CP denote eastern- and central-Pacific El Niño, and EX denotes extreme El Niño. MYEN and MYLN denote multi-year events. Panel (h): Peak temperature anomalies versus peak longitudes for simulated and observed events. Panels (i–m): State reconstruction from partial measurements. Test A uses $T_{C}$ and $T_{E}$ to reconstruct $h_{W}$ , $u$ , and $\tau$ . Test B swaps the observed and reconstructed variables. Panels (n, o): Correlations between ensemble-mean forecasts and observations at lead times of one to twelve months over the held-out 2015–2024 period, compared with persistence.

Figure 5 : Structural simplification of agent-built ENSO models. (a) Nonlinear dynamics and stochastic forcing as two established explanations of ENSO asymmetry, in an excerpt from Vialard et al. [29] . (b) Two ablation examples. $L_{i}$ collects constant and linear coupling terms, $G_{i}(t)$ is the seasonal growth or damping rate of the variable itself ( $G_{i}^{(2)}$ its semiannual part), $B_{i}(t)$ is the seasonal coupling to $h_{W}$ , $N_{i}$ denotes nonlinear terms, and $H$ is an internal heat-content variable of GPT-6 Astra mapped to $h_{W}$ . Other letters are parameters, seasonal when written with $t$ . $dW$ denotes increments of a Wiener process, scaled by the state-dependent $\sigma_{i}$ in Claude Fable 5.1 (multiplicative noise) and by the time-dependent matrix $\Sigma(t)$ in GPT-6 Astra (additive noise). (c) Classification of the nine models that exceed or approach the reference by their higher-scoring viable reduction, with no family assigned in advance, recovering the two model families debated for ENSO.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

System	Final checkpoint	Best checkpoint
Claude Fable 5.1	0.515	0.531
Kimi K3	0.479	0.500
GPT-5.6-Sol	0.443	0.466
GPT-6 Astra	0.442	0.475
Claude Opus 5	0.440	0.508
GPT-5.5	0.403	0.404

Appendix

Table 4 : Composite scores of the final and best checkpoints of every system. The best checkpoint is identified after the run by scoring all checkpoints with the hidden graders. Six systems exceed the reference by final checkpoints and eight by best checkpoints.

System	Scaffold	Route
Claude Fable 5.1	Claude Code 2.1.270	Subscription
Kimi K3	Claude Code 2.1.223	OpenRouter (Moonshot AI)
GPT-5.6-Sol	Codex 0.146.1	Subscription
GPT-6 Astra	Codex 0.154.0	Subscription
Claude Opus 5	Claude Code 2.1.223	Subscription
GPT-5.5	Codex 0.146.1	Subscription

Appendix

Table 5 : Scaffolds and routes of the twelve systems. Systems routed through OpenRouter use one pinned provider each, given in parentheses, with no fallback.

System	Route	Calls	Prompt tokens	Cached	Cost (USD)
Kimi K3	OpenRouter	954	120.3 M	99.2%	43.69
GLM-5.2	OpenRouter	669	65.4 M	98.0%	21.48
Qwen3-Max	OpenRouter	3,250	308.1 M	95.3%	132.37
DeepSeek V4 Pro	OpenRouter	589	48.7 M	93.2%	7.67
MiniMax-M3	OpenRouter	1,011	83.1 M	98.2%	5.97
DeepSeek V4.1 Flash	OpenRouter	752	62.0 M	98.4%	2.43

Appendix

Table 6 : Model calls, prompt tokens, share of prompt tokens served from the provider’s cache, and cost of the main run of each system. Subscription systems pay a fixed fee. The usage of GPT-5.6-Sol was not recorded. For DeepSeek V4.1 Flash, tokens are counted from the transcript and the cost is the change in account usage over its batch, in which it was the only system routed through OpenRouter.

Symbol	Value	Symbol	Value
$a_{11}$	$-0.9170$	$a_{13}$	$-0.4020$
$a_{14}$	$0.2600$	$a_{15}$	$0.1140$
$a_{21}$	$-0.1640$	$a_{23}$	$0.09686$
$a_{24}$	$-0.1500$	$a_{25}$	$-0.07482$
$a_{26}$	$-0.3000$	$a_{27}$	$-1.184$
$a_{28}$	$0.004840$	$T_{s,h}$	$0.4000$

Appendix

Table 11

Symbol	Definition or value
$a_{12}(t)$	$0.8500+\allowbreak 0.1700c_{1}+\allowbreak 0.4250s_{1}$
$a_{22}(t)$	$-0.1640+\allowbreak 0.1044c_{1}-0.08738s_{1}$
$a_{32}(t)$	$0.1073+\allowbreak 0.01866c_{1}+\allowbreak 0.03000s_{1}$
$a_{33}(t)$	$-0.2500+\allowbreak 0.09000c_{1}-0.2400s_{1}$
$a_{44}(t)$	$-0.1510-0.006228c_{1}-0.2700s_{1}-0.1200c_{2}-0.1500s_{2}$
$\sigma_{u}(t)$	$0.09400\,\allowbreak(1-0.1000c_{1}+\allowbreak 0.03000s_{1}-0.1500c_{2}+\allowbreak 0.1500s_{2})$

Appendix

Table 12

Symbol	Value	Symbol	Value
$\widetilde{a}_{11}$	$-0.8523968$	$\widetilde{a}_{12}$	$0.9358338$
$\widetilde{a}_{13}$	$-0.4152634$	$\widetilde{a}_{14}$	$0.2980273$
$\widetilde{a}_{15}$	$0.1134559$	$\widetilde{a}_{21}$	$-0.1834501$
$\widetilde{a}_{22}$	$-0.2025177$	$\widetilde{a}_{23}$	$0.1094091$
$\widetilde{a}_{24}$	$-0.2023363$	$\widetilde{a}_{25}$	$-0.07096293$
$\widetilde{a}_{31}$	$0.1524918$	$\widetilde{a}_{32}$	$0.1193508$

Appendix

Table 13

Symbol	Definition or value
$\widetilde{a}_{33}$	$-0.3096332+\allowbreak 0.03899662c_{1}-0.2438837s_{1}$
$\widetilde{a}_{44}$	$-0.1736369-0.02538335c_{1}-0.3242342s_{1}-0.123943c_{2}-0.1380168s_{2}$

Appendix

Table 14

		Composite		Best checkpoint
Condition	Access	Best	Final	Stat.	Dyn.	Pred.
Offline (formal)	None	0.531	0.515	0.741	0.632	0.297
Years anonymized	None	0.482 ( $\downarrow$ 0.049)	0.463 ( $\downarrow$ 0.052)	0.656 ( $\downarrow$ 0.085)	0.576 ( $\downarrow$ 0.056)	0.281 ( $\downarrow$ 0.016)
Web access 1	Literature, data	0.475 ( $\downarrow$ 0.056)	0.386 ( $\downarrow$ 0.129)	0.638 ( $\downarrow$ 0.103)	0.572 ( $\downarrow$ 0.060)	0.280 ( $\downarrow$ 0.017)
Web access 2	Literature	0.528 ( $\downarrow$ 0.003)	0.406 ( $\downarrow$ 0.109)	0.798 ( $\uparrow$ 0.057)	0.556 ( $\downarrow$ 0.076)	0.306 ( $\uparrow$ 0.009)

Appendix

Table 11 : Claude Fable 5.1 under four conditions, each run once for six hours of eligible time. Best is the highest-scoring checkpoint and final is the last one, and the three component scores are for the best checkpoint. Parentheses give the change relative to the offline run (green for an increase, red for a decrease), and bold marks the best value in each column.

Explore similar work

Benchmarking AI Agents for Addressing Scientific Challenges Across Scales
Jun 10, 2026Tianyu Liu, Allen Xin Wang, Antonia Panescu +30Scientific Discovery Agentic Benchmarks
Yale University, CT, USA · The Pennsylvania State University, PA, USA · Northeastern University, MA, USA +7
Science sandboxes measure the scientific capability of AI agents
Aug 31, 2026Arya S. Rao, Rodrigo I. Castro, Sager J. Gosai +8Scientific Discovery Sandbox
The Broad Institute of MIT and Harvard; Cambridge, MA 02142, USA. · Department of Biomedical Informatics, Harvard Medical School; Boston, MA 02115, USA. · The Jackson Laboratory; Bar Harbor, ME 04609, USA. +10
AgentIdeaBench: Benchmarking Scientific Ideation in the Agent Era
Sep 7, 2026Yunxiang Mo, Tianshi Zheng, Yisen Gao +7Scientific Ideation Artificial Intelligence Scientists
Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China · NVIDIA AI Technology Center (NVAITC), NVIDIA, Santa Clara, USA

Jun 10, 2026, cs.AIJ/K move · Enter open · S save

Benchmarking AI Agents for Addressing Scientific Challenges Across Scales

Tianyu Liu, Allen Xin Wang, Antonia Panescu, Lisa Xinyi Chen+29

Yale University, CT, USA · The Pennsylvania State University, PA, USA · Northeastern University, MA, USA+7

Scientific Discovery Agentic Benchmarks Artificial Intelligence Agents Exploration

Open paper AskPDF arXiv

AI agents are increasingly being developed to accelerate scientific discovery, yet their practical capabilities in real research settings remain poorly understood. Existing benchmarks for AI agents rarely capture the complexity, heterogeneity, and extended reasoning required by scientific work, whereas benchmarks for scientific tasks often reduce research to static, direct problems and provide limited support for interactive evaluation. Here, we introduce SciAgentArena, a systematic benchmark for evaluating AI agents in real-world scientific research scenarios drawn from emerging needs across multiple domains. SciAgentArena comprises approximately 200 tasks with stepwise verification and an interactive, agent-agnostic environment for assessing diverse AI agents. Using this benchmark, we find that current agents can contribute effectively to well-specified data-analysis workflows, particularly when the task structure and evaluation criteria are clear. However, their performance remains uneven across scientific contexts: agents struggle to generate genuinely novel insights, sustain self-directed exploration, and formulate robust solutions for open-ended research questions. We further characterize common failure modes across agents and identify opportunities for improving their reliability, autonomy, and scientific reasoning. Together, SciAgentArena provides a practical framework for measuring progress in AI agents for science and for guiding the design of future agents capable of addressing complex scientific challenges. Full codes, tasks, and datasets can be accessed via this link: https://sciagentarena.github.io/.

Abstract

Explore similar work

Benchmarking AI Agents for Addressing Scientific Challenges Across Scales

Science sandboxes measure the scientific capability of AI agents

AgentIdeaBench: Benchmarking Scientific Ideation in the Agent Era