Automating ML for Science: Can Frontier Agents Climb Scientific Hills in the Wild?

Ming Zhong; Stacy Li; Nicholas Carlini; Matthew Jagielski

cs.AIOct 20, 2026

Automating ML for Science: Can Frontier Agents Climb Scientific Hills in the Wild?

Authors: Ming Zhong, Stacy Li, Nicholas Carlini, Matthew Jagielski

Organizations: Anthropic Fellows Program · Anthropic

Abstract

Frontier coding agents are increasingly framed as end-to-end automated researchers, and ML for science offers a demanding test of that promise: improving a scientific model often requires domain insight beyond routine ML engineering. We introduce Agent4Sci, a long-horizon testbed for measuring how far today's agents can improve the published best models on scientific ML benchmarks, spanning 20 benchmarks across 11 domains. Rather than ranking agents on a leaderboard, Agent4Sci asks what kinds of progress agents can make on these tasks under in-the-wild conditions, and where they fall short. Empirically, today's frontier agents advance the published best on every benchmark, with Claude Opus reaching 10.4% mean relative improvement and no run flagged for academic misconduct. These gains, however, come almost entirely from a bag of ML-engineering tricks rather than research-level insight. Trajectory analysis surfaces three recurring limitations in this setting: novelty miscalibration, resource under-utilization, and rapid collapse into local engineering sweeps. Lightweight plug-in skills for research ideation, experiment management, and scheduled checklists raise Claude Opus from 10.4% to 20.1% and Codex from 4.4% to 8.1%, while yielding more diverse, domain-relevant interventions. Agent4Sci gives the community a way to track and push frontier agents from ML engineering toward domain insight on real-world ML-for-science problems. Code and data are available on GitHub (https://github.com/maszhongming/agent4sci) and Hugging Face (https://huggingface.co/datasets/MingZhong/agent4sci).

Figures & tables

Panel of Figure 1 — Figure 1: Scientific hill-climbing gives an agent a paper, codebase, data, and target metric, then asks it to improve that metric under a fixed resource budget. On Agent4Sci , frontier agents push past the published best on every benchmark, with Claude Opus reaching 10.4% mean relative improvement while most gains come from ML-engineering routines.

Figure 2: Agent4Sci evaluation protocol. Each task includes a standard reproduction folder with standardized training scripts, evaluation scripts, and data splits. Runs separate the hill-climbing agent from held-out evaluation and auditing to prevent evaluation gaming.

Benchmark	Domain	Task	Modality	Train Time	Best Agent	$\Delta\%$
ATAT [ 9 ]	Astronomy	Transient classification	Time series	4.5 h $\times$ 5	Sonnet	1.5
[0.5pt/3pt] CarbonGlobe [ 60 ]	Ecology	Carbon forecasting	Time series	4.5 h	Opus	14.7
[0.5pt/3pt] CheMixHub [ 46 ]	Chemistry	Olfactory similarity	Mixture	3 min
		Fuel octane (MON)		1 min
		Polymer conductivity		45 min	Gemini	17.4
[0.5pt/3pt] CosmoBench [ 20 ]	Cosmology	Cosmo $\sigma_{8}$ regression	Tree	20 min

Table 1: The 25 tasks across 20 benchmarks in Agent4Sci , with each benchmark’s top default agent and mean relative improvement

\Delta\%

over the reproduced published best result. Training time is the cost of one run on

1\times

H200, and “

\times N

” indicates the metric averages over

N

random seeds or folds.

Panel of Figure 3 — Figure 3: Hill-climbing methods across 25 scientific tasks. Frontier agents mostly use a shared ML-engineering toolbox, while problem-specific interventions remain rare.

Panel of Figure 4 — Figure 4: Per-run trajectory statistics for the three analyzed agent configurations on Agent4Sci . Claude Opus stays active for the full 12-hour budget, Codex commits early to a long-running script and then waits for it to finish, and Gemini oscillates between the two paradigms.

Benchmarks	Metrics	Baseline	Claude Opus 4.6		Opus + Skills		Codex (GPT-5.4)		Codex + Skills
Benchmarks	Metrics	Baseline	Abs.	$\Delta$ %	Abs.	$\Delta$ %	Abs.	$\Delta$ %	Abs.	$\Delta$ %
ATAT	Avg Macro F1 $\blacktriangle$	82.79	83.47	0.8	85.58	3.4	83.47	0.8	82.69	-0.1
CarbonGlobe	RMSE $\blacktriangledown$	0.604	0.515	14.7	0.391	35.3	0.612	-1.3	0.496	17.9
CheMixHub	MAE $\blacktriangledown$	1.679	1.388	11.9	1.338	23.1	1.506	15.4	1.439	16.7
CosmoBench	R 2 /Acc $\blacktriangle$	0.604	0.631	4.8	0.643	6.9	0.627	3.0	0.643	7.7
DART-Eval	Spearman $\blacktriangle$	0.544	0.577	6.1	0.598	9.9	0.547	0.6	0.551	1.3

Table 2: Effect of plug-in skills on hill-climbing performance. Each “ + Skills ” column reports the same agent run with our plug-in skills enabled. Metrics are flagged with

\blacktriangle

\blacktriangledown

for higher-/lower-is-better. Bold marks the highest

\Delta\%

per benchmark. Values are task-averaged for multi-task benchmarks. The last row reports the mean across all 20 benchmarks.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Paper Title	Venue
ATAT [ 9 ]	ATAT: Astronomical Transformer for Time Series and Tabular Data	Astronomy & Astrophysics 2024
[0.5pt/3pt] CarbonGlobe [ 60 ]	CarbonGlobe: A Global-Scale, Multi-Decade Dataset and Benchmark for Carbon Forecasting in Forest Ecosystems	NeurIPS 2025 D & B
[0.5pt/3pt] CheMixHub [ 46 ]	CheMixHub: Datasets and Benchmarks for Chemical Mixture Property Prediction	NeurIPS 2025 D & B
[0.5pt/3pt] CosmoBench [ 20 ]	CosmoBench: A Multiscale, Multiview, Multitask Cosmology Benchmark for Geometric Deep Learning	NeurIPS 2025 D & B
[0.5pt/3pt] DART-Eval [ 42 ]	DART-Eval: A Comprehensive DNA Language Model Evaluation Benchmark on Regulatory DNA	NeurIPS 2024 D & B
[0.5pt/3pt] ESCAPE [ 39 ]	A Standardized Benchmark for Multilabel Antimicrobial Peptide Classification	NeurIPS 2025 D & B

Appendix

Table 3: Source publication and venue for each benchmark. D & B denotes Datasets & Benchmarks.

Panel of Figure 5 — Figure 5: NeurIPS 2025 filtering case study, screening stages. 5(a) Four-stage funnel from 497 accepted papers to 10 admitted benchmarks. 5(b) Per-criterion pass rates over the 207 scientific-domain papers, used as a Stage II view of which criteria are hardest to meet. 5(c) Distribution of how many of the seven criteria each of the 207 papers satisfies under verified labels, used as a Stage III view of how many papers reach the 7/7 bar.

Panel of Figure 6 — Figure 6: NeurIPS 2025 filtering case study, human-review stages. 6(a) Per-criterion correction waterfall during manual verification of the 70 automated 7/7 candidates, ending at 40 verified 7/7 candidates. 6(b) Rejection-reason pictogram for the 40 verified candidates entering deep review, with 10 selected and 30 rejected by primary reason.

Panel of Figure 7 — Figure 7: Standardized project layout for each Agent4Sci task. The hill-climbing agent has access to all components except the test split.

Benchmark	Metric	Paper	Ours	Baseline Method
ATAT	Avg Macro F1 $\blacktriangle$	83.50	82.79	ATAT, 5-fold CV
CarbonGlobe	RMSE $\blacktriangledown$	0.571	0.604	DeepED
CheMixHub	MAE $\blacktriangledown$	1.638	1.679	Best-per-task models, 5-fold CV
CosmoBench	R 2 /Acc $\blacktriangle$	0.604	0.604	GNN, $k$ -NN, and velocity GNN baselines
DART-Eval	Spearman $\blacktriangle$	0.515	0.544	Nucleotide Transformer with full fine-tuning
ESCAPE	mAP $\blacktriangle$	72.10	71.80	ESCAPE, 2-fold CV $\times$ 3 seeds

Appendix

Table 4: Per-benchmark comparison between paper-reported numbers ( Paper ) and our reproductions ( Ours ); higher-/lower-is-better metrics are flagged with

\blacktriangle

\blacktriangledown

, and the Ours column provides the baseline used throughout the paper.

Figure 8: Hill-climbing instructions issued to the hill-climbing agent at the start of every run. Project-specific paths are shown as placeholders.

Figure 9: Evaluation instructions issued to the evaluation agent after the hill-climbing phase ends. The evaluation agent is the only stage with access to the held-out test split.

Figure 10: Auditing instructions issued to the auditing agent after evaluation. The four cheating-pattern categories mirror the evaluation-gaming behaviors identified in the pilot study (§ 2.2 ).

Benchmarks	Metrics	Baseline	Gemini 3.1 Pro		Codex (GPT-5.4)		Claude Sonnet 4.6		Claude Opus 4.6
Benchmarks	Metrics	Baseline	Abs.	$\Delta$ %	Abs.	$\Delta$ %	Abs.	$\Delta$ %	Abs.	$\Delta$ %
ATAT	Avg Macro F1 $\blacktriangle$	82.79	82.60	-0.2	83.47	0.8	84.07	1.5	83.47	0.8
CarbonGlobe	RMSE $\blacktriangledown$	0.604	0.629	-4.1	0.612	-1.3	0.529	12.4	0.515	14.7
CheMixHub	MAE $\blacktriangledown$	1.679	1.355	17.4	1.506	15.4	1.439	8.9	1.388	11.9
CosmoBench	R 2 /Acc $\blacktriangle$	0.604	0.630	3.1	0.627	3.0	0.625	2.3	0.631	4.8
DART-Eval	Spearman $\blacktriangle$	0.544	0.547	0.6	0.547	0.6	0.591	8.6	0.577	6.1

Appendix

Table 5: Results on Agent4Sci . We report each agent’s absolute score ( Abs. ) and relative improvement over the reproduction Baseline (

\Delta

%, positive is better). Values are task-averaged for multi-task benchmarks. Bold marks the best agent per benchmark, and the last row reports the mean across all 20 benchmarks in total.

Panel of Figure 11 — Figure 11: Per-benchmark Pareto trade-offs for Claude Opus 4.6 against Opus 4.6 + Skills. Arrows point from the no-skill final method to the with-skill final method; the horizontal dashed line marks zero relative improvement, and the vertical dashed line in 11(b) marks the reproduction inference time. Both x-axes are log-scaled.

	Unrestricted					Restricted
Benchmark	Paper	Run 1	Run 2	Run 3	Mean $\pm$ Std	HPO-only
ATAT	0.8	0.5	0.9	-0.3	0.5 $\pm$ 0.5	0.2
DART-Eval	6.1	5.8	6.4	6.2	6.1 $\pm$ 0.3	2.7
TIDMAD	7.4	7.7	7.1	7.8	7.5 $\pm$ 0.3	3.5
CheMixHub	11.9	12.4	10.7	11.3	11.6 $\pm$ 0.7	1.4
Mars-Bench	46.4	45.1	47.6	44.3	45.9 $\pm$ 1.5	4.3

Appendix

Table 6: Run-to-run variance and HPO-only ablation of Claude Opus 4.6 (without skills). Each entry is the relative improvement

\Delta\%

over the reproduction baseline. Paper is the run reported in Table 5 , Mean

\pm

Std is computed over the four unrestricted runs, and HPO-only restricts the agent to hyperparameter search under the same budget (Appendix C.4 ).

Treatment of the reproduced baseline	Count	Benchmarks
Explicit HPO search (grid, random, or Bayesian)	7	CheMixHub, CosmoBench, GEO-Bench-2, Mars-Bench, MOTIVE, PFDelta, SR-CACO-2
Tuned on validation	3	ATAT, MassSpecGym, ProteinGym (via marginal likelihood)
Fixed hyperparameters only	10	CarbonGlobe, DART-Eval, ESCAPE, GTPBD, LITHOS, NeuroGraph, TIDMAD, TreeFinder, UniSite, gRNAde

Appendix

Table 7: How each source paper treats the baseline we reproduce. Ten of the twenty starting points are already-tuned configurations rather than untuned defaults.

Figure 12: Research skill, invoked at the start of every run. The persona-driven literature review and tier-based experiment planning push the agent away from incremental hyperparameter sweeps and toward the high-risk, high-reward ideas a domain expert would find surprising.

Figure 13: Experiment skill, invoked after research. Tier-1 experiments run first and saturate every available GPU, while runtime optimization, smoke-testing, and structured triage of failed runs keep the GPU budget productive throughout the run.

Figure 14: Scheduled-checklist skill, triggered hourly through a tool-call hook. The four-part diagnostic forces the agent to detect drift toward incrementalism, audit running jobs, re-plan strategy when ideas stall, and update its task list before the next checklist tick.

Panel of Figure 15 — Figure 15: Hill-climbing trajectories on PFDelta. Lower validation PBL is better.

Panel of Figure 16 — Figure 16: Hill-climbing trajectories on the two Mars-Bench sub-tasks: terrain segmentation (top pair) and cone detection (bottom pair). Higher validation IoU/mAP is better.

Panel of Figure 17 — Figure 17: Hill-climbing trajectories on MassSpecGym. Higher validation Hit@1 is better.

Panel of Figure 18 — Figure 18: Hill-climbing trajectories on GEO-Bench-2. Higher validation mIoU is better.

Panel of Figure 19 — Figure 19: Hill-climbing trajectories on CarbonGlobe. Lower validation RMSE is better.

Panel of Figure 20 — Figure 20: Three-day hill-climbing trajectories on PFDelta 2000 (lower validation values indicate better performance). Green dots mark recorded improvements to the running-best validation metric; gray dots mark discarded experiments.

	Claude Opus 4.6 + Skills	Codex (GPT-5.4) + Skills
Experiments	60	253 (of $\sim$ 1226 launched)
Approach	Switch to GNS-S, scale hidden dim 20 $\to$ 128, continued training with decreasing LR	CANOS-PF + SmoothL1 + PCGrad multi-task, then dense GPU Newton-Raphson with condition-aware adaptive step policy
Baseline PBL	1.180	1.157
Best PBL	0.139	0.006
Improvement	88%	99.5%
Batched latency	3.3 ms/sample	$\sim$ 54 ms/sample (NN + NR)

Appendix

Table 8: Three-day PFDelta 2000 outcomes for the two skill-enabled agents. Both agents have skills enabled and run with no human intervention; figures show improvement trajectories.

Panel of Figure 21 — Figure 21: Three-day hill-climbing trajectories on CarbonGlobe (lower validation values indicate better performance). Green dots mark recorded improvements to the running-best validation metric; gray dots mark discarded experiments.

	Claude Opus 4.6 + Skills	Codex (GPT-5.4) + Skills
Training experiments	35	$\sim$ 80–100
[0.5pt/3pt] Approach	BPTT-SA curriculum autoregressive	$\sim$ 80–100 diverse architecture checkpoints
	training: progressively train LSTM	(Informer, PatchTST, iTransformer, $\ldots$ ), then
	on its own predictions to fix	cellwise additive greedy blending over $\sim$ 277
	40-year exposure bias	stages from 1449 candidate predictions
[0.5pt/3pt] Improvement on val	12% (0.604 $\to$ 0.531)	37% (0.604 $\to$ 0.379, ensemble)

Appendix

Table 9: Three-day CarbonGlobe outcomes. Codex’s heavy ensembling improves validation by a wide margin but overfits the validation set; held-out test performance regresses below the baseline.

AI Agents for Scientific Discovery

AI agents for scientific discovery use language models, tools, and experiments to generate, test, and refine scientific knowledge.

See all 284 papers

Benchmarking AI Agents for Addressing Scientific Challenges Across Scales
Jun 10, 2026Tianyu Liu, Allen Xin Wang, Antonia Panescu +30AI for Science Scientific Agent Benchmarks
Yale University, CT, USA · The Pennsylvania State University, PA, USA · Northeastern University, MA, USA +7
SciResearcher: Scaling Deep Research Agents for Frontier Scientific Reasoning
May 2, 2026Tianshi Zheng, Rui Wang, Xiyun Li +5AI for Science Deep Research Agents
HKUST · CUHK · Tencent AI Lab
Autonomous Science Bench: Evaluating AI Agents for Autonomous Scientific Discovery
Oct 13, 2026Yibo Wen, Han LiuAutonomous Scientific Discovery Scientific Agent Benchmarks
Northwestern University

Jun 10, 2026J/K move · Enter open · S save

Benchmarking AI Agents for Addressing Scientific Challenges Across Scales

Tianyu Liu, Allen Xin Wang, Antonia Panescu, Lisa Xinyi Chen+29

Yale University, CT, USA · The Pennsylvania State University, PA, USA · Northeastern University, MA, USA+7

AI for Science Scientific Agent Benchmarks AI Agent Evaluation AI Agent Benchmarks Autonomous Scientific Discovery

Open paper AskPDF arXiv

AI agents are increasingly being developed to accelerate scientific discovery, yet their practical capabilities in real research settings remain poorly understood. Existing benchmarks for AI agents rarely capture the complexity, heterogeneity, and extended reasoning required by scientific work, whereas benchmarks for scientific tasks often reduce research to static, direct problems and provide limited support for interactive evaluation. Here, we introduce SciAgentArena, a systematic benchmark for evaluating AI agents in real-world scientific research scenarios drawn from emerging needs across multiple domains. SciAgentArena comprises approximately 200 tasks with stepwise verification and an interactive, agent-agnostic environment for assessing diverse AI agents. Using this benchmark, we find that current agents can contribute effectively to well-specified data-analysis workflows, particularly when the task structure and evaluation criteria are clear. However, their performance remains uneven across scientific contexts: agents struggle to generate genuinely novel insights, sustain self-directed exploration, and formulate robust solutions for open-ended research questions. We further characterize common failure modes across agents and identify opportunities for improving their reliability, autonomy, and scientific reasoning. Together, SciAgentArena provides a practical framework for measuring progress in AI agents for science and for guiding the design of future agents capable of addressing complex scientific challenges. Full codes, tasks, and datasets can be accessed via this link: https://sciagentarena.github.io/.

Abstract

AI Agents for Scientific Discovery

Explore similar work

Benchmarking AI Agents for Addressing Scientific Challenges Across Scales

SciResearcher: Scaling Deep Research Agents for Frontier Scientific Reasoning

Autonomous Science Bench: Evaluating AI Agents for Autonomous Scientific Discovery