DeltaML-Bench: Evaluating Machine Learning Agents on Real-World Research Repositories

Josias Moukpe; Priyanka Aryal; Matthew Kenney

cs.LGAug 20, 2026

DeltaML-Bench: Evaluating Machine Learning Agents on Real-World Research Repositories

Authors: Josias Moukpe, Priyanka Aryal, Matthew Kenney

Organizations: Algorithmic Research Group

Abstract

Autonomous agents for machine learning experimentation must navigate heterogeneous repositories, repair training pipelines, and evaluate candidate improvements under realistic compute constraints. Existing benchmarks only partially capture these conditions. We introduce DeltaML-Bench, a benchmark comprising 48 tasks sourced from research papers that require agents to improve published baselines within imperfect, open-source repositories. We evaluate GPT-5 and Claude Sonnet 4 with a standard Modular agent and a search-based ARG scaffolding. In the 4 x 6h allocation, ARG raises GPT-5's per-run success rate from 9.4% to 33.9%; in the 2 x 12h allocation, GPT-5 ARG reaches 49.0%. Modular configurations exhibit specification gaming rates as high as 47.9%, while no gaming is observed in the evaluated ARG configurations. These results indicate that scaffolding design and integrity checks are important considerations when deploying agents for autonomous ML experimentation.

Figures & tables

	4@6h		2@12h
Model	Modular	ARG	Modular	ARG
Claude Sonnet 4	24.5	30.2	22.9	19.8
GPT-5	9.4	33.9 *	15.6	49.0 *

Table 1: Success Rate (%) comparing models and agent scaffoldings across compute budgets, averaged across all 48 tasks. Bold indicates best; * indicates significantly better (Wilcoxon,

p<0.05

	4@6h		2@12h
Model	Modular	ARG	Modular	ARG
Claude Sonnet 4	3.2	6.2 *	1.8	5.4
GPT-5	2.6	10.3 *	2.7	10.1 *

Table 2: Average Normalized Score (%) comparing models and agent scaffoldings. This represents the magnitude of improvement over baselines. Bold indicates best; * indicates significantly better (Wilcoxon,

p<0.05

	4@6h		2@12h
Model	Modular	ARG	Modular	ARG
Claude Sonnet 4	33.3	0.0 *	47.9	0.0 *
GPT-5	10.9	0.0 *	9.4	0.0 *

Table 3: Integrity analysis: Specification Gaming Rate (%) comparing models and agent scaffoldings. Bold indicates best; * indicates significantly better (Wilcoxon,

p<0.05

Figure 1: Task-level effect of ARG relative to Modular. Points are mean paired differences over the 48 task-level aggregates; error bars are paired 95% bootstrap intervals from 100,000 task resamples. Positive values favor ARG for success rate and normalized score, whereas negative values favor ARG for specification gaming. The intervals describe variation across tasks and do not estimate seed-level uncertainty.

Figure 2: Concentration of normalized improvement across the 48 task-level averages. The left panel reports the share of total positive improvement contributed by the five highest-scoring tasks; the right panel reports the median task-level normalized score. Solid bars denote ARG and hatched bars denote Modular. These descriptive statistics show how strongly the means in Table 2 depend on a small subset of tasks.

	Claude Sonnet 4		GPT-5
Domain	Modular	ARG	Modular	ARG
NLP	0.0	100.0	25.0	100.0
Tabular/Other	25.0	52.8 *	25.0	58.3 *
Time Series	25.0	18.8	6.2	40.6 *
Computer Vision	27.2	26.1	6.5	23.9 *
Graph/Molecular	17.9	17.9	0.0	17.9

Table 4: Success Rate (%) by ML domain, model, and scaffolding. Configuration: 4 runs at 6 hours. Bold indicates best; * indicates significantly better (Wilcoxon,

p<0.05

	Claude Sonnet 4		GPT-5
Domain	Modular	ARG	Modular	ARG
NLP	100.0	0.0	0.0	0.0
Tabular/Other	30.6	0.0 *	8.3	0.0
Time Series	25.0	0.0	12.5	0.0
Computer Vision	34.8	0.0 *	10.9	0.0 *
Graph/Molecular	32.1	0.0 *	14.3	0.0

Table 5: Specification Gaming Rate (%) by ML domain, model, and scaffolding. Configuration: 4 runs at 6 hours. Bold indicates best; * indicates significantly better (Wilcoxon,

p<0.05

Figure 3: Depth versus breadth under equal scheduled compute. Both allocations provide 24 agent-hours per task, but 4 $\times$ 6h uses four attempts whereas 2 $\times$ 12h uses two. The left panel shows per-run success; the right shows observed task coverage (at least one successful run). Because duration and attempt count change together, this is a descriptive comparison and does not identify the causal effect of a longer run.

	4@6h		2@12h
Model	Modular	ARG	Modular	ARG
Claude Sonnet 4	1.2	1.7	1.6	0.8 *
GPT-5	4.2	2.4 *	6.2	3.4 *

Table 6: Efficiency analysis: Average time usage (hours), averaged across all 48 tasks. Bold indicates best (lower is more efficient); * indicates significantly better (Wilcoxon,

p<0.05

	4@6h		2@12h
Model	Modular	ARG	Modular	ARG
Claude Sonnet 4	1.1	9.0	1.2	2.9
GPT-5	11.2	10.3	13.8	16.1

Table 7: Efficiency analysis: Average token usage (millions), averaged across all 48 tasks. Bold indicates best (lower is more efficient); * indicates significantly better (Wilcoxon,

p<0.05

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

		4 runs at 6h				2 runs at 12h
		Claude Sonnet 4		GPT-5		Claude Sonnet 4		GPT-5
#	Task ID	Mod.	ARG	Mod.	ARG	Mod.	ARG	Mod.	ARG
1	pwc_5_datasets_code_cl	0	25	0	25	0	0	0	50
2	pwc_astock_srl_factors	50	75	75	100	50	50	100	100
3	pwc_btad_urd	100	0	0	0	0	0	0	0
4	pwc_california_housing_binary_diffusion	0	75	75	100	100	100	50	100

Appendix

Table 8: Detailed Success Rate (%) per task across models and scaffolding. Results averaged over 4 runs at 6h and 2 runs at 12h.

		4 runs at 6h				2 runs at 12h
		Claude Sonnet 4		GPT-5		Claude Sonnet 4		GPT-5
#	Task ID	Mod.	ARG	Mod.	ARG	Mod.	ARG	Mod.	ARG
1	pwc_5_datasets_code_cl	0	0	0	0	100	0	0	0
2	pwc_astock_srl_factors	50	0	0	0	50	0	0	0
3	pwc_btad_urd	100	0	0	0	100	0	0	0
4	pwc_california_housing_binary_diffusion	0	0	25	0	0	0	0	0

Appendix

Table 9: Detailed Specification Gaming Rate (%) per task across models and scaffolding. Results averaged over 4 runs at 6h and 2 runs at 12h.

		4 runs at 6h				2 runs at 12h
		Claude Sonnet 4		GPT-5		Claude Sonnet 4		GPT-5
#	Task ID	Mod.	ARG	Mod.	ARG	Mod.	ARG	Mod.	ARG
1	pwc_5_datasets_code_cl	0.0	0.665	0.0	0.477	0.0	0.0	0.0	0.954
2	pwc_astock_srl_factors	10.360	16.118	13.476	42.587	2.916	5.342	8.599	43.896
3	pwc_btad_urd	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
4	pwc_california_housing_binary_diffusion	34.572	27.017	25.929	37.088	34.572	37.123	17.286	36.690

Appendix

Table 10: Detailed Average Score per task across models and scaffolding. Results averaged over 4 runs at 6h and 2 runs at 12h.

		4 runs at 6h				2 runs at 12h
		Claude Sonnet 4		GPT-5		Claude Sonnet 4		GPT-5
#	Task ID	Mod.	ARG	Mod.	ARG	Mod.	ARG	Mod.	ARG
1	pwc_5_datasets_code_cl	1.780	0.613	5.430	0.568	2.150	0.685	6.690	0.520
2	pwc_astock_srl_factors	0.688	0.523	0.525	0.520	0.790	0.515	0.820	0.530
3	pwc_btad_urd	0.903	0.505	4.380	0.513	2.185	0.505	6.685	0.585
4	pwc_california_housing_binary_diffusion	0.723	0.895	3.318	0.520	2.015	0.545	3.715	0.690

Appendix

Table 11: Detailed Average Time per task across models and scaffolding (hours). Results averaged over 4 runs at 6h and 2 runs at 12h.

		4 runs at 6h				2 runs at 12h
		Claude Sonnet 4		GPT-5		Claude Sonnet 4		GPT-5
#	Task ID	Mod.	ARG	Mod.	ARG	Mod.	ARG	Mod.	ARG
1	pwc_5_datasets_code_cl	1.93	6.51	10.32	1.89	1.47	2.96	24.56	1.76
2	pwc_astock_srl_factors	0.48	8.97	2.04	2.60	0.65	7.55	1.12	6.42
3	pwc_btad_urd	1.02	7.57	17.26	2.70	0.94	7.22	22.62	2.05
4	pwc_california_housing_binary_diffusion	1.36	2.23	10.80	3.67	1.71	2.81	13.73	2.70

Appendix

Table 12: Detailed Average Tokens used per task across models and scaffolding (millions).

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,494 papers

ResearchGym: Evaluating Language Model Agents on Real-World AI Research
Feb 16, 2026Aniketh Garikaparthi, Manasi Patwardhan, Arman CohanAI Agent Benchmarks Autonomous Research Agents
TCS Research · Yale University
ML-AutoResearch: Training Machine Learning Research Agents with Automatically Generated Environments
Mar 17, 2026Ziyang Cai, Amir Saeidi, Harkirat BehlAutonomous Research Agents LLM Agent Training
Arizona State University · Microsoft Research
FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics
May 17, 2026Qiran Zou, Hou Hei Lam, Wenhao Zhao +11AI Agents for Scientific Discovery Inference-Time Search
National University of Singapore · Tsinghua University · University of Minnesota +2

Feb 16, 2026J/K move · Enter open · S save

ResearchGym: Evaluating Language Model Agents on Real-World AI Research

Aniketh Garikaparthi, Manasi Patwardhan, Arman Cohan

TCS Research · Yale University

AI Agent Benchmarks Autonomous Research Agents Automated Agent Evaluation Long-Horizon LLM Agents AI Agent Reliability

Open paper AskPDF arXiv

We introduce ResearchGym, a benchmark and execution environment for evaluating AI agents on end-to-end research. To instantiate this, we repurpose five oral and spotlight papers from ICML, ICLR, and ACL. From each paper's repository, we preserve the datasets, evaluation harness, and baseline implementations but withhold the paper's proposed method. This results in five containerized task environments comprising 39 sub-tasks in total. Within each environment, agents must propose novel hypotheses, run experiments, and attempt to surpass strong human baselines on the paper's metrics. In a controlled evaluation of an agent powered by GPT-5, we observe a sharp capability--reliability gap. The agent improves over the provided baselines from the repository in just 1 of 15 evaluations (6.7%) by 11.5%, and completes only 26.5% of sub-tasks on average. We identify recurring long-horizon failure modes, including impatience, poor time and resource management, overconfidence in weak hypotheses, difficulty coordinating parallel experiments, and hard limits from context length. Yet in a single run, the agent surpasses the solution of an ICML 2025 Spotlight task, indicating that frontier agents can occasionally reach state-of-the-art performance, but do so unreliably. We additionally evaluate proprietary agent scaffolds including Claude Code (Opus-4.5) and Codex (GPT-5.2) which display a similar gap. ResearchGym provides infrastructure for systematic evaluation and analysis of autonomous agents on closed-loop research.

Abstract

AI Agent Benchmarks

Explore similar work

ResearchGym: Evaluating Language Model Agents on Real-World AI Research

ML-AutoResearch: Training Machine Learning Research Agents with Automatically Generated Environments

FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics