cs.AIMar 17, 2026

ML-AutoResearch: Training Machine Learning Research Agents with Automatically Generated Environments

Authors: Ziyang Cai, Amir Saeidi, Harkirat Behl

Organizations: Arizona State University · Microsoft Research

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

With the advent of AI agents, automated scientific discovery is becoming an increasingly plausible goal. However, training agents to autonomously execute the engineering-heavy labor of machine learning (ML) research requires massive, process-level supervision. Existing static benchmarks omit critical intermediate steps such as debugging and incremental reasoning, and manual data collection is prohibitively expensive. To overcome this data bottleneck, we introduce ML-AutoResearch (ML-AR), a scalable pipeline for automatically generating synthetic, end-to-end ML research tasks. Each task defines a complete research cycle, including problem specification, dataset selection, baseline implementation, and iterative improvement. To ensure realism and executability, tasks are grounded in real-world datasets and refined via an automated self-debugging procedure without requiring human supervision. We construct a large-scale dataset of teacher trajectories on these synthetic tasks to train student agents via supervised fine-tuning. We evaluate the resulting agents across 3 diverse ML research benchmarks. Our comprehensive experiments across 2 distinct model families and 3 model sizes demonstrate that training on ML-AR trajectories yields consistent and significant capability gains. Fine-tuning improves the Aggregation Under the Performance (AUP) by up to 9% and substantially boosts overall pass rates, highlighting robust out-of-domain generalization.

Figures & tables

Figure 1: Generalizability of ML-AutoResearch Models. Performance comparison with the base instruction model across the MLGym and AutoResearch benchmarks.
Figure 2: Overview of the ML-AutoResearch framework . Stage 1: Generate diverse research topic ideas. Stage 2: Verify each research topic by identifying a corresponding real-world dataset and discard topics for which no suitable dataset exists. Stage 3: Generate baseline code and evaluation pipelines for agents to solve the machine learning tasks. Stage 4: Execute tasks on a computational cluster, collect interaction trajectories, and select the best-performing trajectories.
Figure 3: Demonstration of Research Topic Diversity. Overview of the diversity of topics generated during topic sampling.
GPT-5 outcome across rolloutsTasksShare
Consistently improves baseline17531.5%
Improves in some rollouts18533.3%
Valid, but never improves12322.2%
No valid solution7213.0%
Total verified tasks555100.0%
Table 1: Difficulty distribution of verified tasks. Outcomes are measured across up to 256 GPT-5 trajectories per task. The distribution spans tasks that GPT-5 improves reliably, improves only intermittently, or cannot improve or solve.
Figure 4: Dataset Statistics. Overview of the training set constructed by the ML-AutoResearch framework across different settings.
Table 6
ModelBest-run AUPGain over base
Qwen3-14B1.0555—
Original ML-AR-14B1.0740+1.8%+1.8\%
ML-AR-14B-Mixed1.1297+7.0%+7.0\%
Table 4: Effect of additional data at 14B. Seed-42 best-run AUP. ML-AR-14B-Mixed combines the original GPT-5 trajectories with additional GPT-5.4 trajectories and improves by 5.2% over the original ML-AR-14B.
SizeModelPer-seed AUPMean AUPRel. gain
  42123456  
8BQwen31.57550.63310.85231.0203±0.40271.0203_{\scriptscriptstyle\pm 0.4027}–
8BML-AR1.74640.55220.85291.0505±0.50721.0505_{\scriptscriptstyle\pm 0.5072}+3.0%+3.0\%
14BQwen31.71290.62180.91211.0823±0.46141.0823_{\scriptscriptstyle\pm 0.4614}–
14BML-AR1.74030.63530.97641.1173±0.46201.1173_{\scriptscriptstyle\pm 0.4620}+3.2%+3.2\%
Table 5: Open-weight trajectory teacher. MLGym AUP by evaluation seed and aggregate performance after fine-tuning on trajectories generated by Qwen3-32B. The verified task environments are held fixed.
Figure 5: Ablation Studies Comparison. Performance comparison of ML-AutoResearch agents across different settings. Refer to A.5 for additional details.
Similarity thresholdMLGymMLAgentBenchCombined
>0.6>0.66/555 (1.1%)0/555 (0.0%)6/555 (1.1%)
>0.7>0.70/555 (0.0%)0/555 (0.0%)0/555 (0.0%)
Table 6: Semantic train–test similarity. Number of synthetic tasks whose maximum cosine similarity to an evaluation task exceeds each threshold. Task descriptions are encoded with nvidia/llama-embed-nemotron-8b .
Figure 6: Distribution of Action Types. Comparison of the distribution of different action types across agents.
Figure 7: Analysis Results. Left: Comparison of ML-AutoResearch performance with other models based on pass rate, indicating how often agents outperform baseline results. Right: Overview of error categories across different agents.
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

ModelMean scorePer-seed score
  42123456
Qwen3-4B0.000±0.0000.000_{\scriptscriptstyle\pm 0.000}0.0000.0000.000
ML-AR-4B0.101±0.1000.101_{\scriptscriptstyle\pm 0.100}0.1030.0000.200
Qwen3-8B0.044±0.0770.044_{\scriptscriptstyle\pm 0.077}0.0000.1330.000
ML-AR-8B0.215±0.2120.215_{\scriptscriptstyle\pm 0.212}0.4250.2190.000
Qwen3-14B0.126±0.1270.126_{\scriptscriptstyle\pm 0.127}0.1240.0000.254
Appendix
Table 9: MLAgentBench results by evaluation seed. Mean normalized score and standard deviation across evaluation seeds 42, 123, and 456, together with each seed-level score.
ModelQwen3ML-ARRel. gain
8B0.871±0.0120.871_{\scriptscriptstyle\pm 0.012}0.910±0.0400.910_{\scriptscriptstyle\pm 0.040}+4.4%+4.4\%
14B0.904±0.0250.904_{\scriptscriptstyle\pm 0.025}0.940±0.0350.940_{\scriptscriptstyle\pm 0.035}+4.0%+4.0\%
Appendix
Table 10: Robustness across training seeds. Mean AUP across independently trained checkpoints using seeds 42, 1234, and 5678, with each checkpoint evaluated using three independent evaluation seeds. The 14B result uses the original ML-AR dataset rather than the mixed-data model.
Asset classExample assets in this workLicenses/terms usedNotes
DatasetsHugging Face datasets used for synthetic task generation and evaluationApache-2.0, MIT, CC0-1.0, CC-BY-4.0, CC-BY-SA-4.0, CC-BY-SA-3.0, CC-BY-SA-3.0+GFDL, BSD/BSD-3-Clause, CC-BY-NC-SA-4.0, CC-BY-NC-ND-4.0, AFL-3.0, ODC-BY, C-UDAPer-dataset terms are respected and attributed
Open modelsQwen family student modelsQwen model licenses/terms from the corresponding model cardsVersion-specific model terms apply
API modelsGPT teacher/evaluator APIsOpenAI API Terms of Use and associated service policiesAPI use only; no model weight redistribution
Benchmarks/codebasesMLGym, MLAgentBench, AutoResearch benchmark codeRepository-specific OSS licenses (including MIT/Apache-2.0/BSD-style where applicable)We follow each repository’s LICENSE file
Agent framework/toolsSWE-agent and related toolingRepository-specific OSS licenses (including MIT/Apache-2.0/BSD-style where applicable)We follow each repository’s LICENSE file
Appendix
Table 16: Licenses and terms of use for external assets used in this work.
ItemSetting
Base modelsQwen3-4B, Qwen3-8B, Qwen3-14B
TokenizerMatching Qwen tokenizer for each base model
OptimizerAdamW
Learning rate1.0×10−51.0\times 10^{-5} (4B), 5.0×10−65.0\times 10^{-6} (8B), 2.0×10−52.0\times 10^{-5} (14B) on H100 configs
LR scheduleWarmupDecayLR with linear warmup; warmup steps 100 (4B/8B), 200 (14B)
Weight decay1.0×10−41.0\times 10^{-4} (4B/8B), 1.0×10−11.0\times 10^{-1} (14B config)
Appendix
Table 17: Training and data-construction settings used by the SFT pipeline.
More in this topic

Autonomous Research Agents

Autonomous research agents use language models and tools to search literature, design and run experiments, analyze results, and support scientific discovery.

See all 318 papers

Explore similar work

CardsList
Aug 20, 2026

DeltaML-Bench: Evaluating Machine Learning Agents on Real-World Research Repositories

Autonomous agents for machine learning experimentation must navigate heterogeneous repositories, repair training pipelines, and evaluate candidate improvements under realistic compute constraints. Existing benchmarks only partially capture these conditions. We introduce DeltaML-Bench, a benchmark comprising 48 tasks sourced from research papers that require agents to improve published baselines within imperfect, open-source repositories. We evaluate GPT-5 and Claude Sonnet 4 with a standard Modular agent and a search-based ARG scaffolding. In the 4 x 6h allocation, ARG raises GPT-5's per-run success rate from 9.4% to 33.9%; in the 2 x 12h allocation, GPT-5 ARG reaches 49.0%. Modular configurations exhibit specification gaming rates as high as 47.9%, while no gaming is observed in the evaluated ARG configurations. These results indicate that scaffolding design and integrity checks are important considerations when deploying agents for autonomous ML experimentation.

Josias Moukpe, Priyanka Aryal, Matthew Kenney

Algorithmic Research Group

AI Agent BenchmarksLLM Agent EvaluationAutonomous ExperimentationAutonomous Research AgentsAI Agents for Scientific Discovery
AskPDF
Feb 16, 2026

ResearchGym: Evaluating Language Model Agents on Real-World AI Research

We introduce ResearchGym, a benchmark and execution environment for evaluating AI agents on end-to-end research. To instantiate this, we repurpose five oral and spotlight papers from ICML, ICLR, and ACL. From each paper's repository, we preserve the datasets, evaluation harness, and baseline implementations but withhold the paper's proposed method. This results in five containerized task environments comprising 39 sub-tasks in total. Within each environment, agents must propose novel hypotheses, run experiments, and attempt to surpass strong human baselines on the paper's metrics. In a controlled evaluation of an agent powered by GPT-5, we observe a sharp capability--reliability gap. The agent improves over the provided baselines from the repository in just 1 of 15 evaluations (6.7%) by 11.5%, and completes only 26.5% of sub-tasks on average. We identify recurring long-horizon failure modes, including impatience, poor time and resource management, overconfidence in weak hypotheses, difficulty coordinating parallel experiments, and hard limits from context length. Yet in a single run, the agent surpasses the solution of an ICML 2025 Spotlight task, indicating that frontier agents can occasionally reach state-of-the-art performance, but do so unreliably. We additionally evaluate proprietary agent scaffolds including Claude Code (Opus-4.5) and Codex (GPT-5.2) which display a similar gap. ResearchGym provides infrastructure for systematic evaluation and analysis of autonomous agents on closed-loop research.

Aniketh Garikaparthi, Manasi Patwardhan, Arman Cohan

TCS Research · Yale University

AI Agent BenchmarksAutonomous Research AgentsAutomated Agent EvaluationLong-Horizon LLM AgentsAI Agent Reliability
AskPDF
Jul 19, 2026

Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer

Auto Research uses language-model agents to propose, implement, and evaluate machine-learning changes in a closed loop, but is usually judged by its terminal pipeline. A terminal score cannot reveal which technical decision produced a gain or distinguish a reusable discovery from a change adapted to development feedback. We introduce intervention-centered Auto Research, which validates research decisions rather than only final artifacts and makes their reliability measurable. Feature, Model, Representation, and Data axes are searched independently with inner five-fold feedback. Each axis winner is frozen before an outer-holdout matrix compares all alternatives on evidence the loop never sees. Across 701 agent-executed attempts spanning ten Matbench endpoints, outer evidence confirms the selected intervention on nine of ten endpoints and preserves 89.3% of non-tied intervention orderings. It also rejects an aggregate Representation gain that inner feedback endorsed. The resulting matrix reveals an information-dependent hierarchy. Composition-only tasks support several routes to improvement, whereas structure-informed tasks favor local geometry features and complementary tree ensembles. A subsequent compatibility test combines already frozen Feature and Model code without further search or tuning and raises mean outer-holdout improvement from 19.0% to 26.3%. By validating decisions rather than only artifacts, this design turns adaptive search into reusable evidence wherever agents propose executable alternatives against a fixed evaluator.

Jingjie Ning, Xiaochuan Li, Shanshan Zhong +2

School of Computer Science, Carnegie Mellon University · 2DP Technology

Automated Algorithm DiscoveryAutomated EvaluationAutonomous Research Agents
AskPDF