With the advent of AI agents, automated scientific discovery is becoming an increasingly plausible goal. However, training agents to autonomously execute the engineering-heavy labor of machine learning (ML) research requires massive, process-level supervision. Existing static benchmarks omit critical intermediate steps such as debugging and incremental reasoning, and manual data collection is prohibitively expensive. To overcome this data bottleneck, we introduce ML-AutoResearch (ML-AR), a scalable pipeline for automatically generating synthetic, end-to-end ML research tasks. Each task defines a complete research cycle, including problem specification, dataset selection, baseline implementation, and iterative improvement. To ensure realism and executability, tasks are grounded in real-world datasets and refined via an automated self-debugging procedure without requiring human supervision. We construct a large-scale dataset of teacher trajectories on these synthetic tasks to train student agents via supervised fine-tuning. We evaluate the resulting agents across 3 diverse ML research benchmarks. Our comprehensive experiments across 2 distinct model families and 3 model sizes demonstrate that training on ML-AR trajectories yields consistent and significant capability gains. Fine-tuning improves the Aggregation Under the Performance (AUP) by up to 9% and substantially boosts overall pass rates, highlighting robust out-of-domain generalization.
Figures & tables
Figure 1: Generalizability of ML-AutoResearch Models. Performance comparison with the base instruction model across the MLGym and AutoResearch benchmarks.
Figure 2: Overview of the ML-AutoResearch framework . Stage 1: Generate diverse research topic ideas. Stage 2: Verify each research topic by identifying a corresponding real-world dataset and discard topics for which no suitable dataset exists. Stage 3: Generate baseline code and evaluation pipelines for agents to solve the machine learning tasks. Stage 4: Execute tasks on a computational cluster, collect interaction trajectories, and select the best-performing trajectories.
Figure 3: Demonstration of Research Topic Diversity. Overview of the diversity of topics generated during topic sampling.
GPT-5 outcome across rollouts
Tasks
Share
Consistently improves baseline
175
31.5%
Improves in some rollouts
185
33.3%
Valid, but never improves
123
22.2%
No valid solution
72
13.0%
Total verified tasks
555
100.0%
Table 1: Difficulty distribution of verified tasks. Outcomes are measured across up to 256 GPT-5 trajectories per task. The distribution spans tasks that GPT-5 improves reliably, improves only intermittently, or cannot improve or solve.
Figure 4: Dataset Statistics. Overview of the training set constructed by the ML-AutoResearch framework across different settings.
Table 6
Model
Best-run AUP
Gain over base
Qwen3-14B
1.0555
—
Original ML-AR-14B
1.0740
+1.8%
ML-AR-14B-Mixed
1.1297
+7.0%
Table 4: Effect of additional data at 14B. Seed-42 best-run AUP. ML-AR-14B-Mixed combines the original GPT-5 trajectories with additional GPT-5.4 trajectories and improves by 5.2% over the original ML-AR-14B.
Size
Model
Per-seed AUP
Mean AUP
Rel. gain
42
123
456
8B
Qwen3
1.5755
0.6331
0.8523
1.0203±0.4027
–
8B
ML-AR
1.7464
0.5522
0.8529
1.0505±0.5072
+3.0%
14B
Qwen3
1.7129
0.6218
0.9121
1.0823±0.4614
–
14B
ML-AR
1.7403
0.6353
0.9764
1.1173±0.4620
+3.2%
Table 5: Open-weight trajectory teacher. MLGym AUP by evaluation seed and aggregate performance after fine-tuning on trajectories generated by Qwen3-32B. The verified task environments are held fixed.
Figure 5: Ablation Studies Comparison. Performance comparison of ML-AutoResearch agents across different settings. Refer to A.5 for additional details.
Similarity threshold
MLGym
MLAgentBench
Combined
>0.6
6/555 (1.1%)
0/555 (0.0%)
6/555 (1.1%)
>0.7
0/555 (0.0%)
0/555 (0.0%)
0/555 (0.0%)
Table 6: Semantic train–test similarity. Number of synthetic tasks whose maximum cosine similarity to an evaluation task exceeds each threshold. Task descriptions are encoded with nvidia/llama-embed-nemotron-8b .
Figure 6: Distribution of Action Types. Comparison of the distribution of different action types across agents.
Figure 7: Analysis Results. Left: Comparison of ML-AutoResearch performance with other models based on pass rate, indicating how often agents outperform baseline results. Right: Overview of error categories across different agents.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Mean score
Per-seed score
42
123
456
Qwen3-4B
0.000±0.000
0.000
0.000
0.000
ML-AR-4B
0.101±0.100
0.103
0.000
0.200
Qwen3-8B
0.044±0.077
0.000
0.133
0.000
ML-AR-8B
0.215±0.212
0.425
0.219
0.000
Qwen3-14B
0.126±0.127
0.124
0.000
0.254
Appendix
Table 9: MLAgentBench results by evaluation seed. Mean normalized score and standard deviation across evaluation seeds 42, 123, and 456, together with each seed-level score.
Model
Qwen3
ML-AR
Rel. gain
8B
0.871±0.012
0.910±0.040
+4.4%
14B
0.904±0.025
0.940±0.035
+4.0%
Appendix
Table 10: Robustness across training seeds. Mean AUP across independently trained checkpoints using seeds 42, 1234, and 5678, with each checkpoint evaluated using three independent evaluation seeds. The 14B result uses the original ML-AR dataset rather than the mixed-data model.
Asset class
Example assets in this work
Licenses/terms used
Notes
Datasets
Hugging Face datasets used for synthetic task generation and evaluation
Autonomous research agents use language models and tools to search literature, design and run experiments, analyze results, and support scientific discovery.