cs.AIMar 31, 2026
SaveTowards End-to-End Automation of AI Research
Organizations: Sakana AI, Tokyo, Japan · University of British Columbia, Vancouver, Canada · Vector Institute, Toronto, Canada · FLAIR, University of Oxford, Oxford, UK
Abstract
The automation of science is a long-standing ambition in the field of AI. While the community has made significant progress in automating individual components of the scientific process, a system that autonomously navigates the entire research lifecycle -- from conception to publication -- has remained out of reach. Here, we present the strongest demonstration to date toward automating the entire process end-to-end. We present The AI Scientist, which creates research ideas, writes code, runs experiments, plots and analyzes data, writes the entire scientific manuscript and performs its own peer review. Its ideas, execution, and presentation are of sufficient quality to produce a manuscript generated by an AI system that passes the first round of peer review at a major machine learning conference workshop. The workshop has an acceptance rate of 70 percent. Our system leverages modern foundation models within a complex agentic system. We evaluate The AI Scientist in two settings: a focused mode using human-provided code templates as an initial scaffold to conduct research on a specific topic, and a template-free, open-ended mode that leverages agentic search for wider scientific exploration. Both settings produce diverse ideas and automatically test, report on, and evaluate them. This achievement demonstrates AI's growing capacity for scientific contribution and signifies a potential paradigm shift in how research is conducted. As with any impactful new technology, there could be significant risks, including taxing overwhelmed review systems and adding noise to scientific literature. However, if developed responsibly, such autonomous systems could greatly accelerate scientific discovery.
Figures & tables
Figure 1: The AI Scientist Workflow . A . The AI Scientist consists of distinct phases covering automated idea generation, tree-based experimentation, manuscript writing, and reviewing. The experimentation phase employs agentic tree search to generate and refine code implementations. This is structured into four stages: (1) initial investigation, (2) hyperparameter tuning, (3) research agenda execution, and (4) ablation studies. From one experimentation stage to the next, the best-performing checkpoint is selected to seed the next stage of tree search. B . Paper quality consistently improves with the underlying model release date (as judged by the Automated Reviewer), suggesting consistent future improvements with improving foundation models. The observed correlation is statistically significant ( -value ). Shaded regions represent the standard error. Full experimental details, including model versions and replication counts, are provided in Supplementary Section A.2.9 . C . The Automated Reviewer achieves performance comparable to human reviewers validated by openly available decisions from past conferences ( Table 1 ). Error bars represent the 95% bootstrapped confidence intervals.
| Reviewer | Balanced Acc. ( ) | Accuracy ( ) | F1 Score ( ) | AUC ( ) | FPR ( ) | FNR ( ) |
| Human (NeurIPS) | 0.66 | 0.73 | 0.49 | 0.65 | 0.17 | 0.52 |
| Years prior to knowledge cutoff ( ) | ||||||
| Random Decision | 0.50 | 0.54 | 0.47 | 0.52 | 0.47 | 0.43 |
| Always Reject | 0.50 | 0.65 | 0.00 | 0.50 | 0.00 | 1.00 |
| Automated Reviewer | 0.69 0.04 | 0.65 0.10 | 0.62 0.09 | 0.69 0.09 | 0.45 0.10 | 0.17 0.08 |
| Year after knowledge cutoff (2025) | ||||||
Table 1: Performance comparison of human reviewers (NeurIPS 2021 consistency experiment ( Beygelzimer et al., 2021 ) ) and the Automated Reviewer, evaluated on papers published prior to ( ) and after (2025) the knowledge cutoff. The Automated Reviewer achieves performance superior or comparable to human reviewer consistency in key metrics such as F1 Score, AUC, and Balanced Accuracy, even for data beyond the knowledge cutoff, highlighting its robustness and reliability across different time periods. Error margins denote the 95% bootstrapped confidence intervals. Arrows indicate if it is better for a score to be higher ( ) or lower ( ). Supplementary Section A.3.2 explains each metric and comparison in detail.
Figure 2: A . The research experimentation phase is visualized as a four-stage process. A preliminary baseline code implementation is first constructed (Stage 1) and refined by tuning hyperparameters (Stage 2). The resultant code serves as a starting point for executing the research agenda via agentic tree search (Stage 3), followed by ablation experiments (Stage 4). Full details of the agentic tree search process are provided in the methods. B . A real example of The AI Scientist’s tree search with node annotations outlining the conducted experiments across the four different stages. C . Scaling the number of experimentation nodes in the agentic tree search shows improvements for deeper test-time search budgets. Error bars represent the standard error. Full experimental details are provided in Supplementary Section A.2.9 .
Figure 3: Selected sections from a paper generated by The AI Scientist that was accepted via peer review at a top-tier machine learning conference workshop. This paper received peer review scores of 6 (weak accept), 7 (accept), and 6 (weak accept) before meta-review and ranked among the top 45% of submitted workshop papers, marking the first instance of a fully AI-generated paper successfully navigating peer review at a top-tier conference workshop. The AI Scientist wrote the entire paper, from the title through to the references and supplementary information. Shown are example sections of the paper it produced, namely portions of the abstract, methods, results (including data visualizations), and references. The full paper contains other typical paper sections, such as an introduction, a motivation, and a conclusion. The paper format follows the specific template provided by the workshop. The complete paper is available in Supplementary Section C.2 .
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| Category | Hyperparameter | Value |
| Idea Generation | Number of Idea Reflections | 3 |
| Number of Novelty Search Rounds (Semantic Scholar) | 10 | |
| Experiment Execution | Max Experiments | 5 |
| Max Experiment Re-attempts (on error) | 4 | |
| Experiment Timeout | 7200 seconds | |
| Plotting Timeout | 600 seconds |
Table 2: Key Hyperparameters for the Template-Based version of The AI Scientist.
| Component/Task | Model Used | Max Tokens | Temperature |
| Code Generation | Claude 3.5 Sonnet | 8,192 | 0.5 |
| Code Generation | Claude Sonnet 4 | 20,000 | 1.0 |
| LLM/VLM Feedback Agent | GPT-4o | 8,192 | 0.5 |
| Summary Report Agent | GPT-4o | 8,192 | 1.0 |
| Code Critic | o3 | 100,000 | 1.0 |
Table 3: LLM and VLM Models and Parameters for the template-free version of The AI Scientist.
| Reviewer | Balanced Acc. ( ) | Accuracy ( ) | F1 Score ( ) | AUC ( ) | FPR ( ) | FNR ( ) |
| Human (NeurIPS) 1 | 0.66 | 0.73 | 0.49 | 0.65 | 0.17 | 0.52 |
| Random Decision | 0.50 | 0.54 | 0.47 | 0.52 | 0.47 | 0.43 |
| Always Reject | 0.50 | 0.65 | 0.00 | 0.50 | 0.00 | 1.00 |
| Ablation with different models | ||||||
| Claude 3.5 Sonnet | 0.63 0.12 | 0.50 0.10 | 0.57 0.09 | 0.61 0.10 | 0.74 0.08 | 0.03 0.04 |
| GPT-4o | 0.54 0.12 | 0.68 0.09 | 0.20 0.08 | 0.54 0.10 | 0.03 0.04 | 0.88 0.06 |
Table 5: Ablation results for Automated Reviewer showing how the choice of the underlying LLM, prompt variants, and auxiliary techniques (VLM, Reflexion, few-shot examples, ensembling) affect six evaluation metrics: balanced accuracy, accuracy, F1 score, AUC, false positive rate (FPR), and false negative rate (FNR). The combination of o4-mini with a 5-ensemble meta-review achieves the best overall performance.
| Final Paper Selected | Topic Area | # Ideas | # Experiments | # Write-Ups |
| Compositional Regularization | Core ML | 25* 1 | 8 1 | 24 1 |
| Label Noise Calibration | Core ML | 25* 1 | 6 1 | 7 1 |
| Pest Detection | Applied ML | 136 1 | 16 1 | 6 1 |
Table 6: Generation and Write-up Statistics. *Rows 1 and 2 share the same pool of 25 generated ideas. The symbol denotes “Generated Selected” for the number of ideas, experiments, and write-ups.
Figure 4: Examples paper writeup failures: (a) citation errors, (b) figure display errors, (c) violations of paper length expectations (the workshop limits papers to four pages), and (d) duplicated figures between the main text and appendix.
Figure 5: Distribution of automated review scores for papers generated by the template-based version of The AI Scientist. Violin plots show the distribution of overall scores assigned by the Automated Reviewer to papers generated across three domains (Diffusion Modeling, Language Modeling and Grokking Analysis) and four foundation models. Scores on the y-axis refer to NeurIPS ratings ( NeurIPS, 2022 ) , which range from 1 (Very Strong Reject) to 5 (Borderline Accept). Although there is no exact mapping between conference and workshop scores, papers rated 4 ((Borderline Reject)) in conference reviews are generally considered borderline-eligible for workshop acceptance.
| LLM | Total | Compiled | Mean | Max | Total |
| Ideas | Papers | Score | Score | Cost ($) | |
| Claude Sonnet 3.5 | 51 | 37 | 2.97 | 4.0 | 250 |
| GPT-4o | 51 | 16 | 2.81 | 3.0 | 300 |
| DeepSeek Coder | 51 | 28 | 2.61 | 3.0 | 10 |
| Llama-3.1 405B | 51 | 20 | 1.3 | 2.0 | 120 |
Table 7: Performance of the template-based version of The AI Scientist for Diffusion Modeling experiments. Metrics include total ideas generated, completed papers, mean and max automated review scores (NeurIPS scale), and approximate total cost in USD.
| LLM | Total | Compiled | Mean | Max | Total |
| Ideas | Papers | Score | Score | Cost ($) | |
| Claude Sonnet 3.5 | 52 | 19 | 2.89 | 3.0 | 250 |
| GPT-4o | 52 | 15 | 2.60 | 3.0 | 300 |
| DeepSeek Coder | 52 | 19 | 2.63 | 3.0 | 10 |
| Llama-3.1 405B | 52 | 19 | 1.16 | 2.0 | 120 |
Table 8: Performance of the template-based version of The AI Scientist for Language Modeling experiments. Metrics as in Table 7 .
| LLM | Total | Compiled | Mean | Max | Total |
| Ideas | Papers | Score | Score | Cost ($) | |
| Claude Sonnet 3.5 | 51 | 25 | 3.4 | 5.0 | 250 |
| GPT-4o | 51 | 12 | 2.67 | 3.0 | 300 |
| DeepSeek Coder | 51 | 32 | 2.81 | 4.0 | 10 |
| Llama-3.1 405B | 51 | 27 | 1.11 | 2.0 | 120 |
Table 9: Performance of the template-based version of The AI Scientist for Grokking Analysis experiments. Metrics as in Table 7 .
Figure 6: Preview of the “Adaptive Dual-Scale Denoising” paper which was entirely autonomously generated by the template-based version of The AI Scientist. The full paper can be viewed in Section D.1.1 .
| Title | Workshop Result | Materials |
| Compositional Regularization: Unexpected Obstacles in Enhancing Neural Network Generalization | Accepted (Score: 6.33) | See Section D.2.1 , GitHub Repository |
| Unveiling the Impact of Label Noise on Model Calibration in Deep Learning | Rejected | See Section D.2.2 , GitHub Repository |
| Real-world Challenges in Pest Detection using Deep Learning: an Investigation into Failures and Solutions | Rejected | See Section D.2.3 , GitHub Repository |
Table 10: Overview of AI-Generated Workshop Submissions.
Figure 7: Example of the data-generating function used in the experiments.
Figure 8: The generated model class shows an embedding layer, a single LSTM layer, and a linear layer head.
Figure 9: The generated training loop shows the loss function as well as the proposed regularization.
Figure 10: The generated accuracy calculation function uses regression to match an output with a ground truth.
Figure 11: Temperature scaling implementation.
Figure 12: Noisy dataset class implementation.
Figure 13: The implementation for Expected Calibration Error.
Figure 14: Domain discriminator and multi-dataset training loop.
Figure 15: Environment noise simulation implementation.
Figure 16: Environmental Robustness Score calculation.