Towards End-to-End Automation of AI Research

Yutaro Yamada; Robert Tjarko Lange; Cong Lu; Chris Lu; Shengran Hu; Jakob Foerster; David Ha; Jeff Clune

cs.AIMar 31, 2026

Towards End-to-End Automation of AI Research

Authors: Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Chris Lu, Shengran Hu, Jakob Foerster, David Ha, Jeff Clune

Organizations: Sakana AI, Tokyo, Japan · University of British Columbia, Vancouver, Canada · Vector Institute, Toronto, Canada · FLAIR, University of Oxford, Oxford, UK

Abstract

The automation of science is a long-standing ambition in the field of AI. While the community has made significant progress in automating individual components of the scientific process, a system that autonomously navigates the entire research lifecycle -- from conception to publication -- has remained out of reach. Here, we present the strongest demonstration to date toward automating the entire process end-to-end. We present The AI Scientist, which creates research ideas, writes code, runs experiments, plots and analyzes data, writes the entire scientific manuscript and performs its own peer review. Its ideas, execution, and presentation are of sufficient quality to produce a manuscript generated by an AI system that passes the first round of peer review at a major machine learning conference workshop. The workshop has an acceptance rate of 70 percent. Our system leverages modern foundation models within a complex agentic system. We evaluate The AI Scientist in two settings: a focused mode using human-provided code templates as an initial scaffold to conduct research on a specific topic, and a template-free, open-ended mode that leverages agentic search for wider scientific exploration. Both settings produce diverse ideas and automatically test, report on, and evaluate them. This achievement demonstrates AI's growing capacity for scientific contribution and signifies a potential paradigm shift in how research is conducted. As with any impactful new technology, there could be significant risks, including taxing overwhelmed review systems and adding noise to scientific literature. However, if developed responsibly, such autonomous systems could greatly accelerate scientific discovery.

Figures & tables

Figure 1: The AI Scientist Workflow . A . The AI Scientist consists of distinct phases covering automated idea generation, tree-based experimentation, manuscript writing, and reviewing. The experimentation phase employs agentic tree search to generate and refine code implementations. This is structured into four stages: (1) initial investigation, (2) hyperparameter tuning, (3) research agenda execution, and (4) ablation studies. From one experimentation stage to the next, the best-performing checkpoint is selected to seed the next stage of tree search. B . Paper quality consistently improves with the underlying model release date (as judged by the Automated Reviewer), suggesting consistent future improvements with improving foundation models. The observed correlation is statistically significant ( $p$ -value $<0.00001$ ). Shaded regions represent the standard error. Full experimental details, including model versions and replication counts, are provided in Supplementary Section A.2.9 . C . The Automated Reviewer achieves performance comparable to human reviewers validated by openly available decisions from past conferences ( Table 1 ). Error bars represent the 95% bootstrapped confidence intervals.

Reviewer	Balanced Acc. ( $\uparrow$ )	Accuracy ( $\uparrow$ )	F1 Score ( $\uparrow$ )	AUC ( $\uparrow$ )	FPR ( $\downarrow$ )	FNR ( $\downarrow$ )
Human (NeurIPS)	0.66	0.73	0.49	0.65	0.17	0.52
Years prior to knowledge cutoff ( $2017-2024$ )
Random Decision	0.50	0.54	0.47	0.52	0.47	0.43
Always Reject	0.50	0.65	0.00	0.50	0.00	1.00
Automated Reviewer	0.69 $\pm$ 0.04	0.65 $\pm$ 0.10	0.62 $\pm$ 0.09	0.69 $\pm$ 0.09	0.45 $\pm$ 0.10	0.17 $\pm$ 0.08
Year after knowledge cutoff (2025)

Table 1: Performance comparison of human reviewers (NeurIPS 2021 consistency experiment ( Beygelzimer et al., 2021 ) ) and the Automated Reviewer, evaluated on papers published prior to (

2017-2024

) and after (2025) the knowledge cutoff. The Automated Reviewer achieves performance superior or comparable to human reviewer consistency in key metrics such as F1 Score, AUC, and Balanced Accuracy, even for data beyond the knowledge cutoff, highlighting its robustness and reliability across different time periods. Error margins denote the 95% bootstrapped confidence intervals. Arrows indicate if it is better for a score to be higher (

\uparrow

) or lower (

\downarrow

). Supplementary Section A.3.2 explains each metric and comparison in detail.

Figure 2: A . The research experimentation phase is visualized as a four-stage process. A preliminary baseline code implementation is first constructed (Stage 1) and refined by tuning hyperparameters (Stage 2). The resultant code serves as a starting point for executing the research agenda via agentic tree search (Stage 3), followed by ablation experiments (Stage 4). Full details of the agentic tree search process are provided in the methods. B . A real example of The AI Scientist’s tree search with node annotations outlining the conducted experiments across the four different stages. C . Scaling the number of experimentation nodes in the agentic tree search shows improvements for deeper test-time search budgets. Error bars represent the standard error. Full experimental details are provided in Supplementary Section A.2.9 .

Figure 3: Selected sections from a paper generated by The AI Scientist that was accepted via peer review at a top-tier machine learning conference workshop. This paper received peer review scores of 6 (weak accept), 7 (accept), and 6 (weak accept) before meta-review and ranked among the top 45% of submitted workshop papers, marking the first instance of a fully AI-generated paper successfully navigating peer review at a top-tier conference workshop. The AI Scientist wrote the entire paper, from the title through to the references and supplementary information. Shown are example sections of the paper it produced, namely portions of the abstract, methods, results (including data visualizations), and references. The full paper contains other typical paper sections, such as an introduction, a motivation, and a conclusion. The paper format follows the specific template provided by the workshop. The complete paper is available in Supplementary Section C.2 .

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Hyperparameter	Value
Idea Generation	Number of Idea Reflections	3
Idea Generation	Number of Novelty Search Rounds (Semantic Scholar)	10
Experiment Execution	Max Experiments	5
	Max Experiment Re-attempts (on error)	4
	Experiment Timeout	7200 seconds
	Plotting Timeout	600 seconds

Appendix

Table 2: Key Hyperparameters for the Template-Based version of The AI Scientist.

Component/Task	Model Used	Max Tokens	Temperature
Code Generation	Claude 3.5 Sonnet	8,192	0.5
Code Generation	Claude Sonnet 4	20,000	1.0
LLM/VLM Feedback Agent	GPT-4o	8,192	0.5
Summary Report Agent	GPT-4o	8,192	1.0
Code Critic	o3	100,000	1.0

Appendix

Table 3: LLM and VLM Models and Parameters for the template-free version of The AI Scientist.

Table 7

Reviewer	Balanced Acc. ( $\uparrow$ )	Accuracy ( $\uparrow$ )	F1 Score ( $\uparrow$ )	AUC ( $\uparrow$ )	FPR ( $\downarrow$ )	FNR ( $\downarrow$ )
Human (NeurIPS) 1	0.66	0.73	0.49	0.65	0.17	0.52
Random Decision	0.50	0.54	0.47	0.52	0.47	0.43
Always Reject	0.50	0.65	0.00	0.50	0.00	1.00
Ablation with different models
Claude 3.5 Sonnet	0.63 $\pm$ 0.12	0.50 $\pm$ 0.10	0.57 $\pm$ 0.09	0.61 $\pm$ 0.10	0.74 $\pm$ 0.08	0.03 $\pm$ 0.04
GPT-4o	0.54 $\pm$ 0.12	0.68 $\pm$ 0.09	0.20 $\pm$ 0.08	0.54 $\pm$ 0.10	0.03 $\pm$ 0.04	0.88 $\pm$ 0.06

Appendix

Table 5: Ablation results for Automated Reviewer showing how the choice of the underlying LLM, prompt variants, and auxiliary techniques (VLM, Reflexion, few-shot examples, ensembling) affect six evaluation metrics: balanced accuracy, accuracy, F1 score, AUC, false positive rate (FPR), and false negative rate (FNR). The combination of o4-mini with a 5-ensemble meta-review achieves the best overall performance.

Final Paper Selected	Topic Area	# Ideas	# Experiments	# Write-Ups
Compositional Regularization	Core ML	25* $\to$ 1	8 $\to$ 1	24 $\to$ 1
Label Noise Calibration	Core ML	25* $\to$ 1	6 $\to$ 1	7 $\to$ 1
Pest Detection	Applied ML	136 $\to$ 1	16 $\to$ 1	6 $\to$ 1

Appendix

Table 6: Generation and Write-up Statistics. *Rows 1 and 2 share the same pool of 25 generated ideas. The symbol

\to

denotes “Generated

\to

Selected” for the number of ideas, experiments, and write-ups.

Figure 4: Examples paper writeup failures: (a) citation errors, (b) figure display errors, (c) violations of paper length expectations (the workshop limits papers to four pages), and (d) duplicated figures between the main text and appendix.

Figure 5: Distribution of automated review scores for papers generated by the template-based version of The AI Scientist. Violin plots show the distribution of overall scores assigned by the Automated Reviewer to papers generated across three domains (Diffusion Modeling, Language Modeling and Grokking Analysis) and four foundation models. Scores on the y-axis refer to NeurIPS ratings ( NeurIPS, 2022 ) , which range from 1 (Very Strong Reject) to 5 (Borderline Accept). Although there is no exact mapping between conference and workshop scores, papers rated 4 ((Borderline Reject)) in conference reviews are generally considered borderline-eligible for workshop acceptance.

LLM	Total	Compiled	Mean	Max	Total
	Ideas	Papers	Score	Score	Cost ($)
Claude Sonnet 3.5	51	37	2.97	4.0	$\sim$ 250
GPT-4o	51	16	2.81	3.0	$\sim$ 300
DeepSeek Coder	51	28	2.61	3.0	$\sim$ 10
Llama-3.1 405B	51	20	1.3	2.0	$\sim$ 120

Appendix

Table 7: Performance of the template-based version of The AI Scientist for Diffusion Modeling experiments. Metrics include total ideas generated, completed papers, mean and max automated review scores (NeurIPS scale), and approximate total cost in USD.

LLM	Total	Compiled	Mean	Max	Total
	Ideas	Papers	Score	Score	Cost ($)
Claude Sonnet 3.5	52	19	2.89	3.0	$\sim$ 250
GPT-4o	52	15	2.60	3.0	$\sim$ 300
DeepSeek Coder	52	19	2.63	3.0	$\sim$ 10
Llama-3.1 405B	52	19	1.16	2.0	$\sim$ 120

Appendix

Table 8: Performance of the template-based version of The AI Scientist for Language Modeling experiments. Metrics as in Table 7 .

LLM	Total	Compiled	Mean	Max	Total
	Ideas	Papers	Score	Score	Cost ($)
Claude Sonnet 3.5	51	25	3.4	5.0	$\sim$ 250
GPT-4o	51	12	2.67	3.0	$\sim$ 300
DeepSeek Coder	51	32	2.81	4.0	$\sim$ 10
Llama-3.1 405B	51	27	1.11	2.0	$\sim$ 120

Appendix

Table 9: Performance of the template-based version of The AI Scientist for Grokking Analysis experiments. Metrics as in Table 7 .

Figure 6: Preview of the “Adaptive Dual-Scale Denoising” paper which was entirely autonomously generated by the template-based version of The AI Scientist. The full paper can be viewed in Section D.1.1 .

Title	Workshop Result	Materials
Compositional Regularization: Unexpected Obstacles in Enhancing Neural Network Generalization	Accepted (Score: 6.33)	See Section D.2.1 , GitHub Repository
Unveiling the Impact of Label Noise on Model Calibration in Deep Learning	Rejected	See Section D.2.2 , GitHub Repository
Real-world Challenges in Pest Detection using Deep Learning: an Investigation into Failures and Solutions	Rejected	See Section D.2.3 , GitHub Repository

Appendix

Table 10: Overview of AI-Generated Workshop Submissions.

Figure 7: Example of the data-generating function used in the experiments.

Figure 8: The generated model class shows an embedding layer, a single LSTM layer, and a linear layer head.

Figure 9: The generated training loop shows the loss function as well as the proposed regularization.

Figure 10: The generated accuracy calculation function uses regression to match an output with a ground truth.

Figure 11: Temperature scaling implementation.

Panel of Figure 12 — Figure 12: Noisy dataset class implementation.

Figure 13: The implementation for Expected Calibration Error.

Panel of Figure 14 — Figure 14: Domain discriminator and multi-dataset training loop.

Figure 15: Environment noise simulation implementation.

Figure 16: Environmental Robustness Score calculation.

Explore similar work

May 18, 2026

AI for Auto-Research: Roadmap & User Guide

AI-assisted research is crossing a threshold: fully automated systems can now generate research papers for as little as $15, while long-horizon agents can execute experiments, draft manuscripts, and simulate critique with minimal human input. Yet this productivity frontier exposes a deeper integrity problem: under scientific pressure, even frontier LLMs still fabricate results, miss hidden errors, and fail to judge novelty reliably. Studying developments through April 2026, we present an end-to-end analysis of AI across the complete research lifecycle, organized into four epistemological phases: Creation (idea generation, literature review, coding & experiments, tables & figures), Writing (paper writing), Validation (peer review, rebuttal & revision), and Dissemination (posters, slides, videos, social media, project pages, and interactive agents). We identify a sharp, stage-dependent boundary between reliable assistance and unreliable autonomy: AI excels at structured, retrieval-grounded, and tool-mediated tasks, but remains fragile for genuinely novel ideas, research-level experiments, and scientific judgment. Generated ideas often degrade after implementation, research code lags far behind pattern-matching benchmarks, and end-to-end autonomous systems have not yet consistently reached major-venue acceptance standards. We further show that greater automation can obscure rather than eliminate failure modes, making human-governed collaboration the most credible deployment paradigm. Finally, we provide a structured taxonomy, benchmark suite, and tool inventory, cross-stage design principles, and a practitioner-oriented playbook, with resources maintained at our project page.

Lingdong Kong, Xian Sun, Wei Chow +17

Awesome AI Auto-Research Team

AI-Assisted Scientific Research Scientific Workflow Automation Human-AI Collaboration Autonomous Research Agents LLM Agent Reliability

Ask PDF

May 22, 2026

AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discovery

Scientific research is being reshaped by AI systems that move beyond isolated assistance toward longer-horizon workflows spanning literature grounding, hypothesis generation, experimentation, validation, reporting, and revision. This shift marks a transition from task-level AI for science to workflow-level research automation. Yet current systems remain fragmented, differing in autonomy, domain scope, execution environment, validation mechanism, and human oversight, while still struggling with evidence preservation, reproducibility, weak-direction rejection, provenance tracking, cross-domain robustness, and accountable scientific closure. This survey examines these developments through AutoResearch, defined as the developmental spectrum of AI-powered scientific workflow automation. Within it, Vibe Research denotes the human-steered region of prompt-based assistance and human-verified execution, whereas emerging AI-led systems coordinate larger portions of the discovery loop without achieving robust autonomy. We analyze how research systems redistribute control, evidence, execution, validation, and accountability across workflows and organize the field around five workflow conditions: literature and research grounding; hypothesis formation and planning; experimentation and tool use; feedback, validation, and review; and reporting and knowledge communication. We further synthesize AI scientist systems, mixed-initiative co-research frameworks, benchmarks, domain deployments, and open-source infrastructures. Finally, we propose five evaluation dimensions--novelty, validity, impact, reliability, and provenance--and show that AutoResearch autonomy is domain-conditioned, being more credible in structured, executable, and rapidly verifiable settings but limited in embodied, delayed, heterogeneous, ethical, or institutionally accountable contexts.

Guiyao Tie, Jiawen Shi, Dingjie Song +20

Huazhong University of Science and Technology · Lehigh University · Wuhan University +13

AI Agents for Scientific Discovery Scientific Workflow Automation Human-AI Collaboration Autonomous Research Agents

Ask PDF

Sep 17, 2026

ScientistTwo: Pioneering the Human Knowledge Frontier with Autonomous AI

Scientific discovery is defined by the ability to identify the boundaries of existing knowledge and venture into unexplored territory. The ultimate vision for AI in science is problem-driven autonomous research: given a fundamental challenge by a human expert, the AI independently navigates the scientific landscape, uncovers theoretical and empirical bottlenecks, and systematically expands the frontier of knowledge. In this paper, we introduce ScientistTwo, a fully autonomous multi-agent framework designed to realize this vision. Specifically, ScientistTwo takes an initial problem as input, establishes state-of-the-art baselines, formulates novel hypotheses, and coordinates specialized agents to orchestrate an end-to-end discovery cycle without human intervention. Moreover, the framework rigorously conducts experiments using diverse datasets and metrics, refines methodologies through automated ablation studies, and validates research findings via a closed-loop simulated peer-review rebuttal engine. To evaluate ScientistTwo's capabilities against the highest standards of human scientific achievement, we benchmark it across papers accepted at top-tier conferences such as ICLR, ICML, and NeurIPS. As a result, ScientistTwo autonomously generates expert-level, publishable papers and fully verified, executable codebases. Its solutions consistently outperform human state-of-the-art models, and achieve higher average review ratings than human-authored papers under automated AI review agents. These results show that ScientistTwo is not merely an assistive tool but an autonomous scientific pioneer capable of pushing the frontiers of human discovery. Project website: https://scientist-two.github.io/

Jaehyun Nam, Jinsung Yoon, Yanzhou Pan +4

Google Cloud AI Research · University of Waterloo

Scientific Hypothesis Generation Multi-Agent Systems Autonomous Scientific Discovery Autonomous Research Agents

Ask PDF