cs.CLApr 7, 2026

AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discovery

Authors: Yu Li, Chenyang Shao, Xinyang Liu, Ruotong Zhao, Peijie Liu, Hongyuan Su, Zhibin Chen, Qinglong Yang, 

Organizations: Department of Electronic Engineering, BNRist, Tsinghua University · Zhongguancun Academy · Peking University · University of Science and Technology of China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Artificial intelligence research increasingly depends on prolonged cycles of reproduction, debugging, and iterative refinement to achieve State-Of-The-Art (SOTA) performance, creating a growing need for systems that can accelerate the full pipeline of empirical model optimization. In this work, we introduce AutoSOTA, an end-to-end automated research system that advances the latest SOTA models published in top-tier AI papers to reproducible and empirically improved new SOTA models. We formulate this problem through three tightly coupled stages: resource preparation and goal setting; experiment evaluation; and reflection and ideation. To tackle this problem, AutoSOTA adopts a multi-agent architecture with eight specialized agents that collaboratively ground papers to code and dependencies, initialize and repair execution environments, track long-horizon experiments, generate and schedule optimization ideas, and supervise validity to avoid spurious gains. We evaluate AutoSOTA on recent research papers collected from eight top-tier AI conferences under filters for code availability and execution cost. Across these papers, AutoSOTA achieves strong end-to-end performance in both automated replication and subsequent optimization. Specifically, it successfully discovers 105 new SOTA models that surpass the original reported methods, averaging approximately five hours per paper. Case studies spanning LLM, NLP, computer vision, time series, and optimization further show that the system can move beyond routine hyperparameter tuning to identify architectural innovation, algorithmic redesigns, and workflow-level improvements. These results suggest that end-to-end research automation can serve not only as a performance optimizer, but also as a new form of research infrastructure that reduces repetitive experimental burden and helps redirect human attention toward higher-level scientific creativity.

Figures & tables

Figure 1: Task Definition of AutoSOTA: An End-to-End Automated Research Framework
Figure 2: Overall Framework of AutoSOTA
Figure 3: Overall Framework of the Resource Acquisition Process
Figure 4: Overall Framework of the Rubric Construction Process
Figure 5: Overall Framework of the Replication Process
Figure 6: Overall Framework of the Reflection & Ideation Process
IDPaper TitleConferenceImprovement
1SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing [ 11 ]NeurIPS20255.90%
2Pinet: Optimizing hard-constrained neural networks with orthogonal projection layers [ 12 ]ICLR202616.72%
3Discount Model Search for Quality Diversity Optimization in High-Dimensional Measure Spaces [ 13 ]ICLR20267.32%
4Decentralized Attention Fails Centralized Signals: Rethinking Transformers for Medical Time Series [ 14 ]ICLR20264.45%
5Temporal Sparse Autoencoders: Leveraging the Sequential Nature of Language for Interpretability [ 15 ]ICLR20262.25%
6PhySense: Sensor Placement Optimization for Accurate Physics Sensing [ 16 ]NeurIPS20254.16%
Table 1: The Main Experiment of AutoSoTA
Figure 7: Comparison of Average Performance Improvements across Major Research Categories, illustrating the average performance enhancement of the original papers versus the AutoSOTA framework across five key domains: CV, ML, NLP, Optimization, and TimeSeries. To better visualize the relative gains across diverse scales of improvement, the vertical axis represents values processed via a log⁡10\log_{10} transformation ( y′=log⁡10(y)y^{\prime}=\log_{10}(y) ).
Figure 8: Duo-dimensional statistical analysis of AutoSOTA optimization outcomes
Figure 9: Illustration of the LLM case study.
Figure 10: Illustration of the NLP case study.
Figure 11: Illustration of the Biology case study.
Figure 12: Illustration of the CV case study.
Figure 13: Illustration of the time series case study.
Figure 14: Illustration of the Optimization case study.
More in this topic

Autonomous Experimentation

Autonomous experimentation uses AI, active learning, and robotics to design, run, and interpret experiments with less human intervention.

See all 148 papers

Explore similar work

CardsList
Jul 19, 2026

Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer

Auto Research uses language-model agents to propose, implement, and evaluate machine-learning changes in a closed loop, but is usually judged by its terminal pipeline. A terminal score cannot reveal which technical decision produced a gain or distinguish a reusable discovery from a change adapted to development feedback. We introduce intervention-centered Auto Research, which validates research decisions rather than only final artifacts and makes their reliability measurable. Feature, Model, Representation, and Data axes are searched independently with inner five-fold feedback. Each axis winner is frozen before an outer-holdout matrix compares all alternatives on evidence the loop never sees. Across 701 agent-executed attempts spanning ten Matbench endpoints, outer evidence confirms the selected intervention on nine of ten endpoints and preserves 89.3% of non-tied intervention orderings. It also rejects an aggregate Representation gain that inner feedback endorsed. The resulting matrix reveals an information-dependent hierarchy. Composition-only tasks support several routes to improvement, whereas structure-informed tasks favor local geometry features and complementary tree ensembles. A subsequent compatibility test combines already frozen Feature and Model code without further search or tuning and raises mean outer-holdout improvement from 19.0% to 26.3%. By validating decisions rather than only artifacts, this design turns adaptive search into reusable evidence wherever agents propose executable alternatives against a fixed evaluator.

Jingjie Ning, Xiaochuan Li, Shanshan Zhong +2

School of Computer Science, Carnegie Mellon University · 2DP Technology

Automated Algorithm DiscoveryAutomated EvaluationAutonomous Research Agents
AskPDF
Mar 25, 2026

AI-Supervisor: Autonomous AI Research Supervision via a Persistent Research World Model

Existing automated research systems operate as stateless, linear pipelines -- generating outputs without maintaining any persistent understanding of the research landscape they navigate. They process papers sequentially, propose ideas without structured gap analysis, and lack mechanisms for agents to verify, challenge, or refine each other's findings. We present AI-Supervisor, a multi-agent orchestration framework where specialized agents provide end-to-end AI research supervision driven by human interests -- from literature review through gap discovery, method development, evaluation, and paper writing -- through autonomous exploration and self-correcting updates of research knowledge. Unlike sequential pipelines, AI-Supervisor maintains a continuously evolving Research World Model, implemented as a Knowledge Graph, that captures methods, benchmarks, known limitations, and unexplored gaps, serving as shared memory across all agents and enabling agents to explore and build upon a structured understanding of the research landscape. The framework introduces three architectural contributions: (1) structured gap discovery that decomposes methods into core modules, validates their performance across benchmarks, and maps the specific gaps each module creates; (2) self-correcting discovery loops that probe why modules succeed on certain problems and fail on others, whether benchmarks carry hidden biases, and whether evaluation protocols remain adequate for emerging challenges; and (3) self-improving development loops governed by cross-domain mechanism search that iteratively targets failing modules by finding solutions from other scientific fields. All agents operate under a consensus mechanism where independent findings are corroborated before being committed to the Research World Model.

Yunbo Long

Autonomous Research AgentsAI Agents for Scientific DiscoveryMulti-Agent LLM Systems
AskPDF
May 18, 2026

AI for Auto-Research: Roadmap & User Guide

AI-assisted research is crossing a threshold: fully automated systems can now generate research papers for as little as $15, while long-horizon agents can execute experiments, draft manuscripts, and simulate critique with minimal human input. Yet this productivity frontier exposes a deeper integrity problem: under scientific pressure, even frontier LLMs still fabricate results, miss hidden errors, and fail to judge novelty reliably. Studying developments through April 2026, we present an end-to-end analysis of AI across the complete research lifecycle, organized into four epistemological phases: Creation (idea generation, literature review, coding & experiments, tables & figures), Writing (paper writing), Validation (peer review, rebuttal & revision), and Dissemination (posters, slides, videos, social media, project pages, and interactive agents). We identify a sharp, stage-dependent boundary between reliable assistance and unreliable autonomy: AI excels at structured, retrieval-grounded, and tool-mediated tasks, but remains fragile for genuinely novel ideas, research-level experiments, and scientific judgment. Generated ideas often degrade after implementation, research code lags far behind pattern-matching benchmarks, and end-to-end autonomous systems have not yet consistently reached major-venue acceptance standards. We further show that greater automation can obscure rather than eliminate failure modes, making human-governed collaboration the most credible deployment paradigm. Finally, we provide a structured taxonomy, benchmark suite, and tool inventory, cross-stage design principles, and a practitioner-oriented playbook, with resources maintained at our project page.

Lingdong Kong, Xian Sun, Wei Chow +17

Awesome AI Auto-Research Team

AI-Assisted Scientific ResearchScientific Workflow AutomationHuman-AI CollaborationAutonomous Research AgentsLLM Agent Reliability
AskPDF