AI-Supervisor: Autonomous AI Research Supervision via a Persistent Research World Model

Yunbo Long

cs.AIMar 25, 2026

AI-Supervisor: Autonomous AI Research Supervision via a Persistent Research World Model

Authors: Yunbo Long

Abstract

Existing automated research systems operate as stateless, linear pipelines -- generating outputs without maintaining any persistent understanding of the research landscape they navigate. They process papers sequentially, propose ideas without structured gap analysis, and lack mechanisms for agents to verify, challenge, or refine each other's findings. We present AI-Supervisor, a multi-agent orchestration framework where specialized agents provide end-to-end AI research supervision driven by human interests -- from literature review through gap discovery, method development, evaluation, and paper writing -- through autonomous exploration and self-correcting updates of research knowledge. Unlike sequential pipelines, AI-Supervisor maintains a continuously evolving Research World Model, implemented as a Knowledge Graph, that captures methods, benchmarks, known limitations, and unexplored gaps, serving as shared memory across all agents and enabling agents to explore and build upon a structured understanding of the research landscape. The framework introduces three architectural contributions: (1) structured gap discovery that decomposes methods into core modules, validates their performance across benchmarks, and maps the specific gaps each module creates; (2) self-correcting discovery loops that probe why modules succeed on certain problems and fail on others, whether benchmarks carry hidden biases, and whether evaluation protocols remain adequate for emerging challenges; and (3) self-improving development loops governed by cross-domain mechanism search that iteratively targets failing modules by finding solutions from other scientific fields. All agents operate under a consensus mechanism where independent findings are corroborated before being committed to the Research World Model.

Figures & tables

Figure 1: The Research World Model ecosystem. Left: distributed knowledge network of multiple RWMs sharing verified findings. Center: internal structure of one RWM with uncertainty-annotated edges carrying performance metrics. Right: bidirectional interaction with physical world—literature, code, compute, and community—enabling active exploration over passive generation.

System	Curiosity			Gap	Cross-	Self-	RWM	Cons.
System	Driven	Lit	Repro	Probing	Domain	Imp.	RWM	Cons.
AI Scientist v2	×	×	×	×	×	✓	×	×
AI-Researcher	×	✓	×	×	×	✓	×	×
Agent Lab	×	×	×	×	×	✓	×	×
SciAgents	×	×	×	×	×	×	$\dagger$	×
ResearchAgent	×	✓	×	×	×	✓	×	×

Table 1: Comparison of AI-Supervisor with existing systems. Curiosity : starts from user interest, no expertise needed. Lit : systematic literature search. Repro : baseline reproduction. Gap : empirically verified gap probing. Cross : cross-domain search. Self-Imp. : self-correcting quality-gated loops. RWM : persistent evolving Research World Model with uncertainty. Cons. : multi-agent consensus. † Static pre-built KG, not persistent or evolving.

Figure 2: AI-Supervisor as a dynamic DAG with the Persistent Research World Model $\mathcal{W}$ at center. All agent teams read from and write to $\mathcal{W}$ . The consensus mechanism (right) implements Eq. 4 . The self-correcting loop (bottom) implements Eq. 7 . Red dashed arrows show review-triggered backward routing.

Category	Criteria
Novelty	New gap from our experiments (not “nobody tested X”)
	Novel formulation with mathematical grounding
	Surprising insight that changes field understanding
Performance	Beats $\geq$ 2 published baselines on their metrics
	Statistical significance: $p<0.001$ , $n\geq 50$ , 3 seeds
	Ablation: each component removal causes measurable drop

Table 2: Quality gate checklist

Q(m)

(Eq. 7 ). All 10 criteria must pass for method finalization.

Condition	Gaps/Task	Precision	Recall	Best Align
AI-Supervisor (RWM)	5.0	0.807	1.000	4.44
LLM-only brainstorm	4.9	0.679	0.926	4.15
Divergent-convergent	2.0	0.755	0.926	4.04

Table 3: Experiment 1: Gap discovery quality on 27 Scientist-Bench tasks across 5 AI domains. AI-Supervisor achieves the highest best alignment (4.44) and precision (0.807) with perfect recall.

Condition	Quality Gate	Iters	Cross-domain	Std
AI-Supervisor (full loop)	8.0/10	1.4	5/5	0.0
Single-pass	8.0/10	1.0	0/5	0.0
Tree search, same domain	7.4/10	2.4	0/5	0.5
Cross-domain, no loop	5.6/10	1.0	3/5	1.2

Table 4: Experiment 2: Method development quality on 5 curated gaps. Quality gate = criteria passed out of 10. AI-Supervisor achieves the highest gate score with lowest variance. Cross-domain search without a quality-gated loop produces the weakest results.

Condition	Gaps	Cross-ins.	Cross-conn.	Verified	KG Growth
AI-Supervisor (persistent RWM)	15	3/3	16	13	7 $\to$ 13 $\to$ 19
Isolated runs (fresh RWM)	15	0/3	0	0	7, 4, 9
Context-window memory	15	2/3	0	0	—
Static world model	15	0/3	0	0	0, 0, 0

Table 5: Experiment 3: Knowledge persistence across 3 sequential projects. Cross-connections = structural links between projects found in the KG. Cross-insights = projects where prior knowledge informed gap discovery. Only the persistent KG achieves both structural connections AND cross-project insights.

Agents	Gaps/Task	Best Alignment	Mean Alignment
1	6.2	4.10	2.97
3	4.2	4.10	3.39
5	4.3	4.00	3.23
7	3.9	4.00	3.19

Table 6: Scalability experiment: more agents produce fewer but more focused gaps while maintaining alignment quality. The consensus filter tightens with agent count.

Condition	Best Align	Mean Align	Precision
Individual (best agent)	3.67	3.16	0.240
Union (all agents merged)	3.67	3.13	0.227
AI-Supervisor consensus	3.67	3.27	0.297

Table 7: Experiment 5: Consensus improves both mean alignment and precision over individual agents and naive union. The 2-round protocol with shared visibility produces higher-quality gaps.

Gap	Cross+Mech	Within	Naive Cross	Best
Safe RL	21 /25	16	11	A
Deepfake detection	19 /25	15	10	A
LLM alignment	21 /25	16	11	A
GNN over-smoothing	23 /25	17	12	A
Few-shot learning	19 /25	14	10	A
Average	20.6 /25	15.6	10.8	5/5

Table 8: Method novelty comparison on 5 gaps. Cross-domain with mechanism analysis wins all 5 gaps, scoring 32% higher than within-domain and 91% higher than naive cross-domain. The mechanism analysis step is critical — naive cross-domain (just “borrow a technique”) is the worst approach.

Dimension	Cross+Mech	Within	Naive Cross
Mathematical novelty	4.0	3.0	2.0
Mechanism grounding	4.6	3.6	2.6
Theoretical depth	3.8	2.8	2.0
Differentiation	4.0	3.0	2.0
Reviewer score	4.2	3.2	2.2

Table 9: Per-dimension breakdown of method novelty scores (1–5 scale).

System	Cost/Run	GPU	Stages	LLM
AI Scientist v1	$\sim$ $15	Yes	3	Claude Sonnet 3.5
Agent Lab (GPT-4o)	$2.33	Config.	3	GPT-4o
Agent Lab (o1-preview)	$13.10	Config.	3	o1-preview
AI-Researcher	N/R	Yes	3	Gemini 2.5 Pro
MLR-Copilot	N/R	Yes	2	GPT-4
AI-Supervisor (efficient)	$8–16	No	5	Qwen-72B

Table 10: Cost comparison. AI-Supervisor covers more stages at comparable cost with no GPU requirement.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Phase	API Calls	Est. Cost
Phase 0: Supervision	3–5	$0.50
Phase 1: Literature (6–12 agents)	20–40	$1–3
Phase 2a: World Model Build (20 agents)	60–80	$3–5
Phase 2b: Gap Probing (2 rounds $\times$ 5+1)	15–20	$1–2
Phase 3: Development (3 iters $\times$ 031–034)	30–50	$2–4
Phases 4–7: Eval + Paper + Review	20–30	$1–2

Appendix

Table 11: AI-Supervisor cost breakdown per phase (Qwen-72B-Instruct).

Autonomous Research Agents

Autonomous research agents use language models and tools to search literature, design and run experiments, analyze results, and support scientific discovery.

See all 318 papers

May 18, 2026

AI for Auto-Research: Roadmap & User Guide

AI-assisted research is crossing a threshold: fully automated systems can now generate research papers for as little as $15, while long-horizon agents can execute experiments, draft manuscripts, and simulate critique with minimal human input. Yet this productivity frontier exposes a deeper integrity problem: under scientific pressure, even frontier LLMs still fabricate results, miss hidden errors, and fail to judge novelty reliably. Studying developments through April 2026, we present an end-to-end analysis of AI across the complete research lifecycle, organized into four epistemological phases: Creation (idea generation, literature review, coding & experiments, tables & figures), Writing (paper writing), Validation (peer review, rebuttal & revision), and Dissemination (posters, slides, videos, social media, project pages, and interactive agents). We identify a sharp, stage-dependent boundary between reliable assistance and unreliable autonomy: AI excels at structured, retrieval-grounded, and tool-mediated tasks, but remains fragile for genuinely novel ideas, research-level experiments, and scientific judgment. Generated ideas often degrade after implementation, research code lags far behind pattern-matching benchmarks, and end-to-end autonomous systems have not yet consistently reached major-venue acceptance standards. We further show that greater automation can obscure rather than eliminate failure modes, making human-governed collaboration the most credible deployment paradigm. Finally, we provide a structured taxonomy, benchmark suite, and tool inventory, cross-stage design principles, and a practitioner-oriented playbook, with resources maintained at our project page.

Lingdong Kong, Xian Sun, Wei Chow +17

Awesome AI Auto-Research Team

AI-Assisted Scientific Research Scientific Workflow Automation Human-AI Collaboration Autonomous Research Agents LLM Agent Reliability

Ask PDF

Jul 29, 2026

Can AI agents conduct open-ended AI research? Early evidence from two case studies

Forecasts of explosive AI progress hinge on AI agents automating AI research. But evidence on whether agents can carry out open-ended AI research is thin. Current evaluations either test agents on narrow, verifiable tasks, which excludes open-ended research, or submit AI-generated papers to blind peer review, which is overstretched, stochastic, and suffers from poor review quality. We introduce a third way to measure progress towards AI R&D automation. An agent takes on the central, open-ended research question of a high-quality unpublished paper, and the paper's original authors grade its output. We call these shadow evaluations. We ran shadow evaluations on two unpublished NeurIPS 2026 submissions, giving frontier agents six days and thousands of dollars of compute. The agents completed all of the engineering without human help, yet could not make substantial progress towards answering the research questions. As a result, both papers were unambiguously rejected by the authors. We identify five recurring failure modes: poor judgment about the bar for publishable research, uncreative responses to shortcomings in the research design, ineffective backtracking from dead ends, poor resource awareness, and instruction drift. A robustness check with a second model and scaffold reproduced these failures. We release the expert reviews, survey responses, agent repositories, and logs. Our results provide early evidence that today's agents can do the engineering of AI research, but struggle with critical parts of the research lifecycle.

Peter Kirgis, Sayash Kapoor, Andrew Schwartz +21

Princeton University · Cornflower Labs · UK AI Security Institute +8

AI-Assisted Scientific Research AI Agent Evaluation Autonomous Research Agents LLM Agent Reliability

Ask PDF

Sep 17, 2026

ScientistTwo: Pioneering the Human Knowledge Frontier with Autonomous AI

Scientific discovery is defined by the ability to identify the boundaries of existing knowledge and venture into unexplored territory. The ultimate vision for AI in science is problem-driven autonomous research: given a fundamental challenge by a human expert, the AI independently navigates the scientific landscape, uncovers theoretical and empirical bottlenecks, and systematically expands the frontier of knowledge. In this paper, we introduce ScientistTwo, a fully autonomous multi-agent framework designed to realize this vision. Specifically, ScientistTwo takes an initial problem as input, establishes state-of-the-art baselines, formulates novel hypotheses, and coordinates specialized agents to orchestrate an end-to-end discovery cycle without human intervention. Moreover, the framework rigorously conducts experiments using diverse datasets and metrics, refines methodologies through automated ablation studies, and validates research findings via a closed-loop simulated peer-review rebuttal engine. To evaluate ScientistTwo's capabilities against the highest standards of human scientific achievement, we benchmark it across papers accepted at top-tier conferences such as ICLR, ICML, and NeurIPS. As a result, ScientistTwo autonomously generates expert-level, publishable papers and fully verified, executable codebases. Its solutions consistently outperform human state-of-the-art models, and achieve higher average review ratings than human-authored papers under automated AI review agents. These results show that ScientistTwo is not merely an assistive tool but an autonomous scientific pioneer capable of pushing the frontiers of human discovery. Project website: https://scientist-two.github.io/

Jaehyun Nam, Jinsung Yoon, Yanzhou Pan +4

Google Cloud AI Research · University of Waterloo

Scientific Hypothesis Generation Multi-Agent Systems Autonomous Scientific Discovery Autonomous Research Agents

Ask PDF

Abstract

Autonomous Research Agents

Explore similar work

AI for Auto-Research: Roadmap & User Guide

Can AI agents conduct open-ended AI research? Early evidence from two case studies

ScientistTwo: Pioneering the Human Knowledge Frontier with Autonomous AI