cs.AINov 5, 2026

Scientist World: Grow a Research World for AI-Scientists

Authors: Taiming Lu, Sachin Konan, Zhuang Liu

Organizations: Princeton University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

AI-Scientists arrived, but a world for them to grow in did not. Human scientists take theirs for granted −- centuries of open problems and shared ideas, waiting in the scientific literature. To AI-Scientists, that world is readable but not runnable, so we build Scientist World for them: every research paper becomes an executable environment. Our autonomous pipeline extracts each paper's open-ended research question and reproduces its proposed method as the baseline; an AI-Scientist entering the environment is evaluated against what the human authors achieved. We release the first 1,000 verified environments. With the world as an evaluation, we found that frontier AI-Scientists can already surpass the human researchers 74% of the time. As a record of how AI does science, it contains 6,400 hours of research traces, with every idea, experiment, failure, and success. As a training ground, we are able to raise (+158%) a small model to match frontier models in research ideation. At last, AI-Scientists' own world has arrived, grown from and growing with science itself.

Figures & tables

Figure 1: We introduce Scientist World. Each research paper becomes an executable environment, where an AI-Scientist ’s Frontier-Normalized Improvement (FNI) is above zero when its method improves on the paper’s reproduced one. We test how well AI-Scientists improve existing research (left) and innovate new research from scratch (right). We show the best model of each lab plus the next best overall; full leaderboards are in Figure 12 .
Figure 2: Where Scientist World stands. Prior AI-Scientist environments sit on the trade-off surface we describe; Scientist World reaches the corner where scale, evolve, and execution meet.
Figure 3: Scientist World is three things at once. One set of environments, grown from real papers, serves all three.
Figure 4: Builders turn a paper into an environment. Construction runs like an assembly line: a published paper enters at one end, and a reusable research environment leaves the other. Specialized builders take turns on a single shared workspace, each refining what the previous one built and offloading heavy computation to a shared cluster. A quality gate guards the line, sending unfinished work back for another pass, so only what clears verification is accepted.
Figure 5: Using compute. A builder submits a job to the shared pool, sleeps, and is woken with the logs; its GPU budget is enforced outside the loop.
Figure 6: Anatomy of an environment. A produced environment is a single self-contained workspace, laid out so the problem stays separate from the solution. The briefing , data , and scoring fix what to solve and how it is judged without referring to any one method, which keeps the task open to future approaches and stops a score from depending on the implementation. The method is the paper’s source method reproduced, and the scores it reaches are the reference we compare against, so a new method’s final score is the improvement it makes over them.
Figure 7: Selecting 1,000 papers from 124,816 indexed. In a first pass (a), we drop papers that are low quality or are mainly a type of theory, a benchmark, a dataset, or a single finding, which leaves 21,354. In (b), we look at these more closely and favor work that is both high quality and compute-friendly , keeping the best 1,000. In (c), we check the result: the papers we picked (blue) are clearly stronger and more reproducible than the pool we started from (grey).
per papermediantotal
compute jobs76239,506
GPU-hours0.42,286
CPU-hours628,338
agent turns730947,056
input tokens28M51B
output tokens163k231M
Table 1: Per-paper and total build cost. Median and the total cost over 1,000 papers.
Figure 9
Figure 9: An example environment. This environment turns a Transformer architecture paper into a constrained improvement task: raise ImageNet accuracy without changing scale, pretraining, or data. The scripts make the task runnable, the source and baseline implementations anchor the comparison, and the scoring matrix defines progress as weighted improvement over the reproduced source method.
Figure 10: Environments’ openness. From fully prescribed (0) to fully open (100).
Figure 11: What the environments ask for. Each task names, in its own words, what should get better.
Figure 12: Leaderboards. Mean FNI by model in improve mode (top) and innovate mode (bottom), averaged over visible and held-out splits. Zero matches the reproduced source method; higher is better. Improve mode gives models the source method, while innovate mode requires them to design from scratch. We use xhigh effort for all models.
Figure 13: The two abilities are loosely tied. Each model’s rank at improving against its rank at innovating.
Figure 14: Effort versus outcome, and research quality. (a) each run’s score against the compute hours (top) and agent turns (bottom) behind it, pooling both modes; effort explains almost none of the variance in score ( R2=0.00R^{2}=0.00 , 0.020.02 ). (b) research quality by dimension, judged apart from the score. Newest model per family, colored by provider.
recordaverage
idea revisions3.7
method revisions6
experiments run25
commits35
hours of work5.4
Table 4: Dataset statistics. Each experiment is stored with its log; hours of work is the scientist’s end-to-end running wall-clock time.
Figure 15: Raised in the world. Median held-out FNI of each stage’s ideas, implemented identically; after GRPO it reaches the frontier cohort’s anchor.
Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 16: System design overview. Hive supplies execution capacity for CPU and GPU jobs; Nexus accounts for the budget and resource boundary of each attempt; Swarm maintains one paper-level trajectory across roles, restarts, and handoffs; and Workers connect different agent scaffolds through the same system interface. Together, these layers let many agents iterate on open-ended papers while preserving the evidence, budget state, and handoff context.
Figure 17: Hive execution overview. A job request enters Hive with a command, resource target, working directory, and time limit. Admission queues the request, dispatch routes it to test, CPU, or GPU capacity, and monitor records the return code, runtime, and logs that form the job record.
Figure 18: Nexus operations overview. Nexus stores the job, project, budget, and review state needed to translate agent requests into Hive actions and to translate Hive execution evidence back into agent decisions.
Figure 19: A Nexus project turns resource policy into explicit accept/reject decisions. The example shows how budget, concurrency, and allowed resource pools gate agent requests before they reach Hive.
Figure 20: Swarm coordination overview. Swarm drives a pool of paper workspaces in parallel. Each paper moves through an ordered role workflow—reproduce, design, package, verify—whose transitions are driven by wake events: worker start, job submit, job update, validation failure, and role transition. Progress is graded against a milestone ladder, from none up to full reproduction, and the milestone gate decides whether a paper’s artifact is accepted.
eventshown to agentkept in trajectory
iteration startpaper, workspace, milestone, budgetmodel, role, starting checkpoint
job runningstreaming logs and remaining budget, to continue or cancelthe continue or cancel decision
job finishedlogs, exit code, and remaining budgetoutcome, compute used, next action
build failedthe build log, to fix and resubmitfailure cause, fix attempts
budget exhaustedbudget spent, no further jobs, wrap up current workfinal spend, the milestone the agent wraps up at
validation failvalidation or milestone-gate failureresulting status, revert attempts
Appendix
Table 5: Swarm wakes a builder only at decision-changing events. Rather than holding a session open for the whole reproduction, Swarm re-engages the builder only when new information changes what it should do next. Each row gives the triggering event, what the agent is shown, and the state Swarm writes into the paper’s trajectory.
rolemain responsibilitygate to continue
reproducerimplements the paper method and tries to reproduce the central claim.reach core_claim , or stop or revert.
designerconverts the reproduction into gym targets, constraints, and scoring.scores must validate
packagerconsolidates scripts, environment files, and download logic.workspace needs to be able to rebuild.
verifieraudits whether the final evidence supports the claimed milestone.unsupported claims have their milestone lowered.
Appendix
Table 7: Role separation keeps one workspace auditable across builder handoffs. The role split separates reproduction, environment design, packaging, and verification while keeping all roles on the same workspace and evidence trail.
Figure 21: Swarm workflow stages. The standard workflow expands reproduction, design, packaging, and verification into repeated builder passes; each row carries the model profile, gate, GPU budget, and possible revert target.
Figure 22: Worker runtime overview. Different agent scaffolds connect through one uniform turn interface. Worker runs the agent under a container runtime with capped resources and a read-only validator, cycling through a turn loop of render prompt, agent acts, and submit jobs. Each turn ends by selecting the next action in action.yaml , which the system reads before waking the agent for the following turn.
Figure 23: The action.yaml that ends a turn. The agent writes it to request the next job or to stop, and the system reads it after every turn.
Figure 24: Builder overview. From a paper (in PDF and Markdown) and an empty starter workspace, builders build a reusable environment along three tracks: a problem definition that states the task and its constraints without leaking a solution, a reproduction of the paper’s method whose code and experiments recover the central claim, and an evaluation harness with visible and hidden slices that turns outputs into scores.
conditionwhat the builder is asked to do
new paperread the paper and reproduce its method from a full briefing
continued iterationaudit the previous workspace and push it further, starting one milestone below the prior claim
job runningread the streaming logs and decide whether to let the job continue or cancel it
job finished, budget leftinspect the results, record progress, and plan the next job
job finished, budget spentstop submitting and wrap up the milestone, scoring files, and environment
build failedread the build log, fix the container definition, and resubmit
Appendix
Table 8: Each turn is prompted for its condition. The first turn is a full role briefing; later turns receive a short prompt for whatever event woke the builder, keeping it on task across a long unattended run.
grouppurpose
briefing/states the task and how success is measured, while keeping the paper’s method out of the problem files
method/ , baseline/the reproduced implementation, with the baselines used to interpret its score
data/ , eval/the data interface and a method-agnostic evaluator that any future method must run against
scoring/the scoring contract: how each metric counts, what an improvement means, and what must stay fixed
scripts/ , environment/everything needed to rebuild the environment and rerun the method from a fresh clone
root filesthe current action, the claimed milestone, progress and deviation notes, and the automatic checks
Appendix
Table 9: Each directory has one responsibility. The workspace keeps task context, method code, evaluation, scoring, execution, and record state apart, so a later scientist can reuse it without the original conversation.
Figure 26: The scoring contract, held in two files. scoring/reference.json fixes the weights, metric directions, coefficients, and the paper’s reported numbers; scoring/scores.json holds the numbers the reproduction actually produced, under the same keys.
Figure 27: Each design choice targets a concrete failure mode. The workspace is structured to avoid solution leakage, method-coupled scoring, unrebuildable dependencies, evaluator overfitting, and unsupported claims.
groupfields recordeduse
reproduction summaryfinal milestone and status, scaffold and model, workflow role and step, and turn, run, and iteration countsattributes the result to the builders and role that produced it
compute and costGPU and CPU hours, the GPU budget, jobs run, wall-clock duration, tokens, and dollar costseparates a scientific failure from an execution failure, and compares spend
components presentwhether the container, method, evaluator, scripts, reference, and data are all in placeconfirms the workspace is complete enough to rerun
per-builder recordsone entry per builder, with its turns, its jobs, and why it stoppedreconstructs the result across handoffs
Appendix
Table 10: The finished workspace records how it was produced. manifest.json ties each accepted artifact to the builders, compute, and components behind it, so it stays auditable after the session ends.
checkwhat it verifies
scoring schemareference.json parses, experiment weights sum to one, and each experiment declares a primary metric, its metrics, and its results; every metric declares a direction and a coefficient; and every method is typed as a baseline or a proposed method, with at least one proposed
no paper-agreement metricsmetric names may not encode closeness to the paper ( deviation , error_vs_ , agreement_with , and similar); the score must measure the scientific quantity, not fidelity to the publication
scores match the schemascores.json uses the same experiments, methods, and metric names as reference.json , and every value is numeric
reproduction sanityeach reproduced primary-metric value stays close to the paper’s: no sign flip, under a 2×2\times magnitude gap, and under a 0.30.3 absolute gap for bounded metrics
evaluator independenceeval/ and data/ never import from method/ , so the evaluator works for any method, not only the reproduced one
fresh-clone integrityevery local import in eval/ , method/ , and baseline/ resolves to a git-tracked file, and no committed code references the machine-local shared/ cache
Appendix
Table 11: What validate.py checks before a workspace is accepted. The validator enforces the scoring schema, blocks scores that measure agreement with the paper, and verifies that the workspace is independent of its own method, rebuildable from a fresh clone, and free of train/test shortcuts.
Figure 28: Screening preserves nearly all unique papers before deeper analysis. Counts are computed from the selection snapshot used by the export. Shares after the unique-index row use 124,816 as the denominator.
Figure 29: The source pool is largest in cs.AI and cs.LG before deduplication. These are category-listing records before ID-level deduplication; a single paper can contribute to multiple rows.
Figure 30: Fast screens keep broad coverage while identifying metric-method papers as the dominant type. The left side summarizes 123,980 fast quality scores; the right side summarizes 124,018 fast paper-type labels.
Figure 31: Structured-analysis score distributions across the 21,354 analyzed papers. Each panel gives the share of papers at each integer score from 0 to 10: the five reproduction-difficulty sub-scores (coding, compute, time to run, data or model, and overall), and the paper-level quality and problem-novelty scores. Dashed lines mark the means. Difficulty scores concentrate in the low-to-middle range, while quality and problem novelty center higher.
Figure 32: Feasibility filters reduce 21,354 analyzed papers to 8,987 candidates. The feasible candidates are ranked by quality-feasibility utility before the final category-balanced selection.
Figure 33: Temperature-balanced sampling keeps large areas prominent while preserving smaller categories. The final 1,000-paper selection is sampled from 8,987 feasible candidates across 18 canonical categories.
Figure 34: What environments ask, and the work it takes. Each environment states what should get better and what work that takes; phrases are grouped into families that emerge from data; ribbon mass is the share of environments.
evaluation stagecontract enforced by the harness
scaffoldcreate a fresh per-run workspace and remove files that would leak hidden tests or reference scores
main phaselet the scientist edit the method and submit compute jobs until it writes train-phase scores and declares completion
validationcheck that every required proposed-method metric is present; missing cells trigger bounded repair turns
held-out testmaterialize the hidden evaluator only after the method is frozen, then run it through Nexus
normalizationconvert heterogeneous metric improvements into per-paper scores before averaging into a leaderboard
Appendix
Table 12: Evaluation separates iteration, validation, hidden testing, and normalization into auditable stages. The staged harness keeps scientist behavior inspectable and comparable.
Figure 35: How often scientists beat the paper. Distribution of held-out improvement over the paper’s source method, per mode. Colored bars (right of zero) beat the reproduced method; the grey tail regressed. The percentage is the share of runs that improved.
Figure 36: Training gains carry over to held-out test. Training-phase score against held-out test score, per mode, with the y=xy=x line. Points hug the diagonal; the shaded quadrant holds the few runs that gained on training but regressed on test.
Figure 37: Effort versus outcome. Within-paper normalized score against GPU-hours (top row) and agent turns (bottom row), shown separately for the Improve and Innovate modes. Scores are min-max normalized within each (paper, mode); each line is a least-squares fit in log-effort. Every panel is flat ( ∣ρ∣≤0.22|\rho|\leq 0.22 ).
resourcecountrole
A100 GPU nodes16general accelerator workloads and pilot experiments
H100 GPU nodes16high-throughput training and evaluation jobs
H200 GPU nodes16memory-intensive or largest-scale reproduction runs
CPU nodes4low-compute, CPU-only setup and validation jobs
Appendix
Table 13: Hive exposes heterogeneous capacity for accelerator-heavy and CPU-only experiments. The counts describe the compute capacity exposed in Hive.
Figure 47
More in this topic

Autonomous Research Agents

Autonomous research agents use language models and tools to search literature, design and run experiments, analyze results, and support scientific discovery.

See all 338 papers

Explore similar work

CardsList
Mar 31, 2026

Towards End-to-End Automation of AI Research

The automation of science is a long-standing ambition in the field of AI. While the community has made significant progress in automating individual components of the scientific process, a system that autonomously navigates the entire research lifecycle -- from conception to publication -- has remained out of reach. Here, we present the strongest demonstration to date toward automating the entire process end-to-end. We present The AI Scientist, which creates research ideas, writes code, runs experiments, plots and analyzes data, writes the entire scientific manuscript and performs its own peer review. Its ideas, execution, and presentation are of sufficient quality to produce a manuscript generated by an AI system that passes the first round of peer review at a major machine learning conference workshop. The workshop has an acceptance rate of 70 percent. Our system leverages modern foundation models within a complex agentic system. We evaluate The AI Scientist in two settings: a focused mode using human-provided code templates as an initial scaffold to conduct research on a specific topic, and a template-free, open-ended mode that leverages agentic search for wider scientific exploration. Both settings produce diverse ideas and automatically test, report on, and evaluate them. This achievement demonstrates AI's growing capacity for scientific contribution and signifies a potential paradigm shift in how research is conducted. As with any impactful new technology, there could be significant risks, including taxing overwhelmed review systems and adding noise to scientific literature. However, if developed responsibly, such autonomous systems could greatly accelerate scientific discovery.

Yutaro Yamada, Robert Tjarko Lange, Cong Lu +5

Sakana AI, Tokyo, Japan · University of British Columbia, Vancouver, Canada · Vector Institute, Toronto, Canada +1

AI-Assisted Scientific ResearchAutonomous Scientific DiscoveryAutonomous Research AgentsScientific Workflow Automation
AskPDF
Aug 13, 2026

OmniScientist: An Omni-Modal Omni-Discipline AI Scientist

Recent advances in foundation models have enabled AI scientists to automate increasingly complete research workflows, from hypothesis generation and code execution to manuscript preparation. Yet workflow coverage alone does not provide access to the full evidence on which scientific discovery depends. Existing systems typically reason over text, code, labels, or precomputed summaries, leaving scientifically decisive spatial, temporal, cross-channel, and procedural relations unavailable to the agent. We introduce OmniScientist, an end-to-end, omni-modal AI scientist that conducts multidisciplinary research directly from heterogeneous raw evidence. A perception layer and 3 autonomous agents for ideation, experiment, and writeup operate within a deterministic pipeline, allowing observations to shape research questions, experimental decisions, and final claims throughout the research lifecycle. By running idea, rigour, and claim checks in code, the system enforces novelty screening, statistical validity, execution provenance, and numerical traceability. We evaluate OmniScientist on 36 real-data cases spanning 5 discipline families, 4 families of scientific evidence, and modalities including images, signals, audio, video, 3-D structures, trajectories, tables, formulae, and graphs. The system completes the full path from raw data to a compiled manuscript in all 36 cases and achieves a mean overall paper score of 6.3 with the reference reasoning backbone. In paired comparisons against a blind variant that receives only precomputed scalar features, direct perception improves all 7 evaluation dimensions and wins 85% of head-to-head judgments. These results show that lifecycle-wide perception is essential for evidence-grounded scientific discovery and provides a practical path toward broadly capable AI scientists.

Bobo Li, Hao Fei, Tianjie Ju +2

1National University of Singapore · University of Oxford

AI Agents for Scientific DiscoveryScientific DiscoveryAutonomous Scientific DiscoveryEvidence-Grounded GenerationScientific Workflow Automation
AskPDF
Oct 29, 2026

Letting AI Formulate the Research Question: Autonomous Research Agents

Systems such as The AI Scientist have shown that AI agents can carry out and write up research, but in those demonstrations humans chose the research area and selected the outputs to show. We study letting the AI formulate the research question itself, and we report every run. Our system gives an off-the-shelf coding agent an instruction document and two skills: one for formulating the question, in which the agent diverges from recent technologies into problems of other fields, screens its candidates for known failure modes and obtains a critique from a different model; and one for peer review, in which a reviewer from another model family reruns the study from its records before the agent revises the paper. Over 13 development runs, changes to the one or two sentences that set the research area coincided with large shifts in what the candidate questions and the chosen papers were about, which led us to the area of a new technology applied to a societal problem. We froze the system after a pilot run and report five consecutive runs: the pilot, counted by a disclosed amendment to the pre-registration, and four pre-registered runs. All five runs went from question formulation to a revised, re-computable paper, with local computation on a laptop and no research input from humans, and external AI reviewers reading only the papers judged them at or near workshop level. We release the system, the five papers and the records needed to reproduce them.

Ko Fujimura

Otsuma Women’s University, Faculty of Social Information Studies

Autonomous Research AgentsResearch Idea Generation
AskPDF