cs.AINov 5, 2026
SaveScientist World: Grow a Research World for AI-Scientists
Organizations: Princeton University
Abstract
AI-Scientists arrived, but a world for them to grow in did not. Human scientists take theirs for granted centuries of open problems and shared ideas, waiting in the scientific literature. To AI-Scientists, that world is readable but not runnable, so we build Scientist World for them: every research paper becomes an executable environment. Our autonomous pipeline extracts each paper's open-ended research question and reproduces its proposed method as the baseline; an AI-Scientist entering the environment is evaluated against what the human authors achieved. We release the first 1,000 verified environments. With the world as an evaluation, we found that frontier AI-Scientists can already surpass the human researchers 74% of the time. As a record of how AI does science, it contains 6,400 hours of research traces, with every idea, experiment, failure, and success. As a training ground, we are able to raise (+158%) a small model to match frontier models in research ideation. At last, AI-Scientists' own world has arrived, grown from and growing with science itself.
Figures & tables
Figure 1: We introduce Scientist World. Each research paper becomes an executable environment, where an AI-Scientist ’s Frontier-Normalized Improvement (FNI) is above zero when its method improves on the paper’s reproduced one. We test how well AI-Scientists improve existing research (left) and innovate new research from scratch (right). We show the best model of each lab plus the next best overall; full leaderboards are in Figure 12 .
Figure 2: Where Scientist World stands. Prior AI-Scientist environments sit on the trade-off surface we describe; Scientist World reaches the corner where scale, evolve, and execution meet.
Figure 3: Scientist World is three things at once. One set of environments, grown from real papers, serves all three.
Figure 4: Builders turn a paper into an environment. Construction runs like an assembly line: a published paper enters at one end, and a reusable research environment leaves the other. Specialized builders take turns on a single shared workspace, each refining what the previous one built and offloading heavy computation to a shared cluster. A quality gate guards the line, sending unfinished work back for another pass, so only what clears verification is accepted.
Figure 5: Using compute. A builder submits a job to the shared pool, sleeps, and is woken with the logs; its GPU budget is enforced outside the loop.
Figure 6: Anatomy of an environment. A produced environment is a single self-contained workspace, laid out so the problem stays separate from the solution. The briefing , data , and scoring fix what to solve and how it is judged without referring to any one method, which keeps the task open to future approaches and stops a score from depending on the implementation. The method is the paper’s source method reproduced, and the scores it reaches are the reference we compare against, so a new method’s final score is the improvement it makes over them.
Figure 7: Selecting 1,000 papers from 124,816 indexed. In a first pass (a), we drop papers that are low quality or are mainly a type of theory, a benchmark, a dataset, or a single finding, which leaves 21,354. In (b), we look at these more closely and favor work that is both high quality and compute-friendly , keeping the best 1,000. In (c), we check the result: the papers we picked (blue) are clearly stronger and more reproducible than the pool we started from (grey).
| per paper | median | total |
|---|---|---|
| compute jobs | 76 | 239,506 |
| GPU-hours | 0.4 | 2,286 |
| CPU-hours | 6 | 28,338 |
| agent turns | 730 | 947,056 |
| input tokens | 28M | 51B |
| output tokens | 163k | 231M |
Table 1: Per-paper and total build cost. Median and the total cost over 1,000 papers.
Figure 9: An example environment. This environment turns a Transformer architecture paper into a constrained improvement task: raise ImageNet accuracy without changing scale, pretraining, or data. The scripts make the task runnable, the source and baseline implementations anchor the comparison, and the scoring matrix defines progress as weighted improvement over the reproduced source method.
Figure 10: Environments’ openness. From fully prescribed (0) to fully open (100).
Figure 11: What the environments ask for. Each task names, in its own words, what should get better.
Figure 12: Leaderboards. Mean FNI by model in improve mode (top) and innovate mode (bottom), averaged over visible and held-out splits. Zero matches the reproduced source method; higher is better. Improve mode gives models the source method, while innovate mode requires them to design from scratch. We use xhigh effort for all models.
Figure 13: The two abilities are loosely tied. Each model’s rank at improving against its rank at innovating.
Figure 14: Effort versus outcome, and research quality. (a) each run’s score against the compute hours (top) and agent turns (bottom) behind it, pooling both modes; effort explains almost none of the variance in score ( , ). (b) research quality by dimension, judged apart from the score. Newest model per family, colored by provider.
| record | average |
|---|---|
| idea revisions | 3.7 |
| method revisions | 6 |
| experiments run | 25 |
| commits | 35 |
| hours of work | 5.4 |
Table 4: Dataset statistics. Each experiment is stored with its log; hours of work is the scientist’s end-to-end running wall-clock time.
Figure 15: Raised in the world. Median held-out FNI of each stage’s ideas, implemented identically; after GRPO it reaches the frontier cohort’s anchor.
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 16: System design overview. Hive supplies execution capacity for CPU and GPU jobs; Nexus accounts for the budget and resource boundary of each attempt; Swarm maintains one paper-level trajectory across roles, restarts, and handoffs; and Workers connect different agent scaffolds through the same system interface. Together, these layers let many agents iterate on open-ended papers while preserving the evidence, budget state, and handoff context.
Figure 17: Hive execution overview. A job request enters Hive with a command, resource target, working directory, and time limit. Admission queues the request, dispatch routes it to test, CPU, or GPU capacity, and monitor records the return code, runtime, and logs that form the job record.
Figure 18: Nexus operations overview. Nexus stores the job, project, budget, and review state needed to translate agent requests into Hive actions and to translate Hive execution evidence back into agent decisions.
Figure 19: A Nexus project turns resource policy into explicit accept/reject decisions. The example shows how budget, concurrency, and allowed resource pools gate agent requests before they reach Hive.
Figure 20: Swarm coordination overview. Swarm drives a pool of paper workspaces in parallel. Each paper moves through an ordered role workflow—reproduce, design, package, verify—whose transitions are driven by wake events: worker start, job submit, job update, validation failure, and role transition. Progress is graded against a milestone ladder, from none up to full reproduction, and the milestone gate decides whether a paper’s artifact is accepted.
| event | shown to agent | kept in trajectory |
|---|---|---|
| iteration start | paper, workspace, milestone, budget | model, role, starting checkpoint |
| job running | streaming logs and remaining budget, to continue or cancel | the continue or cancel decision |
| job finished | logs, exit code, and remaining budget | outcome, compute used, next action |
| build failed | the build log, to fix and resubmit | failure cause, fix attempts |
| budget exhausted | budget spent, no further jobs, wrap up current work | final spend, the milestone the agent wraps up at |
| validation fail | validation or milestone-gate failure | resulting status, revert attempts |
Table 5: Swarm wakes a builder only at decision-changing events. Rather than holding a session open for the whole reproduction, Swarm re-engages the builder only when new information changes what it should do next. Each row gives the triggering event, what the agent is shown, and the state Swarm writes into the paper’s trajectory.
| role | main responsibility | gate to continue |
|---|---|---|
| reproducer | implements the paper method and tries to reproduce the central claim. | reach core_claim , or stop or revert. |
| designer | converts the reproduction into gym targets, constraints, and scoring. | scores must validate |
| packager | consolidates scripts, environment files, and download logic. | workspace needs to be able to rebuild. |
| verifier | audits whether the final evidence supports the claimed milestone. | unsupported claims have their milestone lowered. |
Table 7: Role separation keeps one workspace auditable across builder handoffs. The role split separates reproduction, environment design, packaging, and verification while keeping all roles on the same workspace and evidence trail.
Figure 21: Swarm workflow stages. The standard workflow expands reproduction, design, packaging, and verification into repeated builder passes; each row carries the model profile, gate, GPU budget, and possible revert target.
Figure 22: Worker runtime overview. Different agent scaffolds connect through one uniform turn interface. Worker runs the agent under a container runtime with capped resources and a read-only validator, cycling through a turn loop of render prompt, agent acts, and submit jobs. Each turn ends by selecting the next action in action.yaml , which the system reads before waking the agent for the following turn.
Figure 23: The action.yaml that ends a turn. The agent writes it to request the next job or to stop, and the system reads it after every turn.
Figure 24: Builder overview. From a paper (in PDF and Markdown) and an empty starter workspace, builders build a reusable environment along three tracks: a problem definition that states the task and its constraints without leaking a solution, a reproduction of the paper’s method whose code and experiments recover the central claim, and an evaluation harness with visible and hidden slices that turns outputs into scores.
| condition | what the builder is asked to do |
|---|---|
| new paper | read the paper and reproduce its method from a full briefing |
| continued iteration | audit the previous workspace and push it further, starting one milestone below the prior claim |
| job running | read the streaming logs and decide whether to let the job continue or cancel it |
| job finished, budget left | inspect the results, record progress, and plan the next job |
| job finished, budget spent | stop submitting and wrap up the milestone, scoring files, and environment |
| build failed | read the build log, fix the container definition, and resubmit |
Table 8: Each turn is prompted for its condition. The first turn is a full role briefing; later turns receive a short prompt for whatever event woke the builder, keeping it on task across a long unattended run.
| group | purpose |
|---|---|
| briefing/ | states the task and how success is measured, while keeping the paper’s method out of the problem files |
| method/ , baseline/ | the reproduced implementation, with the baselines used to interpret its score |
| data/ , eval/ | the data interface and a method-agnostic evaluator that any future method must run against |
| scoring/ | the scoring contract: how each metric counts, what an improvement means, and what must stay fixed |
| scripts/ , environment/ | everything needed to rebuild the environment and rerun the method from a fresh clone |
| root files | the current action, the claimed milestone, progress and deviation notes, and the automatic checks |
Table 9: Each directory has one responsibility. The workspace keeps task context, method code, evaluation, scoring, execution, and record state apart, so a later scientist can reuse it without the original conversation.
Figure 26: The scoring contract, held in two files. scoring/reference.json fixes the weights, metric directions, coefficients, and the paper’s reported numbers; scoring/scores.json holds the numbers the reproduction actually produced, under the same keys.
Figure 27: Each design choice targets a concrete failure mode. The workspace is structured to avoid solution leakage, method-coupled scoring, unrebuildable dependencies, evaluator overfitting, and unsupported claims.
| group | fields recorded | use |
|---|---|---|
| reproduction summary | final milestone and status, scaffold and model, workflow role and step, and turn, run, and iteration counts | attributes the result to the builders and role that produced it |
| compute and cost | GPU and CPU hours, the GPU budget, jobs run, wall-clock duration, tokens, and dollar cost | separates a scientific failure from an execution failure, and compares spend |
| components present | whether the container, method, evaluator, scripts, reference, and data are all in place | confirms the workspace is complete enough to rerun |
| per-builder records | one entry per builder, with its turns, its jobs, and why it stopped | reconstructs the result across handoffs |
Table 10: The finished workspace records how it was produced. manifest.json ties each accepted artifact to the builders, compute, and components behind it, so it stays auditable after the session ends.
| check | what it verifies |
|---|---|
| scoring schema | reference.json parses, experiment weights sum to one, and each experiment declares a primary metric, its metrics, and its results; every metric declares a direction and a coefficient; and every method is typed as a baseline or a proposed method, with at least one proposed |
| no paper-agreement metrics | metric names may not encode closeness to the paper ( deviation , error_vs_ , agreement_with , and similar); the score must measure the scientific quantity, not fidelity to the publication |
| scores match the schema | scores.json uses the same experiments, methods, and metric names as reference.json , and every value is numeric |
| reproduction sanity | each reproduced primary-metric value stays close to the paper’s: no sign flip, under a magnitude gap, and under a absolute gap for bounded metrics |
| evaluator independence | eval/ and data/ never import from method/ , so the evaluator works for any method, not only the reproduced one |
| fresh-clone integrity | every local import in eval/ , method/ , and baseline/ resolves to a git-tracked file, and no committed code references the machine-local shared/ cache |
Table 11: What validate.py checks before a workspace is accepted. The validator enforces the scoring schema, blocks scores that measure agreement with the paper, and verifies that the workspace is independent of its own method, rebuildable from a fresh clone, and free of train/test shortcuts.
Figure 28: Screening preserves nearly all unique papers before deeper analysis. Counts are computed from the selection snapshot used by the export. Shares after the unique-index row use 124,816 as the denominator.
Figure 29: The source pool is largest in cs.AI and cs.LG before deduplication. These are category-listing records before ID-level deduplication; a single paper can contribute to multiple rows.
Figure 30: Fast screens keep broad coverage while identifying metric-method papers as the dominant type. The left side summarizes 123,980 fast quality scores; the right side summarizes 124,018 fast paper-type labels.
Figure 31: Structured-analysis score distributions across the 21,354 analyzed papers. Each panel gives the share of papers at each integer score from 0 to 10: the five reproduction-difficulty sub-scores (coding, compute, time to run, data or model, and overall), and the paper-level quality and problem-novelty scores. Dashed lines mark the means. Difficulty scores concentrate in the low-to-middle range, while quality and problem novelty center higher.
Figure 32: Feasibility filters reduce 21,354 analyzed papers to 8,987 candidates. The feasible candidates are ranked by quality-feasibility utility before the final category-balanced selection.
Figure 33: Temperature-balanced sampling keeps large areas prominent while preserving smaller categories. The final 1,000-paper selection is sampled from 8,987 feasible candidates across 18 canonical categories.
Figure 34: What environments ask, and the work it takes. Each environment states what should get better and what work that takes; phrases are grouped into families that emerge from data; ribbon mass is the share of environments.
| evaluation stage | contract enforced by the harness |
|---|---|
| scaffold | create a fresh per-run workspace and remove files that would leak hidden tests or reference scores |
| main phase | let the scientist edit the method and submit compute jobs until it writes train-phase scores and declares completion |
| validation | check that every required proposed-method metric is present; missing cells trigger bounded repair turns |
| held-out test | materialize the hidden evaluator only after the method is frozen, then run it through Nexus |
| normalization | convert heterogeneous metric improvements into per-paper scores before averaging into a leaderboard |
Table 12: Evaluation separates iteration, validation, hidden testing, and normalization into auditable stages. The staged harness keeps scientist behavior inspectable and comparable.
Figure 35: How often scientists beat the paper. Distribution of held-out improvement over the paper’s source method, per mode. Colored bars (right of zero) beat the reproduced method; the grey tail regressed. The percentage is the share of runs that improved.
Figure 36: Training gains carry over to held-out test. Training-phase score against held-out test score, per mode, with the line. Points hug the diagonal; the shaded quadrant holds the few runs that gained on training but regressed on test.
Figure 37: Effort versus outcome. Within-paper normalized score against GPU-hours (top row) and agent turns (bottom row), shown separately for the Improve and Innovate modes. Scores are min-max normalized within each (paper, mode); each line is a least-squares fit in log-effort. Every panel is flat ( ).
| resource | count | role |
|---|---|---|
| A100 GPU nodes | 16 | general accelerator workloads and pilot experiments |
| H100 GPU nodes | 16 | high-throughput training and evaluation jobs |
| H200 GPU nodes | 16 | memory-intensive or largest-scale reproduction runs |
| CPU nodes | 4 | low-compute, CPU-only setup and validation jobs |
Table 13: Hive exposes heterogeneous capacity for accelerator-heavy and CPU-only experiments. The counts describe the compute capacity exposed in Hive.
More in this topic
Autonomous Research Agents
Autonomous research agents use language models and tools to search literature, design and run experiments, analyze results, and support scientific discovery.
See all 338 papers