Humanity has long sought to explain natural phenomena. Reliable prediction of experimental outcomes is a central test of our understanding of the processes that produce them. If artificial general intelligence (AGI) is to surpass human scientific capabilities, it must also predict experimental outcomes more reliably than humans. Such capabilities could accelerate scientific discovery across disciplines, help address humanity's most pressing challenges, and improve human well-being. We introduce Science Prediction Universe (SPU), a benchmark of hidden experimental outcomes with 100 public verified tasks from chemistry, materials science, biology and medicine, and 16 private tasks from unpublished wet-lab experiments. Given an experimental setup and candidate conditions, an agent works with any resources it needs to rank the candidates by a measured outcome. We evaluate thirteen agent configurations spanning twelve frontier models across two harnesses (Codex and Claude Code). Current agents remain far from saturating wet-lab outcome prediction, with substantial performance disparities across models: on the public benchmark, configurations separate into two distinct tiers, with task-adjusted Spearman correlations ranging from 0.227 for the lowest-performing configuration to 0.478 for the best. Analysis of 1,155 scored public trajectories (227,103 tool calls) suggests that the bottleneck lies in scientific judgment rather than tool execution: agents rely on shared monotone priors, agreeing far more with one another than with experimental reality (0.52 vs. 0.33), and frequently compute trivial physical or chemical proxies that do not influence the final outcome.
Figures & tables
Figure 1: Configurations ranked by mean score on all scored tasks. Solid bars are means over the tasks, among the 100 public and 16 private tasks, on which the configuration has a scored run (Scored tasks). Hatched bars are means over the 28 shared tasks, the 27 public tasks and one private task on which all thirteen configurations have a scored run; the far-right column ranks the configurations on these tasks. Claude Opus 5.5 and Claude Fable 5.1 ended most of their runs in safety refusals, mainly on chemistry, biology and medicine tasks, so they are scored on fewer tasks and only 28 tasks are shared by all thirteen configurations, against 94 shared by the other eleven. Lines are 95% bootstrap intervals over tasks, with the same resampled tasks for every configuration on the shared tasks. A perfect ranking scores 1, and random rankings average 0. GPT-6 Astra ran in Codex at medium and max reasoning effort, GPT-6.1 Sol in Codex at max effort, and the other models in Claude Code at max effort.
Figure 2: The SPU setting, shown with one official run. Left, the task: rank 13 copper-catalyzed deaminative arylations of one amine-borane, which differ only in the base additive, by measured 1 H NMR yield. Middle, GPT-6 Astra at max reasoning effort worked in Codex for 6.6 of the 8 available hours, reaching the web only through the audited service and revising its ranking twice. Right, the files the run submitted. Bottom, the measured yields stay in a separate grading container and are never shown to the agent; this submission scores 0.75, above the task’s chance threshold of 0.47.
Figure 3: Coverage of the Science Prediction Universe. a , The 116 tasks by discipline and subfield; sector angles are proportional to task counts, and hatching marks private tasks. b , Sources of the tasks: 85 papers in 11 Nature Portfolio journals, published online between January 2025 and August 2026, for the 100 public tasks, and four series of unpublished wet-lab experiments for the 16 private tasks; journals with one source paper are pooled. c , Number of experiments in each task on a logarithmic scale, with open circles for private tasks and black ticks at discipline medians.
Figure 4: Public Task Construction. More than ten experts in five fields built the public tasks in four steps. Step 2 avoids two kinds of comparison (crossed charts): orderings that rise or fall strictly with one varied parameter, and values that differ only slightly. In Step 3, each task is reviewed independently, and a task with a blocking issue is fixed and reviewed again (dashed arrow). Step 4 selects tasks by pilot scores, mainly those of GPT-6 Astra; the 100 squares on the right are the public tasks, 29 giving every subfield a task and 71 chosen by composite score.
Figure 5: Outcomes of the official runs on the public tasks before exclusion. a , Outcome of each of the 1,300 official runs, with tasks grouped by discipline and ordered within each by mean score. Valid submissions and recovered files are scored, and all other runs are excluded. b , The grader’s scores of all 1,300 runs, with every run without a valid file at −1 .
Figure 6: Files rejected for formatting errors before and after recovery. a , The 94 formatting-rejected files by how they depart from the required JSON schema, with a schematic of each type (departures shaded) and the files from each configuration. b , Scores of the recovered files and of the valid submissions of the same seven configurations; numbers under the columns count runs. c , Recovered files minus valid submissions within each configuration, with each run’s score taken relative to the mean of the other configurations on the same task; the weighted mean weights each configuration by its recovered files. Lines are 95% bootstrap intervals.
Figure 7: Refusals in the official runs on the public tasks. a , Share of the 100 public tasks refused by each configuration; every refusal by Opus 5.5, Fable 5.1, Opus 5 and Sonnet 5.5 carries the [bio] label, and GPT-6.1 Sol’s two were flagged by its provider for possible biological risk. b , Share of the tasks in each discipline refused by Opus 5.5 and Fable 5.1.
Figure 8: Mean score of each configuration under the two scoring rules that penalize a missing or formatting-rejected file. a , Formatting-rejected files and runs without a submission scored at −1 . b , Formatting-rejected files scored by their recovered rankings, with runs without a submission still at −1 . Both rules cover the public and the private tasks and exclude refusals; Runs gives the number of runs behind each mean, and lines are 95% bootstrap intervals over tasks.
Figure 9: Coverage and scores on the public tasks. a , Raw mean (open circles) and task-adjusted mean (filled circles), with 95% bootstrap intervals over tasks; n is the number of scored tasks. b , Scores of all scored runs, with medians and interquartile ranges. c , Share of scored tasks whose score exceeds the task’s chance threshold, with Wilson 95% intervals; the dashed line marks the 5% expected from random rankings. d , Number of tasks scored for both configurations of each pair.
Configuration
Scored tasks
Task-adjusted mean [95% CI]
Raw mean [95% CI]
Above chance
Opus 5.5
44
0.478 [0.395, 0.564]
0.491 [0.401, 0.578]
23 (52%)
Astra Medium
100
0.431 [0.373, 0.486]
0.431 [0.373, 0.487]
52 (52%)
Opus 5
99
0.424 [0.361, 0.487]
0.422 [0.359, 0.484]
51 (52%)
Fable 5.1
30
0.421 [0.326, 0.508]
0.453 [0.351, 0.551]
14 (47%)
Astra Max
100
0.409 [0.352, 0.466]
0.409 [0.352, 0.466]
50 (50%)
Sonnet 5.5
97
0.393 [0.328, 0.457]
0.388 [0.323, 0.450]
46 (47%)
Table 1: Results on the public tasks. Configurations are ordered by task-adjusted mean, and brackets give 95% bootstrap intervals over tasks. Above chance counts the scored tasks whose score exceeds the task’s chance threshold (Section 2.5 ).
Figure 10: Scores on the private tasks. Score of each configuration on each private task, grouped into the VHH (eight tasks), reaction-condition (three), electrolyte (two) and Magrolimab (three) series. Gray cells mark excluded runs: refusals, one timeout and one information-policy stop.
Configuration
Scored tasks
Mean [95% CI]
Above chance
P vs random (Holm)
Astra Medium
12
0.297 [0.056, 0.531]
5 of 12
0.0020
Astra Max
12
0.322 [0.083, 0.556]
4 of 12
0.0013
GPT-6.1 Sol
11
0.346 [0.091, 0.579]
5 of 11
0.0013
Opus 5.5
3
0.420 [ −0.286 , 0.927]
1 of 3
0.045
Opus 5
14
0.405 [0.191, 0.588]
6 of 14
0.0003
Sonnet 5.5
11
0.391 [0.138, 0.631]
5 of 11
0.0005
Table 2: Results on the private tasks. Rows follow the configuration order of Section 3.1 , and brackets give 95% bootstrap intervals over tasks, descriptive only for the four configurations scored on fewer than 12 tasks (Opus 5.5, Fable 5.1, GPT-6.1 Sol and Sonnet 5.5). Above chance counts the scored tasks whose score exceeds the task’s chance threshold, as in Table 1 ; the last column gives the Holm-corrected one-sided P for a mean above that of uniformly random rankings, a different comparison.
Configuration
Scored tasks
Input (M tokens)
Cache reads
Output (M tokens)
Run time (h)
Cost per scored run (USD) [95% CI]
Total, 100 runs (USD)
Opus 5.5
44
25.6
93%
0.49
2.74
24.11 [19.67, 28.85]
1,075
Astra Medium
100
5.1
96%
0.05
0.46
9.59 [6.99, 13.05]
959
Opus 5
99
35.4
96%
0.46
3.81
37.44 [31.59, 43.61]
3,717
Fable 5.1
30
10.8
80%
0.15
1.55
36.42 [24.81, 49.60]
1,158
Astra Max
100
30.4
95%
0.34
2.90
65.91 [53.22, 79.84]
6,591
Sonnet 5.5
97
20.7
94%
0.49
2.63
9.80 [8.57, 11.09]
954
Table 3: Estimated API cost on the public tasks. Rows follow Table 1 . Token counts, run time and cost are means over the scored official runs of each configuration, and the last column sums the cost of all 100 official runs. Input counts all prompt tokens, Cache reads is the share of input read from the prompt cache, and Output includes reasoning tokens. Costs use the list prices of Table 7 , with cache reads and writes billed at their own prices; brackets give 95% bootstrap intervals over runs.
Figure 11: Cost–performance Pareto efficiency curve on the public tasks. Each point represents one configuration, plotting estimated mean API cost per scored run (Appendix E.6 ) against task-adjusted mean score, with 95% bootstrap intervals. Filled markers connected by the dashed line define the cost–efficiency Pareto frontier. Opus 5.5 and Fable 5.1 are evaluated on their 44 and 30 scored tasks.
Figure 12: Five repeated runs of Astra Max on 29 public tasks. One task was drawn at random from each subfield, and Astra Max ran five more times on each. a , Scores of the repeated runs and of the official run on each task. b , Official score against the mean of the scored repeated runs. c , Agreement between two repeated runs against agreement between a repeated run and the official runs of the other twelve configurations. Dashed lines mark equality.
Figure 13: Guarding against shortcuts. a , What the agent can reach, what the audited service records and what stays in the grading container. b , Score of the order in which the experiments are presented on each public task, against the task’s chance threshold, with the range of the configurations’ task-adjusted means. c , Indicators of memory and network use, as percentages of each configuration’s scored runs (for the source DOI, of the scored runs whose task has one). d , Difference in score between the runs flagged by the identifier detector and the unflagged runs of the same configuration, both measured against the other configurations on the same task, with 95% intervals; the band shows the pooled difference.
Figure 14: How agents spend a run. a , Share of time spent on each activity along a run, for the 1,155 scored official runs on the public tasks; each run is normalized to its own length and counts equally, the shares are smoothed with a Gaussian kernel of s.d. 2% of the run, and the activities are defined in Appendix E.7 . b , Median and interquartile range of the point in the run at which each milestone is reached.
Figure 15: Methods and information sources. a , Share of the 1,155 scored official runs that used each method family, over all runs and for each configuration; the families are defined in Appendix E.7 . b , The same share by discipline. c , Source class of the 24,789 URLs fetched through the audited fetch tool. d , Share of a run’s search queries that use each operator, averaged over the 1,002 runs that searched.
Figure 16: How the submitted rankings are produced. a , Where the submitted order of each of 4,230 groups first appears in the trajectory. b , Point in the run at which the first complete ranking was written, with medians (bars) and, on the right, the share of all runs that wrote it in the first half. c , On 37 tasks, the absolute Spearman correlation between a numeric experimental condition and the submitted orders against that for the measured order; the dashed line marks equality. d , Share of groups whose submitted order is monotone in the condition ( ∣ρ∣≥0.95 ), with ticks for the measured order.
Figure 17: What relates to score. a , Score difference between runs with and without each method or behavior, with configuration and task fixed effects (Appendix E.7 ); filled markers have P<0.05 before correction, and none remains below 0.05 after Holm correction within each group. b , Expected Spearman correlation stated in 63 reports against the score of the run; the dashed line marks equality. c , Score of the submission minus that of the first complete ranking, for the 317 runs whose submission differs from it; diamonds are means. Lines are 95% intervals from resampling tasks.
Figure 18: Tasks that separate the configurations. a , Lowest and highest score among the configurations on each public task, with the mean, ordered by the standard deviation of the scores. b , Spread between configurations on a task against the spread between the repeated runs of Astra Max on the same task. c , Astra Max’s official score minus the mean of the other configurations, against the same difference computed from its repeated runs. d , Correlation between the per-task scores of two configurations, for the 55 pairs among the eleven configurations with the widest coverage; the dashed line marks the value expected without a configuration-by-task interaction. e , On the hydrogel task, the correlation between each submitted order and five monomer fractions, with the measured order at the top. Dashed lines in b and c mark equality, and the solid line in c is a least-squares fit.
Figure 19: What varies between repeated runs. a , Share of the comparison groups that received the same order in every scored repeat of their task, by the number of items in the group. b , Order submitted for each group of an underwater adhesive task in its five repeats, with the items shaded by their place in the most common order of the group, and the score of each repeat. c , How many of the repeats of each task used each method family, and the difference in score between the repeats of a task that used a family and those that did not, for families used by some but not all repeats on at least four tasks. d , Scores of the repeats on the tasks where some repeats stopped under the information policy, and the mean score of the stopped repeats minus that of the others, with the mean over these tasks and its 95% bootstrap interval.
Figure 20: Scores by discipline. a , Mean score of each public task over the configurations scored on it, by discipline, with the raw mean of the discipline and its mean chance threshold. b , Raw mean score of each configuration in each discipline, with the number of scored tasks. c , Mean score of Astra Medium, Opus 5, Astra Max and Sonnet 5.5 minus that of DeepSeek, Qwen3.8, GLM-5.3, Doubao, HY4 and Kimi-K3.1-Pro on the same task, by discipline; below, the largest minus the smallest of the four differences, against its values when the disciplines of the tasks are shuffled (grey). d , Rank correlation between the discipline scores of the eleven configurations for the six pairs of disciplines (points) and their mean (line), against random splits of the task set into blocks of the same sizes (grey). Error bars in a and c are 95% intervals.
Figure 21: Scores and revisions on selected tasks. a , Score of every scored official run on the tasks examined in detail, grouped by discipline. b , Score of the first complete ranking and of the submission, for those of these runs whose submission differs from it; numbers count the flipped pairs that moved into the measured order.
Figure 22: Use of the GPU. a , Share of runs at each level of GPU use, by configuration. b , Main workload of the runs that used the GPU, by discipline. c , Share of the runs that started a workload hitting each kind of GPU error. d , Median run time of runs that used the GPU and of the others, by configuration. e , Difference between runs that used the GPU and the other runs of the same configuration, after subtracting the mean of the other configurations on the same task, with 95% intervals.
Figure 23: Configuration profiles. a , Working habits of each configuration, as medians over its runs or, for task lists, subagents and background jobs, as the percentage of runs that used them; colors show z scores across the thirteen configurations, and task lists and subagents are not available in Codex. b , Confusion matrix of a random forest that predicts a run’s configuration from its behavior, for the 861 runs of the ten configurations under Claude Code (five-fold cross-validation).
Figure 24: The same model at two reasoning efforts. a , Difference in score between Astra Medium and Astra Max on each of the 100 public tasks. b , Mean difference under four ways of scoring Max, with 95% bootstrap intervals. c , Ratio of Max to Medium on the same task for run time, output tokens, code written and web searches, and the share of runs that used a GPU. d , Score of Max’s first complete ranking, of its own submission and of Medium’s submission on the tasks where that draft is visible, and the draft minus each submission, with 95% intervals. e , Share of the item pairs flipped after the draft that moved into the measured order, against the share expected from random flips at the same distance in that draft, and the difference, with 95% intervals.
Benchmark
Domain
Output
Wet-lab outcome
Agentic compute
Hours-long runs
Open web
Humanity’s Last Exam [ Phan et al., 2025 ]
Multi-discipline
Closed-ended QA
✗
✗
✗
✗
BrainBench [ Luo et al., 2024 ]
Neuroscience
2-way choice
✓
✗
✗
✗
SciPredict [ Sehwag et al., 2026 ]
Phys./Chem./Bio.
Answer / value
✓
✗
✗
✗
ProteinGym [ Notin et al., 2023 ]
Protein fitness
Spearman ranking
✓
✗
✗
✗
AssayBench † [ De Brouwer et al., 2026 ]
CRISPR screens
Gene ranking
✓
△
✗
✗
ScienceAgentBench [ Chen et al., 2025 ]
Data-driven science
Program
✗
✓
✗
✗
Table 4: SPU compared with related evaluations. Wet-lab outcome : the target is a measured experimental result. Agentic compute : the model runs code and scientific software of its own choosing. Hours-long runs : a single run may last hours (Terminal-Bench 4.0: 8 h limit). Open web : retrieval is allowed during the run ( △ : allowed with an instruction not to look up answers, but not audited). ✓ yes, △ partly, ✗ no, – not stated in the release. † AssayBench evaluates agent baselines alongside direct LLM prediction.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
Chemistry (39)
Materials science (29)
Biology (20)
Medicine (12)
All (100)
Opus 5.5
0.606 (14)
0.475 (26)
0.185 (4)
–
0.491 (44)
Astra Medium
0.451 (39)
0.398 (29)
0.384 (20)
0.520 (12)
0.431 (100)
Opus 5
0.453 (39)
0.394 (28)
0.315 (20)
0.565 (12)
0.422 (99)
Fable 5.1
0.416 (6)
0.502 (21)
0.191 (3)
–
0.453 (30)
Astra Max
0.409 (39)
0.461 (29)
0.335 (20)
0.406 (12)
0.409 (100)
Sonnet 5.5
0.374 (39)
0.393 (28)
0.370 (19)
0.459 (11)
0.388 (97)
Appendix
Table 5: Raw mean score by discipline on the public tasks. Each cell averages a configuration’s scored official runs in one discipline, with the number of scored tasks in parentheses; the header gives the number of public tasks in each discipline, and – marks a discipline without a scored task. Rows follow Table 1 .
Configuration
Rejected
Refusal
No file
Mean at −1
After recovery
Scored mean
Astra Medium
0
0
0
0.431
0.431
0.431
Astra Max
0
0
0
0.409
0.409
0.409
GPT-6.1 Sol
0
2
4
0.259
0.259
0.308
Opus 5
18
1
0
0.168
0.421
0.422
Sonnet 5.5
0
1
2
0.347
0.347
0.388
DeepSeek
3
0
0
0.221
0.254
0.254
Appendix
Table 6: Outcomes and the grader’s scoring on the public tasks. Each configuration has 100 official runs. Mean at −1 is the grader’s default convention, which scores every run without a valid file at −1 ; the next column replaces the formatting-rejected files by their recovered scores, and Scored mean is the raw mean over the runs that remain. Five runs left a valid file before refusing, and the grader scored those files rather than giving them −1 . Two runs of GPT-6.1 Sol stopped under the information policy with an empty list of rankings, which the grader also scores at −1 ; they are counted under No file, as is the run of GPT-6.1 Sol that the memory limit killed after the grader had accepted its file. Kimi-K3.1-Pro’s five runs without a file are three information-policy stops and two runs that ended without a submission, and Sonnet 5.5’s two also ended without a submission.
Figure 25: Paired comparisons on the public tasks. a , Upper triangle, mean paired difference (row minus column) on the tasks scored for both configurations, with bold boxes for pairs that pass Holm correction (two-sided Wilcoxon signed-rank test); lower triangle, difference detectable with 80% power at α=0.05/78 , the strictest level the Holm correction applies; pairs whose detectable difference exceeds 0.304 are in italics in both triangles. b , Observed against detectable difference for each pair. c , Number of configurations detectably below (right) or above (left) each configuration. d , Per-task difference between Astra Medium and Astra Max, with the uncorrected Wilcoxon P . e , Scores of Opus 5.5 and Fable 5.1 on their 30 shared tasks.
Figure 26: Run-to-run variation and rank stability on the public tasks. a , Standard deviation of Astra Max scores across the repeated runs on each of 29 tasks. b , Reliability of a task mean over k runs. c , Shares of the variance of single-run task scores due to tasks, configurations, run-to-run noise and the configuration-by-task interaction. d , Difference detectable with 80% power at an uncorrected α=0.05 , against the number of shared tasks. e , Detectable difference at the same power and level, for a fixed number of runs spent on new tasks or on repeated runs of 100 tasks. f , Ranks of the thirteen task-adjusted means in 10,000 simulated reruns, with 95% intervals and the observed ranks (ticks).
Model
Input
Cache read
Cache write
Output
GPT-6 Astra
10.00
1.00
12.50
50.00
GPT-6.1 Sol
2.00
0.10
2.50
10.00
Claude Opus 5.5
4.00
0.20
5.00 (8.00)
20.00
Claude Opus 5
5.00
0.50
6.25 (10.00)
25.00
Claude Sonnet 5.5
2.00
0.10
2.50 (4.00)
10.00
DeepSeek-V4.1-Flash-0910
0.15
0.003
n/a
0.60
Appendix
Table 7: List prices used for the cost estimate. USD per million tokens, checked on 20 October 2026 (GPT-6.1 Sol and Claude Opus 5 on 23 October, Kimi-K3.1-Pro on 26 October, Claude Sonnet 5.5 on 29 October). GPT-6.1 Sol: short-context rates; 26 of its 22,608 requests exceed 272K input tokens and are billed at the long-context rates (4.00, 0.20, 5.00 and 15.00). Cache writes are billed at the five-minute rate; the one-hour rate is given in parentheses where it differs. n/a: the provider reports no cache writes, and tokens not read from the cache are billed at the input price. DeepSeek: off-peak rates. Qwen3.8: international rates for explicit caching. Doubao: converted from yuan at 6.72 CNY per USD.
Figure 27: Agreement between configurations. a , Agreement between configurations on each public task, the mean Spearman correlation between the orders of two different configurations, against the mean score of these submissions; the dashed line marks equality. b , Mean score of each configuration minus that of the consensus of the other configurations on the same tasks. c , Spearman correlation, across the runs of each configuration, between a run’s agreement with that consensus and its score. Lines are 95% bootstrap intervals over tasks.
Figure 28: Scores on individual runs. a , Measured rank against submitted rank for five official runs scoring about 0, 0.2, 0.4, 0.6 and 0.8, where rank 1 is the lowest. b , The same for one run per discipline at the median score of 0.40. c , Share of experiment pairs in the measured order against score for all 1,155 scored public runs, with the median run circled. d , By configuration, mean share of pairs in the measured order and share of runs whose top-ranked experiment is the measured top, with 95% bootstrap intervals over tasks.
Accelerating scientific discovery requires the identification of which experiments would yield the best outcomes before committing resources to costly physical validation. While existing benchmarks evaluate LLMs on scientific knowledge and reasoning, their ability to predict experimental outcomes - a task where AI could significantly exceed human capabilities - remains largely underexplored. We introduce SciPredict, a benchmark comprising 405 tasks derived from recent empirical studies in 33 specialized sub-fields of physics, biology, and chemistry. SciPredict addresses two critical questions: (a) can LLMs predict the outcome of scientific experiments with sufficient accuracy? and (b) can such predictions be reliably used in the scientific research process? Evaluations reveal fundamental limitations on both fronts. Model accuracies are 14-26% and human expert performance is ≈20%. Although some frontier models exceed human performance model accuracy is still far below what would enable reliable experimental guidance. Even within the limited performance, models fail to distinguish reliable predictions from unreliable ones, achieving only ≈20% accuracy regardless of their confidence or whether they judge outcomes as predictable without physical experimentation. Human experts, in contrast, demonstrate strong calibration: their accuracy increases from ≈5% to ≈80% as they deem outcomes more predictable without conducting the experiment. SciPredict establishes a rigorous framework demonstrating that superhuman performance in experimental science requires not just better predictions, but better awareness of prediction reliability. For reproducibility all our data and code are provided at https://github.com/scaleapi/scipredict
Can AI systems trained on the existing scientific record forecast the advances that will follow? We introduce PreScience, a dataset and benchmark for scientific forecasting built around 98K recent AI research papers, together with companion papers covering author publication histories and citation links, yielding 502K papers in total. The resulting paper records include titles, abstracts, disambiguated author identities, influential references, topic labels, citation trajectories, and metadata snapshotted to respect temporal cutoffs. We instantiate seven exemplar tasks: five paper-anchored tasks -- contribution generation, collaborator prediction, prior work selection, citation count prediction, and future combination prediction -- and two aggregate topic trend forecasting variants. We develop baselines ranging from simple heuristics and embedding methods to frontier language models and agentic systems, and introduce LACER, an LLM-based metric for evaluating similarity of generated contribution descriptions that agrees better with human judgments than existing metrics. Finally, we compose task models to generate a 12-month synthetic corpus and find that the resulting papers are systematically less diverse and less novel than human-authored research from the same period. We release the PreScience dataset (https://huggingface.co/datasets/allenai/prescience) and code (https://github.com/allenai/prescience).
Anirudh Ajith, Amanpreet Singh, Jay DeYoung +7
Allen Institute for Artificial Intelligence, Seattle, WA, USA · Knowledge Lab, University of Chicago, Chicago, IL, USA · School of Computer Science and Engineering, Hebrew University of Jerusalem, Israel +1
Scientific progress depends not only on finding solutions, but on learning the rules that explain why they work and using that understanding to design better experiments. We introduce science sandboxes, a framework for studying this capability in AI agents through repeated cycles of experimentation, feedback, and hypothesis revision. Science sandboxes invite an agent to query the natural world in different ways, ranging from "wet" physical experiments, to "damp" predictive models trained on empirical data, to "dry" invented rules. By establishing a common experimental loop and a protocol for evaluating agents within it, science sandboxes allow assessment of both quantitative performance on specific metrics and qualitative scientific reasoning, across a spectrum of empirical verifiability. Here, we instantiate this framework in two biological settings, models of regulatory genomics and protein fitness prediction, and examine the capabilities of frontier agents. Across these settings, we could see when agents successfully optimized a quantitative metric without understanding the rules underlying the system. In particular, their scientific reasoning deteriorated when they encountered systems whose rules fell outside familiar biological priors. By highlighting such failure modes, science sandboxes make the frontier of scientific capability measurable and provide a controlled setting in which to study and ultimately expand it.
Arya S. Rao, Rodrigo I. Castro, Sager J. Gosai +8
The Broad Institute of MIT and Harvard; Cambridge, MA 02142, USA. · Department of Biomedical Informatics, Harvard Medical School; Boston, MA 02115, USA. · The Jackson Laboratory; Bar Harbor, ME 04609, USA. +10