cs.AIApr 12, 2026

SciPredict: Can LLMs Predict the Outcomes of Scientific Experiments in Natural Sciences?

Authors: Udari Madhushani Sehwag, Elaine Lau, Haniyeh Ehsani Oskouie, Shayan Shabihi, Erich Liang, Andrea Toledo, Guillermo Mangialardi, Sergio Fonrouge, 

Organizations: Scale AI · University of California, Los Angeles · Human Frontier Collective, Scale AI · University of Maryland · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Accelerating scientific discovery requires the identification of which experiments would yield the best outcomes before committing resources to costly physical validation. While existing benchmarks evaluate LLMs on scientific knowledge and reasoning, their ability to predict experimental outcomes - a task where AI could significantly exceed human capabilities - remains largely underexplored. We introduce SciPredict, a benchmark comprising 405 tasks derived from recent empirical studies in 33 specialized sub-fields of physics, biology, and chemistry. SciPredict addresses two critical questions: (a) can LLMs predict the outcome of scientific experiments with sufficient accuracy? and (b) can such predictions be reliably used in the scientific research process? Evaluations reveal fundamental limitations on both fronts. Model accuracies are 14-26% and human expert performance is ≈\approx20%. Although some frontier models exceed human performance model accuracy is still far below what would enable reliable experimental guidance. Even within the limited performance, models fail to distinguish reliable predictions from unreliable ones, achieving only ≈\approx20% accuracy regardless of their confidence or whether they judge outcomes as predictable without physical experimentation. Human experts, in contrast, demonstrate strong calibration: their accuracy increases from ≈\approx5% to ≈\approx80% as they deem outcomes more predictable without conducting the experiment. SciPredict establishes a rigorous framework demonstrating that superhuman performance in experimental science requires not just better predictions, but better awareness of prediction reliability. For reproducibility all our data and code are provided at https://github.com/scaleapi/scipredict

Figures & tables

Figure 1: Key findings of SciPredict. Frontier models exhibit fundamental gaps in accuracy and calibration robustness in scientific experiment outcome prediction. We highlight four key failure modes using a representative subset of SOTA models: Claude O4.5 (Claude Opus 4.5), OpenAI GPT-5.2, Gemini 3P (Gemini 3 Pro), Llama 3.3 (Meta Llama 3.3 70B), and Qwen 3 235B. (a) Providing expert-curated background knowledge (BK) consistently boosts performance over No Background Knowledge (NBK). (b) Accuracy generally degrades when moving from multiple-choice questions (MCQ) to questions requiring free-form answers to Numerical value questions. (c) Unlike Human Experts (dahsed lines), models show poor calibration in SciPredict tasks; the accuracy of the models’ answers to tasks do not correlate with their self-reported Confidence and perceived task prediction Feasibility. (d) Model performance varies across different domains. The Avg field shown represents the weighted average of scores according to the number of questions per domain.
Figure 2: LLM-enhanced efficient scientific research workflow. The figure illustrates how LLM-powered experimental outcome prediction can be integrated into the scientific research process. Phase 1 involves ideation and experimental design through literature review and hypothesis formulation. Phase 2 represents a fast, low-cost prediction loop where LLMs predict experimental outcomes and identify high-potential experiments for physical validation, which researchers then review for plausibility. Based on this evaluation, researchers either proceed to Phase 3 (resource planning and experiment setup) and Phase 4 (empirical validation through physical experimentation), or archive the idea for later consideration. This workflow demonstrates how reliable LLM predictions could accelerate scientific discovery by filtering suboptimal experimental directions before committing to costly empirical validation.
Figure 3: SciPredict curation pipeline. The benchmark construction involves four integrated stages: (Top-Left) Data Collection from preprint repositories ensuring a post-March 2025 cutoff to prevent data leakage; (Top-Middle) Expert Annotation where domain specialists convert raw papers into MCQ, numerical, and free-form prediction tasks; (Bottom-Left) Task Structure enforcement, ensuring every sample includes granular fields such as experimental setup, measurements, and expert-curated background knowledge; and (Bottom-Middle) Quality Control/Assurance , combining iterative expert review with deterministic and LLM-based verifications. An example outcome prediction task is shown on the Right.
Figure 4: Accuracy with and without background knowledge. Accuracy (%) of each evaluated model under two input conditions: (a) W/o background knowledge : the model receives only the experimental setup, measurements, and the question; (b) W/ background knowledge : the same information as previous case with the addition of expert-curated background knowledge.
Figure 5: Restating known facts in context enhances model performance. We present a scatter plot comparing model accuracy given full expert-curated background (BK, y-axis) versus a filtered version of such background (FBK, x-axis) in context (see definitions under §Sec. 4.3 ). Each point represents a model, colored by family. The dashed diagonal indicates equal performance under both context conditions (y = x). Marker size indicating the percentage of the tasks’ background knowledge bullet points the model already knew ( >70%>70\% for all). Most models lie above the dashed identity line ( y=xy=x ), showing that explicitly restating already known required background knowledge in the context yields higher accuracy than relying on originally learned knowledge alone.
Figure 6: Human vs self-generated background knowledge. Evaluated accuracy (%) for the models under the four prediction conditions defined in §Sec. 4.3 . BK generally yields the highest accuracy, while SBK frequently degrades accuracy relative to NBK , indicating that models fail to reliably generate useful predictive context. Furthermore, SABK rarely improves upon BK , suggesting that adding synthetic information likely introduces noise or misleading cues even when the correct expert background information is available in context.
Figure 7: Models are poorly calibrated in self-reported confidence, difficulty, and feasibility, whereas human calibration correlated with accuracy. The top row plots empirical accuracy (y-axis) against model-provided confidence/difficulty/feasibility metrics. Each circle represents a model at a particular confidence/difficulty/feasibility level, and its color corresponds to the percentage of the number of questions assigned to that level by that model. The expected trends (accuracy rising with confidence/feasibility and falling with difficulty) are weak and non-monotonic. The bottom row shows the prediction accuracy of models and humans plotted against the human calibration metrics.The bottom-left subplot shows human accuracy with significant positive correlation with reated confidence and feasibility levels, while showing a significant negative correlation with the rated difficulty as expected. The bottom-middle and bottom-right subplots reveal that model accuracy recovers the expected trends when plotted against human-rated difficulty and feasibility levels, confirming that human judgment provides a superior signal for task outcome predictability compared to models’ self-reports. Circle colors correspond to the percentage of the number of total questions assigned to that calibration level (x-axis) by each model (top row) or human expert (bottom row). Each circle corresponds to a distinct model.
Figure 8: Analysis of model errors. We errors into a hierarchical taxonomy spanning five top-level (in black background) categories and 16 specific error types. The heatmap shows the percentage of incorrect responses containing each error type for each evaluated model. Error categories (as defined in Tab. 8 ) progress from surface-level issues (e.g., Comprehension & Scope) to deeper reasoning failures (e.g., Logical & Reasoning flaws) to fundamental scientific deficiencies (Scientific Rigor flaws). Models can exhibit multiple error types simultaneously, so accumulative percentage scores within top-level categories may exceed 100%100\% . SciPredict tasks contribute to top-level category percentages if flagged with at least one underlying error type. Fig. 13 shows the same chart only for tasks with human rated feasibility∈{4,5}feasibility\in\{4,5\} .
Figure 9: Accuracy is highly sensitive to the question format. Answer format drives large swings in model accuracy under identical prediction tasks. We evaluate each model in the NBK setting across four response formats: multiple-choice questions (MCQ), free-form (FF), numerical value (NUM), and MCQs rewritten into matched free-form prompts (MCQ →\rightarrow FF). Accuracy is generally the highest for MCQs, lower for free-form, and lowest for numerical predictions, and drops systematically when converting MCQs to free-form, showing that reported performance depends heavily on how the prediction is requested.
Figure 10: Domain specific accuracy. Heatmap of model accuracy (%) on benchmark questions, broken down by scientific domain (Biology, Physics, Chemistry). Results are provided for the evaluated models and human baseline. Overall, frontier models achieve the highest accuracies, but performance varies by domain; Chemistry tends to be the most challenging subset, and several models (including the human baseline) exhibit performance degradation on Chemistry relative to Biology/Physics. The Avg shown represents the weighted average respective to number of questions per domain.
Figure 11: Model accuracy on SciPredict correlates with performance on the HLE benchmark. Benchmark performance correlates with general hard-reasoning performance. This is a scatter plot of each evaluated model’s accuracy on SciPredict in the no-background-knowledge (NBK) setting (y axis) versus its HLE text-only accuracy (x-axis). The solid line shows a linear fit and the shaded region indicates the corresponding confidence bands. Overall, SciPredict NBK accuracy exhibits a moderate positive correlation with HLE performance (Pearson r ≈\approx 0.46), suggesting that broader reasoning capability explains some-but not all-variance in empirical outcome prediction.
Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 12: Diversity of the experts recruited for benchmark construction and human baseline. Top left : A plot of the highest degree distribution of experts recruited for benchmark construction. Top center : A plot of the domain expertise of experts recruited for benchmark construction. Top right : A plot of the highest degree distribution of experts recruited for human baseline. Bottom left : A heatmap of the countries of origin of experts recruited for benchmark construction. Bottom right : A heatmap of the countries of origin of experts recruited for human baseline.
Task DomainSubdomainHuman Baseline Subfields
PhysicsAll PhysicsAdvanced Chemical Engineering, Applied And Interdisciplinary Physics, Applied Physics And Interdisciplinary, Chemical Engineering, Classical And Mechanical Physics, Condensed Matter And Materials, Electromagnetism And Optics, Engineering Physics, High-energy And Nuclear Physics, Radiophysics & Electronics, Theoretical Physics, Zoology
 Condensed Matter & Materials PhysicsAdvanced Chemical Engineering, Applied Physics And Interdisciplinary, Chemical Engineering, Condensed Matter And Materials, Electromagnetism And Optics, Engineering Physics, Radiophysics & Electronics
 Materials ChemistryCondensed Matter And Materials, Engineering Physics
 Optics, Photonics & Laser PhysicsApplied Physics And Interdisciplinary, Condensed Matter And Materials, Electromagnetism And Optics, Engineering Physics, Radiophysics & Electronics, Zoology
 High-Energy / Nuclear / Particle PhysicsEngineering Physics, High-energy And Nuclear Physics, Radiophysics & Electronics, Theoretical Physics, Zoology
 Applied & Instrumentation PhysicsApplied And Interdisciplinary Physics, Applied Physics And Interdisciplinary, Classical And Mechanical Physics, Condensed Matter And Materials, Electromagnetism And Optics, Engineering Physics, High-energy And Nuclear Physics, Radiophysics & Electronics
Appendix
Table 1: Subfield expertise of human annotators, grouped by the task domains (Physics, Chemistry, Biology) and subdomains.
FieldSubfieldCount
PhysicsCondensed Matter & Materials Physics33
 Materials Chemistry17
 Optics, Photonics & Laser Physics16
 High-Energy / Nuclear / Particle Physics15
 Applied & Instrumentation Physics13
 Quantum & Atomic Physics10
Appendix
Table 2: Task distribution by scientific subfield: number of tasks per Biology, Physics, and Chemistry subdomain.
ModelExperimental SetupChemistryBiologyPhysicsAll Domains
Accuracy (%)Calibration (1-5)Accuracy (%)Calibration (1-5)Accuracy (%)Calibration (1-5)Accuracy (%)Calibration (1-5)
Conf.Diff.Feas.Conf.Diff.Feas.Conf.Diff.Feas.Conf.Diff.Feas.
Gemini 3-proNBK26.14±5.4026.14\pm 5.404.37±0.014.37\pm 0.013.56±0.033.56\pm 0.033.55±0.043.55\pm 0.0424.47±1.2424.47\pm 1.244.38±0.024.38\pm 0.023.44±0.013.44\pm 0.013.55±0.093.55\pm 0.0926.00±0.0026.00\pm 0.004.56±0.024.56\pm 0.023.39±0.013.39\pm 0.013.71±0.103.71\pm 0.1025.27±1.9225.27\pm 1.924.42±0.014.42\pm 0.013.46±0.013.46\pm 0.013.59±0.063.59\pm 0.06
BK28.43±0.9828.43\pm 0.984.39±0.034.39\pm 0.033.52±0.023.52\pm 0.023.50±0.073.50\pm 0.0727.26±0.7527.26\pm 0.754.39±0.024.39\pm 0.023.40±0.073.40\pm 0.073.59±0.033.59\pm 0.0326.33±2.0826.33\pm 2.084.56±0.014.56\pm 0.013.25±0.063.25\pm 0.063.89±0.063.89\pm 0.0627.33±0.7927.33\pm 0.794.43±0.014.43\pm 0.013.40±0.043.40\pm 0.043.64±0.023.64\pm 0.02
SBK28.43±0.0028.43\pm 0.004.43±0.004.43\pm 0.003.45±0.003.45\pm 0.003.71±0.003.71\pm 0.0026.11±0.0026.11\pm 0.004.45±0.004.45\pm 0.003.19±0.003.19\pm 0.003.89±0.003.89\pm 0.0021.00±0.0021.00\pm 0.004.63±0.004.63\pm 0.003.24±0.003.24\pm 0.004.00±0.004.00\pm 0.0025.43±0.0025.43\pm 0.004.49±0.004.49\pm 0.003.27±0.003.27\pm 0.003.87±0.003.87\pm 0.00
SABK30.39±0.0030.39\pm 0.004.48±0.004.48\pm 0.003.42±0.003.42\pm 0.003.79±0.003.79\pm 0.0025.12±0.0025.12\pm 0.004.44±0.004.44\pm 0.003.20±0.003.20\pm 0.003.78±0.003.78\pm 0.0027.00±0.0027.00\pm 0.004.68±0.004.68\pm 0.003.14±0.003.14\pm 0.004.01±0.004.01\pm 0.0026.91±0.0026.91\pm 0.004.51±0.004.51\pm 0.003.24±0.003.24\pm 0.003.84±0.003.84\pm 0.00
Appendix
Table 3: Domain-wise performance across LLM families. Different versions of Gemini, OpenAI, Claude, Llama, Qwen, and DeepSeek evaluated on Chemistry, Biology, Physics , and All Domains . Conf. := Confidence Score; Diff. := Difficulty Level; Feas. := Feasibility Score.
ModelExperimental SetupMCQNumericalFree form
Accuracy (%)Calibration (1-5)Accuracy (%)Calibration (1-5)Accuracy (%)Calibration (1-5)
Conf.Diff.Feas.Conf.Diff.Feas.PartialFullConf.Diff.Feas.
Gemini 3-proNBK36.21±1.7836.21\pm 1.784.42±0.034.42\pm 0.033.45±0.013.45\pm 0.013.89±0.063.89\pm 0.0612.80±2.5812.80\pm 2.584.19±0.054.19\pm 0.054.01±0.034.01\pm 0.032.46±0.112.46\pm 0.1137.70±1.6537.70\pm 1.6522.39±2.4522.39\pm 2.454.62±0.014.62\pm 0.013.01±0.033.01\pm 0.034.19±0.094.19\pm 0.09
BK42.39±1.5542.39\pm 1.554.46±0.014.46\pm 0.013.36±0.053.36\pm 0.053.96±0.023.96\pm 0.0212.80±1.3612.80\pm 1.364.16±0.034.16\pm 0.034.03±0.024.03\pm 0.022.47±0.062.47\pm 0.0636.04±0.7836.04\pm 0.7821.12±2.3321.12\pm 2.334.62±0.034.62\pm 0.032.90±0.052.90\pm 0.054.25±0.034.25\pm 0.03
SBK37.04±0.0037.04\pm 0.004.52±0.004.52\pm 0.003.17±0.003.17\pm 0.004.15±0.004.15\pm 0.0010.71±0.0010.71\pm 0.004.19±0.004.19\pm 0.003.98±0.003.98\pm 0.002.77±0.002.77\pm 0.0038.90±0.0038.90\pm 0.0023.66±0.0023.66\pm 0.004.71±0.004.71\pm 0.002.78±0.002.78\pm 0.004.47±0.004.47\pm 0.00
SABK41.36±0.0041.36\pm 0.004.55±0.004.55\pm 0.003.17±0.003.17\pm 0.004.15±0.004.15\pm 0.0012.50±0.0012.50\pm 0.004.24±0.004.24\pm 0.003.91±0.003.91\pm 0.002.75±0.002.75\pm 0.0035.90±0.0035.90\pm 0.0021.37±0.0021.37\pm 0.004.69±0.004.69\pm 0.002.76±0.002.76\pm 0.004.39±0.004.39\pm 0.00
Appendix
Table 4: Question-format performance across LLM families. Different versions of Gemini, OpenAI, Claude (Opus/Sonnet), Llama, Qwen, and DeepSeek evaluated across question formats. Conf. := Confidence Score; Diff. := Difficulty Level; Feas. := Feasibility Score.
ModelExp.ConfidenceDifficultyFeasibility
L1L2L3L4L5L1L2L3L4L5L1L2L3L4L5
Acc (%)# Que.Acc (%)# Que.Acc (%)# Que.Acc (%)# Que.Acc (%)# Que.Acc (%)# Que.Acc (%)# Que.Acc (%)# Que.Acc (%)# Que.Acc (%)# Que.Acc (%)# Que.Acc (%)# Que.Acc (%)# Que.Acc (%)# Que.  
Gemini 3-proNBK0.00±0.000.00\pm 0.000.33----24.70±1.6224.70\pm 1.62233.6726.11±4.0426.11\pm 4.04171.00--23.09±1.6723.09\pm 1.67107.0028.21±18.2928.21\pm 18.2912.3326.26±1.7726.26\pm 1.77278.0016.98±2.8716.98\pm 2.877.6710.22±12.0610.22\pm 12.0615.3320.87±3.4920.87\pm 3.49122.67--31.04±1.1231.04\pm 1.12142.0025.06±2.6925.06\pm 2.69125.00
BK--100.00±0.00100.00\pm 0.000.33--24.85±0.1724.85\pm 0.17229.3330.42±2.2630.42\pm 2.26175.330.00±0.000.00\pm 0.000.6728.26±2.4128.26\pm 2.41119.3318.15±7.0618.15\pm 7.0612.3327.71±0.4527.71\pm 0.45264.6717.04±5.1317.04\pm 5.138.0022.30±9.6622.30\pm 9.6611.0020.98±2.2920.98\pm 2.29124.00--30.71±0.9730.71\pm 0.97134.6730.33±1.9730.33\pm 1.97135.33
SBK--0.00±0.000.00\pm 0.001.00--23.65±0.0023.65\pm 0.00203.0027.36±0.0027.36\pm 0.00201.000.00±0.000.00\pm 0.002.0026.95±0.0026.95\pm 0.00141.0040.00±0.0040.00\pm 0.0015.0024.07±0.0024.07\pm 0.00241.0016.67±0.0016.67\pm 0.006.0012.50±0.0012.50\pm 0.008.0017.53±0.0017.53\pm 0.0097.00--30.37±0.0030.37\pm 0.00135.0026.67±0.0026.67\pm 0.00165.00
SABK------23.74±0.0023.74\pm 0.00198.0029.95±0.0029.95\pm 0.00207.000.00±0.000.00\pm 0.001.0029.73±0.0029.73\pm 0.00148.0033.33±0.0033.33\pm 0.0012.0024.58±0.0024.58\pm 0.00240.0050.00±0.0050.00\pm 0.004.0014.29±0.0014.29\pm 0.007.0022.86±0.0022.86\pm 0.00105.00--26.77±0.0026.77\pm 0.00127.0030.12±0.0030.12\pm 0.00166.00
Appendix
Table 5: Performance across LLM families by confidence, difficulty, and feasibility levels. Different versions of Gemini, OpenAI, Claude, Llama, Qwen, and DeepSeek evaluated across different levels of confidence, difficulty, and feasibility scores.
ModelExp.ConfidenceDifficultyFeasibility
L1L2L3L4L5L1L2L3L4L5L1L2L3L4L5
Acc (%)# Que.Acc (%)# Que.Acc (%)# Que.Acc (%)# Que.Acc (%)# Que.Acc (%)# Que.Acc (%)# Que.Acc (%)# Que.Acc (%)# Que.Acc (%)# Que.Acc (%)# Que.Acc (%)# Que.Acc (%)# Que.Acc (%)# Que.  
Gemini 3-proNBK17.33±1.1517.33\pm 1.155022.57±2.4122.57\pm 2.419626.45±0.6326.45\pm 0.639226.76±4.1626.76\pm 4.1614741.67±5.7741.67\pm 5.772036.67±5.7736.67\pm 5.771030.26±3.8730.26\pm 3.876529.84±4.2729.84\pm 4.2712421.51±0.4121.51\pm 0.4114117.95±2.3517.95\pm 2.356520.00±0.0020.00\pm 0.006520.74±1.9620.74\pm 1.9613520.41±5.4020.41\pm 5.404928.97±3.2428.97\pm 3.2410741.50±5.8941.50\pm 5.8949
BK18.00±2.0018.00\pm 2.005025.32±3.3525.32\pm 3.357926.52±3.4726.52\pm 3.478828.69±0.9728.69\pm 0.9715843.33±3.3343.33\pm 3.333041.67±3.6141.67\pm 3.611625.98±0.8525.98\pm 0.856833.89±1.7333.89\pm 1.7312023.56±1.5723.56\pm 1.5713321.08±1.7021.08\pm 1.706820.00±3.0820.00\pm 3.086521.48±0.7421.48\pm 0.7413525.17±5.1425.17\pm 5.144931.15±2.7031.15\pm 2.7010746.94±0.0046.94\pm 0.0049
Claude Opus 4.5NBK14.00±2.0014.00\pm 2.005022.22±3.9422.22\pm 3.949625.72±3.4925.72\pm 3.499222.68±1.7122.68\pm 1.7114740.00±5.0040.00\pm 5.002020.00±10.0020.00\pm 10.001027.18±2.3527.18\pm 2.356527.96±2.0327.96\pm 2.0312421.28±0.7121.28\pm 0.7114113.85±3.0813.85\pm 3.086512.31±1.5412.31\pm 1.546519.01±1.1319.01\pm 1.1313514.97±2.3614.97\pm 2.364927.10±1.6227.10\pm 1.6210747.62±1.1847.62\pm 1.1849
BK23.33±4.1623.33\pm 4.165022.78±3.3522.78\pm 3.357922.73±3.4122.73\pm 3.418829.32±1.4629.32\pm 1.4615848.89±1.9248.89\pm 1.923029.17±3.6129.17\pm 3.611639.22±0.8539.22\pm 0.856830.00±0.8330.00\pm 0.8312023.06±1.1523.06\pm 1.1513318.63±1.7018.63\pm 1.706815.38±0.0015.38\pm 0.006521.48±0.7421.48\pm 0.7413521.09±1.1821.09\pm 1.184932.40±2.1632.40\pm 2.1610754.42±4.7154.42\pm 4.7149
Appendix
Table 6: Performance across LLM families by human-rated confidence, difficulty, and feasibility levels. Different versions of Gemini, OpenAI, Claude, Llama, Qwen, and DeepSeek evaluated across different levels of human-rated confidence, difficulty, and feasibility scores.
Model# Corr.# Ques.Acc (%)
Gemini 3-pro1268135093.93
Claude Opus 4.51277134495.01
Claude Sonnet 4.51232131693.62
Claude Opus 4.11228132792.54
Gemini 3-Flash1279135094.74
OpenAI GPT-5.21276135094.52
Appendix
Table 7: Different versions of Gemini, OpenAI, Claude Sonnet, Llama, Qwen, and Deepseek evaluated on their ability to answer questions based on required background knowledge needed to answer questions.
Figure 13: Analysis of model errors for high feasible questions. We employ an LLM judge to systematically classify errors in model predictions according to a hierarchical taxonomy spanning five top-level (in black background) categories and 16 specific error types. The heatmap shows the percentage of incorrect responses containing each error type for each evaluated model. Error categories progress from surface-level issues (Comprehension & Scope) to deeper reasoning failures (Logical & Reasoning Flaws) to fundamental scientific deficiencies (Deficiencies in Scientific Rigor). Models can exhibit multiple error types simultaneously, so accumulative percentage scores within top-level categories may exceed 100%100\% . SciPredict tasks contribute to top-level category percentages if flagged with at least one underlying error type. Error analysis only considers the questions human experts marked as feasible to answer without running the practical experiment. Fig. 8 shows the same chart for all tasks. Tab. 8 provides comprehensive definitions for the error categories.
Main CategorySpecific ErrorDescription
Comprehension & ScopeTask MisinterpretationThe answer addresses a fundamentally different scientific question than the one that was asked.
Constraint ViolationThe answer ignores or violates a specific instruction or constraint mentioned in the question.
Insufficient SpecificityThe answer is overly generic, too high-level, or omits necessary details required to fully address the question.
Irrelevant InformationThe answer includes factually correct information that is not essential and does not help in answering the specific question.
Factual & ExtractionInformation OmissionThe answer fails to use a required piece of data that is explicitly present in the provided materials (e.g., experimental setup, measurements).
Factual ContradictionThe answer misrepresents or directly contradicts facts, values, or relationships that are clearly stated in the provided materials.
Appendix
Table 8: Definitions of prediction error categories and flags.
More in this topic

AI-Assisted Scientific Research

AI-assisted scientific research uses machine learning and AI agents to support discovery, analysis, experimentation, and scientific communication.

See all 253 papers

Explore similar work

CardsList
Oct 15, 2026

Can LLMs predict the outcomes of machine learning experiments?

Recent advances in AI make Recursive Self-Improvement (RSI) increasingly plausible. Large Language Models (LLMs) can already propose, select, and run Machine Learning (ML) experiments. However, the space of possible design choices in ML is vast. An LLM that proposes an experiment needs a reliable intuition of whether it is worth running given the timing constraints of long training runs. In this work, we measure such intuition in current frontier LLMs by evaluating how accurately they predict the impact of self-generated ML experiment ideas before these are put into training runs. To this end, we construct a dataset of 288 LLM-proposed ideas on eight MLE-bench tasks. We then prompt five frontier LLMs (GPT-6 Astra, Claude Opus 5.5, Claude Fable 5.1, Kimi K3, and GLM-5.3) to predict the impact of each idea and compare their predictions with the actual outcomes after training. We report four main findings. First, LLM predictions are substantially better than random chance but far from accurate: even the best model, Claude Opus 5.5, explains only half of the variance in outcomes (R^2 = 0.49). Second, we notice that all tested models systematically underestimate the magnitude of the changes. Third, prediction quality varies widely across tasks. Finally, the choice of the coding harness that actually implements the ideas has little effect on outcomes, and we find no evidence of self-preference. Higher reasoning settings and additional context lead to modest gains. Overall, our results show that current LLMs have useful but limited intuition about which ML ideas will work. We release our dataset to support further study.

Ayush Bansal, Aldo Pareja, Sean Klein +2

Autolab AI · Institute of Foundation Models · MIT

LLM EvaluationAutonomous ExperimentationAutomated Machine Learning
AskPDF
Oct 15, 2026

SciΠΠvot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents

Large language model (LLM) agents can keep scientific experiments running for months with little human supervision, but reliable autonomous research also requires scientific judgment. Evaluating scientific judgment is difficult because research decisions are open-ended and depend on project context. We construct SciΠΠvot, a 248-case benchmark of scientific judgment based on human corrections. The benchmark draws on records from six months of research in which senior researchers ran LLM agents on 26 projects across six scientific fields and repeatedly intervened to correct the agents' research judgments. In each case, an agent chooses between the original LLM plan and the human-corrected alternative, using only the project record available before the correction as context. We compare LLM and human performance and examine the effects of reasoning effort, access to the project record, and reconsideration. At medium reasoning effort with the project record, models systematically prefer the original LLM plan over the human-corrected alternative. Four of five frontier models perform below chance on the selected cases. The highest accuracy in the project condition across the tested models and reasoning effort settings is 63.5%, compared with 73.4% for fresh researchers. At medium effort, every model achieves lower accuracy with the project record than without it, and reconsideration does not consistently improve overall accuracy. Even with a human-corrected alternative available, frontier models struggle with the selected research decisions, while SciΠΠvot supports systematic evaluation of scientific judgment for LLM agents.

Youran Sun, Xingyu Ren, Chugang Yi +15

University of Maryland, College Park · The Chinese University of Hong Kong · Gradient Network +3

LLM Agent EvaluationAI Agent BenchmarksScientific Reasoning in Language ModelsAutonomous Research Agents
AskPDF
Apr 20, 2026

Experiments or Outcomes? Probing Scientific Feasibility in Large Language Models

Scientific feasibility assessment asks whether a claim is consistent with established knowledge and whether experimental evidence could support or refute it. We frame feasibility assessment as a diagnostic reasoning task in which, given a hypothesis, a model predicts feasible or infeasible and justifies its decision. We evaluate large language models (LLMs) under controlled knowledge conditions (hypothesis-only, with experiments, with outcomes, or both) and probe robustness by progressively removing portions of the experimental and/or outcome context. Across multiple LLMs and two datasets, providing outcome evidence is generally more reliable than providing experiment descriptions. Outcomes tend to improve accuracy beyond what internal knowledge alone provides, whereas experimental text can be brittle and may degrade performance when the context is incomplete. These findings clarify when experimental evidence benefits LLM-based feasibility assessment and when it introduces fragility.

Seyedali Mohammadi, Manas Gaur, Francis Ferraro

University of Maryland, Baltimore County, MD, USA

LLM EvaluationScientific Reasoning in Language ModelsLanguage Model Robustness
AskPDF