Do Language Models Know What Compelling Scientific Evidence Looks Like?

Conor Igoe; Gabriel Grand; Zane Koch; Albert Bou; Eli Moss; Michaela M. Hinks; Samuel G. Rodriques; Andrew White

cs.LGOct 29, 2026

Do Language Models Know What Compelling Scientific Evidence Looks Like?

Authors: Conor Igoe, Gabriel Grand, Zane Koch, Albert Bou, Eli Moss, Michaela M. Hinks, Samuel G. Rodriques, Andrew White,

Organizations: Edison Scientific Inc., San Francisco, CA, USA · MIT CSAIL, Cambridge, MA, USA

Abstract

Expert scientists excel at weighing empirical findings from multiple sources that vary in both statistical reliability and epistemic relevance. To what extent are language models, which are increasingly driving autonomous discovery systems, capable of such evidence deliberation? We introduce EDB-Bio, a scientific reasoning benchmark curated from real-world data in three biological domains: drug-perturbation transcriptomics, CRISPR screens, and human genome-wide association studies (GWAS). Each of EDB-Bio's 96k episodes presents a sequence of experimental results from published studies and asks the model to predict the probability that a target hypothesis is true. Out-of-the-box, open-weight and frontier models alike demonstrate systematic underconfidence and attain Brier scores comparable to a trivial base-rate forecaster. Post-training against strictly proper scoring rules significantly improves both calibration and discrimination. This calibration pays off downstream: in a sequential experiment selection task, our Ibis0 model is 2.5 times more efficient than the best frontier model. Throughout, quantifying evidence weights as Bayes factors reveals how models update their beliefs over time, whether they agree with expert biologists on relevance, and where they exhibit inconsistencies and biases. Our work extends calibration training and uncertainty analysis to a realistic scientific forecasting setting with applications to human cancer biology. We release all code, data, and trained models.

Figures & tables

Figure 1: Evidence deliberation task. EDB-Bio asks models to reason about a scientific hypothesis $h$ given a sequence of empirical evidence. Each evidence stub is derived from published data in one of three biological domains ( Table 1 ). In the example here, each experiment $x$ is an RNA-Seq study from Tahoe-100M [ 10 ] in which a drug perturbs a gene in a particular cell line. The result $y$ reports a DESeq2 assay readout, including an effect (log 2 fold-change) and its significance (adjusted $p$ -value). Left: In each episode, the hypothesis $h$ and binary support label $\theta$ are defined by a held-out target stub , which is not shown to the model. Right: Given $h$ , the model sees a sequence of supporting stubs from studies on related drugs, cell lines, and genes and must verbalize its belief $p(\theta)$ after each. By comparing sequential belief states, we can measure which stubs models find compelling based on the strength of their updates.

Table 1: (A) Comparison of the three EDB-Bio domains: each instantiates the same evidence-deliberation task with a distinct assay, hypothesis, support criterion, and evidence modality. (B) Five overarching design principles, which are applied to each domain.

Figure 2: Do language models produce calibrated & accurate beliefs when presented with evidence? Zero-shot LLMs (Qwen3, GPT-5.6-sol, and Opus 5) underperform a constant

p=0.5

forecast on Brier score (dotted line, left ). This is primarily due to poor calibration: Models that sit above the diagonal ( middle ) are systematically underconfident , despite demonstrating above-chance discrimination ( right ). Explicitly instructing models to use a prior of 0.5, matching the base rate, helps somewhat (dashed bars), but post-training on proper-score objectives (Ibis0 variants) improves both calibration and discrimination substantially.

Figure 3: How do language models verbalize beliefs? Histograms of every stated

p(\theta)

produced as evidence accrues on the sensitivity set (

N=13{,}056

per model). Out of the box, Qwen3 (thinking) and the frontier models are underconfident; post-training recenters the mean near the

0.5

support rate. Models that verbalize a probability in text produce fewer than

100

distinct values, whereas the scalar head of Ibis0 (classifier) produces a smooth distribution with

18

–

50\times

finer resolution.

Figure 4: Do models weigh each piece of evidence at the right magnitude? Left: Odds ratio of support

\psi

against weight of evidence

w

; calibrated updates lie on the diagonal (

\beta=1

). Middle: the slope

\beta

of each curve;

\beta<1

indicates overreaction, meaning updates are larger than justified. Post-training brings Ibis0 (non-thinking) closest to calibrated. Right: Evidence weights

w

on Tahoe, normalized by episode mean and grouped by biological relevance. A model that follows the expert hierarchy should show monotonic reduction from left to right. Instead, all models under-weight evidence from non-matching drugs that share a mechanism of action with the target hypothesis [2], and all but Ibis0 over-weight genome-wide aggregate summaries [4], which are only indirectly-related to the target.

Figure 5: Experiment selection under the EIG objective. Left: Belief in the held-out conclusion,

p(\theta=\theta^{\ast}\mid I_{k})

, versus experiments performed, pooled over domains (

\pm

s.e.m.). EIG-simulate, which uses Ibis0, converges more quickly than both Opus 5 (under naive prompting) and a random policy. Right: Information gained from the first experiment,

\mathbb{H}[p(\theta)]-\mathbb{H}[p(\theta\mid I_{1})]

(bits,

95\%

CI), pooled over domains. EIG-simulate and EIG-exemplar extract

{\sim}2.5\times

the information of the best prompted model.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

	Tahoe	DepMap	GWAS
P1 Targets	Mitochondrial, ribosomal, pseudogene and housekeeping-lncRNA genes excluded, as are near-zero base-mean measurements	Pan-essential and non-expressed genes excluded; targets drawn from the $2{,}480$ selective -dependency genes (essential in $2$ – $50\%$ of lines)	MHC/HLA variants excluded (long-range linkage disequilibrium makes single-variant labels unreliable), as are heavily replicated “textbook” pairs
P2 Diversity	$49$ cell lines, $179$ drugs across mechanisms, $13{,}083$ genes; balanced within every cell line	$30$ lineages, thousands of genes, at most $16$ episodes per gene	Thousands of genes and traits across EFO parent categories, with per-gene and per-trait caps; one episode per unique (SNP, trait) pair
P3 Hierarchy	Within-assay: same drug, other line $\succ$ same mechanism $\succ$ other drug, same line $\succ$ aggregate summary $\succ$ unrelated control	Cross-modal: same gene, same lineage $\succ$ same gene, other lineage $\succ$ co-essential partner $\succ$ drug targeting $G$ $\succ$ molecular state of $G$ in $C$ $\succ$ unrelated control	Locus and trait proximity: other SNP in gene, same trait $\succ$ same gene, related trait $\succ$ target SNP, related trait $\succ$ target SNP, unrelated trait $\succ$ pathway neighbor $\succ$ unrelated control
P4 Sufficiency	Significance-rich ( $\geq 4$ significant stubs), -moderate ( $1$ – $3$ ), or -poor (none)	Rich, moderate or poor in dependency-supporting evidence: dependent related lines, over-expression, amplification, damaging mutation, drug sensitivity	Rich, moderate or poor by capping significant related stubs. Non-significant related evidence is scarce in a catalog of significant hits, so poor pools fill largely with controls
P5 Leakage	No pool stub shares the target’s exact (drug, cell line) pair	Same-gene stubs come only from other lines; the molecular state of $G$ in $C$ is a different assay from the held-out fitness readout, so it is evidence rather than leakage	The exact (SNP, trait) pair is excluded. A different SNP in high linkage disequilibrium carries nearly the same signal; filtering it needs an LD reference panel and is deferred

Appendix

Table 2: How each EDB-Bio design principle is instantiated in the three domains. Full criteria and audits are in §§ A.3 to A.5 .

	Primary	Sensitivity
Episodes per domain	$2{,}000$	$256$
Episodes, all three domains	$6{,}000$	$768$
Context lengths per episode	$1$ , from $k\in\{2,\dots,16\}$	$17$ , all of $k\in\{0,\dots,16\}$
Rows per domain	$2{,}000$	$4{,}352$
Rows, all three domains	$6{,}000$	$13{,}056$
Models evaluated	$10$	$7$

Appendix

Table 3: Primary and sensitivity test-set configurations. Both are published in the testsets/ folder of EdisonScientific/EDB-Bio as six configs (two sets

\times

three domains).

Domain	Measurement key	Measurement space	Exposed as stubs	Coverage	Test overlap
Tahoe	(drug, cell line, gene)	$114{,}750{,}993$	$450{,}307$	$0.39\%$	$11$ ( $0.6\%$ )
DepMap	(gene, cell line)	$2{,}411{,}366$	$365{,}141$	$15.1\%$	$202$ ( $10.1\%$ )
GWAS	(SNP, trait)	$198{,}001^{*}$	$183{,}166$	$92.5\%$	$1{,}065$ ( $53.2\%$ )

Appendix

Table 4: Overlap between test targets and training evidence. Measurement space is the number of possible target keys; exposed as stubs is the number of distinct measurements that appear as an evidence stub in the first

16

positions of any training or validation pool; coverage is their ratio; test overlap is the number of the

2{,}000

test episodes per domain whose held-out measurement is among the exposed stubs. ∗ GWAS has no product space, so the entry is the number of distinct (SNP, trait) associations that appear anywhere in the dataset, and coverage is relative to that set.

Figure 6: Metrics with and without the overlapping test episodes. Solid bars use the full

2{,}000

-episode primary test set in each domain; hatched bars use the filtered dataset, which removes the episodes whose held-out measurement appears in a training or validation pool ( Table 4 ). Rows are metrics and columns are domains, with the equal-weight macro-average on the right. Labels above a bar pair give the filtered value minus the full value where the difference reaches

0.005

Run	Training domains	Selection metric	Best value	Step
Ibis0 (classifier)	Tahoe, DepMap, GWAS	in-domain BCE $\downarrow$	$0.4785$	$2400$
Ibis0 (non-thinking)	Tahoe, DepMap, GWAS	eval reward $\uparrow$	$0.8304$	$4200$
Ibis0 (thinking)	Tahoe, DepMap, GWAS	eval reward $\uparrow$	$0.8168$	$2400$
Ibis0 (classifier), Tahoe held out	DepMap, GWAS	in-domain BCE $\downarrow$	$0.4390$	$1700$
Ibis0 (classifier), DepMap held out	Tahoe, GWAS	in-domain BCE $\downarrow$	$0.4794$	$1300$
Ibis0 (classifier), GWAS held out	Tahoe, DepMap	in-domain BCE $\downarrow$	$0.5096$	$1800$

Appendix

Table 5: Training runs and selected checkpoints. All runs start from Qwen3-30B-A3B . The Ibis0 runs use the same optimizer settings and differ in objective and training mixture; the outcome simulator used in § 6 is trained separately ( App. B ). In-domain BCE is averaged over the trained domains only, so it is not comparable across rows with different mixtures.

Model	$n$	Brier Score ( $\downarrow$ )	sMECE ( $\downarrow$ )	AUROC ( $\uparrow$ )	Reliability ( $\downarrow$ )	Resolution ( $\uparrow$ )
Qwen3 (non-thinking)	6,000	0.3336 [0.3237, 0.3436]	0.3101 [0.2990, 0.3229]	0.6608 [0.6471, 0.6752]	0.1096 [0.1024, 0.1189]	0.0250 [0.0226, 0.0295]
Tahoe	2,000	0.3634 [0.3459, 0.3827]	0.3367 [0.3201, 0.3589]	0.6039 [0.5805, 0.6279]	0.1277 [0.1140, 0.1452]	0.0147 [0.0111, 0.0204]
DepMap	2,000	0.2860 [0.2715, 0.2995]	0.2773 [0.2575, 0.2978]	0.7454 [0.7239, 0.7671]	0.0854 [0.0754, 0.0977]	0.0485 [0.0414, 0.0577]
GWAS	2,000	0.3513 [0.3333, 0.3694]	0.3162 [0.2966, 0.3391]	0.6333 [0.6080, 0.6571]	0.1156 [0.1022, 0.1314]	0.0120 [0.0089, 0.0176]
Qwen3 (thinking)	6,000	0.2973 [0.2896, 0.3047]	0.2696 [0.2607, 0.2818]	0.7040 [0.6914, 0.7170]	0.0829 [0.0772, 0.0907]	0.0339 [0.0307, 0.0387]
Tahoe	2,000	0.3235 [0.3103, 0.3373]	0.3014 [0.2770, 0.3244]	0.6822 [0.6586, 0.7049]	0.0988 [0.0877, 0.1131]	0.0224 [0.0172, 0.0288]

Appendix

Table 6: Calibration and discrimination on the held-out multi-domain test set, in full.

Figure 7: Confidence in the true label increases as evidence accumulates. Mean probability each model assigns to the true label vs. the number of evidence stubs shown (

256

episodes per domain). The slope reflects integration of new evidence (up to

k=16

stubs; shading marks

k<2

, below the range seen in training). The intercept reflects prior knowledge:

p>0.5

k=0

means that the model predicts the target hypothesis with no evidence . Above-chance a priori predictions on GWAS, most notably by Ibis0 (classifier), reflect non-random association of alleles in high linkage disequilibrium in EDB-Bio (see § A.5 ).

Figure 8: Cumulative evidence-induced log-odds change by domain. Cumulative log Bayes factor toward the observed outcome,

s\,[\operatorname{logit}(p_{k})-\operatorname{logit}(p_{0})]

in decibans, against the number of stubs shown. Each curve is relative to that model’s own

k=0

forecast. Panels show the three domains and their macro-average on a shared

y

axis; bands are

95\%

intervals from within-domain episode resampling.

Figure 9: Do models distinguish supporting vs. refuting evidence? Mean signed

w_{j}

in decibans; positive raises belief in the target hypothesis and negative lowers it. Every model’s mean update has the expected sign for supporting and refuting evidence, and most models place little weight on inconclusive evidence. (GWAS admits no refuting evidence ( § A.2 ), so its panel contrasts genome-wide-significant with sub-threshold associations.)

Figure 10: Do models assign higher weight to more relevant evidence in an expert biological knowledge hierarchy? Each stem shows how strongly a tier of evidence moves a model, relative to that model’s average stub in the same episode (

1\times

is average). Tiers run left to right in the expert-ranked order, so a model that follows the hierarchy shows heights falling from left to right. On DepMap, the models largely follow the expert ordering. On GWAS, models assign more weight to indirect evidence (same SNP) than to more direct evidence (same gene, same trait). On Tahoe, Claude gives disproportionately high weight to aggregate summary evidence over more-specific assay readouts.

Figure 11: Transfer to domains excluded from training. A: We train Ibis0 (classifier) on two domains (rows) and evaluate on a third held-out domain (columns). Outlined diagonal cells are o.o.d. evaluations on the excluded domain and report the raw Brier score and the retention percentage ( Eq. 4 ). B: Retention on the excluded domains, decomposed into Brier score and its Murphy reliability and resolution terms. Boxes show paired-bootstrap $95\%$ intervals and internal lines show point estimates; vertical lines mark Qwen3 ( $0\%$ ) and the all-domain classifier ( $100\%$ ). Reliability retains $58$ – $86\%$ of the full-model improvement, whereas effectively no resolution improvement is retained.

A. Absolute Brier performance
Training mix	Tahoe	DepMap	GWAS	Macro
DepMap + GWAS (Tahoe held out)	0.2535 [0.2426, 0.2641]	0.1641 [0.1539, 0.1739]	0.1483 [0.1367, 0.1599]	0.1886 [0.1825, 0.1945]
Tahoe + GWAS (DepMap held out)	0.2072 [0.1994, 0.2151]	0.2337 [0.2253, 0.2417]	0.1509 [0.1400, 0.1613]	0.1972 [0.1919, 0.2025]
Tahoe + DepMap (GWAS held out)	0.2047 [0.1955, 0.2139]	0.1639 [0.1534, 0.1743]	0.2993 [0.2870, 0.3114]	0.2226 [0.2165, 0.2287]
All domains	0.2063 [0.1971, 0.2157]	0.1713 [0.1602, 0.1821]	0.1537 [0.1416, 0.1654]	0.1771 [0.1707, 0.1831]

Appendix

Table 7: Full cross-domain transfer results. A: Brier score for every classifier training mix and test domain, with

95\%

bootstrap intervals. Bold cells evaluate a domain excluded from training; other per-domain cells are descriptive in-domain evaluations. Macro is the equal-domain average and is bootstrapped directly as a stratified macro statistic. B: Exact results on the excluded domain for Brier score and its Murphy reliability and resolution terms. The Qwen3 and full-model references denote the released checkpoint and the all-domain classifier, respectively. Loss vs. full is signed so that positive values indicate worse performance: leave-one-out minus full for Brier and reliability, and full minus leave-one-out for resolution. Gain retained places the leave-one-out classifier between Qwen3 (

0\%

) and the full model (

100\%

). Loss and retention intervals use paired bootstraps over identical episodes.

As reported
Model	$n$	Brier Score ( $\downarrow$ )	sMECE ( $\downarrow$ )	AUROC ( $\uparrow$ )	Reliability ( $\downarrow$ )	Resolution ( $\uparrow$ )
Qwen3 (non-thinking)	6,000	0.3336 [0.3237, 0.3436]	0.3101 [0.2990, 0.3229]	0.6608 [0.6471, 0.6752]	0.1096 [0.1024, 0.1189]	0.0250 [0.0226, 0.0295]
Qwen3 (thinking)	6,000	0.2973 [0.2896, 0.3047]	0.2696 [0.2607, 0.2818]	0.7040 [0.6914, 0.7170]	0.0829 [0.0772, 0.0907]	0.0339 [0.0307, 0.0387]
Ibis0 (non-thinking)	6,000	0.1933 [0.1888, 0.1977]	0.0316 [0.0258, 0.0450]	0.7729 [0.7616, 0.7846]	0.0020 [0.0017, 0.0038]	0.0579 [0.0540, 0.0631]
Ibis0 (thinking)	6,000	0.2069 [0.2025, 0.2111]	0.0327 [0.0296, 0.0431]	0.7329 [0.7209, 0.7450]	0.0023 [0.0021, 0.0044]	0.0448 [0.0413, 0.0500]
Ibis0 (classifier)	6,000	0.1771 [0.1707, 0.1831]	0.0658 [0.0588, 0.0762]	0.8195 [0.8099, 0.8295]	0.0075 [0.0065, 0.0104]	0.0790 [0.0753, 0.0849]

Appendix

Table 8: Cross-fit Platt recalibration of every evaluated system: a two-parameter affine map on the logit, fitted out-of-fold within each domain. Because it requires labelled target-domain data, it bounds how much of each gap a post-hoc fix can close and is not a deployable method. The trained systems are also recalibrated so that the two blocks are comparable.

Model	$n$	Brier Score ( $\downarrow$ )	sMECE ( $\downarrow$ )	AUROC ( $\uparrow$ )	Reliability ( $\downarrow$ )	Resolution ( $\uparrow$ )
Qwen3 (non-thinking)	6,000	0.2315 [0.2274, 0.2353]	0.0603 [0.0417, 0.0657]	0.6580 [0.6363, 0.6698]	0.0043 [0.0027, 0.0075]	0.0214 [0.0180, 0.0266]
Tahoe	2,000	0.2451 [0.2419, 0.2487]	0.0656 [0.0383, 0.0743]	0.6034 [0.5508, 0.6223]	0.0029 [0.0006, 0.0078]	0.0048 [0.0007, 0.0138]
DepMap	2,000	0.2092 [0.1987, 0.2186]	0.0703 [0.0469, 0.0968]	0.7431 [0.7218, 0.7654]	0.0082 [0.0041, 0.0154]	0.0486 [0.0410, 0.0588]
GWAS	2,000	0.2401 [0.2348, 0.2443]	0.0450 [0.0165, 0.0634]	0.6274 [0.5936, 0.6502]	0.0018 [0.0003, 0.0051]	0.0109 [0.0051, 0.0177]
Qwen3 (thinking)	6,000	0.2148 [0.2109, 0.2192]	0.0281 [0.0239, 0.0380]	0.7033 [0.6880, 0.7152]	0.0018 [0.0013, 0.0034]	0.0363 [0.0324, 0.0407]
Tahoe	2,000	0.2245 [0.2182, 0.2312]	0.0302 [0.0204, 0.0455]	0.6814 [0.6540, 0.7027]	0.0023 [0.0010, 0.0050]	0.0278 [0.0204, 0.0338]

Appendix

Table 9: Cross-fit Platt recalibration by domain, with the map fitted separately within each of Tahoe, DepMap and GWAS. The as-reported breakdown is in the complete master table.

Figure 12: Ordering effects: Given the same evidence, are models consistent in belief revision? Each point is one model; the schematic at right shows the two manipulations. Position-1 bias (vertical): a single “compelling” evidence stub (supporting and highly-relevant) is placed at varying positions within an otherwise identical context. We report the stated belief when the stub appears first minus the mean stated belief when it appears in the later three positions, in decibans. Positive values indicate that the same stub moves the belief further when it appears first, a form of primacy bias. Belief-consistency violation (horizontal): a matched pair of supporting/refuting stubs is constructed by inverting the direction of effect while keeping all other details the same, and we measure how far the joint update departs from the product of the two single-stub updates. Zero on either axis is the unbiased value, so the lower left is best. Arrows join the two matched pairs that differ only in whether a reasoning trace is verbalized. Across both measures, thinking reduces bias. Ibis0 (classifier) is well calibrated yet shows pronounced bias on both axes, whereas RL post-training improves calibration and reduces the observed biases.

Figure 13: Composition of conflicting evidence. Absolute departure from additive weight-of-evidence updating for a supporting stub and its constructed equal-and-opposite counterpart, macro-averaged over Tahoe and DepMap. Panels group models by training regime and bars are ordered within each panel; the vertical scale is shared. Intervals are

95\%

episode bootstrap intervals with episodes resampled within domain,

256

episodes per domain. Lower values indicate closer agreement with multiplicative combination of Bayes factors.

Model	Position-1 bias (dB)	95% CI
Ibis0 (classifier)	+1.01	$[+0.82,+1.22]$
Qwen3 (non-thinking)	+0.69	$[+0.21,+1.18]$
Qwen3 (thinking)	+0.20	$[+0.11,+0.30]$
Ibis0 (non-thinking)	+0.18	$[+0.10,+0.26]$
GPT-5.6-sol	-0.03	$[-0.29,+0.23]$
Claude Opus 5	-0.27	$[-0.37,-0.18]$

Appendix

Table 10: Dependence of the stated belief on the first evidence position. Stated belief at position

1

minus the mean over positions

4

7

and

10

, in decibans, macro-averaged over the three domains with

256

episodes per domain. Positive values indicate that the same stub moves the belief further when it appears first. Intervals are

95\%

episode bootstrap intervals resampled within domain; bold marks an interval excluding zero.

Figure 14: Dependence of the stated belief on evidence position. The same evidence stub placed at four positions of a ten-stub context, with the remaining slots held to identical neutral padding in identical order. Panels group models that differ in whether a reasoning trace is verbalized. Values are decibans, centred within episode, macro-averaged over the three domains with episodes resampled within domain; bands are

95\%

intervals,

256

episodes per domain. Legend entries give the position-1 bias of Table 10 . A flat line at zero indicates a belief that does not depend on the order of the evidence.

Baseline	Fit data	Coverage	Brier ( $\downarrow$ )	Hard Brier ( $\downarrow$ )
Rhaister prompt-local	Prompt evidence only	85.1%	0.2310	0.4130
Rhaister all-context	Train targets + context	88.5%	0.2295	0.3968
Numeric-only GBM	DESeq2 numeric features	100.0%	0.2206	n/a
All-domain classifier	Three-domain training	100.0%	0.2063	n/a

Appendix

Table 11: Tahoe-only structured baselines on the

2{,}000

-episode test set. Coverage is the fraction of episodes for which a baseline produces a forecast.

Scientific Reasoning in Language Models

Scientific reasoning in language models studies how AI systems use evidence, domain knowledge, and tools to solve problems and support discovery.

See all 154 papers

InfiniteScienceGym: An Unbounded, Procedurally-Generated Benchmark for Scientific Analysis
Apr 14, 2026Oliver Bentham, Vivek SrikumarLLM Evaluation Scientific Agent Benchmarks
Kahlert School of Computing University of Utah
TRACES: A Benchmark for Epistemic Reliability in Scientific Reasoning by LLMs
Aug 11, 2026Valentin Rodionov, Shamil AssylbekovLLM Evaluation LLM Reliability
Intellicat Cleveland, OH 44106
Science Edge Evaluation: SEE the Missing Step Toward Real Scientific Discovery
Aug 7, 2026Taolin Han, Yuchen Zhang, Jinghang Wang +22Scientific Reasoning Scientific Discovery
1Alibaba Group · University of Chinese Academy of Sciences · Taolin Han1,3* +28

Apr 14, 2026J/K move · Enter open · S save

InfiniteScienceGym: An Unbounded, Procedurally-Generated Benchmark for Scientific Analysis

Oliver Bentham, Vivek Srikumar

Kahlert School of Computing University of Utah

LLM Evaluation Scientific Agent Benchmarks Scientific QA Synthetic Benchmark Generation Evidence-Grounded Generation

Open paper AskPDF arXiv

Large language models are emerging as scientific assistants, but evaluating their ability to reason from empirical data remains challenging. Benchmarks derived from published studies and human annotations inherit publication bias, known-knowledge bias, label noise, and substantial storage requirements. We present InfiniteScienceGym, a procedurally generated benchmark of scientific repositories paired with a verifiable question-answering task. From a seed, the simulator deterministically generates a self-contained repository with realistic directory structure, files, and tabular data, and a privileged QA generator produces both answerable and unanswerable questions with exact ground truth. This makes it possible to evaluate evidence-grounded reasoning, abstention, and tool-mediated analysis in a controlled setting without distributing a large static corpus. InfiniteScienceGym complements real scientific benchmarks by targeting blind spots and failure modes that are hard to evaluate using published datasets alone. Evaluating both proprietary and open-weight models, we find that none achieve more than 50% accuracy overall, that recognizing unanswerable questions remains a major weakness, and that stronger models tend to use tools more effectively rather than simply consuming more tokens.

Abstract

Scientific Reasoning in Language Models

Explore similar work

InfiniteScienceGym: An Unbounded, Procedurally-Generated Benchmark for Scientific Analysis

TRACES: A Benchmark for Epistemic Reliability in Scientific Reasoning by LLMs

Science Edge Evaluation: SEE the Missing Step Toward Real Scientific Discovery