Are We Measuring Scientific Intelligence? Rethinking the Evaluation of AI Scientists

Kate Zhang; Yuante Li

cs.AIOct 4, 2026

Are We Measuring Scientific Intelligence? Rethinking the Evaluation of AI Scientists

Authors: Kate Zhang, Yuante Li

Organizations: School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213

Abstract

AI agents can now carry out data-driven scientific analyses end to end, and benchmarks assess them by giving an agent a question and a dataset and scoring its final answer against a fixed key. These benchmarks assume that a correct answer was derived from the supplied data, a property we call evidence grounding. However, an agent can also reach the key from prior knowledge or by ruling out the other options, and a score based on a single run cannot tell these cases apart. We show how to test this assumption and find that it often fails. For each question, we build versions of its data files in which the evidence for the answer is left intact, withdrawn or reversed, check each edit with a pre-registered reference statistic, and run the same agent on every version. We then measure evidence-grounded accuracy, which credits a correct answer only if the agent also responds when the evidence is withdrawn and follows it when it is reversed. We evaluate three agent scaffolds and five models on 18 single-cell questions from BAISBench and four synthetic problems from GeneBench-Pro. On the single-cell questions, the Claude agents are 95% accurate and answer 83% correctly without any data, but their evidence-grounded accuracy is only 41%. Hiding gene names raises the share of runs that follow reversed evidence from 58% to 93% on ten gene tasks, suggesting that prior knowledge competes with the supplied data. The benchmark score and LLM judges can also reward answers that ignore the changed evidence. Measuring scientific intelligence rather than recall therefore requires checking whether answers follow the evidence and whether scores reward them for it.

Figures & tables

Figure 1: Does the answer follow the evidence, and does the score reward it? Left: a conceptual benchmark item under its original, withdrawn and flipped data. The benchmark keeps its original key (A), so an agent that retains A keeps full credit, whereas an agent that follows the flipped evidence and answers B receives zero. Right: mean benchmark score of the Claude agents on our 18 BAIS-SD tasks, graded against the original key. App. B follows a real item through the same data versions.

family	compares	level	correct response	evaluator
			to the change known	tested
outcome scoring	answer vs. key	run	–	no
process scoring	trajectory vs. key	run	–	no
input ablation	with vs. without data	benchmark	–	no
ad hoc perturbation	before vs. after an edit	item	assumed	no
verified counterfactual (ours)	paired data versions	run	verified	yes

Table 1: Families of evaluation for data-driven AI scientists. Only a verified counterfactual tests single answers against a known correct response and also tests the evaluator.

Figure 2: Protocol. (1) Each item is paired with null, withdrawal and flip versions of its data, each verified with a pre-registered reference statistic. (2) The same agent runs on every version in a fresh sandbox. (3) The paired outputs audit both the agent, which should track the evidence, and the evaluator, which should score an evidence-following answer above one that keeps the old key.

		Benchmark accuracy		Invariance		Responsiveness		Blind spot $B$
Bench.	Model	$E$	no data	$I_{\text{rep}}$	$I_{\text{null}}$	$R_{\text{rm}}^{\text{flag}}$	$R_{\text{flip}}$	rm	flip	EGA
BAIS	Opus 5.5	98	85	96	98	41	59	85	33	35
	Sonnet 5	100	89	100	100	31	52	89	41	31
	Haiku 4.5	87	74	96	94	31	72	56	15	56
	GPT-5.6-Sol †	100	81	100	98	31	65	81	28	33
	GPT-5.6-Terra †	94	80	93	94	37	63	78	31	35

Table 2: Accuracy, invariance and responsiveness (%). Claude rows: 18 (BAIS) or 4 (GB-Pro) tasks

\times

3 replicates. Accuracy: official score on

E

and with no data.

I_{\text{rep}}

: agreement of two runs on

E

I_{\text{null}}

: same answer after a null edit.

R^{\text{flag}}_{\text{rm}}

: the agent reports that the withdrawn data do not support an answer.

R_{\text{flip}}

: the answer follows the flip (GB-Pro: passes against the transformed ground truth).

B

: credit against the original key after withdrawal or flip. EGA: Eq. ( 3 ), not defined for GB-Pro. † Codex CLI, 3 replicates each. ‡ Biomni on Sonnet 5, 1 replicate, no separate no-data run.

Figure 3: Agents notice but do not act. What each agent does when the key’s evidence is (a) withdrawn or (b) flipped (% of paired runs, complete replicates only). “Noticed”: the agent reports that the data do not support an answer, yet gives the key. (c) Accuracy on the original data against EGA. The two orderings differ markedly. Colour: scaffold. Marker: model (Biomni runs Sonnet 5).

Figure 4: Where the accuracy comes from. Each correct BAIS-SD run is classified by its paired withdrawal and flip runs (Eq. 4 ). Only the dark-blue segment, runs that react to withdrawal and follow the flip, is evidence-grounded accuracy. Bars sum to accuracy. Complete replicates only.

Figure 5: Gene-name anonymisation. The same 10 gene tasks with every gene symbol replaced by an opaque identifier in data and text (grey: names, blue: anonymised). The no-data prior falls towards chance, accuracy with data barely changes, and the same agents now follow the flipped evidence.

Figure 6: What the score hides, and what repairs it. (a) GeneBench-Pro (Claude models). The grader passes almost no run on either version, yet transform-consistent answers are as frequent as run-to-run agreement. (b) Offering “the provided data are insufficient” cuts key credit on withdrawn evidence for Claude Code agents and leaves intact-data accuracy unchanged. (c) Flip AUC, the probability that a run following the flip outscores one that does not. Under the standard prompt both judges reward ignoring the flip, like the benchmark score. The grounding prompt reverses this.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

agent	runs	$E$	$T_{0}E$	$T_{B}E$	$T_{C}E$	route
Haiku 4.5	3	C	C	C, flagged	D	derived
GPT-5.6-Terra	2	C	C	C, partial support	D	eliminated
GPT-5.6-Terra	1	C	C	C, partial support	C	recalled
Opus 5.5	3	C	C	C, partial support	C	recalled
Sonnet 5	3	C	C	C, partial support	C	recalled
GPT-5.6-Sol	3	C	C	C, partial support	C	recalled

Appendix

Table 3: All runs on the worked example (S23_Q3). Answers on the original data

E

, the null version, the withdrawal (with the agent’s data-support label) and the flip, and the resulting route. The key is C on

E

, and the flip moves the data-supported answer to D. Rows group runs with identical answers. The three replicates of GPT-5.6-Terra split into two rows, and Biomni was run once.

qid	family	key	withdrawal ( $T_{B}$ )	flip ( $T_{C}$ )	new
S09_Q1	abundance	B	equalise option abundances	down-sample Enterocyte	A
S07_Q1	abundance	C	equalise option abundances	down-sample mesothelial cell	D
S35_Q1	abundance	C	equalise option abundances	down-sample Pericyte, Myofibroblast, Matrix_Fibroblast, Smooth_Muscle_Cell, Stromal_Cell	A
S06_Q2	abundance	C	remove cells with cellxgene_cell_type = T cell, natural killer cell, B cell, innate lymphoid cell	remove cells with cellxgene_cell_type = natural killer cell, B cell, innate lymphoid cell	A
S27_Q1	composition change	C	remove cells with Disease group = mild COVID-19, mild COVID-19 (asymptomatic)	down-sample classical monocyte and down-sample natural killer cell	B
S19_Q2	composition change	A	remove cells with disease = normal	down-sample AT1 and down-sample Endothelial cells	B

Appendix

Table 4: BAIS-SD tasks. “new” is the option supported by the flipped data.

problem	withdrawal ( $T_{B}$ )	flip ( $T_{C}$ )	effect on ground truth
Hi-C loop strength	case replicates withheld	case and control swapped	$\Delta$ negated
multi-parent QTL	phenotypes permuted across lines within batch	founder labels permuted	high founder F5 becomes F2
scRNA eQTL	donor genotypes permuted	alleles recoded, $g\to 2-g$	$\beta$ negated
Wright–Fisher selection	locus A’s trajectory replaced by B’s	loci A and B swapped	selected locus becomes B

Appendix

Table 5: GeneBench-Pro problems and interventions. Null versions shuffle line order in every file. The flip changes the ground truth by a known transform.

Figure 7: Evaluator stress test (Claude Code runs). Change in each evaluator’s score under null, withdrawal and flip interventions. The null change should stay near the change between replicates. How large the withdrawal and flip changes should be depends on how many runs follow each edit, so the flip AUC (Table 6 ) is the direct test of whether an evaluator rewards following the evidence.

evaluator	$S_{\text{null}}$	$S_{\text{rm}}$	$S_{\text{flip}}$	flip AUC	$n$
Opus judge	0.06	0.25	0.44	0.12	75
GPT judge	0.05	0.31	0.56	0.10	75
Opus judge, grounding prompt	0.07	0.43	0.32	0.83	75
GPT judge, grounding prompt	0.09	0.51	0.38	0.73	75
benchmark score	0.03	0.21	0.68	0.11	75

Appendix

Table 6: Evaluator stress test by judge on the same runs, the first replicate of the Claude and GPT-5.6-Sol agents graded by all four judge variants (

n

per condition). The standard judge sees the reference answer. The grounding judge is also told that matching the reference is not sufficient. Flip AUC is the probability that a run following the flip outscores one that does not.

family	tasks	acc. $E$	no data	$I_{\text{null}}$	$R^{\text{flag}}_{\text{rm}}$	$B_{\text{rm}}$	$R_{\text{flip}}$	$B_{\text{flip}}$
abundance	4	100	81	100	53	64	67	33
composition change	3	89	67	96	19	78	67	0
expression localisation	3	96	74	96	52	96	81	19
marker gene	6	98	96	98	24	69	59	37
condition DE	2	83	83	94	28	94	17	61

Appendix

Table 7: Results by evidence family (%, Claude models).

Figure 8: Per-task rates (%, mean over Claude models and replicates), grouped by evidence family.

Figure 9: Self-reported confidence (left) and data support (right) by data version.

model	acc. on $E$	$B_{\text{rm}}$	$R_{\text{flip}}$	$B_{\text{flip}}$
Opus 5.5	98 [94, 100]	85 [70, 98]	59 [39, 80]	33 [15, 54]
Sonnet 5	100 [100, 100]	89 [76, 100]	52 [30, 74]	41 [19, 63]
Haiku 4.5	87 [70, 100]	56 [39, 74]	72 [52, 91]	15 [0, 31]
GPT-5.6-Sol	100 [100, 100]	81 [63, 96]	65 [44, 85]	28 [11, 48]
GPT-5.6-Terra	94 [85, 100]	78 [61, 93]	63 [41, 83]	31 [13, 52]

Appendix

Table 8: Per-model estimates with task-clustered 95% bootstrap intervals (%).

	original	null	withdrawn	flipped
agent turns (median)	5	5	6	6
wall time, s (median)	30	27	49	38

Appendix

Table 9: Analysis effort per run (Claude models).

problem (flip)	pass on $E$	$I_{\text{rep}}$	$R^{\text{flag}}_{\text{rm}}$	$R^{\text{rel}}_{\text{flip}}$	$R^{\text{rel}}_{\text{flip}}$
	Claude models				GPT-5.6-Sol
Hi-C loop strength (case $\leftrightarrow$ control)	0	56	78	56	33
multi-parent QTL (founders relabelled)	33	33	0	56	100
scRNA eQTL (allele recoded)	0	56	0	44	67
Wright–Fisher selection (loci swapped)	44	44	67	56	0

Appendix

Table 10: GeneBench-Pro per problem (%). Pass is the official grader against the original ground truth. The Claude models and GPT-5.6-Sol have three replicates per problem.

	accuracy on $E$		no data		withdrawal flagged		credit after withdrawal		flips followed		credit after flip		EGA
agent	named	anon.	named	anon.	named	anon.	named	anon.	named	anon.	named	anon.	named	anon.
Opus 5.5	97	100	90	40	40	40	83	73	60	100	37	0	37	60
Sonnet 5	100	90	97	37	17	53	90	70	43	97	53	3	23	67
Haiku 4.5	87	83	73	3	47	40	63	13	70	83	17	10	57	80
GPT-5.6-Sol	100	93	87	30	20	17	90	50	53	93	43	7	20	57
GPT-5.6-Terra	90	83	87	27	27	30	83	50	53	87	47	13	23	53

Appendix

Table 11: Named and anonymised gene tasks (%, 10 tasks and 3 replicates per agent). For the Claude models combined, the 95% task-bootstrap interval of the difference excludes zero for no-data accuracy, flips followed, credit after withdrawal and after a flip, withdrawal flagged and EGA, and includes zero for accuracy on the original data.

	chooses abstention option			key credit, withdrawn		key credit, flipped
Model	orig	withdrawn	flipped	without	with	without	with
Opus 5.5	0	59	6	85	37	33	19
Sonnet 5	0	46	6	89	48	41	35
Haiku 4.5	0	65	6	56	9	15	9
all	0	57	6	77	31	30	21

Appendix

Table 12: Abstention option (%, Claude models), with the option “The provided data are insufficient to determine the answer” offered or not. Key credit is the official BAIS score against the original key.

case	data	run	ans.	agent’s evidence (verbatim)	bench.	judge
Answer from memory after noticing the gap	withdraw: TMPRSS2 deleted from every file	S00_Q4 Sonnet 5	D	“TMPRSS2 is absent from the provided nasal h5ad (X and raw), so it could not be quantified.” … “the answer is a low-confidence extrapolation from that plus prior knowledge.”	1 (key)	6/2
Same data, abstention offered	withdraw (as above), option “insufficient data” added	S00_Q4 Sonnet 5	E	“TMPRSS2 is absent from the nasal (and bronchial) matrices, both in X and raw” … “so its expression by cell type cannot be measured.”	abstained	–
Same failure in three scaffolds	withdraw: the deep-caries (pulpitis) cells removed	S21_Q2 Opus 5.5	B	“it has no deep caries cells, so upregulation in deep caries can’t be tested.” … “the answer rests on VCAM1’s known role in endothelial activation and leukocyte adhesion.”	1 (key)	7/3
		S21_Q2 GPT-5.6-Sol	B	“the h5ad contains only normal (890 cells) and enamel-caries (2,015 cells) samples and no deep-caries sample”	1 (key)	7/3
		S21_Q2 Biomni (Sonnet 5)	B	“The provided h5ad file contains only ’normal’ and ’enamel caries’ samples (not ’deep dental caries’ as described in the background)”	1 (key)	7/3
Elimination instead of evidence	flip: HES1 and PTPRC expression swapped	S35_Q2 Opus 5.5	B	“they barely express HES1 (mean 0.02, 1% of cells).” … “HES1 was chosen by ruling those out, not because the data show it.”	1 (key)	6/3

Appendix

Table 13: Case studies. “ans.” is the agent’s answer, “bench.” the official score against the original key, and “judge” the Opus judge’s overall score under the standard and the grounding prompt.

Explore similar work

DISCERN: Can AI Agents Work Like Scientists and Guide Discovery?
Sep 27, 2026Nan Huang, Mario Tapia-Pacheco, Kun Zhou +4AI Agent Reliability Scientific Hypothesis Generation
University of California San Diego · Independent Researcher
InfiniteScienceGym: An Unbounded, Procedurally-Generated Benchmark for Scientific Analysis
Apr 14, 2026Oliver Bentham, Vivek SrikumarLLM Evaluation Scientific QA
Kahlert School of Computing University of Utah
Benchmarking AI Agents for Addressing Scientific Challenges Across Scales
Jun 10, 2026Tianyu Liu, Allen Xin Wang, Antonia Panescu +30AI for Science AI Agent Evaluation
Yale University, CT, USA · The Pennsylvania State University, PA, USA · Northeastern University, MA, USA +7

Sep 27, 2026J/K move · Enter open · S save

DISCERN: Can AI Agents Work Like Scientists and Guide Discovery?

Nan Huang, Mario Tapia-Pacheco, Kun Zhou, Yiming Huang+3

University of California San Diego · Independent Researcher

AI Agent Reliability Scientific Hypothesis Generation AI Agent Benchmarks Scientific Reasoning in Language Models Data Analysis Agents

Open paper AskPDF arXiv

Reliable automated research requires agents to vet data, verify analyses, and generate hypotheses grounded in trustworthy evidence, potentially reducing routine scientific workload while allowing scientists to focus on interpretation and discovery. Existing benchmarks often only assess analytical task completion or hypothesis generation separately rather than testing whether reliable evidence supports valid and novel claims. We introduce DISCERN (Data Integrity and Scientific Capability: Evidence, Reasoning, and Novelty), a controlled benchmark on real, publicly available datasets that evaluates three key levels of an automated research workflow. The first two levels test data integrity and analysis verification under confounds and tool traps, while the third tests hypothesis generation and revision under adversarial review, including counterfactual cases in which evidence consistent with real data and documented scientific phenomena conflicts with established expectations, motivating alternative explanations and testable hypotheses. Across 203 tasks, eight life-science tracks, and eight models, DISCERN shows that strong aggregate performance can mask level-specific weaknesses. Agents earn perfect scores in only 60.8% of Level 1, 34.2% of Level 2, and 0.6% of Level 3 evaluations, with penalties attributed to rejection of sound data, failure to carry recognized limitations into conclusions, and wide variation in hypothesis production. Cross-track rankings by token and code use are substantially more stable than rankings by evidence judgment, suggesting greater consistency in computational effort than in evidence-based reasoning. These profiles identify opportunities for supervised scientific assistance, but current agents do not yet demonstrate reliable autonomous analysis or discovery. Code and data: https://huggingface.co/datasets/discern-bench-anon/discern-benchmark

Abstract

Explore similar work

DISCERN: Can AI Agents Work Like Scientists and Guide Discovery?

InfiniteScienceGym: An Unbounded, Procedurally-Generated Benchmark for Scientific Analysis

Benchmarking AI Agents for Addressing Scientific Challenges Across Scales