Identity or Evidence? Auditing Evidence Dependence in Agentic Life-Science Benchmarks

Danai Brilli; Maria Georgila

cs.CVOct 27, 2026

Identity or Evidence? Auditing Evidence Dependence in Agentic Life-Science Benchmarks

Authors: Danai Brilli, Maria Georgila

Organizations: Zo Labs

Abstract

Agentic life-science benchmarks give an agent evidence about a named entity and score an answer against a recorded outcome. That score tests reasoning over evidence only if the evidence, rather than the name, determines the answer. We examine this distinction by varying identity and evidence independently on a clinical-outcome benchmark and published tasks from Biomni and LAB-Bench. Blinding identity matters little when a prompt contains a decisive sequence or passage. It matters a great deal when the prompt is mostly a name and candidate list (0.742 named versus 0.082 blinded; chance 0.093). Across 96 LAB-Bench LitQA2 questions, the gap increases under one fixed passage-redaction scheme (+0.000 to +0.135; per-instance slope +0.147, p= 0.004). On the clinical benchmark, a no-evidence recall probe matches prediction accuracy and recovers a database-specific programme count. So, we argue that benchmarks intended to measure evidence-dependent reasoning should report identity-blinded results, retrieval telemetry, and positive controls.

Figures & tables

Figure 1: The cost of identity blinding varies with available evidence. Left: conditions within a row differ only in entity mentions; numeric evidence is byte-identical. Right: on 96 LitQA2 questions, the supporting passage is redacted in four steps while entity mentions remain intact. The blinding gap grows with redaction (per-instance slope +0.147,

p=0.004

analysis	status	evidence	key limitation
LitQA2: full passage	primary	0.990 named and blinded at 0% redaction	96 questions; one model
LitQA2: redaction gradient	primary	identity-gap slope +0.147, $p=0.004$	one seeded mask
Clinical dossier: blind	supporting	0.531 blinded versus 0.783 as written	semantic context also removed
Clinical dossier: recall	supporting	0.818 all-case accuracy; count $\rho=0.654$	not a per-prediction mechanism test
Deletion controls	diagnostic	controls are inconclusive	no attribution is claimed

Table 1: Interpretive status of the main analyses.

condition	what it changes	what a movement means
AS-WRITTEN	nothing	the benchmark’s own number, under our harness
BLIND	$I\rightarrow$ opaque salted tokens	does the score survive without the name?
DELETE	$E\rightarrow$ a note that there is none	does the score use the evidence at all?
PLACEBO	$E\rightarrow$ another instance’s evidence	if this costs what deletion costs, not content
RECALL	$E$ and the question cut; record asked for	does the agent already know the label?
POSITIVE	a decisive line, right 95% of the time	can the instrument detect evidence?

Table 2: Core audit conditions.

E

is the evidence block and

I

the entity identity. All contrasts are paired, while the synthetic positive control measures the instrument, not the benchmark.

Figure 2: Recall tracks a database-specific count. Among 218 numeric recall responses, the recalled number of programmes rises with the number recorded in the source table (

\rho=0.654

). Medians and interquartile ranges are shown.

Figure 3: Deletion does not isolate dossier content. The own dossier, a matched placebo dossier, and a numerically scrambled dossier all reduce accuracy relative to no dossier. Because the substituted blocks have little independently measured signal, these controls do not identify a content-use effect.

Figure 4: Stated confidence is weakly related to accuracy across conditions. Accuracy spans 63 points while mean confidence spans 12. The false-identity condition is particularly over-confident.

Scientific Agent Benchmarks

See all 36 papers

Are We Measuring Scientific Intelligence? Rethinking the Evaluation of AI Scientists
Oct 4, 2026Kate Zhang, Yuante LiAI Agent Evaluation Scientific Agent Benchmarks
School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213
Auditing Tool Traces in Agentic LLMs: Source Localization and Counterfactual Dependence Testing
Oct 18, 2026Junhao Liu, Tianxiao Li, Yongxian Wu +4LLM Auditing AI Agent Auditing
University of California, Irvine · NEC Labs America
Can Agent Benchmarks Support Their Scores? Evidence-Supported Bounds for Interactive-Agent Evaluation
May 11, 2026Shanshan Gao, Liyi ZhouComputer-Use Agent Benchmarks Benchmark Design
The University of Sydney

Oct 4, 2026J/K move · Enter open · S save

Are We Measuring Scientific Intelligence? Rethinking the Evaluation of AI Scientists

Kate Zhang, Yuante Li

School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213

AI Agent Evaluation Scientific Agent Benchmarks Evidence-Grounded Reasoning AI Agents for Scientific Discovery Scientific QA Counterfactual Evaluation

Open paper AskPDF arXiv

AI agents can now carry out data-driven scientific analyses end to end, and benchmarks assess them by giving an agent a question and a dataset and scoring its final answer against a fixed key. These benchmarks assume that a correct answer was derived from the supplied data, a property we call evidence grounding. However, an agent can also reach the key from prior knowledge or by ruling out the other options, and a score based on a single run cannot tell these cases apart. We show how to test this assumption and find that it often fails. For each question, we build versions of its data files in which the evidence for the answer is left intact, withdrawn or reversed, check each edit with a pre-registered reference statistic, and run the same agent on every version. We then measure evidence-grounded accuracy, which credits a correct answer only if the agent also responds when the evidence is withdrawn and follows it when it is reversed. We evaluate three agent scaffolds and five models on 18 single-cell questions from BAISBench and four synthetic problems from GeneBench-Pro. On the single-cell questions, the Claude agents are 95% accurate and answer 83% correctly without any data, but their evidence-grounded accuracy is only 41%. Hiding gene names raises the share of runs that follow reversed evidence from 58% to 93% on ten gene tasks, suggesting that prior knowledge competes with the supplied data. The benchmark score and LLM judges can also reward answers that ignore the changed evidence. Measuring scientific intelligence rather than recall therefore requires checking whether answers follow the evidence and whether scores reward them for it.

Abstract

Scientific Agent Benchmarks

Explore similar work

Are We Measuring Scientific Intelligence? Rethinking the Evaluation of AI Scientists

Auditing Tool Traces in Agentic LLMs: Source Localization and Counterfactual Dependence Testing

Can Agent Benchmarks Support Their Scores? Evidence-Supported Bounds for Interactive-Agent Evaluation