cs.AINov 1, 2026

CONFIRM: Governing Scientific Reporting for Trustworthy Agentic Neuroimaging

Authors: Yuxiang Wei, Lijing Zhu, Qizhen Lan, Yanteng Zhang, Vince D. Calhoun

Organizations: TReNDS Center, Atlanta, GA, USA · Georgia Institute of Technology, Atlanta, GA, USA · University of Houston–Clear Lake, Houston, TX, USA · McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX, USA · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language model (LLM) agents can autonomously run neuroimaging analyses from research questions to reported findings. Existing agentic systems use tools to check that code runs and reported values match its output, but rely on LLM judgment to decide which results warrant scientific reporting. We show that LLM judges endorse site-confounded findings even when the confound is presented as evidence. We present CONFIRM, a framework that separates running an analysis from deciding which findings to report. CONFIRM encodes each hypothesis as a machine-readable, pre-specified analysis plan that fixes the claim and its pass criteria before computing results. It then filters results through five deterministic gates (search history and multiplicity, confounds, power, analytic stability, and replication) under three configurable reporting standards, from least to most demanding: Screen, Replicate, and Full. Failed claims receive diagnoses that guide follow-up analyses. To evaluate reporting decisions at the claim level, we introduce NeuroClaimBench, comprising 489 executable neuroimaging claims with frozen data, including literature-supported findings and 150 controls designed to violate individual methodological requirements. On NeuroClaimBench, CONFIRM falsely supports at most 3% of controls, compared with 23--38% for the baselines, while recovering up to 61% of literature-supported claims. When auditing an existing agent, CONFIRM reduces false support on controls from 23% to 0% while retaining the agent's literature-supported claims. Its diagnostic feedback also produces more supported revisions than Self-Refine with fewer LLM calls. Together, CONFIRM and NeuroClaimBench show that basing reporting decisions on pre-specified analysis plans and deterministic gates is essential for trustworthy autonomous scientific discovery.

Figures & tables

Figure 1: CONFIRM separates running an analysis from deciding whether its finding may be reported. The LLM turns a question into an analysis plan that is frozen before any statistic is computed. Code runs the analysis, and five gates check the result. The reporting standard, chosen before results are seen, sets which gates must pass, and only Full can report a claim as confirmed . If a required gate fails, the claim is withheld, and the LLM may propose revised analyses that keep the original question. These are tested first on the discovery and replication cohorts and then on holdout data.
What can go wrongGatePasses when (default)Designed control
Many tests were tried and only the best one is reportedSearch history and multiplicityThe effect stays significant after correcting for every test tried ( α=0.05\alpha=0.05 ).Broad-search
A site or scanner difference creates an apparent effectMeasured confoundsNo measured site, scanner, field-strength, or software variable differs between the groups while being left out of the analysis.Site-confounded
The sample is too small to trust the resultPowerThe planned design can detect a modest reference effect, by default d=0.3d=0.3 , with at least 80% power.Underpowered
The result depends on arbitrary analysis choicesAnalytic stabilityAt least 60% of reasonable alternative analyses give a consistent result ( Steegen et al., 2016 ) .–
The effect does not appear again in new dataIndependent replicationThe effect has the same direction and p<0.05p<0.05 in every replication dataset.Nonreplicating
Table 1: Each gate catches one known way a neuroimaging finding can go wrong. The rightmost column lists the designed controls used to benchmark each gate (Section 4 ). Random-label controls lack true signal and trigger multiple gates. Exact mathematical definitions for scalar and brain-wide claims appear in Appendix Table 4 .
Figure 3: CONFIRM establishes a Pareto-optimal reporting frontier. a , Recovery on literature-supported claims in the common scalar set (higher is better) versus false support on designed negative controls (lower is better). The dashed curve connects CONFIRM’s three nested reporting standards (Screen, Replicate, Full). Error bars indicate 95% confidence intervals. b , Accepted controls by failure mode. Each segment is a percentage of all 150 controls (42 site-confounded, 42 broad-search, 42 random-label, 21 underpowered, and 3 nonreplicating).
SettingRec. ↑\uparrowFS ↓\downarrowCtrl. ↓\downarrow
Defaults41110
Gate thresholds
Stability ρ∈{0.5,0.7,0.8}\rho\in\{0.5,0.7,0.8\}41110
Minimum power ∈{0.7,0.9}\in\{0.7,0.9\}41110
Confound α∈{0.01,0.10}\alpha\in\{0.01,0.10\}35–41110
Replication α∈{0.01,0.10}\alpha\in\{0.01,0.10\}37–415–110
Table 2: CONFIRM-Full rejects all designed controls across the tested gate thresholds. Each row changes one setting and keeps all others at their defaults. Rec. is recovery on 51 literature-supported claims, FS is false support on 19 literature-unsupported claims, and Ctrl. is false support on 150 designed controls. Numbers shown are percentages (%). Brain-wide replication settings change the pattern-correlation, replicated-region-fraction, and Dice thresholds, which set how closely a brain-wide pattern must match across cohorts to count as replicated.
MethodLLM callsCandidatesSupportedParentsHoldout
 ↓\downarrowevaluatedon search ↑\uparrow(%) ↑\uparrow(%) ↑\uparrow
Failure-blind retry7653,0734997
Self-Refine1,3012,16154119
CONFIRM diagnosis6532,971701117
Table 3: CONFIRM diagnostics yield more supported revisions with fewer LLM calls. GPT-5.5 revises the same 215 withheld claims for three rounds, with up to five candidates per round. Methods share the validator and multiplicity correction and differ only in failure feedback.
Figure 4: CONFIRM’s diagnosis leads the LLM to a revision supported on holdout data. CONFIRM withholds a claim because the FreeSurfer version, which is associated with age, was omitted. The LLM’s two valid revisions add the version, pass every gate on the search data, and are supported on holdout data. Blue boxes mark steps that pass, and red boxes mark steps that fail.
Figure 5: Adding CONFIRM as a retrofit layer to NeuroClaw-adapted reduces an existing agent’s false support without losing its findings. Solid bars depict a filter where a claim is reported only if both NeuroClaw-adapted and CONFIRM accept it. Hatched bars replace the agent’s reporting decision with CONFIRM’s decision. Metrics are evaluated on the common scalar set: a , Recovery on literature-supported claims (higher is better); b , False support on literature-unsupported claims (lower is better); and c , False support on designed negative controls (lower is better).
Figure 6: CONFIRM diagnoses improve an existing agent’s revisions. NeuroClaw-adapted drafts every claim, and the agent uses feedback to revise only the claims that CONFIRM withheld. Both panels are percentages of the 280 executed drafts. None of the three versions accepts a site-confounded control.
Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

GateDecision ruleDefault
Search history: provenanceThe search and the data used to select the claim are documented, and selection did not use the full dataset.–
Search history: multiplicityScalar: p≤α/Meffp\leq\alpha/M_{\mathrm{eff}} in the locked direction. Brain-wide: at least one region with Benjamini–Hochberg q≤αq\leq\alpha over MeffM_{\mathrm{eff}} hypotheses, in the locked direction.α=0.05\alpha=0.05
Measured confoundsEvery required covariate is declared, and no site, scanner, field-strength, or software variable is both associated with the predictor and omitted from the covariates.α=0.05\alpha=0.05
PowerThe planned design detects δref\delta_{\mathrm{ref}} , a literature value when available and d=0.3d=0.3 otherwise, at the minimum power.power 0.800.80
Analytic stabilityThe fraction of consistent analysis variants (forks) is at least ρ\rho , over 24 forks for scalar claims and 6 for brain-wide claims.ρ=0.60\rho=0.60
ReplicationScalar: locked direction and p<αp<\alpha in every replication cohort. Brain-wide: pattern correlation, replicated-region fraction, and Dice at or above the analysis-plan thresholds, with at least three shared regions and 50% feature coverage.α=0.05\alpha=0.05 ; 0.50.5 , 0.50.5 , 0.00.0
Appendix
Table 4: Exact gate decision rules and default thresholds. This table is the formal counterpart of Table 1 . Analysis plans may specify stricter thresholds.
VerdictAssignment ruleExample attached reasons
confirmedEvery gate passes for the pre-specified analysis plan.All gate states pass.
fragileSearch history or multiplicity requirements, measured-confound checks, or analytic stability fail.unverifiable_search_provenance ; confound_incomplete ; failed stability.
under_poweredThe planned design cannot detect δref\delta_{\mathrm{ref}} at the required power.Reference effect, planned sample size, and achieved power.
non_replicatedThe locked analysis plan fails in a designated replication cohort.Missing compatible outcome, wrong sign, or non-significant effect.
Appendix
Table 5: Report-level verdicts and the reasons attached to withheld claims. The analysis record retains the result of every gate and every reason for withholding a claim, even when a reporting standard displays a single verdict.
SplitSourceExecutableSupportWithholdUnresolved
Scientific3322795119209
External-transfer6260183012
Designed controls15015001500
Total54448969199221
Appendix
Table 6: NeuroClaimBench composition. The Support, Withhold, and Unresolved columns give reference labels among executable claims. Scored metrics use the labeled scientific claims and the designed controls.
Figure 7: Sensitivity to the requirements for literature reference labels. A reference requires two assessors to identify direct evidence for the exact frozen contrast, and 209 executable scientific claims fail this criterion. Reference rules progress from strictest to most permissive, and nn is the number of labeled claims under each rule. Shading marks the two-assessor, exact-match rule. Open squares and filled circles distinguish coincident Replicate and Full results. CONFIRM’s verdicts are fixed, so only the labeled set changes. a , Recovery. b , False support, including four additional literature-unsupported references. The ordering Full ≤\leq Replicate << Screen holds under every reference rule.
EvaluationRecovery ↑\uparrowLiterature false support ↓\downarrowControl false support ↓\downarrow
Matched common-scalar comparison
VERITAS-adapted65 [45, 81]40 [16, 68]38 [30, 46]
NeuroClaw-adapted48 [30, 67]13 [2, 40]23 [16, 30]
Direct LLM judge55 [36, 73]20 [4, 48]29 [22, 37]
Significance filter52 [33, 70]13 [2, 40]28 [21, 36]
CONFIRM-Screen61 [42, 78]40 [16, 68]3 [1, 8]
Appendix
Table 7: Claim-evaluation comparison and CONFIRM-Full scope. In the matched comparison, every method scores the same 31 literature-supported and 15 literature-unsupported claims from the common scalar set and the same 150 designed controls. The last row reports CONFIRM-Full on the strict references of the full literature set, 23 supported and 3 unsupported. Values are percentages of claims supported. Bracketed 95% confidence intervals accompany the matched-comparison rows. Arrows show whether higher ( ↑\uparrow ) or lower ( ↓\downarrow ) is better.
Removed gateΔ\Delta literature recoveryΔ\Delta literature false supportΔ\Delta control false support
Search history: provenance+0+0+0
Search history: multiplicity+0+0+0
Measured confounds+2+0+24
Power+0+0+1
Analytic stability+0+0+0
Replication+8+26+2
Appendix
Table 8: Effect of removing each gate from CONFIRM-Full. Each row reports the increase in support, in percentage points ( Δ\Delta ), when one gate is omitted and all other thresholds are held fixed. Recovery is computed over 51 literature-supported references, literature false support over 19 literature-unsupported references, and control false support over 150 designed controls.
Figure 8: Recovery and false support under each reporting standard. Results on the literature set, for the direct LLM judge and the three CONFIRM standards. Green bars show recovery and red bars show literature false support.
LabelWithheldRepl.Mult.Stab.PowerConf.
Supported59877373203
Unsupported8994717100
Appendix
Table 9: Gates failed by the literature claims that CONFIRM-Full withholds (%). Withheld is the share of each set that CONFIRM-Full withholds, out of 51 literature-supported and 19 literature-unsupported claims. The remaining columns give the share of withheld claims that fail replication (Repl.), multiplicity (Mult.), stability (Stab.), power, and measured confounds (Conf.). A claim can fail several gates, so the rows do not sum to 100.
Holdout outcomeConfirmedWithheld
Evaluated claims65144
Full standard passed165
Replication gate passed4122
Failed gates (overlapping)
Search history / multiplicity6124
Measured confounds04
Appendix
Table 10: Holdout outcomes by initial verdict. Entries count claims. A claim can fail several gates.
RegimeReported (%) ↑\uparrowAdded (%) ↑\uparrowCalls ↓\downarrowHoldout (%) ↑\uparrowControl (%) ↓\downarrow
No feedback28.6n/an/a5.70
Self-critique35.49.51,3576.40
CONFIRM diagnosis37.512.58948.90
Appendix
Table 11: Diagnosis feedback for an existing agent. NeuroClaw-adapted drafts 280 executable claims, CONFIRM evaluates them, and the agent uses feedback to revise only the 200 withheld claims. Reported and Holdout are percentages of the 280 executed drafts, Added is the percentage of withheld claims revised into supported claims, and Control is false support on the 42 site-confounded controls. Both arms share the model, the budget of R=3R=3 rounds and K=5K=5 candidates per round, the validator, and the multiplicity correction. Holdout support uses partitions excluded from revision. Arrows show whether higher ( ↑\uparrow ) or lower ( ↓\downarrow ) is better.
ArmGeneratedRetainedEvaluatedSupportedParents (%)
CONFIRM diagnosis, R=3R=3 , K=5K=53,1873,0592,9717011
Failure-blind retry, R=3R=3 , K=5K=53,2753,1783,073499
Self-Refine, R=3R=3 , K=5K=53,2592,3612,1615411
Designed controls, R=1R=1 , K=2K=233429629300
Designed controls, R=10R=10 , K=10K=1013,37112,31310,71600
Appendix
Table 12: Follow-up search runs. The first three arms revise the 215 claims that CONFIRM-Full withheld, with R=3R=3 rounds and up to K=5K=5 candidates per round, and differ only in the feedback given to the LLM. The last two arms give CONFIRM’s diagnosis for all 150 designed controls, with a small and a large budget. Generated counts candidate analysis plans proposed by the LLM, Retained counts distinct candidates kept after removing duplicates, and Evaluated counts candidates that passed validation and were run. Supported counts candidates that pass every gate on the search data after correcting for all analyses tried. Parents is the percentage of starting claims with at least one supported candidate.
More in this topic

AI Agents for Scientific Discovery

AI agents for scientific discovery use language models, tools, and experiments to generate, test, and refine scientific knowledge.

See all 305 papers

Explore similar work

CardsList
Aug 20, 2026

Bringing analytic rigor to agentic AI for science: The Brain Researcher platform for neuroimaging data analysis

AI agents can execute scientific analyses, but an analytic output becomes a defensible claim only after alternatives are weighed and the claim is limited to what the evidence supports. Agents may reproduce failures including selective analysis, premature declarations of success and optimization of imperfect criteria. We present Brain Researcher, an agentic research harness operating in a neuroimaging researcher's computational environment under rules for admissible analyses, required checks and claim scope. In benchmarks, Brain Researcher increased first-choice tool-selection accuracy across seven models by 70.2 percentage points (23.3% without it versus 93.6% with it) and verifiable grounding from 4.6% to 22.0%. In collaborator-led and self-evolving studies, multiverse analyses exposed analytic-choice sensitivity, and scientific review classified claims as accepted, qualified, revised, blocked, rejected or deferred. By linking decisions to evidence and provenance, Brain Researcher embeds methodological judgment within the workflow, not after it.

Zijiao Chen, Nicholas Lu, Xinhui Li +14

Stanford University, Stanford, CA, USA · Tri-institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS), Georgia State University, Georgia Institute of Technology, Emory University, Atlanta, GA, USA · Inria, CEA, Universit´e Paris-Saclay, Palaiseau, France +3

Data Analysis AgentsAI Agents for Scientific DiscoveryAI Agent ReliabilityNeuroimaging
AskPDF
Apr 23, 2026

Sound Agentic Science Requires Adversarial Experiments

LLM-based agents are rapidly being adopted for scientific data analysis, automating tasks once limited by human time and expertise. This capability is often framed as an acceleration of discovery, but it also accelerates a familiar failure mode, the rapid production of plausible, endlessly revisable analyses that are easy to generate, effectively turning hypothesis space into candidate claims supported by selectively chosen analyses, optimized for publishable positives. Unlike software, scientific knowledge is not validated by the iterative accumulation of code and post hoc statistical support. A fluent explanation or a significant result on a single dataset is not verification. Because the missing evidence is a negative space, experiments and analyses that would have falsified the claim were never run or never published. We therefore propose that non-experimental claims produced with agentic assistance be evaluated under a falsification-first standard: agents should not be used primarily to craft the most compelling narrative, but to actively search for the ways in which the claim can fail.

Dionizije Fa, Marko Culjak

Entropic

AI-Assisted Scientific ResearchLLM Agent EvaluationLLM Agent ReliabilityScientific Claim Verification
AskPDF
Apr 20, 2026

AI scientists produce results without reasoning scientifically

Large language model (LLM)-based systems are increasingly deployed to conduct scientific research autonomously, yet whether their reasoning adheres to the epistemic norms that make scientific inquiry self-correcting is poorly understood. Here, we evaluate LLM-based scientific agents across eight domains, spanning workflow execution to hypothesis-driven inquiry, through more than 25,000 agent runs and two complementary lenses: (i) a systematic performance analysis that decomposes the contributions of the base model and the agent scaffold, and (ii) a behavioral analysis of the epistemological structure of agent reasoning. We observe that the base model is the primary determinant of both performance and behavior, accounting for 41.4% of explained variance versus 1.5% for the scaffold. Across all configurations, evidence is ignored in 68% of traces, refutation-driven belief revision occurs in 26%, and convergent multi-test evidence is rare. The same reasoning pattern appears whether the agent executes a computational workflow or conducts hypothesis-driven inquiry. They persist even when agents receive near-complete successful reasoning trajectories as context, and the resulting unreliability compounds across repeated trials in epistemically demanding domains. Thus, current LLM-based agents execute scientific workflows but do not exhibit the epistemic patterns that characterize scientific reasoning. Outcome-based evaluation cannot detect these failures, and scaffold engineering alone cannot repair them. Until reasoning itself becomes a training target, the scientific knowledge produced by such agents cannot be justified by the process that generated it.

Martiño Ríos-García, Nawaf Alampara, Chandan Gupta +5

Laboratory of Organic and Macromolecular Chemistry (IOMC), Friedrich Schiller University Jena, Humboldstr. 10, 07743 Jena, Germany · Department of Civil Engineering, Indian Institute of Technology Delhi, Hauz Khas, New Delhi 110016, India · School of Interdisciplinary Research, Indian Institute of Technology Delhi, Hauz Khas, New Delhi 110016, India +3

AI Agent ReliabilityScientific ReasoningLLM Agent EvaluationLLM ReliabilityAutonomous Scientific DiscoveryScientific Reasoning in Language Models+2
AskPDF