CONFIRM: Governing Scientific Reporting for Trustworthy Agentic Neuroimaging

Yuxiang Wei; Lijing Zhu; Qizhen Lan; Yanteng Zhang; Vince D. Calhoun

cs.AINov 1, 2026

CONFIRM: Governing Scientific Reporting for Trustworthy Agentic Neuroimaging

Authors: Yuxiang Wei, Lijing Zhu, Qizhen Lan, Yanteng Zhang, Vince D. Calhoun

Organizations: TReNDS Center, Atlanta, GA, USA · Georgia Institute of Technology, Atlanta, GA, USA · University of Houston–Clear Lake, Houston, TX, USA · McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX, USA ·

Abstract

Large language model (LLM) agents can autonomously run neuroimaging analyses from research questions to reported findings. Existing agentic systems use tools to check that code runs and reported values match its output, but rely on LLM judgment to decide which results warrant scientific reporting. We show that LLM judges endorse site-confounded findings even when the confound is presented as evidence. We present CONFIRM, a framework that separates running an analysis from deciding which findings to report. CONFIRM encodes each hypothesis as a machine-readable, pre-specified analysis plan that fixes the claim and its pass criteria before computing results. It then filters results through five deterministic gates (search history and multiplicity, confounds, power, analytic stability, and replication) under three configurable reporting standards, from least to most demanding: Screen, Replicate, and Full. Failed claims receive diagnoses that guide follow-up analyses. To evaluate reporting decisions at the claim level, we introduce NeuroClaimBench, comprising 489 executable neuroimaging claims with frozen data, including literature-supported findings and 150 controls designed to violate individual methodological requirements. On NeuroClaimBench, CONFIRM falsely supports at most 3% of controls, compared with 23--38% for the baselines, while recovering up to 61% of literature-supported claims. When auditing an existing agent, CONFIRM reduces false support on controls from 23% to 0% while retaining the agent's literature-supported claims. Its diagnostic feedback also produces more supported revisions than Self-Refine with fewer LLM calls. Together, CONFIRM and NeuroClaimBench show that basing reporting decisions on pre-specified analysis plans and deterministic gates is essential for trustworthy autonomous scientific discovery.

Figures & tables

Figure 1: CONFIRM separates running an analysis from deciding whether its finding may be reported. The LLM turns a question into an analysis plan that is frozen before any statistic is computed. Code runs the analysis, and five gates check the result. The reporting standard, chosen before results are seen, sets which gates must pass, and only Full can report a claim as confirmed . If a required gate fails, the claim is withheld, and the LLM may propose revised analyses that keep the original question. These are tested first on the discovery and replication cohorts and then on holdout data.

What can go wrong	Gate	Passes when (default)	Designed control
Many tests were tried and only the best one is reported	Search history and multiplicity	The effect stays significant after correcting for every test tried ( $\alpha=0.05$ ).	Broad-search
A site or scanner difference creates an apparent effect	Measured confounds	No measured site, scanner, field-strength, or software variable differs between the groups while being left out of the analysis.	Site-confounded
The sample is too small to trust the result	Power	The planned design can detect a modest reference effect, by default $d=0.3$ , with at least 80% power.	Underpowered
The result depends on arbitrary analysis choices	Analytic stability	At least 60% of reasonable alternative analyses give a consistent result ( Steegen et al., 2016 ) .	–
The effect does not appear again in new data	Independent replication	The effect has the same direction and $p<0.05$ in every replication dataset.	Nonreplicating

Table 1: Each gate catches one known way a neuroimaging finding can go wrong. The rightmost column lists the designed controls used to benchmark each gate (Section 4 ). Random-label controls lack true signal and trigger multiple gates. Exact mathematical definitions for scalar and brain-wide claims appear in Appendix Table 4 .

Figure 3: CONFIRM establishes a Pareto-optimal reporting frontier. a , Recovery on literature-supported claims in the common scalar set (higher is better) versus false support on designed negative controls (lower is better). The dashed curve connects CONFIRM’s three nested reporting standards (Screen, Replicate, Full). Error bars indicate 95% confidence intervals. b , Accepted controls by failure mode. Each segment is a percentage of all 150 controls (42 site-confounded, 42 broad-search, 42 random-label, 21 underpowered, and 3 nonreplicating).

Gate thresholds
Setting	Rec. $\uparrow$	FS $\downarrow$	Ctrl. $\downarrow$
Defaults	41	11	0
Stability $\rho\in\{0.5,0.7,0.8\}$	41	11	0
Minimum power $\in\{0.7,0.9\}$	41	11	0
Confound $\alpha\in\{0.01,0.10\}$	35–41	11	0
Replication $\alpha\in\{0.01,0.10\}$	37–41	5–11	0

Table 2: CONFIRM-Full rejects all designed controls across the tested gate thresholds. Each row changes one setting and keeps all others at their defaults. Rec. is recovery on 51 literature-supported claims, FS is false support on 19 literature-unsupported claims, and Ctrl. is false support on 150 designed controls. Numbers shown are percentages (%). Brain-wide replication settings change the pattern-correlation, replicated-region-fraction, and Dice thresholds, which set how closely a brain-wide pattern must match across cohorts to count as replicated.

Method	LLM calls	Candidates	Supported	Parents	Holdout
	$\downarrow$	evaluated	on search $\uparrow$	(%) $\uparrow$	(%) $\uparrow$
Failure-blind retry	765	3,073	49	9	7
Self-Refine	1,301	2,161	54	11	9
CONFIRM diagnosis	653	2,971	70	11	17

Table 3: CONFIRM diagnostics yield more supported revisions with fewer LLM calls. GPT-5.5 revises the same 215 withheld claims for three rounds, with up to five candidates per round. Methods share the validator and multiplicity correction and differ only in failure feedback.

Figure 4: CONFIRM’s diagnosis leads the LLM to a revision supported on holdout data. CONFIRM withholds a claim because the FreeSurfer version, which is associated with age, was omitted. The LLM’s two valid revisions add the version, pass every gate on the search data, and are supported on holdout data. Blue boxes mark steps that pass, and red boxes mark steps that fail.

Figure 5: Adding CONFIRM as a retrofit layer to NeuroClaw-adapted reduces an existing agent’s false support without losing its findings. Solid bars depict a filter where a claim is reported only if both NeuroClaw-adapted and CONFIRM accept it. Hatched bars replace the agent’s reporting decision with CONFIRM’s decision. Metrics are evaluated on the common scalar set: a , Recovery on literature-supported claims (higher is better); b , False support on literature-unsupported claims (lower is better); and c , False support on designed negative controls (lower is better).

Figure 6: CONFIRM diagnoses improve an existing agent’s revisions. NeuroClaw-adapted drafts every claim, and the agent uses feedback to revise only the claims that CONFIRM withheld. Both panels are percentages of the 280 executed drafts. None of the three versions accepts a site-confounded control.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Gate	Decision rule	Default
Search history: provenance	The search and the data used to select the claim are documented, and selection did not use the full dataset.	–
Search history: multiplicity	Scalar: $p\leq\alpha/M_{\mathrm{eff}}$ in the locked direction. Brain-wide: at least one region with Benjamini–Hochberg $q\leq\alpha$ over $M_{\mathrm{eff}}$ hypotheses, in the locked direction.	$\alpha=0.05$
Measured confounds	Every required covariate is declared, and no site, scanner, field-strength, or software variable is both associated with the predictor and omitted from the covariates.	$\alpha=0.05$
Power	The planned design detects $\delta_{\mathrm{ref}}$ , a literature value when available and $d=0.3$ otherwise, at the minimum power.	power $0.80$
Analytic stability	The fraction of consistent analysis variants (forks) is at least $\rho$ , over 24 forks for scalar claims and 6 for brain-wide claims.	$\rho=0.60$
Replication	Scalar: locked direction and $p<\alpha$ in every replication cohort. Brain-wide: pattern correlation, replicated-region fraction, and Dice at or above the analysis-plan thresholds, with at least three shared regions and 50% feature coverage.	$\alpha=0.05$ ; $0.5$ , $0.5$ , $0.0$

Appendix

Table 4: Exact gate decision rules and default thresholds. This table is the formal counterpart of Table 1 . Analysis plans may specify stricter thresholds.

Verdict	Assignment rule	Example attached reasons
confirmed	Every gate passes for the pre-specified analysis plan.	All gate states pass.
fragile	Search history or multiplicity requirements, measured-confound checks, or analytic stability fail.	unverifiable_search_provenance ; confound_incomplete ; failed stability.
under_powered	The planned design cannot detect $\delta_{\mathrm{ref}}$ at the required power.	Reference effect, planned sample size, and achieved power.
non_replicated	The locked analysis plan fails in a designated replication cohort.	Missing compatible outcome, wrong sign, or non-significant effect.

Appendix

Table 5: Report-level verdicts and the reasons attached to withheld claims. The analysis record retains the result of every gate and every reason for withholding a claim, even when a reporting standard displays a single verdict.

Split	Source	Executable	Support	Withhold	Unresolved
Scientific	332	279	51	19	209
External-transfer	62	60	18	30	12
Designed controls	150	150	0	150	0
Total	544	489	69	199	221

Appendix

Table 6: NeuroClaimBench composition. The Support, Withhold, and Unresolved columns give reference labels among executable claims. Scored metrics use the labeled scientific claims and the designed controls.

Figure 7: Sensitivity to the requirements for literature reference labels. A reference requires two assessors to identify direct evidence for the exact frozen contrast, and 209 executable scientific claims fail this criterion. Reference rules progress from strictest to most permissive, and

n

is the number of labeled claims under each rule. Shading marks the two-assessor, exact-match rule. Open squares and filled circles distinguish coincident Replicate and Full results. CONFIRM’s verdicts are fixed, so only the labeled set changes. a , Recovery. b , False support, including four additional literature-unsupported references. The ordering Full

\leq

Replicate

<

Screen holds under every reference rule.

Matched common-scalar comparison
Evaluation	Recovery $\uparrow$	Literature false support $\downarrow$	Control false support $\downarrow$
VERITAS-adapted	65 [45, 81]	40 [16, 68]	38 [30, 46]
NeuroClaw-adapted	48 [30, 67]	13 [2, 40]	23 [16, 30]
Direct LLM judge	55 [36, 73]	20 [4, 48]	29 [22, 37]
Significance filter	52 [33, 70]	13 [2, 40]	28 [21, 36]
CONFIRM-Screen	61 [42, 78]	40 [16, 68]	3 [1, 8]

Appendix

Table 7: Claim-evaluation comparison and CONFIRM-Full scope. In the matched comparison, every method scores the same 31 literature-supported and 15 literature-unsupported claims from the common scalar set and the same 150 designed controls. The last row reports CONFIRM-Full on the strict references of the full literature set, 23 supported and 3 unsupported. Values are percentages of claims supported. Bracketed 95% confidence intervals accompany the matched-comparison rows. Arrows show whether higher (

\uparrow

) or lower (

\downarrow

) is better.

Removed gate	$\Delta$ literature recovery	$\Delta$ literature false support	$\Delta$ control false support
Search history: provenance	+0	+0	+0
Search history: multiplicity	+0	+0	+0
Measured confounds	+2	+0	+24
Power	+0	+0	+1
Analytic stability	+0	+0	+0
Replication	+8	+26	+2

Appendix

Table 8: Effect of removing each gate from CONFIRM-Full. Each row reports the increase in support, in percentage points (

\Delta

), when one gate is omitted and all other thresholds are held fixed. Recovery is computed over 51 literature-supported references, literature false support over 19 literature-unsupported references, and control false support over 150 designed controls.

Figure 8: Recovery and false support under each reporting standard. Results on the literature set, for the direct LLM judge and the three CONFIRM standards. Green bars show recovery and red bars show literature false support.

Label	Withheld	Repl.	Mult.	Stab.	Power	Conf.
Supported	59	87	73	73	20	3
Unsupported	89	94	71	71	0	0

Appendix

Table 9: Gates failed by the literature claims that CONFIRM-Full withholds (%). Withheld is the share of each set that CONFIRM-Full withholds, out of 51 literature-supported and 19 literature-unsupported claims. The remaining columns give the share of withheld claims that fail replication (Repl.), multiplicity (Mult.), stability (Stab.), power, and measured confounds (Conf.). A claim can fail several gates, so the rows do not sum to 100.

Failed gates (overlapping)
Holdout outcome	Confirmed	Withheld
Evaluated claims	65	144
Full standard passed	16	5
Replication gate passed	41	22
Search history / multiplicity	6	124
Measured confounds	0	4

Appendix

Table 10: Holdout outcomes by initial verdict. Entries count claims. A claim can fail several gates.

Regime	Reported (%) $\uparrow$	Added (%) $\uparrow$	Calls $\downarrow$	Holdout (%) $\uparrow$
No feedback	28.6	n/a	n/a	5.7
Self-critique	35.4	9.5	1,357	6.4
CONFIRM diagnosis	37.5	12.5	894	8.9

Appendix

Table 11: Diagnosis feedback for an existing agent. NeuroClaw-adapted drafts 280 executable claims, CONFIRM evaluates them, and the agent uses feedback to revise only the 200 withheld claims. Reported and Holdout are percentages of the 280 executed drafts, Added is the percentage of withheld claims revised into supported claims, and Control is false support on the 42 site-confounded controls. Both arms share the model, the budget of

R=3

rounds and

K=5

candidates per round, the validator, and the multiplicity correction. Holdout support uses partitions excluded from revision. Arrows show whether higher (

\uparrow

) or lower (

\downarrow

) is better.

Arm	Generated	Retained	Evaluated	Supported	Parents (%)
CONFIRM diagnosis, $R=3$ , $K=5$	3,187	3,059	2,971	70	11
Failure-blind retry, $R=3$ , $K=5$	3,275	3,178	3,073	49	9
Self-Refine, $R=3$ , $K=5$	3,259	2,361	2,161	54	11
Designed controls, $R=1$ , $K=2$	334	296	293	0	0
Designed controls, $R=10$ , $K=10$	13,371	12,313	10,716	0	0

Appendix

Table 12: Follow-up search runs. The first three arms revise the 215 claims that CONFIRM-Full withheld, with

R=3

rounds and up to

K=5

candidates per round, and differ only in the feedback given to the LLM. The last two arms give CONFIRM’s diagnosis for all 150 designed controls, with a small and a large budget. Generated counts candidate analysis plans proposed by the LLM, Retained counts distinct candidates kept after removing duplicates, and Evaluated counts candidates that passed validation and were run. Supported counts candidates that pass every gate on the search data after correcting for all analyses tried. Parents is the percentage of starting claims with at least one supported candidate.

AI Agents for Scientific Discovery

AI agents for scientific discovery use language models, tools, and experiments to generate, test, and refine scientific knowledge.

See all 305 papers

Bringing analytic rigor to agentic AI for science: The Brain Researcher platform for neuroimaging data analysis
Aug 20, 2026Zijiao Chen, Nicholas Lu, Xinhui Li +14Data Analysis Agents AI Agents for Scientific Discovery
Stanford University, Stanford, CA, USA · Tri-institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS), Georgia State University, Georgia Institute of Technology, Emory University, Atlanta, GA, USA · Inria, CEA, Universit´e Paris-Saclay, Palaiseau, France +3
Sound Agentic Science Requires Adversarial Experiments
Apr 23, 2026Dionizije Fa, Marko CuljakAI-Assisted Scientific Research LLM Agent Evaluation
Entropic
AI scientists produce results without reasoning scientifically
Apr 20, 2026Martiño Ríos-García, Nawaf Alampara, Chandan Gupta +5AI Agent Reliability Scientific Reasoning
Laboratory of Organic and Macromolecular Chemistry (IOMC), Friedrich Schiller University Jena, Humboldstr. 10, 07743 Jena, Germany · Department of Civil Engineering, Indian Institute of Technology Delhi, Hauz Khas, New Delhi 110016, India · School of Interdisciplinary Research, Indian Institute of Technology Delhi, Hauz Khas, New Delhi 110016, India +3

Aug 20, 2026J/K move · Enter open · S save

Bringing analytic rigor to agentic AI for science: The Brain Researcher platform for neuroimaging data analysis

Zijiao Chen, Nicholas Lu, Xinhui Li, Jocelyn A. Ricard+13

Stanford University, Stanford, CA, USA · Tri-institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS), Georgia State University, Georgia Institute of Technology, Emory University, Atlanta, GA, USA · Inria, CEA, Universit´e Paris-Saclay, Palaiseau, France+3

Data Analysis Agents AI Agents for Scientific Discovery AI Agent Reliability Neuroimaging

Open paper AskPDF arXiv

AI agents can execute scientific analyses, but an analytic output becomes a defensible claim only after alternatives are weighed and the claim is limited to what the evidence supports. Agents may reproduce failures including selective analysis, premature declarations of success and optimization of imperfect criteria. We present Brain Researcher, an agentic research harness operating in a neuroimaging researcher's computational environment under rules for admissible analyses, required checks and claim scope. In benchmarks, Brain Researcher increased first-choice tool-selection accuracy across seven models by 70.2 percentage points (23.3% without it versus 93.6% with it) and verifiable grounding from 4.6% to 22.0%. In collaborator-led and self-evolving studies, multiverse analyses exposed analytic-choice sensitivity, and scientific review classified claims as accepted, qualified, revised, blocked, rejected or deferred. By linking decisions to evidence and provenance, Brain Researcher embeds methodological judgment within the workflow, not after it.

Abstract

AI Agents for Scientific Discovery

Explore similar work

Bringing analytic rigor to agentic AI for science: The Brain Researcher platform for neuroimaging data analysis

Sound Agentic Science Requires Adversarial Experiments

AI scientists produce results without reasoning scientifically