Bringing analytic rigor to agentic AI for science: The Brain Researcher platform for neuroimaging data analysis

Zijiao Chen; Nicholas Lu; Xinhui Li; Jocelyn A. Ricard; Ce Ju; Huan H. Wang; Christian Kindermann; Jeanette A. Mumford

cs.AIAug 20, 2026

Bringing analytic rigor to agentic AI for science: The Brain Researcher platform for neuroimaging data analysis

Authors: Zijiao Chen, Nicholas Lu, Xinhui Li, Jocelyn A. Ricard, Ce Ju, Huan H. Wang, Christian Kindermann, Jeanette A. Mumford,

Organizations: Stanford University, Stanford, CA, USA · Tri-institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS), Georgia State University, Georgia Institute of Technology, Emory University, Atlanta, GA, USA · Inria, CEA, Universit´e Paris-Saclay, Palaiseau, France · The University of Texas at Austin, Austin, TX, USA ·

Abstract

AI agents can execute scientific analyses, but an analytic output becomes a defensible claim only after alternatives are weighed and the claim is limited to what the evidence supports. Agents may reproduce failures including selective analysis, premature declarations of success and optimization of imperfect criteria. We present Brain Researcher, an agentic research harness operating in a neuroimaging researcher's computational environment under rules for admissible analyses, required checks and claim scope. In benchmarks, Brain Researcher increased first-choice tool-selection accuracy across seven models by 70.2 percentage points (23.3% without it versus 93.6% with it) and verifiable grounding from 4.6% to 22.0%. In collaborator-led and self-evolving studies, multiverse analyses exposed analytic-choice sensitivity, and scientific review classified claims as accepted, qualified, revised, blocked, rejected or deferred. By linking decisions to evidence and provenance, Brain Researcher embeds methodological judgment within the workflow, not after it.

Figures & tables

Figure 1: Brain Researcher: workspace-centric infrastructure for auditable neuroimaging research. Brain Researcher runs inside the researcher’s existing computational environment and exposes neuroimaging analyses as structured, auditable operations: every choice, check, and input is recorded as it happens, so that a frozen record of the completed analysis can be read and audited by someone other than the person who ran it, without re-executing it. (a) The tool ecosystem: established neuroimaging software for preprocessing, modeling, meta-analysis, machine learning, quality control, and reporting, each represented by a machine-readable specification and executed in a version-pinned container. (b) Each specification declares its inputs, outputs, parameters, version, evidence anchors, and validation rules; the rule checker tests every proposed call against these clauses before it runs, and records which clauses passed or failed. (c) The episode workflow: the researcher frames a question and approves the plan at the commitment gate, valid actions are dispatched to version-pinned executors, and the resulting audit bundle, containing the committed plan, tool versions, evidence consulted, artifacts, logs, provenance, and the checks each claim passed, feeds the review layer, which writes condition-tagged claims back to memory. This audit bundle is what makes an analysis auditable: a completed run is a fully inspectable research object rather than a one-off result, exported as a compact claim card a reviewer can reopen field by field. Methodological judgment remains the researcher’s; the system makes it visible at each stage.

Figure 2: BR-KG: provenance-linked semantic integration for grounded, auditable neuroimaging reasoning. BR-KG integrates existing ontologies, repositories, data resources, and literature into a single graph (745,949 nodes, 2,461,469 edges; 2026-07-07 release snapshot) aligned to the OpenNeuro Vocabulary (ONVOC) ( OpenNeuro, 2026 ) , which normalizes heterogeneous terms to shared identifiers so a query resolves consistently across sources. The central graph links neuroimaging concepts (tasks, contrasts, cognitive constructs), neural representations (brain regions, statistical maps), and research resources (datasets, tools) through typed relationships, with literature evidence attached. Crucially, source-backed facts carry explicit provenance (their source, and where available a verbatim supporting quote and grounding label), so a retrieved claim can be traced back to the study and passage that support it rather than taken on trust; coverage is partial and tracked, and this is what makes retrieval here auditable . Downstream panels show the payoff: grounded query answering and multi-hop reasoning over concept-task-map paths, each hop inspectable down to its underlying nodes, edges, and cited evidence, which lets the review layer attach a recommendation’s method-condition checks (cohort, paradigm, preprocessing, statistical model) before it is accepted.

Figure 3: Summary of Brain Researcher effects across quantitative benchmark tasks. Without-BR (gray) and with-BR (blue) benchmark performance. Capability@k is mean coverage of required task capabilities after the first k non-neutral actions. The left column reports Capability@1 and @3 across the seven model variants (Claude Opus 4.8, Codex GPT-5.5, Gemini 3.1 Pro, GLM-5.1, DeepSeek-V4-Pro, Kimi K2.5, Qwen3.6-Plus). Upper-right panels break Capability@1 down by task domain. Lower panels report Handoff score@1 and @3 (whether the first route carries enough information for another agent to continue) and a Gemini 2.5 Flash single-judge safeguard: precision among claims marked grounded (fraction whose cited evidence was both locatable and judged supportive). Correct route/tool@1, the first-action selection accuracy reported in the text (23.3% to 93.6%), is detailed in Supplementary Methods S11.1.1. Metrics are interpreted within panel, as denominators and scoring rules differ across benchmarks.

Figure 4: Multiverse sensitivity and claim-review outcomes across three collaborator episodes. (A–C) Schizophrenia NeuroMark audit: (A) group-mean functional connectivity for controls (HC, $N=181$ ), patients (SZ, $N=182$ ), and their difference across four estimators; (B) NM-H2 (between- versus within-domain) specification curve over the 480-specification multiverse; after sign-aware rescoring, its estimand comprises 24 unique connectivity–confound–domain contrasts, with favorable support at 100% for Pearson and Spearman and 0% for partial correlation and mutual information. This complete estimator partition, rather than the pooled 12-of-24 fraction, is the informative result: NM-H2 is measure-dependent, and the mechanism underlying the partition remains unresolved. (C) Marginal influence of each analytic choice on NM-H2. (D, E) Cocaine-use-disorder episode: (D) multiverse stability of systemic-segregation associations across 36 specifications with SDMA-GLS consensus; (E) single-specification versus multiverse SDMA-GLS maps for five network–outcome pairs. (F) Cross-cultural social cognition: culture-stratified ALE maps contrasting Euro-American trust networks with East Asian social-cognition networks.

Figure 5: Brain Researcher searches 116 HCP prediction pipelines and identifies a workflow that consistently exceeds a matched reference. A. Brain Researcher first evaluated 20 candidate pipelines for Cognition prediction, reaching a best discovery score of

r=.373

. Brain researcher then launched a 96-candidate expansion; 84 candidates returned scores and 12 ended in transport failure. Within the expanded episode, Brain Researcher adapted its proposals to the accumulating results: 27 candidates exceeded the initial search maximum, and the highest discovery score was

r=.487

, obtained with whole-band coherence and ridge regression. Following a selector audit, the researcher froze a related coherence-based workflow for matched evaluation. Across 10 repeated family-grouped

5\times 3

nested-cross-validation runs, this workflow achieved median

r=.332

, compared with

.235

for the matched reference (median

\Delta r=.098

; conditional one-sided

p=.006

), and was higher in all 10 runs. B. The same frozen selected workflow was then refit for each of four additional behavioural outcomes without target-specific retuning. It produced a higher median correlation for every outcome and exceeded the matched reference in 37 of 40 repeat-level comparisons, giving 47 of 50 directional wins across all five outcomes.

Figure 6: Brain Researcher turns an open question about TRIBE into successive tests with new sounds and collections. A. Brain Researcher began by asking how TRIBE changes natural-sound representations from early to late layers. It screened category contrasts by their change in held-out distinguishability (AUC), without choosing a target in advance. Tools–voice changed most, but the pattern varied across collections. Rather than simply following the top-ranked result, Brain Researcher identified speech–tools as a clearer lead: the categories moved closer in later layers while usually keeping the same representational direction. The contrast could also be retested with new, acoustically matched sounds from several collections. The researcher approved this direction and froze the prediction and analysis. B. Brain Researcher then evaluated three non-overlapping 48-item panels. All three showed a smaller speech–tools separation on average in later layers. In 11 of 12 collection-by-panel comparisons, the categories became less separated while retaining the prespecified direction. C. After the pattern recurred across all three panels, Brain Researcher extended the test to four previously unused sound collections. Three of four showed the same geometry, and the late-layer separation was again smaller on average (

\Delta S=-0.198

). In all geometry plots, horizontal position shows the late-minus-early change in normalized separation (

\Delta S

), and vertical position shows late directional alignment (

C

); the upper-left quadrant therefore marks smaller separation with retained direction. A uses fold-specific references, whereas B and C use the frozen speech–tools reference.

Appendix figures & tables47 assets

Supplementary material from the paper’s appendix.

Appendix

Supplementary Figure S1: Detailed Brain Researcher system architecture. The architecture separates the researcher-facing coding-agent harness, the BR-MCP control plane, and the domain execution substrate. The episode flow runs from question intake and typed intent resolution through BR-KG and registry lookup, resource checks, constraint compilation, commitment, execution, review, and memory writeback. The control-plane boxes indicate where typed MCP operations expose admissible actions, while the backend boxes indicate where Python, container, Neurodesk, Slurm/HPC, storage, and provenance services produce environment-observed artifacts. Dataset readiness, tool admissibility, execution success, and claim eligibility are returned by typed system state; the language model interprets, requests, and explains those states.

Paper label	Resolved model identifier	Coding-agent surface
Claude Opus 4.8	claude-opus-4-8	Claude Code ( claude -p )
Codex GPT-5.5	gpt-5.5	Codex CLI ( codex exec )
Gemini 3.1 Pro	google/gemini-3.1-pro-preview	OpenCode ( opencode run )
GLM-5.1	zai-coding-plan/glm-5.1	OpenCode ( opencode run )
DeepSeek-V4-Pro	deepseek/deepseek-v4-pro	OpenCode ( opencode run )
Kimi K2.5	opencode/kimi-k2.5	OpenCode ( opencode run )

Appendix

Table 1: Resolved model identifiers and coding-agent surfaces for the seven benchmarked agents.

Supplementary Figure S2: Human-agent authority boundary in Brain Researcher. The diagram separates researcher authority, language-model assistance, MCP-mediated system authority, and scientific verification across the episode lifecycle. Brain Researcher can search evidence, compare options, prepare recipes, observe execution, assemble review inputs, and record memory candidates. The researcher retains authority over question framing, admissible tradeoffs, commitment-gate approval, interpretation, authorship, and final claim language. System gates prevent the model from inventing registry identifiers, overriding resource blocks, accepting missing artifacts, or promoting claims without a review verdict.

Supplementary Figure S3: Condition-tagged evidence for scoped planning. The workflow begins with an underspecified scientific request and expands it into entities, methods, datasets, assumptions, and validity conditions that can be checked before execution. BR-KG links literature evidence to those conditions with provenance, while resource and tool resolvers determine which conditions are satisfied, missing, controversial, or incompatible in the current episode. Applicability checks convert the resulting condition set into admissible plans, required sensitivity analyses, caveats, or blockers. The purpose is to expose methodological scope before a run is committed and before a positive or negative result can anchor the narrative.

Supplementary Figure S4: BR-KG node-type composition. Node composition summarizes the object families present in the BR-KG snapshot used for retrieval, normalization, and planning support. The chart separates scientific entities such as publications, coordinates, statistical maps, tasks, contrasts, concepts, and brain regions from resource and workflow entities such as datasets, tools, methods, governance records, and prior reviewed claims. Large node families indicate coverage available for traversal and grounding, not evidentiary weight for any individual claim. Detailed counts, release metadata, and source-quality checks are reported in Appendix B.

Supplementary Figure S5: BR-KG relationship-type composition. Relationship composition summarizes the edge families that connect publications, coordinates, maps, tasks, contrasts, concepts, regions, datasets, tools, constraints, and provenance records in the BR-KG snapshot. High-count relationship classes define the main traversal surfaces used for evidence assembly, task-to-construct normalization, map lookup, dataset discovery, and tool compatibility checks. Edge frequencies document graph structure and retrieval affordances; support strength for a scientific claim requires source provenance and review context. Appendix B provides the expanded relationship inventory and release-readiness checks.

Supplementary Figure S6: BR-KG schema topology and provenance flow. The atlas view summarizes how major node classes are connected, which relationship-label pairs dominate the schema, and how source records feed derived graph objects. Dense schema paths identify the routes most often available for structured retrieval, such as publication-to-coordinate-to-region, task-to-construct-to-map, and dataset-to-tool-to-workflow traversals. Provenance flow indicates where source systems, loaders, and curation records enter the graph. The figure documents the substrate available for multi-hop planning and evidence lookup; inferential claims require episode-level review.

Supplementary Figure S7: Representative BR-KG evidence and workflow paths. Example paths illustrate the kinds of traversals available during an episode. Literature peaks can be linked through coordinates, brain regions, task contrasts, and cognitive constructs; saved maps and NeuroVault assets can be linked to provenance and reuse constraints; datasets can be linked to derivative readiness, phenotype availability, and compatible tool or workflow contracts. These paths show how BR-KG turns text-level mentions into typed objects that can be queried, checked, and passed to the registry or review layer. Scientific acceptance occurs later, after execution evidence and review are available.

Supplementary Figure S8: Systematic multiverse and robustness workflow. A candidate claim is expanded into an admissible specification space through evidence assembly, tool-card constraints, resource checks, and researcher commitment. Each branch is either executed, blocked, or rejected under the active constraint vocabulary, preserving both successful artifacts and non-executed alternatives in the run bundle. The resulting landscape is summarized by specification curves, robustness tables, branch-level caveats, and claim-family verdicts. The workflow turns analytic multiplicity into a reviewable robustness boundary with the specification space fixed before claim calibration.

Supplementary Figure S9: Cross-modal, multi-stage neuroimaging execution substrate. Brain Researcher routes heterogeneous neuroimaging workflows through a shared execution layer that coordinates modality-specific tools, analysis stages, provenance capture, and backend selection. The figure summarizes the execution surface covered by Supplementary Methods S7: supported modalities, stable tooling and pipeline families, execution backends, and the shared planner-review-provenance logic that links a selected workflow to observed artifacts.

State	Decision rule or adjudication criterion
Accepted	Holds across the episode-defined majority of admissible analytic specifications, with no single defensible choice reversing it, and passes every check required for the stated claim language.
Qualified	Supported only under explicit, named conditions: support is confined to a subset of specifications or holds only with stated caveats; when an identifiable support partition exists, it is retained alongside the state.
Revised	The target, measurement, or validation plan is changed in response to a failed gate, and the redesigned successor passes the same rules.
Blocked	A resource, design, power, or validity constraint prevents promotion from exploratory to confirmatory, regardless of the point estimate.
Rejected	Fails the applicable null, replication, or support rule.
Deferred	Passes an internal gate but does not yet satisfy eligibility for the next validation tier (for example, while awaiting external replication).

Appendix

Table 2: Claim states and their explicit adjudication criteria. Each evaluated claim in the collaborator and bounded-autonomous cases is assigned exactly one state by the review layer. For prospectively governed episodes, thresholds are committed before the confirmatory test; post-hoc audits instead apply documented criteria to completed artifacts. This is the reporting-level projection of the review verdicts in the table above.

Supplementary Figure S10: From auditable run bundles to cumulative field memory. Completed episodes first produce run bundles containing the committed plan, executed actions, logs, artifacts, validation outcomes, review notes, failures, and caveats. The review layer determines which candidate claims are accepted or explicitly qualified for memory writeback; rejected, blocked, or failed candidates remain recoverable in the run bundle without becoming accepted graph facts. Eligible claims become condition-tagged claim cards with provenance pointers and relation events such as support, contradiction, refinement, conditioning, or supersession. Curated memory cards may later be promoted to BR-KG under a separate governance policy.

Supplementary Figure S11: In-silico experiment design and bounded hypothesis iteration. The bounded workflow starts from a declared research context, evidence state, allowed action vocabulary, budget, validation ladder, expected artifacts, and stopping rule. Candidate in-silico experiments are proposed, materialized, run through lower-resource validation tiers, and either rejected, revised, deferred, or advanced to stronger tests. Reward or search scores are kept separate from scientific acceptance, which requires the predeclared nulls, artifact checks, and review rules to pass. The loop returns reviewable follow-up hypotheses or memory candidates; acceptance remains with verification.

Supplementary Figure S12: Out-of-distribution hypothesis proposal and triage. Candidate hypotheses are generated near sparse, conflicting, or under-connected regions of the current knowledge space. Triage checks feasibility, novelty, evidentiary support, resource availability, and validation requirements before a candidate can enter an episode. Hypothesis generation can propose a direction, but promotion still requires evidence assembly, execution, review, and memory eligibility.

Supplementary Figure S13: Result-to-hypothesis loop closure. Completed outputs, residual uncertainty, failed branches, and condition-dependent findings can seed successor analyses. Every successor must pass the same commitment, execution, review, and memory-writeback path as a new episode.

Label	Model claim (cited to a retrieved source)	Judge’s reason
yes (verified)	Head motion, respiration, arterial CO 2 , blood pressure, autoregulation, and vasomotion can affect BOLD signals and therefore confound resting-state interpretation.	The source explicitly lists each of these factors and how they alter the BOLD signal; the claim is fully supported.
partial (not credited)	Head-motion artifacts are the most frequently reported issue in QC, and outliers should be detected in image-homogeneity and co-registration tests before group analysis.	The source supports the first clause (motion is the most frequent QC issue) but says nothing about the second (the outlier-detection procedure); only a weaker version is grounded.
no_unrelated (not credited)	The spatial-normalization workflow should include visual verification that functional contours align with anatomy before writing deformation fields.	The cited source is about time-series QC (framewise displacement, WM/CSF signals); it does not mention spatial normalization or deformation fields at all.

Appendix

Table 3: Grounding labels, with a representative with-BR item for each. Each model claim is checked, blind to condition, against the source it cites. Only yes (the source fully supports the claim) is counted as verified; partial (supports only a narrower version) and no_unrelated (a real source on a different topic) are not. Claims use the models’ own wording, lightly shortened; judge rationales are condensed. Further not-verified labels ( cannot_judge , malformed , fabricated ) apply to references that cannot be resolved or parsed.

Benchmark	Checked (20%)	Confirmed	To review
Tool routing	12	12	0
NIK answer keys	15	15	0
Grounding judgments	245	234	11
Total	272	261	11

Appendix

Table 4: Human audit of the automated benchmark judges. A reproducible random 20% sample of the scored results across the reported benchmarks, adjudicated by hand against the automated verdicts. Discrepancies were confined to the grounding benchmark and were all one-step severity nuances that never reversed a supported/unsupported call, with the judges erring strict.

Item	Judge	Human	Issue
NIK-BP-H-003	yes	partial	Dead-salmon study and multiple-comparison correction absent from the source; support real but incomplete.
NIK-BP-H-001	no_unrelated	no	Source contradicts the claim, so a negative rather than unrelated.
NIK-BP-H-005	no_unrelated	partial	Source states the exact “multiplicity of methodologic variants” problem the claim describes.
NIK-IN-H-004	no_unrelated	partial	GSR-debate context bears on global-signal content.
NIK-ME-H-004	no_unrelated	partial	Covers subject motion and noise modeling; related but general.
NIK-NK-E-003	no_unrelated	partial	Measures FWER control with no true signal, i.e. false-positive territory.

Appendix

Table 5: The 11 flagged grounding items from the human audit. All fell in the grounding benchmark under the with-BR condition. Nine reflect the judge scoring an arguably partial item as fully unrelated (too strict); one (NIK-BP-H-003) scored an arguably partial item as a full yes (too generous); and one (NIK-BP-H-001) is a relabel within non-support ( no versus no_unrelated ). None moves directly between the opposite hard calls ( yes and no_unrelated ), and the net tendency is strict, so with-BR grounding is if anything under-credited.

Metric	Without BR	With BR	Mean $\Delta$ [95% CI]	$t(6)$	Wilcoxon $p$
Correct route/tool@1	0.233	0.936	$+0.702$ [0.576, 0.829]	13.6	0.016
Capability@1	0.498	0.945	$+0.447$ [0.282, 0.612]	6.6	0.016
Handoff score@1	0.474	0.761	$+0.287$ [0.201, 0.373]	8.2	0.016

Appendix

Table 6: Paired benchmark effect sizes with uncertainty and significance. Unit of analysis is the model (

n=7

); each entry is a mean across the seven model variants, and each model’s score is itself a mean over the 60 tool-calling tasks. The improvement is the mean within-model with-BR minus without-BR difference; the 95% confidence interval is

t

-based across the seven models. Significance is a two-sided paired

t

test [

t(6)

] and an exact two-sided Wilcoxon signed-rank test. All seven models improved on every metric.

Runner/requested route	Exact top-1	Protocol-compliant	Gold available	Run provenance
Claude Code / claude-opus-4-8	54/60 (90.0%)	59/60	59/60	v1 tools-only run
Codex / gpt-5.5	51/60 (85.0%)	60/60	59/60	v1 tools-only run
OpenCode / google/gemini-3.1-pro-preview	54/60 (90.0%)	59/60	59/60	v1 tools-only run
OpenCode / opencode-go/glm-5.1	52/60 (86.7%)	60/60	59/60	v2 tools-only single arm
OpenCode / opencode-go/deepseek-v4-pro	51/60 (85.0%)	58/60	59/60	v2 tools-only single arm
OpenCode / opencode/kimi-k2.5	51/60 (85.0%)	60/60	59/60	v1 tools-only single arm

Appendix

Table 7: Seven-model routing ablation without direct KG calls. All seven rows use the same no-direct-KG exact-label top-1 endpoint; run provenance is shown separately.

Case	Mode	Driving model	BR’s role	Code author	Commitment / audit provenance
NeuroMark schizophrenia (Li & Calhoun)	agent-executed	Cursor / Claude Opus 4.6 (analysis); Codex (external review)	Recorded and reviewed only; analysis ran in an external workspace	X. Li (agent-assisted)	No sealed card; post-hoc audit ( pre_run_commitment_card_found: false )
Cocaine-use-disorder (Ricard & Poldrack)	agent-executed	Claude Code (Claude Sonnet 4.6)	Commitment gate, MCP-mediated cluster execution, and review; bounded null	Agent	Not sealed (collaborator protocol); run bundle + review verdict
Cross-cultural social cognition (Wang)	agent-executed	Claude Code	Reviewed within the episode; gate-blocked as underpowered	Agent	Not sealed; gate-blocked (run bundle + review verdict)
HCP workflow search and transfer	agent-executed	Claude Code / Codex	Bounded search accounting, frozen successor analyses, run-bundle review, and claim-boundary enforcement	Agent	Prospective episode bundles plus frozen successor contracts; retrospective same-cohort inference
TRIBE speech–tools geometry	agent-executed	Claude Code / Codex	Open contrast discovery, researcher-gated successor choice, frozen panel evaluation, and reward-blind closeout	Agent	Discovery record plus frozen recurring-source and new-source successor contracts
Neurosynth working-memory (public reference)	engine-executed	None (deterministic engine)	Executed the sealed episode end to end	BR generator	Sealed pre-run commitment card (hash 4871ea43 …)

Appendix

Table 8: Per-case attribution: execution mode, driving model, and the human/system responsibility split. Every reported analysis case ran in agent-executed mode; the engine-executed full-bundle protocol is demonstrated on the public Neurosynth example (final row). “BR” denotes Brain Researcher.

Outcome	Frozen selected workflow $r$	Matched $r$	Frozen selected workflow $R^{2}$	Matched $R^{2}$	Median $\Delta r$	$r$ wins
Cognition	.332	.235	.107	.009	.098	10/10
Tobacco Use	.242	.126	.057	$-.029$	.100	10/10
Personality–Emotion	.075	.006	$-.019$	$-.058$	.068	9/10
Illicit Drug Use	.122	.008	$-.003$	$-.044$	.117	10/10
Mental Health	.001	$-.062$	$-.047$	$-.046$	.069	8/10

Appendix

Table 9: HCP frozen selected workflow results across five behavioural outcomes. Values are medians over 10 overlapping repeated splits. Matched denotes the locally matched Liu-style nested procedure; wins count repeats in which the frozen selected workflow had the larger pooled out-of-fold correlation. These repeats measure same-cohort stability, not independent replication.

Rank	Category pair	Early AUC	Late AUC	$\Delta$ AUC
1	tools–voice	.625	.250	$-.375$
2	music–speech	.917	.563	$-.354$
3	speech–tools	.979	.646	$-.333$
4	animal–speech	.958	.792	$-.167$
5	speech–voice	.917	.750	$-.167$
6	animal–music	.271	.417	$+.146$

Appendix

Table 10: Complete TRIBE six-category discovery ranking. Negative

\Delta\mathrm{AUC}

denotes weaker category decoding in later than early layers. Values are descriptive results from the same ordered 48-item technical-rerun panel.

Panel	Aggregate	AudioSet Strong	BBC	FreeSound	SoundBible	Outcome
R3	$-.6937$	$-.7664$	$-.8625$	$-.7177$	$-.4284$	bounded support
R4	$-.4467$	$-.5433$	$-.3985$	$-.2749$	$-.5701$	bounded support
R5	$-.5471$	$+.0284$	$-.4885$	$-1.0059$	$-.7225$	bounded support

Appendix

Table 11: TRIBE recurring-source successor panels. Each cell is late-minus-early normalized speech–tools separation (

\Delta S

); negative values indicate contraction in later layers.

Collection	$\Delta S$	$\Delta$ AUC
DCASE Office Live	$-.5883$	$-.0741$
SINGA:PURA	$-.2073$	$-.0278$
SONYC-UST	$+.0856$	$+.0926$
STARSS23	$-.0820$	$-.0463$
Aggregate	$-.1980$	—

Appendix

Table 12: TRIBE frozen new-source endpoint. Negative values are in the predicted direction.

Token component	Tokens	Approx. cost
Output (generation)	0.89M	$13
Cache-creation (cache write)	19.9M	$75
Cache-read	180.2M	$54
Uncached input	0.03M	$<$ $1
Total (single episode)	$\sim$ 201M (raw sum)	$\sim$ $140

Appendix

Table 13: Token usage and approximate API-list-equivalent cost for one collaborator episode (cocaine-use-disorder resting-state case, Claude Sonnet 4.6). Components are reported separately; the raw sum is dominated by cache reads (billed at roughly one-tenth the input rate) and is not the headline quantity. Costs are derived from the per-component token counts at standard list prices (input

3, output

15, cache write

3.75, cache read

0.30 per 1M tokens) and are approximate; actual billing depends on the user’s plan and tier.

Episode (data, $N$ )	Acquisition / preprocessing origin	Features & estimator	Statistical model, inference & correction	Robustness / review criteria	Claim state(s)
NeuroMark / schizophrenia FNC (FBIRN; $N=363$ , 181 control / 182 patient)	NeuroMark 2.2 template-ICA FNC derivatives (framework/collaborator-provided) ( Du et al., 2020 )	5,460 FNC edges, two domain granularities; four connectivity estimators (Pearson, Spearman, partial correlation, mutual information); five dimensionality-reduction methods; four classifiers	Patient-vs-control classification (AUC) and group-difference contrasts; one-sided permutation test with sign-aware acceptance ( $p<0.05$ and $\Delta\,\mathrm{mean}\|d\|>0$ ); Wilcoxon test for loading mass	480-specification multiverse ( $4\times 3\times 5\times 4\times 2$ ); 384 latent-evaluable; 24 unique sign-aware H2 contrasts	NM-H1 qualified; NM-H2 qualified; NM-H3 qualified
Cocaine-use-disorder connectivity (SUDMEX CONN, OpenNeuro ds003346 v1.1.3; $N=138$ )	fMRIPrep derivatives, collaborator-supplied ( Angeles-Valdez et al., 2022 )	Network segregation; atlas varied across specifications; same-data meta-analysis (SDMA-GLS) ( Lefort-Besnard et al., 2025 )	Connectivity–behavior associations; FDR correction (confirmatory all $Z<1.24$ , $q>0.58$ ; exploratory max $Z_{\mathrm{GLS}}=2.18$ , 0/70)	36-specification multiverse (atlas $\times$ framewise-displacement threshold $\times$ confound); 70-combination exploratory screen	5 confirmatory rejected; exploratory screen blocked
Cross-cultural social cognition (coordinate literature; 21 studies, 85 peaks)	Published MNI peak coordinates (no raw imaging data)	Cell-wise activation-likelihood estimation (ALE), NiMARE ( Salo et al., 2023 )	ALE with per-cell $k=6$ – $8$ (below recommended $k\geq 17$ ( Eickhoff et al., 2016 ) ); centroid shift reported as descriptive only	Paradigm-composition imbalance check across four culture-by-relationship cells	mPFC-topology interpretation blocked as exploratory
HCP-YA behavioural prediction (search cohort $N=326$ ; matched comparison $N=244$ , 243 families; internal holdout $N=81$ )	HCP S1200 minimally-preprocessed connectomes and locally reconstructed Liu-component counterparts ( Van Essen et al., 2013 ; Liu et al., 2025 )	116 allocated candidate evaluations; frozen selected workflow using a coherence-magnitude representation with cosine-kernel ridge regression; locally matched Liu-style nested comparator	Ten repeated family-grouped $5\times 3$ nested-CV splits with pooled out-of-fold metrics; family-cluster pointwise bootstrap and conditional one-sided test; weak-FWER transfer analysis	104/116 parent-run scores plus 12 recorded transport failures and separate recovery; frozen selected workflow refits across outcomes; retrospective same-cohort, search-unadjusted sensitivity	The frozen selected workflow exceeded the matched comparator in 10/10 Cognition splits; 47/50 directional wins across five outcomes, but no transfer cell passed weak-FWER; separate holdout did not confirm; no external acceptance
TRIBE speech–tools representation geometry (TRIBE v2; discovery and successor panels of natural sounds)	No subject-level BOLD; model-representation analysis over six sound categories, four recurring collections, and four previously unused collections ( d’Ascoli et al., 2026 )	Early-versus-late encoder decoding and frozen geometric decomposition of normalized separation, orientation, and signed projection	Descriptive 15-pair discovery; three non-overlapping recurring-source panels; 99,999 balanced-label permutations and Holm correction at the frozen four-new-source endpoint	Researcher-authorized post-hoc choice of speech–tools; frozen estimand; item non-overlap; score-blind acoustic balancing; collection-specific and leave-one-out diagnostics	Recurring-source bounded support in 3/3 panels (11/12 cells), followed by unsupported new-source H1 (Holm $p=.396$ ); terminal outcome inconclusive or conflicting; no scientific acceptance

Appendix

Table 14: Analysis-level methods provenance across the five reported episodes (COBIDAS-style). Brain Researcher performs no raw acquisition or image preprocessing; the acquisition/preprocessing column records the origin of the derivatives or coordinates each episode consumed. All remaining columns describe analysis-level choices that Brain Researcher recorded and reviewed, with prospective commitment gates applied where applicable ( S7.4 ); the compute itself was executed by the agent, except in the engine-executed Neurosynth reference. Full per-case records are in Appendices F and L.

Input	Source appendix	Status
Selected plan	Appendix D / E	present
Active constraints	Appendix E	present
Run bundle	Appendix F	present
Artifact manifest	Appendix F	present
Logs/errors/warnings	Appendix F	present
Scorecard and QC outputs	Appendix F	present

Appendix

Table 34

Validity layer	Review question	Example checks
Statistical validity	Is the inference valid?	exchangeability; multiple comparisons; permutation/null
Measurement validity	Are data and QC adequate?	motion; registration; QC; reliability
Construct validity	Does the measure support the construct?	task/construct mapping; reverse inference risk
Generalization validity	Does the result generalize?	leakage; grouped CV; held-out/external support
Claim validity	Is language calibrated?	no causal/biomarker/external claim without evidence

Appendix

Table 35

Lifecycle status	Meaning	How it is reported
Implemented	The rule is operational in the review system for the required metadata fields.	May support an enforced BLOCK or WARN verdict.
Deterministic candidate	The rule can likely be implemented from structured metadata, manifests, or pipeline logs.	Reported as a candidate check unless the episode records enforcement.
Schema-dependent candidate	The rule requires additional review-context fields before deterministic enforcement.	Reported as a missing-field or protocol requirement.
NLP/LLM candidate	The rule requires text interpretation, claim extraction, or semantic comparison.	Reported as assisted review or calibration unless independently verified.
Calibration-only	The rule is used for examples, annotator training, or benchmark calibration.	Does not by itself produce an enforced verdict.

Appendix

Table 36

Rule ID	Severity	Validity layer / reason tag	Default action
feature_selection_outside_cv	BLOCK	generalization / leakage	revise or block
uncorrected_whole_brain_inference	BLOCK	statistical / multiple comparisons	revise or block
same_data_roi_definition	BLOCK	circularity	block
gsr_without_sensitivity	WARN	measurement / controversial choice	sensitivity or caveat
small_sample_biomarker_claim	WARN	claim / claim inflation	downgrade claim
missing_random_seed	WARN	reproducibility / reporting gap	report or fix

Appendix

Table 37

Finding type	Examples	Consequence
BLOCK	leakage outside CV; invalid null; circular ROI; uncorrected whole-brain inference	Cannot accept until resolved.
WARN	external validation missing; GSR unresolved; small sample; missing seed; partial QC	May accept with caveats, sensitivity, or downgraded claim.

Appendix

Table 38

Failure mode	Evidence inspected	Default action	Claim-language effect
Leakage outside cross-validation	Pipeline DAG, fit scope for feature selection, scaling, PCA, harmonization, or confound regression, and fold manifest.	BLOCK until the fitting step is moved inside the training fold.	No generalization, prediction, or biomarker claim.
Circular ROI or feature definition	ROI provenance, localizer source, contrast identity, dataset identity, and whether the same data also test the effect.	BLOCK unless an independent localizer, atlas, or non-circular provenance is supplied.	No confirmatory effect claim.
Invalid whole-brain or map-level inference	Correction method, correction domain, primary threshold, permutation or spatial-null framework, and file space.	BLOCK for uncorrected whole-brain tests, spatial-domain mismatch, or missing spatial null.	Only descriptive or exploratory language remains.
Unresolved controversial preprocessing choice	Preprocessing record and sensitivity coverage for GSR, dynamic FC windows, graph thresholds, HRF model, or harmonization.	WARN with required sensitivity analysis.	Claim is qualified to the tested preprocessing choices.
Missing external or held-out validation	Validation ladder, held-out split, external dataset availability, post-selection correction, and resource-gate status.	WARN or DEFER, depending on whether internal support is sufficient.	No external generalization or deployment claim.
Overstated interpretation	Candidate claim text, evidence type, behavioral covariates, out-of-sample evidence, ablations, and alternatives considered.	WARN with claim downgrade.	Prediction, causal, mechanistic, or reverse-inference language is removed unless directly supported.

Appendix

Table 39

Check	Required by	Claim effect
with/without GSR	controversial-choice rule	qualified if not performed or unstable
parcellation sensitivity	validation ladder	supports robustness level if stable
confound model sensitivity	review rule or soft constraint	caveat if attenuated
permutation/null check	hard statistical validity	accept/block depending on result
post-selection correction	replayed configurations	accept/block depending on result
external dataset validation	L4 ladder	no external claim if deferred

Appendix

Table 40

Artifact class	Required for review?	Review consequence if missing
Scorecard	yes	review not ready or revise
Predictions	yes for prediction	statistical review incomplete
Null-test output	yes if claimed	revise/block
Config manifest	yes	reproducibility warning or revise
QC report	yes or conditional	measurement WARN

Appendix

Table 41

Candidate claim	Overclaim blocked	Allowed language
Model predicts behavior	clinical biomarker	internally validated predictive association
FC is associated with score	causal mechanism	exploratory or condition-qualified association
Result robust to parcellation	external validity	robustness under tested specifications
Null-control passed	biological truth	leakage/null-controlled internal result

Appendix

Table 42

Verdict	Meaning	Next action
accept	Artifacts and claims satisfy active constraints.	Eligible for memory.
accept with qualifications	Result usable only with caveats.	Qualified memory claim.
revise	Fixable issue remains.	Return to planning or execution.
block	Hard validity failure.	Stop or redesign.
reject	Claim unsupported or contradicted.	No writeback.
escalate	Human expert needed.	Pause or route.

Appendix

Table 43

Cases	Scenarios covered	Typical label
C01–C06	Repeated-measures, mixed-design, longitudinal, and spatial-domain errors, plus valid paired-test and motor-task controls.	BLOCK for invalid design or spatial correction; allow for valid controls.
C07–C12	Extreme or unexpected effects, ordinary morphometry effects, motion-confounded small-sample FC, and expected task activation.	WARN for prior conflict or confounding; allow for expected, well-scoped effects.
C13–C24	Whole-brain correction, cluster-threshold, localization, double-dipping, ROI multiplicity, analytic flexibility, pseudoreplication, fixed-effects population claims, exchangeability, and spatial-null examples.	BLOCK for hard inference errors; WARN for soft multiplicity or reporting issues.
C25–C36	Motion reporting, motion imbalance, global signal regression, dynamic FC, graph thresholding, task FC/PPI, EPI distortion, MRIQC, small sample, missing external validation, and multiband QC.	WARN except for high-confidence motion confounds that block inference.
C37–C41	Reverse inference, stimulus generalization, missing behavioral covariates, ICV/TIV omission, and multi-site site confounding.	WARN with claim narrowing or added covariates/sensitivity.
C42–C55	Harmonization, feature selection, standardization, test-set selection, split grouping, permutation, fold-wise error, prediction language, temporal splits, post-hoc layer selection, RSA multiplicity and hierarchy, and missing ICC.	BLOCK for leakage and test-set use; WARN for missing reliability, uncertainty, or permutation support.

Appendix

Table 44

Field	Recommended entry
Episode or campaign ID	ep_hcp_predict_001 / campaign_autonomous_001
Mode	interactive / benchmark / collaborator / bounded autonomous
Gate authority	human researcher / evaluation harness / critic / supervisor
Instruction or task contract	instruction.md / bounded_episode_contract.yaml
Allowed resources	dataset/resource IDs and approved roots
Action vocabulary	search, plan, run, inspect, review, branch, stop

Appendix

Table 45

Contract field	What it records
Target question	The scientific objective for the bounded episode.
Admissible design space	Which analyses and branches are allowed.
Cycle budget	Maximum number of supervisor/critic cycles.
Expected artifacts	Files and records required for verification/review.
Validation ladder	Target and achieved L0-L5 level.
Review rules	BLOCK/WARN rules required for acceptance.

Appendix

Table 46

Record	Recommended fields
Branch-decision record	branch type; reason; proposed action; expected resolution; budget cost
Critic decision	continue / investigate anomaly / broaden / revise / escalate / terminate
Harbor verifier output	artifact manifest present; required statistic present; schema valid; command completed; reward or partial credit
Validation-ladder status	highest achieved level and unmet next gate
Termination status	completed / blocked / budget exhausted / escalated / terminated

Appendix

Table 47

Surface	Unit	What is evaluated
Tool-calling benchmark	60 neuroimaging task manifests; seven models per condition; 420 model-item trajectories per condition.	Whether a model maps a natural-language neuroimaging request to the correct analysis tool or executable route before execution.
Routing ablation without direct KG calls	Same 60-task manifest; seven model routes; 420 route–task episodes.	Exact-label top-1 route selection when direct KG-named calls are unavailable.
Evidence-citation benchmark	76-item open-ended NeuroimageKnowledge manifest; descriptive Results aggregate reported on a 50-question analysis set over all generated evidence-basis rows.	Whether model-generated neuroimaging claims cite evidence that can be located and judged supportive by three condition-blind LLM judges (Claude Opus 4.8, Codex GPT-5.5, and Gemini 3.1 Pro). All reference-bearing rows were sent to all three judges; a row entered the numerator when at least two judges voted verified, while every generated evidence-basis row remained in the denominator.
Collaborator cases	Three active scientific questions from collaborating researchers.	Whether Brain Researcher returns bounded claim records rather than unqualified findings.
Bounded self-evolving campaigns	Two predeclared episode contracts.	Whether fixed validation gates reject, defer, or revise claims during exploratory search.

Appendix

Table 48

Field	Entry
Conditions	Without-BR baseline; with-BR condition.
Model variants	Claude Code / Claude Opus 4.8; Codex GPT-5.5; Gemini 3.1 Pro; GLM-5.1; DeepSeek-V4-Pro; Kimi K2.5; Qwen3.6-Plus.
Tool-calling items	Natural-language neuroimaging requests requiring an analysis tool call or executable route with the required inputs available before execution.
Evidence-citation item families	Open-ended neuroimaging review questions covering preprocessing, statistical inference, functional connectivity, multivariate decoding, meta-analysis, and reproducibility.
Isolation policy	Benchmark runs use fixed task manifests, fixed scoring contracts, and isolated memory partitions.

Appendix

Table 49

Metric	Numerator	Denominator
Capability@k	Fraction of required task capabilities covered by the selected call or route after the first k non-neutral actions.	Model-item trajectories in the tool-calling benchmark; values are averaged as coverage scores.
Correct route/tool@k	Tasks that reach exact task success under the capability scorer after the first k non-neutral actions.	All tool-calling model-item trajectories.
Restricted-interface exact-label top-1	Requested routes with an exact top-1 label match under route_search $\rightarrow$ tool_search ; protocol violations count as errors.	All 60 task-route episodes; gold candidate availability is reported separately as an oracle-coverage diagnostic.
Handoff score@k	Selected call or route contains enough information for a receiving agent to continue the analysis after the first k non-neutral actions.	Tool-calling trajectories with parsed actions.
verified_groundedness_rate	Evidence items whose cited evidence is both locatable and judged supportive.	All evidence-basis rows generated for each question, followed by an equal-weight mean across the 50 questions.
verified_among_claimed_grounded	Model-claimed grounded items whose cited evidence is both locatable and judged supportive.	Model-claimed grounded items.

Appendix

Table 50

Metric	Without BR	With BR	Direction
Capability@1 (all tool-calling trajectories)	0.498	0.945	Higher
Correct route/tool@1	0.233	0.936	Higher
Handoff score@1	0.474	0.761	Higher
verified_groundedness_rate (three-judge majority; 50-question mean)	0.046	0.220	Higher
verified_groundedness_rate (Gemini 3.1 Pro recovered judge; 50-question mean)	0.0435	0.2097	Higher
verified_groundedness_rate (Codex GPT-5.5 judge; 50-question mean)	0.0389	0.3088	Higher

Appendix

Table 51

Case	Evaluation object	Reported outcome boundary
NeuroMark schizophrenia FNC	FBIRN cohort, N = 363; 480-specification multiverse over connectivity, confound, dimensionality-reduction, classifier, and domain granularity.	NM-H1, NM-H2, and NM-H3 all recorded as qualified (NM-H2 favorable in 12 of 24 contrasts after sign-aware rescoring); support depended on analytic regime.
Cocaine-use-disorder connectivity	SUDMEX CONN / OpenNeuro ds003346, N = 138; 36-specification multiverse over atlases, framewise-displacement thresholds, and confound strategies.	Five prespecified network-systemic-segregation associations rejected under SDMA-GLS (all Z $<$ 1.24, FDR q $>$ 0.58); exploratory 70-combination screen yielded no FDR-surviving effects (max $Z_{\mathrm{GLS}}=2.18$ ; 0/70 surviving).
Cross-cultural social cognition	21 published studies, 85 MNI coordinates, four culture-by-relationship ALE cells with k = 6–8 cell-level entries.	Mechanistic mPFC-topology interpretation blocked as exploratory because cell counts were below recommended ALE stability thresholds and paradigm composition was imbalanced.

Appendix

Table 52

Campaign	Predeclared controls	Claim-status outcome
HCP-YA workflow search and transfer	Two staged search bundles with all 116 slots retained in the denominator (104 scored in parent runs; 12 transport failures); frozen selected workflow designated before a locally matched comparison; repeated family-grouped nested CV; frozen selected workflow refits across four additional outcomes; separate internal holdout.	The frozen selected workflow exceeded the matched comparator in 10/10 Cognition splits and in 37/40 additional-outcome splits, but the result remained retrospective and same-cohort, no transfer cell survived weak-FWER correction, and the separate holdout did not confirm the comparator claim. No external scientific acceptance.
TRIBE speech–tools geometry	Complete 15-pair six-category discovery; researcher-authorized post-hoc contrast choice; frozen geometry estimand; three non-overlapping recurring-source panels; score-blind acoustic balancing and Holm-corrected permutation inference on four previously unused collections.	Three recurring-source panels met bounded support (11/12 collection cells in the predicted direction), but the frozen four-new-source H1 was not supported after correction (Holm $p=.396$ ). Terminal outcome inconclusive or conflicting; no scientific acceptance.

Appendix

Table 53

Data Analysis Agents

Data analysis agents use language models and tools to explore datasets, run analyses, and produce answers or reports grounded in evidence.

See all 106 papers

BrainPilot: Automating Brain Discovery with Agentic Research
Jul 16, 2026Haoxuan Li, Tianci Gao, Jianhe Li +13AI for Science LLM Agent Orchestration
College of AI, Tsinghua University · 2Shanghai Qizhi Institute · 3Business School, Renmin University of China +8
DISCERN: Can AI Agents Work Like Scientists and Guide Discovery?
Sep 27, 2026Nan Huang, Mario Tapia-Pacheco, Kun Zhou +4AI Agent Reliability Scientific Hypothesis Generation
University of California San Diego · Independent Researcher
The Agentic Garden of Forking Paths
Jul 1, 2026Jiacheng Miao, Jonathan K Pritchard, James ZouAI Agents for Scientific Discovery Scientific Reproducibility
Department of Biomedical Data Science, Stanford University · Department of Genetics, Stanford University · Department of Biology, Stanford University +2

Jul 16, 2026J/K move · Enter open · S save

BrainPilot: Automating Brain Discovery with Agentic Research

Haoxuan Li, Tianci Gao, Jianhe Li, Yang Fan+12

College of AI, Tsinghua University · 2Shanghai Qizhi Institute · 3Business School, Renmin University of China+8

AI for Science LLM Agent Orchestration Multi-Agent Collaboration AI Agent Benchmarks Autonomous Research Agents

Open paper AskPDF arXiv

Understanding the brain increasingly depends on integrating evidence across scales, modalities, and disciplines. Addressing a single research question therefore requires a coordinated sequence of operations, from surveying prior work to executing analyses and interpreting results in light of domain knowledge. AI agents promise to accelerate this process, but current agents lack domain expertise in brain science, may fabricate claims, drift during multi-step reasoning, and offer few defined points for expert intervention. These failures are especially costly in brain science, where conclusions feed into downstream scientific claims and depend on laboratory-specific expertise and careful human judgment. We present BrainPilot a fully open-source multi-agent system that accelerates brain science research with traceable logs and agent-verified results. A principal investigator (PI) agent coordinates specialist agents grounded in curated domain knowledge: a unified brain science knowledge base containing 7{,}233 indexed items and a skill library of 72 reusable methodology units across seven research domains. Every major step is recorded in the Graph of Trace, an auditable record that links subgoals, tool use, evidence, and claims and allows researchers to follow and inspect the workflow. An Auditor agent further integrates fabrication checking into the workflow. For evaluation, we run three brain science tasks from Agents' Last Exam, introduce our own benchmark, BrainPilotBench-v0, and present additional end-to-end case studies. Across these evaluations, BrainPilot with an open-source backbone model attains performance comparable to state-of-the-art agent framework with less costs.

Abstract

Data Analysis Agents

Explore similar work

BrainPilot: Automating Brain Discovery with Agentic Research

DISCERN: Can AI Agents Work Like Scientists and Guide Discovery?

The Agentic Garden of Forking Paths