BrainTRACE: Tracing Longitudinal, Multimodal, and Volumetric Evidence in Brain MRI Clinical Reasoning

Qizhen Lan; Mengchen Fan; Hang Zhang; Jingwei Duan; Moule Lin; Jialin Chen; Baocheng Geng; Xiaoqian Jiang

cs.CVOct 5, 2026

BrainTRACE: Tracing Longitudinal, Multimodal, and Volumetric Evidence in Brain MRI Clinical Reasoning

Authors: Qizhen Lan, Mengchen Fan, Hang Zhang, Jingwei Duan, Moule Lin, Jialin Chen, Baocheng Geng, Xiaoqian Jiang

Organizations: University of Texas Health Science Center at Houston · University of Alabama at Birmingham · University of Pittsburgh · The University of Texas MD Anderson Cancer Center · University of Dublin, Trinity College · Yale University

Abstract

Brain MRI interpretation is a longitudinal clinical reasoning problem: radiologists compare serial studies, integrate information across MRI sequences, localize findings within volumetric anatomy, and translate this evidence into report-grounded assessments. Existing medical VQA and 3D imaging benchmarks capture important parts of this workflow, but often evaluate brain MRI through isolated images, static volumes, or ungrounded report-style answers, thereby obscuring failures in the evidence chain that support clinical validity. We introduce BrainTRACE, a report-grounded benchmark for evaluating whether vision-language models can trace the evidence structure required for longitudinal brain MRI interpretation. BrainTRACE contains 7,273 scored VQA instances derived from 1,778 longitudinal patients, 7,299 MRI studies, and approximately 29k co-registered 3D MRI sequence volumes. The benchmark is organized by five levels of clinical reasoning, from acquisition recognition to case-level synthesis, and by evidence demands covering longitudinal comparison, report-grounded references, multi-sequence integration, and volumetric spatial evidence. BrainTRACE supports rendered inputs compatible with standard VLM interfaces, a 3D-evidence condition, and a decomposed case-reasoning track that audits six steps in a longitudinal evidence chain. Evaluation of 20 VLM configurations shows that current systems can identify isolated visual cues but rarely compose them into grounded longitudinal interpretations. We release the benchmark specification, evaluation lists, scoring implementation, scoring rubrics, and audit-record format to support reproducible progress in brain MRI VLM evaluation.

Figures & tables

Figure 1: Positioning of BrainTRACE relative to representative medical VLM/VQA benchmarks. BrainTRACE makes longitudinal brain MRI follow-up the central evaluation unit, requiring models to integrate serial, multi-sequence, volumetric, and report-grounded clinical evidence.

Panel of Table 1 — Table 1: Positioning representative medical VLM/VQA benchmarks relative to BrainTRACE along four evidence attributes. Longitudinal : longitudinal or temporal visual evidence. Multi-seq. : item-level multi-sequence evidence. Volumetric : native-volume or explicit volumetric evidence. Clinical : reasoning depth beyond visual recognition. $\checkmark$ = full, $\circ$ = partial, $\times$ = absent.

Figure 2: A running longitudinal case illustrates how BrainTRACE turns one patient record into localization, interval comparison, trajectory, and case synthesis tasks. The case reasoning track further decomposes the longitudinal interpretation into six report-grounded VQA steps.

Figure 3: BrainTRACE constructs report-grounded VQA instances through a human-in-the-loop pipeline that separates hidden clinical references from model-facing visual evidence. Reports and segmentation-derived measurements define answer targets, required facts, contradictions, and scoring rubrics. Models receive only the permitted visual inputs.

Panel of Table 2 — Table 2: Closed-form accuracy (%) is fragmented across clinical subclasses. Subclass columns pool broad-compatible and 3D-evidence items. Overall columns separate tracks.

Panel of Table 3 — Table 3: Open-ended acceptability and case-reasoning remain low even for frontier models; entries are Pass %.

Figure 4: Case-level reasoning fails by attrition. Local step successes rarely survive through all six decomposed VQA steps in the same longitudinal case.

Condition	Evidence retained			Accuracy (%)		Drop (pp)
	Long.	Multi-seq.	Image	Qwen	GPT	Qwen	GPT
Full evidence	$\checkmark$	$\checkmark$	$\checkmark$	50.0	50.0	–	–
Latest timepoint only	–	$\checkmark$	$\checkmark$	12.5	26.7	-37.5	-23.3
T2 sequence only	$\checkmark$	–	$\checkmark$	29.2	38.3	-20.8	-11.7
No image	–	–	–	15.0	9.2	-35.0	-40.8

Table 4: Evidence-channel ablations reduce accuracy on matched diagnostic items. All ablation rows use

n=120

with a constructed 50.0% full-evidence baseline, so drops are comparable across conditions, but conditional on the balanced sample.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Table 5 — Table 5: Existing resources cover important parts of medical VQA, temporal imaging, 3D imaging, and brain-MRI evaluation, but do not simultaneously instantiate the four evidence attributes targeted by BrainTRACE. Scale is reported in each source’s native unit for provenance and is not a normalized dataset-size comparison. $\checkmark$ = full, $\circ$ = partial, $\times$ = absent.

Figure 5: The retained MR-RATE subset provides the longitudinal follow-up structure required by BrainTRACE. Every retained patient has at least three MRI studies, allowing report-grounded questions to be tied to serial imaging rather than only single-visit findings.

Figure 6: Report-keyword audits indicate that the retained cohort is not a single-pathology collection. Groups are non-mutually exclusive report-text keyword audits used for cohort characterization, not official diagnostic prevalence labels.

Figure 7: The retained cohort is not restricted to a single demographic or acquisition slice. Patient-level demographics and follow-up depth show broad age coverage, balanced sex distribution, and substantial multi-year follow-up. Sequence availability is high for T1w, T2w, FLAIR, and SWI, while study-level acquisition metadata spans multiple scanner vendors and both 1.5T and 3.0T field strengths. Percentages are patient-level except for vendor and field strength, which are study-level; the figure describes retained-cohort breadth rather than population prevalence.

Figure 8: Scored VQA instances span five clinical-operation levels, three evidence conditions, and the evidence attributes used to audit longitudinal, multi-sequence, volumetric, and clinically interpretive reasoning.

Panel of Table 6 — Table 6: Task-family design matrix defines evaluated denominators and intended evidence demands. Broad, 3D, and Case give evaluated VQA-instance counts. L = longitudinal, M = multi-sequence, V ${}_{\!p}$ = pseudo-3D volumetric, V ${}_{\!n}$ = native-volume volumetric, and C = clinical reasoning beyond surface perception. $\checkmark$ = required, $\circ$ = partial or compatible, and $\times$ = not required; $\checkmark$ ∗ marks a central purpose.

Rank	Model	Pass	QS	Crit.	BLEU4	ROUGE-L	chrF++	BERTScore
1	GPT-5.4	19.8	2.55	22.2	0.065	0.198	0.281	0.043
2	Qwen3-VL-30B	14.4	2.27	36.6	0.039	0.193	0.273	0.149
3	Gemini 2.5 Pro	14.2	2.28	32.9	0.040	0.183	0.267	0.144
4	Qwen3-VL-8B	13.8	2.29	29.0	0.081	0.202	0.272	0.157
5	Qwen3-VL-4B	11.8	2.24	28.0	0.070	0.190	0.263	0.134
6	GPT-5-mini	10.6	2.07	40.3	0.056	0.176	0.264	0.102

Appendix

Table 7: The full broad-compatible open-ended leaderboard is ranked by factual-audit Pass %, while traditional text metrics are reported only as secondary comparability diagnostics. QS is Mean Quality Score and Crit. is Critical Error %; BLEU4, ROUGE-L, chrF++, and BERTScore are mean scores over the same broadQA open-ended items in their native scales. Bold marks the best value in each metric column and underlining marks the second-best value; lower is better for Crit., and higher is better for all other metric columns.

Figure 9: Three-way inter-annotator agreement analysis evaluates the factual-audit judge as a scoring instrument rather than treating it as an unchecked automatic metric. The audit covers the case-reasoning track ( $n=60$ decomposed VQA steps). (A,B) Pairwise agreement on the 1–5 quality scale and the critical-error binary. (C) Per-step $\kappa_{w}$ . (D) Per-model $\kappa_{w}$ .

Figure 10: Cross-family judge audit measures sensitivity to the judge family under the released factual scoring rule. Each cell contains the same stratified sample size ( $n=52$ ). Alternative Gemini and Claude judges preserve low absolute pass rates, while the agreement panel shows where the release judge aligns most strongly with cross-family alternatives.

Figure 11: Bootstrap confidence intervals summarize uncertainty in the headline empirical pattern without adding a second leaderboard narrative. Thick bands show 95% bootstrap confidence intervals and points show estimates. Panel D uses paired bootstrap intervals on matched ablation samples; all intervals remain below zero. Intervals use nonparametric percentile bootstrap over raw scored records or judge-step records, matching the protocol summarized in Appendix C.7 .

Ranks 1–10				Ranks 11–20
Rank	Model	Closed [95% CI]	Pass	Rank	Model	Closed [95% CI]	Pass
1	GPT-5.4	42.2 [39.6, 45.1]	15.6	11	Qwen2.5-VL-7B	33.2 [30.5, 36.0]	3.9
2	GPT-5	38.3 [35.4, 41.0]	11.7	12	Gemini 2.5 Pro	32.0 [29.3, 34.7]	13.0
3	GPT-5-mini	37.5 [34.7, 40.3]	10.4	13	M3D-LaMed	29.9 [27.0, 32.6]	0.0
4	Qwen3-VL-4B	35.4 [32.6, 37.9]	5.2	14	HuatuoGPT-Vision-7B	29.1 [26.4, 31.7]	0.0
5	Lingshu-7B	35.4 [32.6, 38.1]	3.9	15	MedGemma-4B-it	18.9 [16.7, 21.4]	3.9

Appendix

Table 8: Paper-reported 3D-evidence results provide compact numeric provenance for the main 3D columns and the representative CI visualization. Closed-form entries give exact accuracy (%) with bootstrap confidence intervals over the 1,123 closed-form items. Open-ended entries report Pass %; dashes indicate that released factual-audit records are unavailable for that configuration. Non-native-volume VLMs use 16-slice axial renderings; M3D-LaMed and RadFM use their native-volume interfaces, so this table is an interface comparison on shared clinical targets.

Figure 12: Evaluation choice changes the apparent model ranking. Each cell reports a model’s rank on one evaluation axis, with lower ranks better; the side bars summarize each model’s rank spread across axes. The figure is a sensitivity diagnostic, not a replacement for the per-axis results.

Figure 13: Chance-calibrated difficulty shows the drop from recognition to longitudinal reasoning after correcting for level-specific answer floors. CDS sets each level’s random baseline to 0 and perfect accuracy to 100; plum cells are below chance, neutral cells are near chance, and teal cells are above chance. Model names use the same family colors as Figure 12 .

Level	Task template	Failure mode	Error type	Images
L1	one_sentence_image_description	Reversed laterality (left $\to$ right)	wrong_laterality	1
L2	morphology_and_boundary	Missed lesion / false negative	missing_finding	1
L3	open_interval_change_at_site	Decrease reported as increase	wrong_direction	6
L4	open_trajectory_summary	Mislocated + wrong trajectory	wrong_anatomy, wrong_direction	3
L5	full_impression_3to5_sentences	Wrong diagnostic category	wrong_diagnosis	6

Appendix

Table 9: The case atlas covers distinct clinically material failure modes across the five task levels.

Figure 14: L1: Image and acquisition recognition ( L1.6 one_sentence_image_description ). The model matches the required facts in the success case but reverses left/right laterality in the failure case. This is a clinically material contradiction under the factual-audit rule even when the rest of the description is fluent.

Figure 15: L2: Anatomical localization and single-study finding recognition ( L2.6 morphology_and_boundary ). Both cases use a single axial T2W input and receive the same model response. The success case is compatible with the reference, whereas the failure case omits a 52 mm left peritrigonal lesion with ill-defined borders; the omission is a false negative rather than a wording difference.

Figure 16: L3: Disease-specific interpretation and interval change ( L3.6 open_interval_change_at_site ). With longitudinal multi-sequence inputs (2 timepoints $\times$ 3 sequences), the model identifies stable lesions in the success case but reverses the interval direction in the failure case, reporting increased size when the metastasis decreased from 13 mm to 8 mm.

Figure 17: L4: Longitudinal trajectory or quantitative change assessment ( L4.4 open_trajectory_summary ). Across three serial T2W studies, the model traces a decreasing trajectory in the success case. In the failure case, it mislocalizes the finding to the cerebellar hemisphere rather than the right occipital region and misreads an increasing trajectory as stable.

Figure 18: L5: Case-level and integrated clinical judgment ( L5.1 full_impression_3to5_sentences ). Both cases present 3 timepoints $\times$ 2 sequences (T2W + SWI). The model identifies stable status with no new findings in the success case, but assigns the wrong diagnostic category in the failure case by interpreting progressive bone metastases as resolving hemorrhage.

Explore similar work

May 19, 2026cs.CV

NeuroQA: A Large-Scale Image-Grounded Benchmark for 3D Brain MRI Understanding

We present NeuroQA, a large-scale benchmark for visual question answering in 3D brain magnetic resonance imaging (MRI), with 56,953 QA pairs from 12,977 subjects across 12 datasets. It spans ages 5-104 and five clinical domains: Alzheimer's, Parkinson's, tumors, white matter disease, and neurodevelopment. Unlike prior medical Visual Question Answering (VQA) efforts that operate on 2D slices or rely on narrow diagnostic labels, NeuroQA pairs every item with a full 3D volume. It evaluates 11 clinically grounded reasoning skills across Yes/No, multiple-choice, and open-ended formats. Of the 203 templates, 131 are image-grounded (answerable from a 3-plane viewer) and 72 are image-informed (ground truth from quantitative volumetry or clinical instruments). To remove text-only shortcuts, we apply answer-distribution refinement, reducing closed-format text-only accuracy from

>

80% to 44.6%; image necessity is assessed separately through an image-grounding protocol released with the benchmark. A 38-rule deterministic pipeline and two rounds of expert review verify every QA pair against FreeSurfer measurements, metadata, or radiology report fields, with zero same-subject contradictions across templates. We conduct a clinician evaluation in which two clinicians independently assess 100 frozen test items on a three-plane viewer. On closed-format (Yes/No + multiple-choice) test-public items, the best zero-shot vision-language model and a supervised 3D CNN baseline reach 47.5% and 43.7% accuracy respectively, both below the 49.4% text-only majority-template floor. NeuroQA adopts a two-tier release with public QA pairs for open-access datasets and reproducible generation scripts for datasets restricted by data use agreements (DUAs), plus subject-level splits, a held-out private test set, and an online leaderboard.

Mohammad H. Abbasi, Favour Nerrise, Shaurnav Ghosh +12

Stanford University · 1Stanford University, Stanford, CA, USA

Healthcare 3D Medical Imaging Medical VQA Medical Image Grounding 3D VQA Medical Image Benchmarks+1

Ask PDF

Apr 17, 2026cs.CV

Beyond a Single Frame: Multi-Frame Spatially Grounded Reasoning Across Volumetric MRI

Spatial reasoning and visual grounding are core capabilities for vision-language models (VLMs), yet most medical VLMs produce predictions without transparent reasoning or spatial evidence. Existing benchmarks also evaluate VLMs on isolated 2D images, overlooking the volumetric nature of clinical imaging, where findings can span multiple frames or appear on only a few slices. We introduce Spatially Grounded MRI Visual Question Answering (SGMRI-VQA), a 41,307-pair benchmark for multi-frame, spatially grounded reasoning on volumetric MRI. Built from expert radiologist annotations in the fastMRI+ dataset across brain and knee studies, each QA pair includes a clinician-aligned chain-of-thought trace with frame-indexed bounding box coordinates. Tasks are organized hierarchically across detection, localization, counting/classification, and captioning, requiring models to jointly reason about what is present, where it is, and across which frames it extends. We benchmark 10 VLMs and show that supervised fine-tuning of Qwen3-VL-8B with bounding box supervision consistently improves grounding performance over strong zero-shot baselines, indicating that targeted spatial supervision is an effective path toward grounded clinical reasoning.

Lama Moukheiber, Caleb M. Yeung, Haotian Xue +3

Georgia Institute of Technology · Harvard University · Georgetown University

3D Spatial Reasoning 3D Medical Imaging Medical VQA Visually Grounded Reasoning Medical Image Grounding Medical Image Benchmarks

Ask PDF

Aug 13, 2026cs.CV

How Good are Foundation Models in Longitudinal MRI Disease Progression Reasoning?

Magnetic Resonance Imaging (MRI) interpretation is fundamental to clinical decision-making, requiring radiologists to integrate multi-view anatomical planes across sequential timepoints while precisely localizing interval changes. However, existing vision-language benchmarks remain confined to single-timepoint, single-view interpretation, failing to capture the temporal-spatial reasoning essential to radiologic practice. We introduce the Time-Aware Multi-View MRI Benchmark, an evaluation framework unifying multi-view anatomical input, temporal reasoning across longitudinal scans, and structured localization guidance. The benchmark comprises 3,920 expert-verified question-answer pairs derived from 890 patients across over 3,200 longitudinal MRI timepoints, drawn from seven clinical cohorts covering glioblastoma, neurodegeneration, vestibular schwannoma, and brain metastases, in open-ended, multiple-choice, and binary formats, requiring models to identify anatomical regions of maximal change, characterize progression across sequences and views, and provide structured guidance specifying boundaries, imaging features, and confounders. Experiments across 16 vision-language models reveal moderate temporal alignment but systematic failure on change direction recognition and volumetric quantification, while multi-view inputs improve spatial localization yet degrade temporal reasoning in compact architectures. Our benchmark provides a systematic framework for evaluating progression tracking, interval change localization, and temporal ordering, which are essential for clinical deployment. Code, evaluation splits, and the dataset are available at: https://github.com/wafaAlghallabi/Time-Aware-MRI.

Wafa Al Ghallabi, Ritesh Thawkar, Sara Ghaboura +6

Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Department of Health Abu Dhabi, Abu Dhabi, UAE · Fatima College of Health Sciences, Abu Dhabi, UAE +1

Radiology VLMs Medical Image Analysis Longitudinal Medical Image Analysis Medical Image Localization Medical Image Benchmarks

Ask PDF