cs.CVOct 5, 2026
SaveBrainTRACE: Tracing Longitudinal, Multimodal, and Volumetric Evidence in Brain MRI Clinical Reasoning
Organizations: University of Texas Health Science Center at Houston · University of Alabama at Birmingham · University of Pittsburgh · The University of Texas MD Anderson Cancer Center · University of Dublin, Trinity College · Yale University
Abstract
Brain MRI interpretation is a longitudinal clinical reasoning problem: radiologists compare serial studies, integrate information across MRI sequences, localize findings within volumetric anatomy, and translate this evidence into report-grounded assessments. Existing medical VQA and 3D imaging benchmarks capture important parts of this workflow, but often evaluate brain MRI through isolated images, static volumes, or ungrounded report-style answers, thereby obscuring failures in the evidence chain that support clinical validity. We introduce BrainTRACE, a report-grounded benchmark for evaluating whether vision-language models can trace the evidence structure required for longitudinal brain MRI interpretation. BrainTRACE contains 7,273 scored VQA instances derived from 1,778 longitudinal patients, 7,299 MRI studies, and approximately 29k co-registered 3D MRI sequence volumes. The benchmark is organized by five levels of clinical reasoning, from acquisition recognition to case-level synthesis, and by evidence demands covering longitudinal comparison, report-grounded references, multi-sequence integration, and volumetric spatial evidence. BrainTRACE supports rendered inputs compatible with standard VLM interfaces, a 3D-evidence condition, and a decomposed case-reasoning track that audits six steps in a longitudinal evidence chain. Evaluation of 20 VLM configurations shows that current systems can identify isolated visual cues but rarely compose them into grounded longitudinal interpretations. We release the benchmark specification, evaluation lists, scoring implementation, scoring rubrics, and audit-record format to support reproducible progress in brain MRI VLM evaluation.
Figures & tables
Figure 1: Positioning of BrainTRACE relative to representative medical VLM/VQA benchmarks. BrainTRACE makes longitudinal brain MRI follow-up the central evaluation unit, requiring models to integrate serial, multi-sequence, volumetric, and report-grounded clinical evidence.
Table 1: Positioning representative medical VLM/VQA benchmarks relative to BrainTRACE along four evidence attributes. Longitudinal : longitudinal or temporal visual evidence. Multi-seq. : item-level multi-sequence evidence. Volumetric : native-volume or explicit volumetric evidence. Clinical : reasoning depth beyond visual recognition. = full, = partial, = absent.
Figure 2: A running longitudinal case illustrates how BrainTRACE turns one patient record into localization, interval comparison, trajectory, and case synthesis tasks. The case reasoning track further decomposes the longitudinal interpretation into six report-grounded VQA steps.
Figure 3: BrainTRACE constructs report-grounded VQA instances through a human-in-the-loop pipeline that separates hidden clinical references from model-facing visual evidence. Reports and segmentation-derived measurements define answer targets, required facts, contradictions, and scoring rubrics. Models receive only the permitted visual inputs.
Table 2: Closed-form accuracy (%) is fragmented across clinical subclasses. Subclass columns pool broad-compatible and 3D-evidence items. Overall columns separate tracks.
Table 3: Open-ended acceptability and case-reasoning remain low even for frontier models; entries are Pass %.
Figure 4: Case-level reasoning fails by attrition. Local step successes rarely survive through all six decomposed VQA steps in the same longitudinal case.
| Condition | Evidence retained | Accuracy (%) | Drop (pp) | ||||
|---|---|---|---|---|---|---|---|
| Long. | Multi-seq. | Image | Qwen | GPT | Qwen | GPT | |
| Full evidence | 50.0 | 50.0 | – | – | |||
| Latest timepoint only | – | 12.5 | 26.7 | -37.5 | -23.3 | ||
| T2 sequence only | – | 29.2 | 38.3 | -20.8 | -11.7 | ||
| No image | – | – | – | 15.0 | 9.2 | -35.0 | -40.8 |
Table 4: Evidence-channel ablations reduce accuracy on matched diagnostic items. All ablation rows use with a constructed 50.0% full-evidence baseline, so drops are comparable across conditions, but conditional on the balanced sample.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Table 5: Existing resources cover important parts of medical VQA, temporal imaging, 3D imaging, and brain-MRI evaluation, but do not simultaneously instantiate the four evidence attributes targeted by BrainTRACE. Scale is reported in each source’s native unit for provenance and is not a normalized dataset-size comparison. = full, = partial, = absent.
Figure 5: The retained MR-RATE subset provides the longitudinal follow-up structure required by BrainTRACE. Every retained patient has at least three MRI studies, allowing report-grounded questions to be tied to serial imaging rather than only single-visit findings.
Figure 6: Report-keyword audits indicate that the retained cohort is not a single-pathology collection. Groups are non-mutually exclusive report-text keyword audits used for cohort characterization, not official diagnostic prevalence labels.
Figure 7: The retained cohort is not restricted to a single demographic or acquisition slice. Patient-level demographics and follow-up depth show broad age coverage, balanced sex distribution, and substantial multi-year follow-up. Sequence availability is high for T1w, T2w, FLAIR, and SWI, while study-level acquisition metadata spans multiple scanner vendors and both 1.5T and 3.0T field strengths. Percentages are patient-level except for vendor and field strength, which are study-level; the figure describes retained-cohort breadth rather than population prevalence.
Figure 8: Scored VQA instances span five clinical-operation levels, three evidence conditions, and the evidence attributes used to audit longitudinal, multi-sequence, volumetric, and clinically interpretive reasoning.
Table 6: Task-family design matrix defines evaluated denominators and intended evidence demands. Broad, 3D, and Case give evaluated VQA-instance counts. L = longitudinal, M = multi-sequence, V = pseudo-3D volumetric, V = native-volume volumetric, and C = clinical reasoning beyond surface perception. = required, = partial or compatible, and = not required; ∗ marks a central purpose.
| Rank | Model | Pass | QS | Crit. | BLEU4 | ROUGE-L | chrF++ | BERTScore |
|---|---|---|---|---|---|---|---|---|
| 1 | GPT-5.4 | 19.8 | 2.55 | 22.2 | 0.065 | 0.198 | 0.281 | 0.043 |
| 2 | Qwen3-VL-30B | 14.4 | 2.27 | 36.6 | 0.039 | 0.193 | 0.273 | 0.149 |
| 3 | Gemini 2.5 Pro | 14.2 | 2.28 | 32.9 | 0.040 | 0.183 | 0.267 | 0.144 |
| 4 | Qwen3-VL-8B | 13.8 | 2.29 | 29.0 | 0.081 | 0.202 | 0.272 | 0.157 |
| 5 | Qwen3-VL-4B | 11.8 | 2.24 | 28.0 | 0.070 | 0.190 | 0.263 | 0.134 |
| 6 | GPT-5-mini | 10.6 | 2.07 | 40.3 | 0.056 | 0.176 | 0.264 | 0.102 |
Table 7: The full broad-compatible open-ended leaderboard is ranked by factual-audit Pass %, while traditional text metrics are reported only as secondary comparability diagnostics. QS is Mean Quality Score and Crit. is Critical Error %; BLEU4, ROUGE-L, chrF++, and BERTScore are mean scores over the same broadQA open-ended items in their native scales. Bold marks the best value in each metric column and underlining marks the second-best value; lower is better for Crit., and higher is better for all other metric columns.
Figure 9: Three-way inter-annotator agreement analysis evaluates the factual-audit judge as a scoring instrument rather than treating it as an unchecked automatic metric. The audit covers the case-reasoning track ( decomposed VQA steps). (A,B) Pairwise agreement on the 1–5 quality scale and the critical-error binary. (C) Per-step . (D) Per-model .
Figure 10: Cross-family judge audit measures sensitivity to the judge family under the released factual scoring rule. Each cell contains the same stratified sample size ( ). Alternative Gemini and Claude judges preserve low absolute pass rates, while the agreement panel shows where the release judge aligns most strongly with cross-family alternatives.
Figure 11: Bootstrap confidence intervals summarize uncertainty in the headline empirical pattern without adding a second leaderboard narrative. Thick bands show 95% bootstrap confidence intervals and points show estimates. Panel D uses paired bootstrap intervals on matched ablation samples; all intervals remain below zero. Intervals use nonparametric percentile bootstrap over raw scored records or judge-step records, matching the protocol summarized in Appendix C.7 .
| Ranks 1–10 | Ranks 11–20 | ||||||
|---|---|---|---|---|---|---|---|
| Rank | Model | Closed [95% CI] | Pass | Rank | Model | Closed [95% CI] | Pass |
| 1 | GPT-5.4 | 42.2 [39.6, 45.1] | 15.6 | 11 | Qwen2.5-VL-7B | 33.2 [30.5, 36.0] | 3.9 |
| 2 | GPT-5 | 38.3 [35.4, 41.0] | 11.7 | 12 | Gemini 2.5 Pro | 32.0 [29.3, 34.7] | 13.0 |
| 3 | GPT-5-mini | 37.5 [34.7, 40.3] | 10.4 | 13 | M3D-LaMed | 29.9 [27.0, 32.6] | 0.0 |
| 4 | Qwen3-VL-4B | 35.4 [32.6, 37.9] | 5.2 | 14 | HuatuoGPT-Vision-7B | 29.1 [26.4, 31.7] | 0.0 |
| 5 | Lingshu-7B | 35.4 [32.6, 38.1] | 3.9 | 15 | MedGemma-4B-it | 18.9 [16.7, 21.4] | 3.9 |
Table 8: Paper-reported 3D-evidence results provide compact numeric provenance for the main 3D columns and the representative CI visualization. Closed-form entries give exact accuracy (%) with bootstrap confidence intervals over the 1,123 closed-form items. Open-ended entries report Pass %; dashes indicate that released factual-audit records are unavailable for that configuration. Non-native-volume VLMs use 16-slice axial renderings; M3D-LaMed and RadFM use their native-volume interfaces, so this table is an interface comparison on shared clinical targets.
Figure 12: Evaluation choice changes the apparent model ranking. Each cell reports a model’s rank on one evaluation axis, with lower ranks better; the side bars summarize each model’s rank spread across axes. The figure is a sensitivity diagnostic, not a replacement for the per-axis results.
Figure 13: Chance-calibrated difficulty shows the drop from recognition to longitudinal reasoning after correcting for level-specific answer floors. CDS sets each level’s random baseline to 0 and perfect accuracy to 100; plum cells are below chance, neutral cells are near chance, and teal cells are above chance. Model names use the same family colors as Figure 12 .
| Level | Task template | Failure mode | Error type | Images |
|---|---|---|---|---|
| L1 | one_sentence_image_description | Reversed laterality (left right) | wrong_laterality | 1 |
| L2 | morphology_and_boundary | Missed lesion / false negative | missing_finding | 1 |
| L3 | open_interval_change_at_site | Decrease reported as increase | wrong_direction | 6 |
| L4 | open_trajectory_summary | Mislocated + wrong trajectory | wrong_anatomy, wrong_direction | 3 |
| L5 | full_impression_3to5_sentences | Wrong diagnostic category | wrong_diagnosis | 6 |
Table 9: The case atlas covers distinct clinically material failure modes across the five task levels.
Figure 14: L1: Image and acquisition recognition ( L1.6 one_sentence_image_description ). The model matches the required facts in the success case but reverses left/right laterality in the failure case. This is a clinically material contradiction under the factual-audit rule even when the rest of the description is fluent.
Figure 15: L2: Anatomical localization and single-study finding recognition ( L2.6 morphology_and_boundary ). Both cases use a single axial T2W input and receive the same model response. The success case is compatible with the reference, whereas the failure case omits a 52 mm left peritrigonal lesion with ill-defined borders; the omission is a false negative rather than a wording difference.
Figure 16: L3: Disease-specific interpretation and interval change ( L3.6 open_interval_change_at_site ). With longitudinal multi-sequence inputs (2 timepoints 3 sequences), the model identifies stable lesions in the success case but reverses the interval direction in the failure case, reporting increased size when the metastasis decreased from 13 mm to 8 mm.
Figure 17: L4: Longitudinal trajectory or quantitative change assessment ( L4.4 open_trajectory_summary ). Across three serial T2W studies, the model traces a decreasing trajectory in the success case. In the failure case, it mislocalizes the finding to the cerebellar hemisphere rather than the right occipital region and misreads an increasing trajectory as stable.
Figure 18: L5: Case-level and integrated clinical judgment ( L5.1 full_impression_3to5_sentences ). Both cases present 3 timepoints 2 sequences (T2W + SWI). The model identifies stable status with no new findings in the success case, but assigns the wrong diagnostic category in the failure case by interpreting progressive bone metastases as resolving hemorrhage.