LoV3D: Grounding Cognitive Prognosis Reasoning in Longitudinal 3D Brain MRI via Regional Volume Assessments
Authors: Zhaoyang Jiang, Zhizhong Fu, David McAllister, Yunsoo Kim, Honghan Wu
Organizations: School of Health & Wellbeing, University of Glasgow, Glasgow, UK · School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China · Institute of Health Informatics, University College London, London, UK
Longitudinal brain MRI is essential for characterizing the progression of neurological diseases such as Alzheimer's disease assessment. However, current deep-learning tools fragment this process: classifiers reduce a scan to a label, volumetric pipelines produce uninterpreted measurements, and vision-language models (VLMs) may generate fluent but potentially hallucinated conclusions. We present LoV3D, a pipeline for training 3D vision-language models, which reads longitudinal T1-weighted brain MRI, produces a region-level anatomical assessment, conducts longitudinal comparison with the prior scan, and finally outputs a three-class diagnosis (Cognitively Normal, Mild Cognitive Impairment, or Dementia) along with a synthesized diagnostic summary. The stepped pipeline grounds the final diagnosis by enforcing label consistency, longitudinal coherence, and biological plausibility, thereby reducing the risks of hallucinations. The training process introduces a clinically-weighted Verifier that scores candidate outputs automatically against normative references derived from standardized volume metrics, driving Direct Preference Optimization without a single human annotation. On a subject-level held-out ADNI test set (479 scans, 258 subjects), LoV3D achieves 93.7% three-class diagnostic accuracy (+34.8% over the no-grounding baseline), 97.2% on two-class diagnosis accuracy (+4% over the SOTA) and 82.6% region-level anatomical classification accuracy (+33.1% over VLM baselines). Zero-shot transfer yields 95.4% on MIRIAD (100% Dementia recall) and 82.9% three-class accuracy on AIBL, confirming high generalizability across sites, scanners, and populations. Code is available at https://github.com/Anonymous-TEVC/LoV-3D.
Figures & tables
Figure 1: Model paradigms and the LoV3D training pipeline. Top : Published MRI classifiers produce only a two-class diagnostic label from MRI inputs. Middle and Bottom : multimodal VLM models that take a MRI and text prompt, and produce an output including (3-class) classification and a detailed LLM output. Specifically, Middle : 3D medical VLMs (RadFM, M3D-LaMed) and an ungrounded ablation (same pipeline, encoder warmed via diagnosis classification instead of regional regression). Bottom : LoV3D training introduces a three-stage learning paradigm to ground the VLM’s cognitive prognosis reasoning. Stage 0 warms the encoder via a regional volume regression task; Stages 1a–b learn structured clinical reasoning through projector alignment and LoRA SFT for a multi-task learning (selecting relevant brain regions; classifying region deterioration categories; the longitudinal progression classification); Stage 2 refines outputs via Verifier-guided DPO (dashed border marks optimized modules) for a more reasonable diagnostic summary generation. LoV3D produces both a classification result and structured LLM output (right) comprising imaging observations, chain-of-thought reasoning, verifiable facts, and a diagnostic summary. Fire/snowflake icons denote trainable/frozen modules.
Method
DX (%)
F 1
κ
Region (%)
AD/CN (%)
Classification baselines
Liu et al. [ 17 ] ‡
—
—
—
—
91.1
Lian et al. [ 15 ] ‡
—
—
—
—
90.3
Zhang et al. [ 28 ] ‡
—
—
—
—
93.2
ResNet-50 (ours) ∗
58.9
58.1
.461
—
87.8
3D medical VLM baselines
Table 1: Comparison on the ADNI test set (258 subjects). DX = three-way diagnostic accuracy, F 1 = macro-averaged F 1 , κ = Cohen’s weighted kappa, Region = mean anatomical accuracy across five AD-signature regions, AD/CN = binary accuracy on the CN and Dementia subset (MCI subjects excluded, MCI predictions mapped to CN). ∗ Same MONAI ResNet-50 backbone with a linear head, trained on the same split. † VLM baselines produce no valid JSON; metrics are from heuristic extraction. ‡ Published results on different ADNI subsets; shown for reference. LoV3D (no-grounding) uses the full LoV3D pipeline but replaces Stage 0 regional volume regression with three-class diagnosis classification. LoV3D is evaluated on all 479 test scans (258 subjects, multiple visits).
Diagnosis
Report Quality
Clinical Safety
Stage
Acc
F 1
κ
Region
BLEU-4
ROUGE-L
F.Abn ↓
F.Sev ↓
1a (projector)
89.1
88.0
.851
79.7
.431
.635
7.2
6.3
1b (+ LoRA)
93.3
92.5
.905
81.6
.354
.558
7.1
4.1
2 (+ DPO)
93.7
93.3
.911
82.6
.584
.763
5.0
2.2
Table 2: Stage-wise progression on the ADNI test set (479 scans). Each stage builds on the previous checkpoint. Region = mean accuracy across five AD-signature regions. F.Abn./F.Sev. = false abnormal/severe rate on normal anatomy ( ↓ = lower is better). Reasoning completeness is 100% and hallucination rate < 1% at all stages.
Dataset
Method
Classes
Subjects
Acc (%)
Bal. Acc (%)
MIRIAD
Yee et al. [ 27 ]
2
69
—
95.7
Lu et al. [ 19 ]
2
69
93.6
94.9
LoV3D (ours)
2
69
95.4
93.2
AIBL
Lteif et al. [ 18 ]
3
661
73.4
—
Batool et al. [ 2 ]
3
661
76.4
61.4
LoV3D (ours)
3
621
82.9
74.2
Table 3: Cross-site generalization. LoV3D is evaluated zero-shot (no fine-tuning) on two independent datasets. Prior work results are reproduced from published numbers on the same datasets used as pure external test sets. MIRIAD baselines were trained on ADNI; AIBL baselines were trained on NACC.
Deep learning has become an important tool for Alzheimer's disease (AD) classification from structural MRI. Many existing studies analyze individual 2D slices extracted from MRI volumes, while clinical neuroimaging practice typically relies on the full three dimensional structure of the brain. From this perspective, volumetric analysis may better capture spatial relationships among brain regions that are relevant to disease progression. Motivated by this idea, this work proposes a multimodal 3D convolutional neural network for AD classification using raw OASIS 1 MRI volumes. The model combines structural T1 information with gray matter, white matter, and cerebrospinal fluid probability maps obtained through FSL FAST segmentation in order to capture complementary neuroanatomical information. The proposed approach is evaluated on the clinically labelled OASIS 1 cohort using 5 fold subject level cross validation, achieving a mean accuracy of 72.34% plus or minus 4.66% and a ROC AUC of 0.7781 plus or minus 0.0365. GradCAM visualizations further indicate that the model focuses on anatomically meaningful regions, including the medial temporal lobe and ventricular areas that are known to be associated with Alzheimer's related structural changes. To better understand how data representation and evaluation strategies may influence reported performance, additional diagnostic experiments were conducted on a slice based version of the dataset under both slice level and subject level protocols. These observations help provide context for the volumetric results. Overall, the proposed multimodal 3D framework establishes a reproducible subject level benchmark and highlights the potential benefits of volumetric MRI analysis for Alzheimer's disease classification.
Md Sifat, Sania Akter, Akif Islam +5
Department of Computer Science and Engineering, University of Rajshahi, Rajshahi, Bangladesh · School of Computer Science and Engineering, University of Aizu, Aizuwakamatsu, Japan · Department of Computer Science and Engineering, Pabna University of Science and Technology, Pabna, Bangladesh
Current medical vision-language models (VLMs) process volumetric brain MRI using 2D slice-based approximations, fragmenting the spatial context required for accurate neuroradiological interpretation. We developed Brain3D, a staged vision-language framework for automated radiology report generation from 3D brain tumor MRI. Our approach inflates a pretrained 2D medical encoder into a native 3D architecture and progressively aligns it with a causal language model through three stages: contrastive grounding, supervised projector warmup, and LoRA-based linguistic specialization. Unlike generalist 3D medical VLMs, Brain3D is tailored to neuroradiology, where hemispheric laterality, tumor infiltration patterns, and anatomical localization are critical. Evaluated on 468 subjects (BraTS pathological cases plus healthy controls), our model achieves a Clinical Pathology F1 of 0.951 versus 0.413 for a strong 2D baseline while maintaining perfect specificity on healthy scans. The staged alignment proves essential: contrastive grounding establishes visual-textual correspondence, projector warmup stabilizes conditioning, and LoRA adaptation shifts output from verbose captions to structured clinical reports\footnote{Our code is publicly available for transparency and reproducibility
Mariano Barone, Francesco Di Serio, Giuseppe Riccio +4
University of Naples Federico II, Department of Electrical Engineering and Information Technology, Italy · Northwestern University, Dept. of Computer Science, McCormick School of Engineering and Applied Science, United States · Pegaso University, Department of Information Science and Technology, Italy
Automated diagnosis of 3D brain CT scans is essential for critical care, yet it remains challenging due to the heavy reliance on manual annotations and the limited semantic understanding of conventional models. While 2D foundation vision-language models (VLMs) have shown remarkable generalization, effectively transferring their representational power to 3D volumes remains an open problem. In this paper, we propose Brain-Adapter, a novel dual-stream multiple instance learning (MIL) framework that leverages pre-trained 2D biomedical VLMs and raw diagnostic reports for robust scan-level multi-label classification. Specifically, we introduce a Text-Conditioned Attention (TCA) mechanism, utilizing raw diagnostic sentences as semantic queries to dynamically align visual cues with specific disease concepts. Concurrently, a parallel visual MIL stream captures global scan characteristics, supervised by structured labels extracted via a Large Language Model (LLM). To ensure representation coherence, a consistency constraint enforces synergy between the two streams. During inference, an Uncertainty-Aware Refinement (UAR) module dynamically calibrates and fuses these dual-stream predictions to resolve ambiguous cases. Extensive experiments demonstrate that our method significantly outperforms state-of-the-art 3D models and standard MIL approaches. By eliminating the reliance on dense annotations, Brain-Adapter provides a highly scalable and clinically viable solution for 3D acute intracranial pathology analysis.
Zhenyu Yi, Zhiyun Song, Yusong Sun +6
School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, China · Department of Computing, Imperial College London, London, UK