Latent Representation Learning from 3D Brain MRI for Latent Analysis and Prediction in Multiple Sclerosis

Trinh Ngoc Huynh; Nguyen Duc Kien; Nguyen Hai Anh; Dinh Tran Hiep; Manuela Vaneckova; Tomas Uher; Jeroen Van Schependom; Stijn Denissen

eess.IVSep 28, 2025

Latent Representation Learning from 3D Brain MRI for Latent Analysis and Prediction in Multiple Sclerosis

Authors: Trinh Ngoc Huynh, Nguyen Duc Kien, Nguyen Hai Anh, Dinh Tran Hiep, Manuela Vaneckova, Tomas Uher, Jeroen Van Schependom, Stijn Denissen,

Organizations: VNU University of Engineering and Technology, Hanoi, Vietnam. · Bach Mai Hospital, Hanoi, Vietnam. · Department of Radiology, First Faculty of Medicine, Charles University, General University Hospital, Prague, Czech Republic. · Department of Neurology and Center of Clinical Neuroscience, First Faculty of Medicine, Charles University, General University Hospital, Prague, Czech Republic. ·

Abstract

Neurological diseases can cause significant cognitive impairment, yet linking brain structure to cognitive function remains challenging due to the limitations of conventional methods and the black-box nature of deep learning. We propose InfoVAE-Med3D, an adaptation of InfoVAE for three-dimensional brain MRI that learns informative latent representations by relaxing the mutual information regularization penalty between inputs and latent representations. We evaluated the method on a healthy control (HC) dataset ( $n=6{,}503$ ) with age and sex information and a multiple sclerosis (MS) dataset ( $n=916$ ) with age, sex, and Symbol Digit Modalities Test (SDMT) scores. Two-dimensional visualizations revealed sex-related clustering and smooth transitions in grey and white matter volumes. Removing the mutual information regularization penalty generally improved MRI reconstruction and brain age and SDMT prediction using support vector regression, outperforming other variational autoencoder variants and achieving competitive performance with deep learning predictors, particularly on the MS dataset. InfoVAE-Med3D provides informative latent representations for investigating brain-related characteristics and supports latent-space analysis and imaging biomarker discovery for cognitive impairment, particularly in MS.

Figures & tables

Figure 1: Overview of the proposed pipeline using InfoVAE-Med3D for latent representation learning, followed by latent-space analysis and cognitive biomarker prediction.

Dataset		HC dataset	MS dataset
Subjects / Sessions		6,503 / 6,503	916 / 2,408
Train / Val / Test		5,202 / 650 / 651	1,929 / 241 / 238
Age	Range	18–97	19–74
Age	Mean $\pm$ SD	$43.70\pm 21.41$	$42.19\pm 9.52$
SDMT	Range	–	16–97
SDMT	Mean $\pm$ SD	–	$58.94\pm 12.02$

Table 1: Summary statistics of the HC and MS datasets.

Panel of Figure 2 — Figure 2: Effect of latent dimensionality on MRI reconstruction quality.

Baseline models
Models	HC dataset			MS dataset
Models	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
AE	0.8065	24.99	0.1395	0.7370	24.73	0.1609
VAE	0.6269	20.38	0.2329	0.5825	20.45	0.2500
BetaVAE	0.6642	21.39	0.2156	0.6097	21.75	0.2362
Our proposed model: InfoVAE-Med3D

Table 2: Reconstruction results for different configurations of the proposed model compared with baseline VAE variants. The best results are highlighted in bold .

Figure 3: Qualitative reconstruction comparison of InfoVAE-Med3D and VAE baselines on two cases from the HC (in the left, blue background) and MS (in the right, yellow background) datasets.

Panel of Figure 4 — Figure 4: Two-dimensional projections of latent representations colored by sex labels for the HC and MS datasets. PCA and PLS Regression projections are shown on the left and right, respectively.

Panel of Figure 5 — Figure 5: Latent representations colored by estimated GM volumes for the HC and MS datasets. PCA and PLS Regression projections are shown on the left and right, respectively.

Panel of Figure 6 — Figure 6: Latent representations colored by estimated WM volumes for the HC and MS datasets. PCA and PLS Regression projections are shown on the left and right, respectively.

Models	HC dataset	MS dataset
Models	Brain age	Brain age	SDMT
RF	8.192 $\pm$ 0.304	6.145 $\pm$ 0.368	8.821 $\pm$ 0.384
LR	6.715 $\pm$ 0.351	6.314 $\pm$ 0.386	10.648 $\pm$ 0.486
KNN	8.829 $\pm$ 0.336	7.313 $\pm$ 0.535	9.713 $\pm$ 0.491
XGBoost	7.234 $\pm$ 0.345	5.929 $\pm$ 0.420	8.853 $\pm$ 0.441
Adaboost	7.707 $\pm$ 0.261	6.055 $\pm$ 0.439	8.877 $\pm$ 0.459

Table 3: Results of cognitive biomarker prediction using different ML models on the HC and MS datasets, reported as MAE (Mean

\pm

SD) over 5-fold cross-validation on the combined training and validation sets. The best results are highlighted in bold .

DL models
Models	HC dataset			MS dataset
	Brain age			Brain age			SDMT
	MAE $\downarrow$	$R^{2}\uparrow$	RMSE $\downarrow$	MAE $\downarrow$	$R^{2}\uparrow$	RMSE $\downarrow$	MAE $\downarrow$	$R^{2}\uparrow$	RMSE $\downarrow$
Average	10.39	0.000	11.98	7.153	0.000	9.129	9.650	-2.274	11.96
ResNet	6.484	0.541	8.221	5.691	0.406	7.112	9.132	0.095	11.42
DenseNet	6.063	0.574	7.901	5.544	0.413	6.978	9.023	0.108	11.35

Table 4: Quantitative results of cognitive biomarker prediction on the HC and MS datasets, comparing latent representations learned by our proposed model, representation-learning baselines, and DL models. The best results are highlighted in bold .

Medical Image Representation Learning

Medical image representation learning builds reusable visual features for diagnosis, segmentation, retrieval, and other clinical imaging tasks.

See all 260 papers

Modality-Aware and Anatomical Vector-Quantized Autoencoding for Multimodal Brain MRI
Apr 6, 2026Mingjie Li, Edward Kim, Yue Zhao +2Vector-Quantized Variational Autoencoder Medical Image Representation Learning
Stanford University
BrainG3N: A Dual-Purpose Tokenizer for Controllable 3D Brain MRI Generation
Jun 17, 2026Max Van Puyvelde, Ibrahim Gulluk, Wim Van Criekinge +1Masked Autoencoders Latent Diffusion Models
Department of Biomedical Data Science, Stanford University School of Medicine · Department of Mathematical Modelling, Statistics & Bioinformatics, Ghent University · Department of Electrical Engineering, Stanford University
VolTA-3D: Self-Supervised Learning for Brain MRI using 3D Volumetric Token Alignment
May 16, 2026Amy Makawana, Abhijeet Parida, Marius George Linguraru +23D ViTs Self-Supervised Pre-Training
Institute of Health Informatics, University College London, London, UK · Sheikh Zayed Institute for Pediatric Surgical Innovation, Children’s National Hospital, Washington, DC USA · School of Medicine and Health Sciences, George Washington University, Washington, DC, USA

Apr 6, 2026J/K move · Enter open · S save

Modality-Aware and Anatomical Vector-Quantized Autoencoding for Multimodal Brain MRI

Mingjie Li, Edward Kim, Yue Zhao, Ehsan Adeli+1

Stanford University

Vector-Quantized Variational Autoencoder Medical Image Representation Learning Brain MRI Neuroimaging Multimodal Disentangled Representation Learning

Open paper AskPDF arXiv

Learning a robust Variational Autoencoder (VAE) is a fundamental step for many deep learning applications in medical image analysis, such as MRI synthesizes. Existing brain VAEs predominantly focus on single-modality data (i.e., T1-weighted MRI), overlooking the complementary diagnostic value of other modalities like T2-weighted MRIs. Here, we propose a modality-aware and anatomically grounded 3D vector-quantized VAE (VQ-VAE) for reconstructing multi-modal brain MRIs. Called NeuroQuant, it first learns a shared latent representation across modalities using factorized multi-axis attention, which can capture relationships between distant brain regions. It then employs a dual-stream 3D encoder that explicitly separates the encoding of modality-invariant anatomical structures from modality-dependent appearance. Next, the anatomical encoding is discretized using a shared codebook and combined with modality-specific appearance features via Feature-wise Linear Modulation (FiLM) during the decoding phase. This entire approach is trained using a joint 2D/3D strategy in order to account for the slice-based acquisition of 3D MRI data. Extensive experiments on two multi-modal brain MRI datasets demonstrate that NeuroQuant achieves superior reconstruction fidelity compared to existing VAEs, enabling a scalable foundation for downstream generative modeling and cross-modal brain image analysis.

Abstract

Medical Image Representation Learning

Explore similar work

Modality-Aware and Anatomical Vector-Quantized Autoencoding for Multimodal Brain MRI

BrainG3N: A Dual-Purpose Tokenizer for Controllable 3D Brain MRI Generation

VolTA-3D: Self-Supervised Learning for Brain MRI using 3D Volumetric Token Alignment