Organizations: University of Science and Technology of China, Hefei, 230026, Anhui, China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, 230000, Anhui, China · Beijing Tiantan Hospital, Capital Medical University, Beijing Neurosurgical Institute, 100071, Beijing, China
Automated Magnetic Resonance Imaging (MRI) image classification provides a crucial non-invasive approach for neurological diagnosis, yet it is fundamentally challenged by inherent acquisition noise and high inter-subject structural similarity, which obscure subtle pathological variations, thereby limiting the performance of deep neural networks. While contrastive learning (CL) offers a promising mitigation strategy, existing paradigms rely on rigid binary partitions that ignore negative sample heterogeneity, and they fail to capture the cross-view structural consistency vital for fine-grained anatomical alignment. To address these barriers, we propose Dual-Branch Hierarchical Contrastive Learning (DBHiCo), a novel framework explicitly designed to collaboratively enforce structural consistency and semantic discrimination. Specifically, DBHiCo leverages a natural anatomical multi-view pair (original and skull-stripped MRI scans), processing them through a dual-branch architecture to seamlessly integrate holistic structural robustness with localized intracranial details. Crucially, we introduce a hierarchical contrastive loss that discards conventional binary constraints in favor of a margin-ranked similarity hierarchy. This mechanism explicitly orders positive and negative pairs based on their structural and semantic difficulty, thereby amplifying the fine-grained semantic discriminability of the learned representations. Extensive experiments across three diverse neurological datasets demonstrate that DBHiCo achieves state-of-the-art performance, consistently outperforming existing MRI multi-view representation learning and advanced contrastive fine-tuning methods.
Figures & tables
Figure 1: Motivation of DBHiCo. (a)-(b) Comparison between the semantic supervised paradigms and ours. Intra-views supervision provides insufficient learning of structural relationships, whereas our method explicitly enforces cross-view and semantic contrastive learning. (c) Similarity analysis across views and classes. MRI images exhibit high similarity issue within the same view, while previous method (e.g., SuperCon [ 23 ] ) disproportionately emphasize semantic discrimination but ignoring structural consistency learning.
Figure 2: An overview of DBHiCo. (a) The pipeline of proposed DBHiCo. A dual-branch pipeline processes original and skull-stripped MRIs with independent encoders, extracting features for classification ( LCE ) and contrastive learning ( LHiCo ). (b) Hierarchical Contrastive Loss framework, which defines positive intra-subject pairs and multi-level negative pairs, enforcing structured similarity constraints to learn discriminative features.
Method (Publications,Years)
Pre-training dataset
Network
Input Size
Medulloblastoma
Choroid
Ependymoma
Glioma
Total
ACC(%)
F1(%)
ACC(%)
F1(%)
ACC(%)
F1(%)
ACC(%)
F1(%)
ACC(%)
F1(%)
MRI Multi-View Representation Learning
DAE (MIDL 2024) [ 60 ]
DAE-10K [ 60 ]
Swin-Unetr
8×224×224
90.40
89.04
55.33
63.01
69.99
67.14
79.80
79.96
79.96
74.79
VoCo (CVPR 2024) [ 62 ]
VoCo-1.6K [ 62 ]
Swin-Unetr
8×224×224
89.69
87.67
71.14
72.91
64.94
68.61
76.28
74.93
79.86
76.03
MiM (TMI 2025) [ 68 ]
MiM-10K [ 68 ]
Swin-ViT
8×224×224
86.76
86.07
75.19
71.38
61.51
65.35
77.71
77.31
78.42
75.03
BrainMVP (CVPR 2025) [ 45 ]
BrainMVP-16K [ 45 ]
Uniformer-small
16×224×224
84.06
81.16
60.47
66.03
60.40
63.14
71.60
70.48
73.95
70.20
Table 1: Experiment results on Tiantan dataset across four tumors. All methods are trained on both original and skull-stripped MRI scans to ensure fairness. The best result is highlighted in bold , while the second-best is marked with an underline .
Figure 3: Grad-CAM [ 48 ] visualizations across four tumor types on the Tiantan dataset. While the baseline SimCLR is prone to background interference and exhibits blurred boundaries, our proposed HiCo effectively captures fine-grained structural details. Furthermore, DBHiCo achieves the most precise localization of anatomical structures and pathological tissues.
Table 3: Experiment results on ADHD200 dataset [ 7 ] across six independent imaging sites.
Figure 4: Evaluation of the loss weight α for LHiCo . (a) AVG; (b) Tiantan-Tumor Dataset; (c) OASIS-I Dataset; (d) ADHD200 Dataset.
Figure 5: Visualization of T-SNE [ 33 ] results. (a)-(b) Feature distributions for SimCLR on OASIS-I and ADHD200, exhibiting noticeable class overlap. (c)-(d) Feature distributions for DBHiCo on the respective datasets, showing enhanced inter-class separability and tighter intra-class compactness.
Contrastive Types
Tiantan Datasets
ACC(%)
PRE(%)
REC(%)
F1(%)
AUC(%)
w/o Dual-Branch & Contrast
81.88
81.23
77.90
79.18
93.81
T1E mod ⟺ T2 mod
81.55
80.09
77.18
77.63
94.56
Strong trans ⟺ Weak trans
82.38
78.63
79.11
79.40
94.60
Brain ⟺ Tumor
84.27
83.05
81.27
81.92
96.10
Original ⟺ Skull-stripped
86.51
84.69
84.18
84.21
96.09
Table 4: Evaluation of model with different contrastive views.
Components
Tiantan
OASIS-I
ADHD200
Dual-Branch
Hierarchical
Projector
ACC(%)
F1(%)
ACC(%)
F1(%)
ACC(%)
F1(%)
✗
✗
✗
82.27
79.18
80.56
73.73
65.97
62.03
✗
✔
✗
83.27
80.91
81.98
77.07
68.45
66.66
✗
✔
✔
84.89
82.17
83.17
78.12
68.73
66.93
✔
✗
✗
82.16
78.50
81.67
75.35
65.99
63.91
✔
✔
✗
85.15
83.01
83.49
79.42
68.02
66.32
Table 5: Evaluation of our DBHiCo’s components.
Method
Loss function LC
Tiantan
OASIS-I
ADHD200
ACC(%)
F1(%)
ACC(%)
F1(%)
ACC(%)
F1(%)
HiCo
Contrastive loss [ 38 ]
83.72
81.13
82.85
74.88
66.57
64.01
Hierarchical Contrastive loss
84.89
82.17
83.17
78.12
68.73
66.93
DBHiCo
Contrastive loss [ 38 ]
83.60
80.64
83.33
76.89
68.12
63.72
Hierarchical Contrastive loss
86.51
84.21
85.32
79.86
69.81
66.78
Table 6: Evaluation of LHiCo whether using hierarchical difficulty.
Loss
Tiantan
OASIS-I
ADHD200
{\color[rgb]{0,0,0}\mathcal{L}_{\mathrm{1}}}
{\color[rgb]{0,0,0}\mathcal{L}_{\mathrm{2}}}
{\color[rgb]{0,0,0}\mathcal{L}_{\mathrm{3}}}
{\color[rgb]{0,0,0}\mathcal{L}_{\mathrm{4}}}
ACC(%)
F1(%)
ACC(%)
F1(%)
ACC(%)
F1(%)
✔
✗
✗
✗
84.09
81.26
83.57
77.65
66.76
64.87
✗
✔
✗
✗
83.72
81.08
82.53
77.29
65.75
63.96
✗
✗
✔
✗
84.68
82.13
81.82
76.32
65.85
63.58
✗
✗
✗
✔
82.40
79.53
81.98
77.74
66.23
62.98
✔
✔
✔
✔
86.51
84.21
85.32
79.86
69.81
66.78
Table 7: Evaluation of the loss terms LHiCo .
Margin
Tiantan
OASIS-I
ADHD200
Uniform
Decay
ACC(%)
F1(%)
ACC(%)
F1(%)
ACC(%)
F1(%)
✔
✗
86.51
84.21
85.32
79.86
68.26
65.58
✔
✔
85.28
83.08
83.65
77.84
66.72
64.25
✗
✗
85.53
82.93
82.94
76.86
69.81
66.78
✗
✔
85.05
82.70
84.37
79.61
68.17
66.00
Table 8: Evaluation of our DBHiCo’s margin-parameter adjustment strategy.
Brain imaging classification is commonly approached from two perspectives: modeling the full image volume to capture global anatomical context, or constructing ROI-based graphs to encode localized and topological interactions. Although both representations have demonstrated independent efficacy, their relative contributions and potential complementarity remain insufficiently understood. Existing fusion approaches are typically task-specific and do not enable controlled evaluation of each representation under consistent training settings. To address this gap, we propose a unified cross-view contrastive framework for joint imaging-ROI representation learning. Our method learns subject-level global (imaging) and local (ROI-graph) embeddings and aligns them in a shared latent space using a bidirectional contrastive objective, encouraging representations from the same subject to converge while separating those from different subjects. This alignment produces comparable embeddings suitable for downstream fusion and enables systematic evaluation of imaging-only, ROI-only, and joint configurations within a unified training protocol. Extensive experiments on the ADHD-200 and ABIDE datasets demonstrate that joint learning consistently improves classification performance over either branch alone across multiple backbone choices. Moreover, interpretability analyses reveal that imaging-based and ROI-based branches emphasize distinct yet complementary discriminative patterns, explaining the observed performance gains. These findings provide principled evidence that explicitly integrating global volumetric and ROI-level representations is a promising direction for neuroimaging-based brain disorder classification. The source code is available at https://anonymous.4open.science/r/imaging-roi-contrastive-152C/.
Self-supervised learning offers a compelling approach for medical imaging, where labeled data are scarce and acquisition costs are high. We present COJEPA, a self-supervised framework for volumetric brain MRI that combines a joint-embedding predictive architecture (JEPA) with a contrastive loss (CO), targeting two complementary properties: local predictivity and global discriminability. The model is trained without labels on T1-weighted structural MRI from two cohorts (HCP-YA and AABC, N=2286, ages 22 to 90), extending I-JEPA to 3D with foreground-aware block masking, a hierarchical convolutional patch embedding, and world-space sinusoidal positional encodings. We evaluate all three objectives across zero-shot twin retrieval, brain tumor segmentation (BraTS 2024), and age regression (OpenBHB). COJEPA achieves the best monozygotic twin recall at rank@1 (0.84), the best finetuning age MAE (2.55 years on OpenBHB 3.0T), and matches CO on BraTS whole-tumor Dice, demonstrating that the combined objective yields representations that are simultaneously discriminative and locally structured.
Fabian Mager, Lars Kai Hansen
Department of Applied Mathematics and Computer Science Technical University of Denmark
Clinical MRI routinely acquires multiple contrast-weighted images of the same anatomy for complementary tissue characterization. However, current accelerated MRI methods typically reconstruct each contrast independently, without fully exploiting shared anatomical information. This work aims to learn anatomical representations invariant to contrast-dependent appearance for reconstruction of accelerated multi-contrast MRI. We propose MAX (MAnifold eXpansion), a subject-specific framework that learns anatomical representations from a single fully sampled reference contrast. To address the under-constrained separation of shared anatomy and contrast-dependent components from a single image, MAX expands the multi-contrast manifold using anatomy-preserving intensity augmentations. A disentangled implicit neural representation models augmented samples using shared spatial coordinates for anatomy and spatially invariant coordinates for contrast appearance. The learned anatomical representation is then fixed, with the contrast representation adapted to the undersampled target data, followed by unrolled refinement. Theoretical analyses further provide insight into the disentangled representation learning and explain how the learned anatomical representation improves the target contrast reconstruction. At R = 8 for brain MRI and R = 6 for knee MRI, MAX achieves the highest mean PSNR and SSIM across all tasks, improving PSNR by more than 1 dB over the strongest baseline for both brain contrasts. MAX more faithfully recovers subtle anatomical and pathological structures and remains robust to inter-contrast motion, structural heterogeneity between reference and target contrasts, and measurement noise. Therefore, MAX provides a general strategy for leveraging high-quality reference scans in accelerated MRI and has the potential to be extended to other reference-assisted MRI inverse problems.
Ruimin Feng, Wanyu Bian, Albert Jang +2
Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Charlestown, Massachusetts, United States · Harvard Medical School, Boston, Massachusetts, United States · Department of Radiology, Massachusetts General Hospital, Charlestown, Massachusetts, United States