Organizations: University of Science and Technology of China, Hefei, 230026, Anhui, China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, 230000, Anhui, China · Beijing Tiantan Hospital, Capital Medical University, Beijing Neurosurgical Institute, 100071, Beijing, China
Automated Magnetic Resonance Imaging (MRI) image classification provides a crucial non-invasive approach for neurological diagnosis, yet it is fundamentally challenged by inherent acquisition noise and high inter-subject structural similarity, which obscure subtle pathological variations, thereby limiting the performance of deep neural networks. While contrastive learning (CL) offers a promising mitigation strategy, existing paradigms rely on rigid binary partitions that ignore negative sample heterogeneity, and they fail to capture the cross-view structural consistency vital for fine-grained anatomical alignment. To address these barriers, we propose Dual-Branch Hierarchical Contrastive Learning (DBHiCo), a novel framework explicitly designed to collaboratively enforce structural consistency and semantic discrimination. Specifically, DBHiCo leverages a natural anatomical multi-view pair (original and skull-stripped MRI scans), processing them through a dual-branch architecture to seamlessly integrate holistic structural robustness with localized intracranial details. Crucially, we introduce a hierarchical contrastive loss that discards conventional binary constraints in favor of a margin-ranked similarity hierarchy. This mechanism explicitly orders positive and negative pairs based on their structural and semantic difficulty, thereby amplifying the fine-grained semantic discriminability of the learned representations. Extensive experiments across three diverse neurological datasets demonstrate that DBHiCo achieves state-of-the-art performance, consistently outperforming existing MRI multi-view representation learning and advanced contrastive fine-tuning methods.
Figures & tables
Figure 1: Motivation of DBHiCo. (a)-(b) Comparison between the semantic supervised paradigms and ours. Intra-views supervision provides insufficient learning of structural relationships, whereas our method explicitly enforces cross-view and semantic contrastive learning. (c) Similarity analysis across views and classes. MRI images exhibit high similarity issue within the same view, while previous method (e.g., SuperCon [ 23 ] ) disproportionately emphasize semantic discrimination but ignoring structural consistency learning.
Figure 2: An overview of DBHiCo. (a) The pipeline of proposed DBHiCo. A dual-branch pipeline processes original and skull-stripped MRIs with independent encoders, extracting features for classification ( LCE ) and contrastive learning ( LHiCo ). (b) Hierarchical Contrastive Loss framework, which defines positive intra-subject pairs and multi-level negative pairs, enforcing structured similarity constraints to learn discriminative features.
Method (Publications,Years)
Pre-training dataset
Network
Input Size
Medulloblastoma
Choroid
Ependymoma
Glioma
Total
ACC(%)
F1(%)
ACC(%)
F1(%)
ACC(%)
F1(%)
ACC(%)
F1(%)
ACC(%)
F1(%)
MRI Multi-View Representation Learning
DAE (MIDL 2024) [ 60 ]
DAE-10K [ 60 ]
Swin-Unetr
8×224×224
90.40
89.04
55.33
63.01
69.99
67.14
79.80
79.96
79.96
74.79
VoCo (CVPR 2024) [ 62 ]
VoCo-1.6K [ 62 ]
Swin-Unetr
8×224×224
89.69
87.67
71.14
72.91
64.94
68.61
76.28
74.93
79.86
76.03
MiM (TMI 2025) [ 68 ]
MiM-10K [ 68 ]
Swin-ViT
8×224×224
86.76
86.07
75.19
71.38
61.51
65.35
77.71
77.31
78.42
75.03
BrainMVP (CVPR 2025) [ 45 ]
BrainMVP-16K [ 45 ]
Uniformer-small
16×224×224
84.06
81.16
60.47
66.03
60.40
63.14
71.60
70.48
73.95
70.20
Table 1: Experiment results on Tiantan dataset across four tumors. All methods are trained on both original and skull-stripped MRI scans to ensure fairness. The best result is highlighted in bold , while the second-best is marked with an underline .
Figure 3: Grad-CAM [ 48 ] visualizations across four tumor types on the Tiantan dataset. While the baseline SimCLR is prone to background interference and exhibits blurred boundaries, our proposed HiCo effectively captures fine-grained structural details. Furthermore, DBHiCo achieves the most precise localization of anatomical structures and pathological tissues.
Table 3: Experiment results on ADHD200 dataset [ 7 ] across six independent imaging sites.
Figure 4: Evaluation of the loss weight α for LHiCo . (a) AVG; (b) Tiantan-Tumor Dataset; (c) OASIS-I Dataset; (d) ADHD200 Dataset.
Figure 5: Visualization of T-SNE [ 33 ] results. (a)-(b) Feature distributions for SimCLR on OASIS-I and ADHD200, exhibiting noticeable class overlap. (c)-(d) Feature distributions for DBHiCo on the respective datasets, showing enhanced inter-class separability and tighter intra-class compactness.
Contrastive Types
Tiantan Datasets
ACC(%)
PRE(%)
REC(%)
F1(%)
AUC(%)
w/o Dual-Branch & Contrast
81.88
81.23
77.90
79.18
93.81
T1E mod ⟺ T2 mod
81.55
80.09
77.18
77.63
94.56
Strong trans ⟺ Weak trans
82.38
78.63
79.11
79.40
94.60
Brain ⟺ Tumor
84.27
83.05
81.27
81.92
96.10
Original ⟺ Skull-stripped
86.51
84.69
84.18
84.21
96.09
Table 4: Evaluation of model with different contrastive views.
Components
Tiantan
OASIS-I
ADHD200
Dual-Branch
Hierarchical
Projector
ACC(%)
F1(%)
ACC(%)
F1(%)
ACC(%)
F1(%)
✗
✗
✗
82.27
79.18
80.56
73.73
65.97
62.03
✗
✔
✗
83.27
80.91
81.98
77.07
68.45
66.66
✗
✔
✔
84.89
82.17
83.17
78.12
68.73
66.93
✔
✗
✗
82.16
78.50
81.67
75.35
65.99
63.91
✔
✔
✗
85.15
83.01
83.49
79.42
68.02
66.32
Table 5: Evaluation of our DBHiCo’s components.
Method
Loss function LC
Tiantan
OASIS-I
ADHD200
ACC(%)
F1(%)
ACC(%)
F1(%)
ACC(%)
F1(%)
HiCo
Contrastive loss [ 38 ]
83.72
81.13
82.85
74.88
66.57
64.01
Hierarchical Contrastive loss
84.89
82.17
83.17
78.12
68.73
66.93
DBHiCo
Contrastive loss [ 38 ]
83.60
80.64
83.33
76.89
68.12
63.72
Hierarchical Contrastive loss
86.51
84.21
85.32
79.86
69.81
66.78
Table 6: Evaluation of LHiCo whether using hierarchical difficulty.
Loss
Tiantan
OASIS-I
ADHD200
{\color[rgb]{0,0,0}\mathcal{L}_{\mathrm{1}}}
{\color[rgb]{0,0,0}\mathcal{L}_{\mathrm{2}}}
{\color[rgb]{0,0,0}\mathcal{L}_{\mathrm{3}}}
{\color[rgb]{0,0,0}\mathcal{L}_{\mathrm{4}}}
ACC(%)
F1(%)
ACC(%)
F1(%)
ACC(%)
F1(%)
✔
✗
✗
✗
84.09
81.26
83.57
77.65
66.76
64.87
✗
✔
✗
✗
83.72
81.08
82.53
77.29
65.75
63.96
✗
✗
✔
✗
84.68
82.13
81.82
76.32
65.85
63.58
✗
✗
✗
✔
82.40
79.53
81.98
77.74
66.23
62.98
✔
✔
✔
✔
86.51
84.21
85.32
79.86
69.81
66.78
Table 7: Evaluation of the loss terms LHiCo .
Margin
Tiantan
OASIS-I
ADHD200
Uniform
Decay
ACC(%)
F1(%)
ACC(%)
F1(%)
ACC(%)
F1(%)
✔
✗
86.51
84.21
85.32
79.86
68.26
65.58
✔
✔
85.28
83.08
83.65
77.84
66.72
64.25
✗
✗
85.53
82.93
82.94
76.86
69.81
66.78
✗
✔
85.05
82.70
84.37
79.61
68.17
66.00
Table 8: Evaluation of our DBHiCo’s margin-parameter adjustment strategy.
Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Charlestown, Massachusetts, United States · Harvard Medical School, Boston, Massachusetts, United States · Department of Radiology, Massachusetts General Hospital, Charlestown, Massachusetts, United States