Multimodal medical image segmentation often faces missing modalities at inference, which induces disagreement among modality experts and makes fusion unstable, particularly on small foreground structures. We propose Consistency Learning of Experts (CLoE), a consistency-driven framework for missing-modality segmentation that preserves strong performance when all modalities are available. CLoE formulates robustness as decision-level expert consistency control and introduces a dual-branch Expert Consistency Learning objective. Modality Expert Consistency enforces global agreement among expert predictions to reduce case-wise drift under partial inputs, while Region Expert Consistency emphasizes agreement on clinically critical foreground regions to avoid background-dominated regularization. We further map consistency scores to modality reliability weights using a lightweight gating network, enabling reliability-aware feature recalibration before fusion. Extensive experiments on BraTS 2020 and MSD Prostate demonstrate that CLoE outperforms state-of-the-art methods in incomplete multimodal segmentation, while exhibiting strong cross-dataset generalization and improving robustness on clinically critical structures.
Figures & tables
Figure 1: Overview of our proposed CLoE. This framework first extracts modality-specific features M x from input volumes, undergoes regularization M pred , and utilizes the ECL module to compute its Modality and Region Expert Consistency scores. Finally, it derives modality reliability weights via a lightweight gating network and performs consistency-driven weighted fusion to achieve robust missing modal segmentation.
#
Methods
T2
T1c
T1
F
T2 T1c
T1c T1
T1 F
T2 T1
T2 F
T1c F
∼ T2
∼ T1c
∼ T1
∼ F
Full
Avg
WT
HeMIS
79.85
64.58
63.01
52.29
84.45
72.50
65.29
82.31
81.56
69.37
73.31
83.03
84.64
85.19
85.19
75.10
RobustSeg
82.20
71.39
71.41
82.87
85.97
76.84
88.10
85.53
88.09
87.33
88.87
89.24
88.68
86.63
89.47
84.17
RFNet
86.05
76.77
77.16
87.32
87.74
81.12
89.73
87.73
89.87
89.89
90.69
90.60
90.68
88.25
91.11
86.98
M³AE
86.10
78.90
79.00
88.00
87.10
80.10
89.60
87.30
90.10
89.50
89.60
90.20
90.50
87.40
90.40
86.90
DC-Seg
86.67
79.36
79.31
88.67
88.57
83.03
90.18
88.29
90.41
90.44
90.98
90.80
91.09
88.99
91.22
87.87
CLoE
87.19
80.08
79.96
88.90
88.63
83.57
90.32
88.42
90.47
90.49
90.92
90.87
91.18
89.06
91.30
88.09
Table 1: Performance comparison (Dice%) with SOTA methods on BraTS 2020.
Figure 2: (a) Visualization of the input modalities. (b) General MedSAM model prediction with bounding box prompt. (c) CLoE predicted segmentation maps.
Multi-modal medical image segmentation leverages complementary diagnostic information, yet fusion can underperform single-modality baselines when spatially aligned inputs differ in quality. Here, "corruption" primarily denotes resolution-induced degradation rather than misalignment or complete modality absence, while synthetic noise is evaluated only as an auxiliary setting. We identify a critical optimization-inference inconsistency: degraded modalities can receive weak training updates yet substantially affect predictions, indicating active interference with fusion. We attribute this failure to resampling-induced feature corruption and optimization bias, where noisy features propagate through skip connections and encourage unreliable modality selection. We therefore propose CoReFuse-Med, a Corruption-aware Rebalanced Fusion framework that suppresses corruption during feature transmission and rebalances modality contributions during high-level fusion. Experiments on EPVS, BraTS, and WMH, including multiple Z-axis slice-retention ratios and an auxiliary noise test, demonstrate improved accuracy and robustness under modality-quality discrepancies. Our code is available at https://github.com/lrever/CoReFuse.
Yuchen Pei, Xiaoyu Hu, Yixiong Zou +5
Central China Normal University Wuhan, China · Huazhong University of Science and Technology Wuhan, China · Guangzhou University of Chinese Medicine Guangzhou, China +2
Multimodal clinical prediction faces three challenges: multiple foundation models (FMs) with complementary strengths per modality, pervasive missing modalities at training and test time, and sample-specific variation in modality contributions. We introduce MedMIX, a multimodal framework that combines intra-modality expert fusion, learned inter-modality fusion, and training-only large--small model collaboration for robust medical prediction under incomplete modalities. Within each modality, MedMIX aggregates complementary embeddings from multiple small expert models; across modalities, it performs learned fusion over available modalities; and during training, it leverages large teacher models to improve deployed representations without additional inference cost. Across three heterogeneous benchmarks (OpenI, MIMIC-IV-MM, and MMIST-ccRCC), MedMIX achieves consistently strong performance while remaining robust under controlled missing-modality perturbations, and further demonstrates sustained robustness under cross-cohort shift on MIMIC-III. These results highlight MedMIX as a practical framework that unifies within-modality expert collaboration, sample-specific cross-modality fusion, and efficient large--small model collaboration while remaining robust to incomplete modalities.
Seungik Cho, Anqi Li, Wei Qiu
Department of Electrical and Computer Engineering Rice University Houston, TX 77005
We introduce a novel uncertainty-aware multimodal segmentation framework that leverages both radiological images and associated clinical text for precise medical diagnosis. We propose a Modality Decoding Attention Block (MoDAB) with a lightweight State Space Mixer (SSMix) to enable efficient cross-modal fusion and long-range dependency modelling. To guide learning under ambiguity, we propose the Spectral-Entropic Uncertainty (SEU) Loss, which jointly captures spatial overlap, spectral consistency, and predictive uncertainty in a unified objective. In complex clinical circumstances with poor image quality, this formulation improves model reliability. Extensive experiments on various publicly available medical datasets, QATA-COVID19, MosMed++, and Kvasir-SEG, demonstrate that our method achieves superior segmentation performance while being significantly more computationally efficient than existing State-of-the-Art (SoTA) approaches. Our results highlight the importance of incorporating uncertainty modelling and structured modality alignment in vision-language medical segmentation tasks. Code: https://github.com/arya-domain/UA-VLS