CoMLP: Cooperatively-Gated MLPs for Fine-Grained Cross-Modal Information Fusion in Medical Image Segmentation
Authors: Mingyuan Meng, Shuchang Ye, Mingjian Li, Zhenyu Zhao, Jinman Kim, Lei Bi
Organizations: Institute of Translational Medicine, Shanghai Jiao Tong University, Shanghai, China · Zhongguancun Academy & Zhongguancun Institute of Artificial Intelligence, Beijing, China · School of Computer Science, the University of Sydney, Sydney, NSW, Australia · China United Network Communications Group Co., Ltd., Beijing, China
Multi-modal medical images and clinical reports provide complementary anatomical, functional, and semantic information for medical image segmentation. Effectively exploiting these heterogeneous sources requires fine-grained cross-modal information fusion that preserves subtle spatial details while capturing semantic dependencies across modalities. Existing fusion approaches frequently rely on cross-attention, whose computational burden increases rapidly with spatial resolution, making dense cross-modal interaction difficult on high-resolution feature maps, particularly for volumetric medical images. In this work, we propose CoMLP, a cooperatively-gated MLP module for fine-grained cross-modal information fusion in medical image segmentation. CoMLP models cross-modal dependencies through cooperative cross-gating, built upon complementary regional and dilated MLP interactions, to capture local and global cross-modal dependencies. We further develop a multi-source fusion architecture in which CoMLP performs both inter-image fusion across imaging modalities and vision-language fusion between visual features and textual reports, enabling heterogeneous information to be integrated without relying on dense cross-attention. Extensive experiments on five medical segmentation benchmarks, covering 2D/3D images, clinical reports, multiple imaging modalities, and diverse anatomical regions, demonstrate consistent improvements over state-of-the-art multi-modal and language-guided segmentation methods. Ablation studies further show that fine-grained interaction at high spatial resolutions and complementary local-global fusion are critical to the performance gains. These results demonstrate the potential of MLP-based interaction as an effective alternative for fine-grained cross-modal information fusion in medical image segmentation.
Figures & tables
Fig. 1: Illustration of CoMLP for fine-grained cross-modal information fusion. (a) CoMLP instantiated for inter-image fusion between two imaging modalities. (b) CoMLP instantiated for vision-language fusion between visual and textual representations. (c) Cooperative cross-gating unit where cross-modal information participates in gate generation to modulate the mainstream representation.
Fig. 2: Illustration of CoMLP-based multi-source fusion architecture. CoMLP is applied throughout the encoder for inter-image fusion between medical imaging modalities and in the decoder for vision-language fusion between visual and textual representations.
TABLE I: Summary of the comparison methods according to supported information sources and fusion mechanisms.
Method
OPC
NPC
Average
Dice (%)
mIoU (%)
Dice (%)
mIoU (%)
Dice (%)
mIoU (%)
General segmentation (concatenated multi-modal images; no report)
U-Net [ 40 ]
74.29
60.95
75.37
61.73
74.83
61.34
U-Net++ [ 41 ]
75.18
61.66
75.88
62.29
75.53
61.98
nnU-Net [ 42 ]
76.23
62.98
76.89
63.22
76.56
63.10
Swin-Unet [ 43 ]
75.95
62.80
76.53
62.84
76.24
62.82
TABLE II: Quantitative comparison on OPC and NPC with multi-modal images and clinical reports.
Method
QaTa-COV19
MosMedData+
Kvasir-SEG
Average
Dice (%)
mIoU (%)
Dice (%)
mIoU (%)
Dice (%)
mIoU (%)
Dice (%)
mIoU (%)
General segmentation (no text)
U-Net [ 40 ]
82.99
70.92
64.60
50.73
82.09
69.62
76.56
63.76
U-Net++ [ 41 ]
83.69
71.96
71.75
58.39
82.06
69.58
79.17
66.64
nnU-Net [ 42 ]
80.42
70.81
72.59
60.36
/
/
/
/
Swin-Unet [ 43 ]
83.60
72.40
66.90
53.10
82.41
72.38
77.64
65.96
TABLE III: Quantitative comparison on QaTa-COV19, MosMedData+, and Kvasir-SEG with uni-modal images and textual descriptions.
Fig. 3: Qualitative comparison on OPC (top) and NPC (bottom). Representative differences between the predicted segmentation masks and ground-truth annotations are highlighted in colored boxes.
Configuration
OPC
NPC
Average
Dice (%)
mIoU (%)
Dice (%)
mIoU (%)
Dice (%)
mIoU (%)
CoMLP ( F1 - F5 )
79.78
65.79
81.24
68.15
80.51
66.97
CoMLP ( F2 - F5 )
77.80
63.71
79.36
65.85
78.58
64.78
CoMLP ( F3 - F5 )
76.32
62.55
77.97
64.21
77.15
63.38
CoMLP ( F4 - F5 )
75.35
61.81
76.84
63.18
76.10
62.50
CoMLP ( F5 )
74.83
61.37
76.25
62.34
75.54
61.86
TABLE IV: Analysis of inter-image fusion on OPC and NPC without using clinical reports.
Configuration
QaTa-COV19
MosMedData+
Kvasir-SEG
Average
Dice (%)
mIoU (%)
Dice (%)
mIoU (%)
Dice (%)
mIoU (%)
Dice (%)
mIoU (%)
CoMLP (Text dim. 768)
91.55
84.37
80.31
67.19
92.85
87.06
88.24
79.54
CoMLP (Text dim. 384)
91.38
84.02
80.15
66.98
92.37
86.29
87.97
79.10
CoMLP (Text dim. 192)
90.79
83.54
79.88
66.35
91.53
84.98
87.40
78.29
CoMLP (Text dim. 96)
90.32
83.12
79.43
65.69
90.89
84.04
86.88
77.62
CoMLP (Text dim. 48)
89.77
82.05
78.76
64.88
90.17
83.35
86.23
76.76
TABLE V: Analysis of vision-language fusion on QaTa-COV19, MosMedData+, and Kvasir-SEG.
Inter-image
Vision-language
Avg. Dice (%)
Avg. mIoU (%)
MA-CA
MA-CA
79.55
65.72
CoMLP
MA-CA
81.34
67.55
MA-CA
CoMLP
80.27
66.14
CoMLP
CoMLP
82.10
68.49
TABLE VI: Unified interaction across heterogeneous information sources on OPC and NPC.
Configuration
Inter-image
Vision-language
Dice (%)
mIoU (%)
Dice (%)
mIoU (%)
Full CoMLP
80.51
66.97
88.24
79.54
w/o Dilated branch
79.57
65.89
84.46
74.54
w/o Regional branch
78.66
64.90
85.95
76.03
TABLE VII: Ablation of the regional and dilated interaction branches.
Operator
F5
F4
F3
F2
F1
Training Memory
Inference Memory
Parameter Number
FLOPs per Case
Avg. Dice (%)
Avg. mIoU (%)
Global-CA
✓
OOM
OOM
OOM
OOM
24GB
5.0GB
24.7M
0.86T
75.81
62.25
MA-CA
✓
✓
✓
✓
OOM
38GB
7.5GB
26.5M
0.98T
78.01
64.06
CoMLP
✓
✓
✓
✓
×
24GB
5.3GB
28.5M
0.90T
78.58
64.78
CoMLP
✓
✓
✓
✓
✓
28GB
5.9GB
30.6M
1.14T
80.51
66.97
TABLE VIII: Computational efficiency analysis of inter-image fusion on OPC/NPC without using clinical reports.
Fig. 4: Hyperparameter analysis of the region size R in CoMLP. (a) and (b) present Dice and mIoU results under different R values, respectively.
Cross-modality medical image segmentation labels anatomy and disease across imaging types despite differences in appearance, resolution, and available annotations.
Multi-modal medical image segmentation leverages complementary diagnostic information, yet fusion can underperform single-modality baselines when spatially aligned inputs differ in quality. Here, "corruption" primarily denotes resolution-induced degradation rather than misalignment or complete modality absence, while synthetic noise is evaluated only as an auxiliary setting. We identify a critical optimization-inference inconsistency: degraded modalities can receive weak training updates yet substantially affect predictions, indicating active interference with fusion. We attribute this failure to resampling-induced feature corruption and optimization bias, where noisy features propagate through skip connections and encourage unreliable modality selection. We therefore propose CoReFuse-Med, a Corruption-aware Rebalanced Fusion framework that suppresses corruption during feature transmission and rebalances modality contributions during high-level fusion. Experiments on EPVS, BraTS, and WMH, including multiple Z-axis slice-retention ratios and an auxiliary noise test, demonstrate improved accuracy and robustness under modality-quality discrepancies. Our code is available at https://github.com/lrever/CoReFuse.
Yuchen Pei, Xiaoyu Hu, Yixiong Zou +5
Central China Normal University Wuhan, China · Huazhong University of Science and Technology Wuhan, China · Guangzhou University of Chinese Medicine Guangzhou, China +2
Multimodal semantic segmentation has shown great potential in leveraging complementary information across diverse sensing modalities. However, existing approaches often rely on carefully designed fusion strategies that either use modality-specific adaptations or rely on loosely coupled interactions, thereby limiting flexibility and resulting in less effective cross-modal coordination. Moreover, these methods often struggle to balance efficient information exchange with preserving the unique characteristics of each modality across different modality combinations. To address these challenges, we propose CrossWeaver, a simple yet effective multimodal fusion framework for arbitrary-modality semantic segmentation. Its core is a Modality Interaction Block (MIB), which enables selective and reliability-aware cross-modal interaction within the encoder, while a lightweight Seam-Aligned Fusion (SAF) module further aggregates the enhanced features. Extensive experiments on multiple multimodal semantic segmentation benchmarks demonstrate that our framework achieves state-of-the-art performance with minimal additional parameters and strong generalization to unseen modality combinations.
Zelin Zhang, Kedi Li, Huiqi Liang +3
The University of Sydney · University of Technology Sydney
In clinical practice, crossmodal information including medical images and tabular data is essential for disease diagnosis. There exists a significant modality gap between these data types, which obstructs advancements in crossmodal diagnostic accuracy. Most existing crossmodal learning (CML) methods primarily focus on exploring relationships among high-level encoder outputs, leading to the neglect of local information in images. Additionally, these methods often overlook the extraction of task-relevant information. In this paper, we propose a novel coarse-to-fine crossmodal learning (CFCML) framework to progressively reduce the modality gap between multimodal images and tabular data, by thoroughly exploring inter-modal relationships. At the coarse stage, we explore the relationships between multi-granularity features from various image encoder stages and tabular information, facilitating a preliminary reduction of the modality gap. At the fine stage, we generate unimodal and crossmodal prototypes that incorporate class-aware information, and establish hierarchical anchor-based relationship mining (HRM) strategy to further diminish the modality gap and extract discriminative crossmodal information. This strategy utilize modality samples, unimodal prototypes, and crossmodal prototypes as anchors to develop contrastive learning approaches, effectively enhancing inter-class disparity while reducing intra-class disparity from multiple perspectives. Experimental results indicate that our method outperforms the state-of-the-art (SOTA) methods, achieving improvements of 1.53% and 0.91% in AUC metrics on the MEN and Derm7pt datasets, respectively. The code is available at https://github.com/IsDling/CFCML.
Tianling Liu, Hongying Liu, Fanhua Shang +3
College of Intelligence and Computing, Tianjin University, Tianjin 300350, China. · Medical School of Tianjin University, Tianjin 300072, China. · Peng Cheng Lab, Shenzhen 518055, China. +3