CoMLP: Cooperatively-Gated MLPs for Fine-Grained Cross-Modal Information Fusion in Medical Image Segmentation

Mingyuan Meng; Shuchang Ye; Mingjian Li; Zhenyu Zhao; Jinman Kim; Lei Bi

cs.CVSep 4, 2026

CoMLP: Cooperatively-Gated MLPs for Fine-Grained Cross-Modal Information Fusion in Medical Image Segmentation

Authors: Mingyuan Meng, Shuchang Ye, Mingjian Li, Zhenyu Zhao, Jinman Kim, Lei Bi

Organizations: Institute of Translational Medicine, Shanghai Jiao Tong University, Shanghai, China · Zhongguancun Academy & Zhongguancun Institute of Artificial Intelligence, Beijing, China · School of Computer Science, the University of Sydney, Sydney, NSW, Australia · China United Network Communications Group Co., Ltd., Beijing, China

Abstract

Multi-modal medical images and clinical reports provide complementary anatomical, functional, and semantic information for medical image segmentation. Effectively exploiting these heterogeneous sources requires fine-grained cross-modal information fusion that preserves subtle spatial details while capturing semantic dependencies across modalities. Existing fusion approaches frequently rely on cross-attention, whose computational burden increases rapidly with spatial resolution, making dense cross-modal interaction difficult on high-resolution feature maps, particularly for volumetric medical images. In this work, we propose CoMLP, a cooperatively-gated MLP module for fine-grained cross-modal information fusion in medical image segmentation. CoMLP models cross-modal dependencies through cooperative cross-gating, built upon complementary regional and dilated MLP interactions, to capture local and global cross-modal dependencies. We further develop a multi-source fusion architecture in which CoMLP performs both inter-image fusion across imaging modalities and vision-language fusion between visual features and textual reports, enabling heterogeneous information to be integrated without relying on dense cross-attention. Extensive experiments on five medical segmentation benchmarks, covering 2D/3D images, clinical reports, multiple imaging modalities, and diverse anatomical regions, demonstrate consistent improvements over state-of-the-art multi-modal and language-guided segmentation methods. Ablation studies further show that fine-grained interaction at high spatial resolutions and complementary local-global fusion are critical to the performance gains. These results demonstrate the potential of MLP-based interaction as an effective alternative for fine-grained cross-modal information fusion in medical image segmentation.

Figures & tables

Fig. 1: Illustration of CoMLP for fine-grained cross-modal information fusion. (a) CoMLP instantiated for inter-image fusion between two imaging modalities. (b) CoMLP instantiated for vision-language fusion between visual and textual representations. (c) Cooperative cross-gating unit where cross-modal information participates in gate generation to modulate the mainstream representation.

Fig. 2: Illustration of CoMLP-based multi-source fusion architecture. CoMLP is applied throughout the encoder for inter-image fusion between medical imaging modalities and in the decoder for vision-language fusion between visual and textual representations.

Method category	Included methods	Uni-modal image	Multi-modal images	Inter-image fusion	Text utilization
General segmentation	U-Net [ 40 ] , U-Net++ [ 41 ] , nnU-Net [ 42 ] , Swin-Unet [ 43 ] , MLP-Unet [ 25 ]	✓	✓	Concat.	None
Language-guided segmentation	LViT [ 3 ] , Ariadne’s [ 11 ] , MAdapter [ 14 ] , ViTexNet [ 32 ] , FMISeg [ 15 ] , TeViA [ 16 ] , ConTEXTual Net 3D [ 4 ]	✓	✓	Concat.	Explicit
Multi-modal image segmentation	SwinCross [ 8 ] , C 2 MAOT [ 9 ] , CIPA [ 10 ]	$\times$	✓	Explicit	None
VLP-based segmentation	CLIP [ 44 ] , GloRIA [ 45 ] , BiomedCLIP [ 46 ]	✓	$\times$	None	Pretrain

TABLE I: Summary of the comparison methods according to supported information sources and fusion mechanisms.

Method	OPC		NPC		Average
	Dice (%)	mIoU (%)	Dice (%)	mIoU (%)	Dice (%)	mIoU (%)
General segmentation (concatenated multi-modal images; no report)
U-Net [ 40 ]	74.29	60.95	75.37	61.73	74.83	61.34
U-Net++ [ 41 ]	75.18	61.66	75.88	62.29	75.53	61.98
nnU-Net [ 42 ]	76.23	62.98	76.89	63.22	76.56	63.10
Swin-Unet [ 43 ]	75.95	62.80	76.53	62.84	76.24	62.82

TABLE II: Quantitative comparison on OPC and NPC with multi-modal images and clinical reports.

Method	QaTa-COV19		MosMedData+		Kvasir-SEG		Average
	Dice (%)	mIoU (%)	Dice (%)	mIoU (%)	Dice (%)	mIoU (%)	Dice (%)	mIoU (%)
General segmentation (no text)
U-Net [ 40 ]	82.99	70.92	64.60	50.73	82.09	69.62	76.56	63.76
U-Net++ [ 41 ]	83.69	71.96	71.75	58.39	82.06	69.58	79.17	66.64
nnU-Net [ 42 ]	80.42	70.81	72.59	60.36	/	/	/	/
Swin-Unet [ 43 ]	83.60	72.40	66.90	53.10	82.41	72.38	77.64	65.96

TABLE III: Quantitative comparison on QaTa-COV19, MosMedData+, and Kvasir-SEG with uni-modal images and textual descriptions.

Fig. 3: Qualitative comparison on OPC (top) and NPC (bottom). Representative differences between the predicted segmentation masks and ground-truth annotations are highlighted in colored boxes.

Configuration	OPC		NPC		Average
	Dice (%)	mIoU (%)	Dice (%)	mIoU (%)	Dice (%)	mIoU (%)
CoMLP ( $F_{1}$ - $F_{5}$ )	79.78	65.79	81.24	68.15	80.51	66.97
CoMLP ( $F_{2}$ - $F_{5}$ )	77.80	63.71	79.36	65.85	78.58	64.78
CoMLP ( $F_{3}$ - $F_{5}$ )	76.32	62.55	77.97	64.21	77.15	63.38
CoMLP ( $F_{4}$ - $F_{5}$ )	75.35	61.81	76.84	63.18	76.10	62.50
CoMLP ( $F_{5}$ )	74.83	61.37	76.25	62.34	75.54	61.86

TABLE IV: Analysis of inter-image fusion on OPC and NPC without using clinical reports.

Configuration	QaTa-COV19		MosMedData+		Kvasir-SEG		Average
	Dice (%)	mIoU (%)	Dice (%)	mIoU (%)	Dice (%)	mIoU (%)	Dice (%)	mIoU (%)
CoMLP (Text dim. 768)	91.55	84.37	80.31	67.19	92.85	87.06	88.24	79.54
CoMLP (Text dim. 384)	91.38	84.02	80.15	66.98	92.37	86.29	87.97	79.10
CoMLP (Text dim. 192)	90.79	83.54	79.88	66.35	91.53	84.98	87.40	78.29
CoMLP (Text dim. 96)	90.32	83.12	79.43	65.69	90.89	84.04	86.88	77.62
CoMLP (Text dim. 48)	89.77	82.05	78.76	64.88	90.17	83.35	86.23	76.76

TABLE V: Analysis of vision-language fusion on QaTa-COV19, MosMedData+, and Kvasir-SEG.

Inter-image	Vision-language	Avg. Dice (%)	Avg. mIoU (%)
MA-CA	MA-CA	79.55	65.72
CoMLP	MA-CA	81.34	67.55
MA-CA	CoMLP	80.27	66.14
CoMLP	CoMLP	82.10	68.49

TABLE VI: Unified interaction across heterogeneous information sources on OPC and NPC.

Configuration	Inter-image		Vision-language
	Dice (%)	mIoU (%)	Dice (%)	mIoU (%)
Full CoMLP	80.51	66.97	88.24	79.54
w/o Dilated branch	79.57	65.89	84.46	74.54
w/o Regional branch	78.66	64.90	85.95	76.03

TABLE VII: Ablation of the regional and dilated interaction branches.

Operator	$F_{5}$	$F_{4}$	$F_{3}$	$F_{2}$	$F_{1}$	Training Memory	Inference Memory	Parameter Number	FLOPs per Case	Avg. Dice (%)	Avg. mIoU (%)
Global-CA	✓	OOM	OOM	OOM	OOM	24GB	5.0GB	24.7M	0.86T	75.81	62.25
MA-CA	✓	✓	✓	✓	OOM	38GB	7.5GB	26.5M	0.98T	78.01	64.06
CoMLP	✓	✓	✓	✓	$\times$	24GB	5.3GB	28.5M	0.90T	78.58	64.78
CoMLP	✓	✓	✓	✓	✓	28GB	5.9GB	30.6M	1.14T	80.51	66.97

TABLE VIII: Computational efficiency analysis of inter-image fusion on OPC/NPC without using clinical reports.

Fig. 4: Hyperparameter analysis of the region size

R

in CoMLP. (a) and (b) present Dice and mIoU results under different

R

values, respectively.

Cross-Modality Medical Image Segmentation

Cross-modality medical image segmentation labels anatomy and disease across imaging types despite differences in appearance, resolution, and available annotations.

See all 72 papers

When Fusion Fails: Corruption-Aware Rebalanced Fusion for Multi-Modal Medical Image Segmentation
Sep 9, 2026Yuchen Pei, Xiaoyu Hu, Yixiong Zou +5Cross-Modality Medical Image Segmentation Multi-Source Image Fusion
Central China Normal University Wuhan, China · Huazhong University of Science and Technology Wuhan, China · Guangzhou University of Chinese Medicine Guangzhou, China +2
CrossWeaver: Cross-modal Weaving for Arbitrary-Modality Semantic Segmentation
Apr 3, 2026Zelin Zhang, Kedi Li, Huiqi Liang +3Multimodal Fusion Semantic Segmentation
The University of Sydney · University of Technology Sydney
CFCML: A Coarse-to-Fine Crossmodal Learning Framework For Disease Diagnosis Using Multimodal Images and Tabular Data
Mar 20, 2026Tianling Liu, Hongying Liu, Fanhua Shang +3Cross-Modal Learning Multimodal Classification
College of Intelligence and Computing, Tianjin University, Tianjin 300350, China. · Medical School of Tianjin University, Tianjin 300072, China. · Peng Cheng Lab, Shenzhen 518055, China. +3

Sep 9, 2026J/K move · Enter open · S save

When Fusion Fails: Corruption-Aware Rebalanced Fusion for Multi-Modal Medical Image Segmentation

Yuchen Pei, Xiaoyu Hu, Yixiong Zou, Dingwen Hu+4

Central China Normal University Wuhan, China · Huazhong University of Science and Technology Wuhan, China · Guangzhou University of Chinese Medicine Guangzhou, China+2

Cross-Modality Medical Image Segmentation Multi-Source Image Fusion Multimodal Robustness Medical Image Segmentation Multimodal Fusion

Open paper AskPDF arXiv

Multi-modal medical image segmentation leverages complementary diagnostic information, yet fusion can underperform single-modality baselines when spatially aligned inputs differ in quality. Here, "corruption" primarily denotes resolution-induced degradation rather than misalignment or complete modality absence, while synthetic noise is evaluated only as an auxiliary setting. We identify a critical optimization-inference inconsistency: degraded modalities can receive weak training updates yet substantially affect predictions, indicating active interference with fusion. We attribute this failure to resampling-induced feature corruption and optimization bias, where noisy features propagate through skip connections and encourage unreliable modality selection. We therefore propose CoReFuse-Med, a Corruption-aware Rebalanced Fusion framework that suppresses corruption during feature transmission and rebalances modality contributions during high-level fusion. Experiments on EPVS, BraTS, and WMH, including multiple Z-axis slice-retention ratios and an auxiliary noise test, demonstrate improved accuracy and robustness under modality-quality discrepancies. Our code is available at https://github.com/lrever/CoReFuse.

Abstract

Cross-Modality Medical Image Segmentation

Explore similar work

When Fusion Fails: Corruption-Aware Rebalanced Fusion for Multi-Modal Medical Image Segmentation

CrossWeaver: Cross-modal Weaving for Arbitrary-Modality Semantic Segmentation

CFCML: A Coarse-to-Fine Crossmodal Learning Framework For Disease Diagnosis Using Multimodal Images and Tabular Data