CMSA-Net: Causal Multi-scale Aggregation with Adaptive Multi-source Reference for Video Polyp Segmentation

Tong Wang; Yaolei Qi; Siwen Wang; Imran Razzak; Guanyu Yang; Yutong Xie

cs.CVFeb 26, 2026

CMSA-Net: Causal Multi-scale Aggregation with Adaptive Multi-source Reference for Video Polyp Segmentation

Authors: Tong Wang, Yaolei Qi, Siwen Wang, Imran Razzak, Guanyu Yang, Yutong Xie

Organizations: Southeast University, China · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE

Abstract

Video polyp segmentation (VPS) is an important task in computer-aided colonoscopy, as it helps doctors accurately locate and track polyps during examinations. However, VPS remains challenging because polyps often look similar to surrounding mucosa, leading to weak semantic discrimination. In addition, large changes in polyp position and scale across video frames make stable and accurate segmentation difficult. To address these challenges, we propose a robust VPS framework named CMSA-Net. The proposed network introduces a Causal Multi-scale Aggregation (CMA) module to effectively gather semantic information from multiple historical frames at different scales. By using causal attention, CMA ensures that temporal feature propagation follows strict time order, which helps reduce noise and improve feature reliability. Furthermore, we design a Dynamic Multi-source Reference (DMR) strategy that adaptively selects informative and reliable reference frames based on semantic separability and prediction confidence. This strategy provides strong multi-frame guidance while keeping the model efficient for real-time inference. Extensive experiments on the SUN-SEG dataset demonstrate that CMSA-Net achieves state-of-the-art performance, offering a favorable balance between segmentation accuracy and real-time clinical applicability.

Figures & tables

Figure 1: (a) Challenge 1: Weak semantic discrimination, caused by the low contrast between polyps and background (case14_6 and case19). (b) Challenge 2: Large spatio-temporal variations across frames (case51_1 and case71_1). (c) Limitation 1: Existing feature fusion methods for polyp semantic learning , which ignore multi-scale information and intrinsic identity relationships. (d) Limitation 2: Current approaches adopt a fixed single-source reference, failing to capture dynamic and diverse spatio-temporal cues. (e) Our Method .

Figure 2: Framework of the proposed CMSA-Net, including the CMA module ( Sec. 2.1 ) with DMR strategy ( Sec. 2.2 ).

Method	Publication	Backbone	SUN-SEG-Easy-Seen (%)						SUN-SEG-Easy-Unseen (%)
Method	Publication	Backbone	$S_{\alpha}\uparrow$	$E_{\phi}^{mn}\uparrow$	$F_{\beta}^{w}\uparrow$	Dice $\uparrow$	IoU $\uparrow$	MAE $\downarrow$	$S_{\alpha}\uparrow$	$E_{\phi}^{mn}\uparrow$	$F_{\beta}^{w}\uparrow$	Dice $\uparrow$	IoU $\uparrow$	MAE $\downarrow$
COSNet [ 16 ]	TPAMI’19	-	84.5	83.6	72.7	73.0	64.8	3.4	65.4	60.0	43.1	42.3	34.2	7.3
PCSA [ 11 ]	AAAI’20	-	85.2	83.5	68.1	70.9	60.4	3.9	68.0	66.0	45.1	45.0	35.3	7.8
2/3D [ 19 ]	MICCAI’20	-	89.5	90.9	81.9	82.9	75.6	2.1	78.6	77.7	65.2	65.6	57.0	4.4
PraNet [ 7 ]	MICCAI’20	Res2Net-50	91.8	94.2	87.7	88.3	82.5	2.0	78.1	78.8	66.3	66.5	58.2	5.2
ACSNet [ 27 ]	MICCAI’20	ResNet-34	92.0	94.2	87.4	88.2	82.8	1.7	77.2	76.6	63.0	63.8	56.4	4.6

Table 1: Quantitative comparison on SUN-SEG-Easy dataset. The best is bold .

Method	Publication	Backbone	SUN-SEG-Hard-Seen (%)						SUN-SEG-Hard-Unseen (%)
Method	Publication	Backbone	$S_{\alpha}\uparrow$	$E_{\phi}^{mn}\uparrow$	$F_{\beta}^{w}\uparrow$	Dice $\uparrow$	IoU $\uparrow$	MAE $\downarrow$	$S_{\alpha}\uparrow$	$E_{\phi}^{mn}\uparrow$	$F_{\beta}^{w}\uparrow$	Dice $\uparrow$	IoU $\uparrow$	MAE $\downarrow$
COSNet [ 16 ]	TPAMI’19	-	78.5	77.2	62.6	63.3	54.1	4.6	67.0	62.7	44.3	43.8	35.3	7.0
PCSA [ 11 ]	AAAI’20	-	77.2	75.9	56.6	58.5	47.9	5.7	68.2	66.0	44.2	45.0	35.1	8.0
2/3D [ 19 ]	MICCAI’20	-	84.9	86.9	75.3	76.4	67.1	3.5	78.6	77.5	63.4	64.4	55.8	4.4
PraNet [ 7 ]	MICCAI’20	Res2Net-50	88.4	91.9	83.1	83.9	76.6	3.1	78.7	80.2	66.7	67.5	58.7	5.3
ACSNet [ 27 ]	MICCAI’20	ResNet-34	87.2	91.0	80.6	82.0	74.8	3.6	76.2	77.6	61.0	62.4	54.7	5.3

Table 2: Quantitative comparison on SUN-SEG-Hard dataset. The best is bold .

Figure 3: Qualitative Comparisons. Upper (case91_2): a sequence of consecutive low-contrast frames; Lower (case32_4): significant variations between adjacent frames.

Efficiency	PraNet	ACSNet	SEPNet	PNS+	SALI	STDDNet (R)	STDDNet (P)	Ours (R)	Ours (P)
GFLOPs $\downarrow$	78.89	130.52	75.12	68.58	86.58	75.17	64.18	73.36	59.42
Param. (M) $\downarrow$	30.50	29.45	25.96	9.8	82.73	38.16	29.67	29.61	25.79
FPS $\uparrow$	42	21	33	64	10	45	36	47	38

Table 3: Comparison of model parameters and inference time (clip = 6).

Methods	SUN-SEG-Easy-Seen					SUN-SEG-Easy-Unseen					SUN-SEG-Hard-Seen					SUN-SEG-Hard-Unseen
Methods	$S_{\alpha}$	$E_{\phi}^{mn}$	$F_{\beta}^{w}$	Dice	IoU	$S_{\alpha}$	$E_{\phi}^{mn}$	$F_{\beta}^{w}$	Dice	IoU	$S_{\alpha}$	$E_{\phi}^{mn}$	$F_{\beta}^{w}$	Dice	IoU	$S_{\alpha}$	$E_{\phi}^{mn}$	$F_{\beta}^{w}$	Dice	IoU
CMSA-Net	95.1	97.5	91.6	92.6	87.6	86.7	90.3	79.3	80.3	72.6	92.3	95.6	87.1	88.9	82.1	87.3	91.0	79.6	81.3	73.7
w/o CMA	92.4	94.8	87.2	88.5	82.1	78.1	79.5	64.4	65.2	56.5	88.5	92.1	81.3	83.1	74.9	76.4	78.2	61.8	62.9	54.4
w/o DMR	93.8	96.2	89.1	90.3	84.7	80.2	81.6	67.5	68.8	60.3	90.4	94.2	84.8	86.6	79.1	79.1	80.2	65.3	67.0	58.4
w/o CMA+DMR	90.6	92.9	84.0	85.3	79.0	72.5	72.1	55.1	55.8	47.6	85.8	89.5	77.7	79.2	71.0	71.6	71.5	53.4	54.8	46.3
w/o Multi-scale	93.2	95.8	88.6	89.7	83.9	83.5	86.9	75.1	76.2	67.4	89.8	93.5	84.2	86.1	78.4	82.9	87.5	74.1	75.8	67.1

Table 4: Ablation study of key components. The best is bold .

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Qualitative Comparisons on consecutive low-contrast frames (case91_2 and case14_6).

Figure 5: Qualitative Comparisons on consecutive low-contrast and significant-variation frames (case51_1 and case74_1).

Causal Attention

See all 15 papers

DepthPolyp: Pseudo-Depth Guided Lightweight Segmentation for Real-Time Colonoscopy
May 15, 2026Zhuoyu Wu, Wenhui Ou, Lexi Zhang +5Endoscopy Image Segmentation
CyPhi AI Lab, Monash University, Malaysia Campus, Malaysia · Department of Electronic & Computer Engineering, Hong Kong University of Science & Technology, Hong Kong, P.R. China · Harbin Institute of Technology, Harbin, P.R. China +1
ASGNet: Adaptive Spectrum Guidance Network for Automatic Polyp Segmentation
Apr 16, 2026Yanguang Sun, Hengmin Zhang, Jianjun Qian +2Medical Image Analysis Polyp Segmentation
PCA Laboratory, Key Laboratory of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China · School of Information Science and Engineering, East China University of Science and Technology, Shanghai, China
ARTEMIS: Agent-guided Reliability-aware Temporal Mask Evolution for Imperfectly Supervised Video Polyp Segmentation
Jun 18, 2026Tong Wang, Siwen Wang, Yaolei Qi +4Video Object Segmentation Semi-Supervised Medical Image Segmentation
Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, Ministry of Education, Jiangsu, China · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE · School of Medicine, Case Western Reserve University, Cleveland, OH, USA

May 15, 2026J/K move · Enter open · S save

DepthPolyp: Pseudo-Depth Guided Lightweight Segmentation for Real-Time Colonoscopy

Zhuoyu Wu, Wenhui Ou, Lexi Zhang, Pei-Sze Tan+4

CyPhi AI Lab, Monash University, Malaysia Campus, Malaysia · Department of Electronic & Computer Engineering, Hong Kong University of Science & Technology, Hong Kong, P.R. China · Harbin Institute of Technology, Harbin, P.R. China+1

Endoscopy Image Segmentation Domain Generalization Polyp Segmentation Medical Image Segmentation Semantic Segmentation

Open paper AskPDF arXiv

Accurate polyp segmentation in colonoscopy is essential for early colorectal cancer detection, yet real-world clinical environments pose persistent challenges such as motion blur, specular reflections, and illumination instability. Most existing methods are optimized on clean benchmark images and suffer noticeable performance degradation when deployed in authentic surgical scenarios. We propose DepthPolyp, a lightweight and robust segmentation framework based on pseudo-depth-guided multi-task learning and efficient feature modulation. The architecture combines hierarchical Ghost factorization for compact feature generation, Interleaved Shuffle Fusion for low-cost cross-scale interaction, and Dynamic Group Gating for adaptive group-wise feature weighting. Extensive experiments demonstrate that DepthPolyp achieves strong cross-dataset generalization when trained on degraded data and evaluated on both clean and noisy target domains, consistently outperforming lightweight baselines and remaining competitive with substantially larger models. In real surgical video evaluation on PolypGen, DepthPolyp achieves better segmentation performance than models up to

20\times

larger while preserving real-time inference speed. With only 3.57M parameters and 0.86 GMACs, the proposed method runs at over 180 FPS on mobile devices, making it well suited for real-time deployment in resource-constrained clinical environments. Code and pretrained weights are available at: https://github.com/ReaganWu/DepthPolyp/

Abstract

Causal Attention

Explore similar work

DepthPolyp: Pseudo-Depth Guided Lightweight Segmentation for Real-Time Colonoscopy

ASGNet: Adaptive Spectrum Guidance Network for Automatic Polyp Segmentation

ARTEMIS: Agent-guided Reliability-aware Temporal Mask Evolution for Imperfectly Supervised Video Polyp Segmentation