A multi-center analysis of deep learning methods for video polyp detection and segmentation

Noha Ghatwary; Pedro Chavarias Solano; Mohamed Ramzy Ibrahim; Adrian Krenzer; Frank Puppe; Stefano Realdon; Renato Cannizzaro; Jiacheng Wang

cs.CVMar 4, 2026

A multi-center analysis of deep learning methods for video polyp detection and segmentation

Authors: Noha Ghatwary, Pedro Chavarias Solano, Mohamed Ramzy Ibrahim, Adrian Krenzer, Frank Puppe, Stefano Realdon, Renato Cannizzaro, Jiacheng Wang,

Organizations: Computer Engineering Department, Arab Academy for Science and Technology,Egypt · AI in Medicine and Surgery Group, School of Computer Science, University of Leeds, Leeds, LS2 9JT, UK · Department of Artificial Intelligence and Knowledge Systems, University of Würzburg, Germany · CRO Centro Riferimento Oncologico IRCCS, Aviano, Italy · Department of Computer Science at School of Informatics, Xiamen University · Div. Intelligent Medical Systems, German Cancer Research Center (DKFZ), Heidelberg, Germany · Hangzhou Hikvision Digital Technology Co.,ltd, Hangzhou, China · The Machine Learning Lab, Ukrainian Catholic University, Lviv, Ukraine · Research Institute of Trustworthy Autonomous Systems, Southern University of Science and Technology, China · Healthcare Technology Innovation Centre, Chennai, India · Internal Medicine Department, College of Medicine, Arab Academy for Science and Technology, Egypt · Faculty of Medicine, University of Alexandria, Alexandria, Egypt · Translational Gastroenterology Unit, John Radcliffe Hospital, University of Oxford, Oxford, UK · Université de Versailles St-Quentin en Yvelines, Hôpital Ambroise Paré, France · Department of Molecular and Clinical Medicine, Sahlgrenska Academy, University of Gothenburg, Sweden · Department of Medicine, Geriatrics and Emergencies, Sahlgrenska University Hospital-Mölndal, Gothenburg Västra Götaland Region, Sweden

Abstract

Colonic polyps are well-recognized precursors to colorectal cancer (CRC), typically detected during colonoscopy. However, the variability in appearance, location, and size of these polyps complicates their detection and removal, leading to challenges in effective surveillance, intervention, and subsequently CRC prevention. The processes of colonoscopy surveillance and polyp removal are highly reliant on the expertise of gastroenterologists and occur within the complexities of the colonic structure. As a result, there is a high rate of missed detections and incomplete removal of colonic polyps, which can adversely impact patient outcomes. Recently, automated methods that use machine learning have been developed to enhance polyps detection and segmentation, thus helping clinical processes and reducing missed rates. These advancements highlight the potential for improving diagnostic accuracy in real-time applications, which ultimately facilitates more effective patient management. Furthermore, integrating sequence data and temporal information could significantly enhance the precision of these methods by capturing the dynamic nature of polyp growth and the changes that occur over time. To rigorously investigate these challenges, data scientists and experts gastroenterologists collaborated to compile a comprehensive dataset that spans multiple centers and diverse populations. This initiative aims to underscore the critical importance of incorporating sequence data and temporal information in the development of robust automated detection and segmentation methods. This study evaluates the applicability of deep learning techniques developed in real-time clinical colonoscopy tasks using sequence data, highlighting the critical role of temporal relationships between frames in improving diagnostic precision.

Figures & tables

Figure 1: Sequence data samples from a range of patients demonstrate both clear frames and those with artifacts. This highlights the significance of temporal context in accurately interpreting sequence data.

Figure 2: (top) A $t$ -SNE plot for the distribution of training and testing set from the six different centers. Each point represents a frame from the multi-center dataset, color-coded according to the center originated from. The distribution highlight the challenges of obtaining a cross-center generalization and the significance of model robustness on various data from different resources. (b) Illustrates the dataset variability in terms of the number of frames from each centre and the number of polyps in these sequences.

Figure 3: Different architectures proposed by various EndoCV2022 participating teams. Each method takes an input image, and the output prediction is generated directly or through an ensemble of networks. Table 1 presents the description of the backbone and features of various networks. Each output prediction for the detection process consists of a bounding box prediction with the class label "polyp", whereas for segmentation, it involves pixel-wise classification, with polyp pixels assigned label 1 and background pixels assigned label 0.

Team Name	Algorithm	Backbone	Nature	Choice Basis	Data Aug.	Loss	Opt.	Code
Task I: Polyp detection
IMed	Two Stage + Detector	ResNest + TCT	Aggregation	Context Speed	Yes	Cascade Loss	SGD + Warmup	N/A
SDS_RBS	YOLO	YOLOv5l6 YOLOv65x6	Ensemble	Accuracy Speed	Yes	CIoU	SGD	N/A
WürzVision	TRA	ResNet50	Temporal Attention	F1-Score	Yes	IoU	SGD	[d2]
Task II: Polyp segmentation
Arrah_htic	XP-Net	U-Net	Knowledge distillation	Accuracy Speed	Yes	DSC Tversky	Adam	[s2]

Table 1: Summary of the detection and segmentation tasks undertaken by the participating teams for the EndoCV2022 challenge. Methods are presented along with their characteristics and the rationale for their selection as considered by the teams. Some codes from teams are accessible for replication purposes.

Figure 4: Qualitative results for the detection task displaying both the ground truth and the outcomes from the participating teams. The first three rows illustrate variations in polyp size and location, while the fourth row features a negative sample with no polyp present.

	Average precision, AP			AP across scales
Teams/Method
	AP mean	AP 50	AP 75	AP small	AP medium	AP large
Arrahhtic ∗	0.117	0.208	0.093	0.001	0.095	0.274
IMED	0.116	0.185	0.141	0.000	0.102	0.256
SDS-RBS	0.334	0.481	0.367	0.038	0.338	0.677
UCU ML Lab ∗	0.146	0.266	0.143	0.006	0.184	0.184

Table 2: Team results for the detection task with average precision AP computed at IoU thresholds 50 (AP 50 ), 75 (AP 75 ), and

[0.50:0.05:0.95]

mean AP (AP mean ). Size wise AP values are also presented. Top-two values for each metrics are highlighted in bold.

Figure 5: Qualitative results for the segmentation task displaying both the ground truth and the outcomes from the participating teams. The first three rows illustrate variations in polyp size and location, while the fourth row features a negative sample with no polyp present, i.e., presence of artefact.

Teams	JC $\uparrow$	DSC $\uparrow$	F2 $\uparrow$	PPV $\uparrow$	Rec $\uparrow$	Acc $\uparrow$	NSD $\uparrow$	MASD $\downarrow$	d_hfd $\downarrow$
Arrahhtic	0.438 $\pm$ 0.42	0.481 $\pm$ 0.44	0.495 $\pm$ 0.45	0.696 $\pm$ 0.39	0.635 $\pm$ 0.44	0.989 $\pm$ 0.01	0.357 $\pm$ 0.39	68.091 $\pm$ 144.90	0.388 $\pm$ 0.22
HeHIK	0.720 $\pm$ 0.35	0.765 $\pm$ 0.35	0.766 $\pm$ 0.36	0.897 $\pm$ 0.21	0.807 $\pm$ 0.33	0.995 $\pm$ 0.01	0.675 $\pm$ 0.35	12.496 $\pm$ 59.07	0.301 $\pm$ 0.22
iMed	0.522 $\pm$ 0.42	0.568 $\pm$ 0.43	0.560 $\pm$ 0.43	0.860 $\pm$ 0.28	0.601 $\pm$ 0.43	0.989 $\pm$ 0.02	0.443 $\pm$ 0.40	32.282 $\pm$ 103.65	0.392 $\pm$ 0.28
lswangxmu	0.747 $\pm$ 0.34	0.787 $\pm$ 0.34	0.796 $\pm$ 0.35	0.893 $\pm$ 0.21	0.843 $\pm$ 0.32	0.995 $\pm$ 0.01	0.715 $\pm$ 0.35	8.917 $\pm$ 38.75	0.282 $\pm$ 0.21
SDS-RBS	0.718 $\pm$ 0.37	0.754 $\pm$ 0.37	0.744 $\pm$ 0.38	0.936 $\pm$ 0.19	0.764 $\pm$ 0.36	0.995 $\pm$ 0.01	0.683 $\pm$ 0.37	11.008 $\pm$ 42.92	0.287 $\pm$ 0.22
UCUML	0.500 $\pm$ 0.41	0.550 $\pm$ 0.43	0.556 $\pm$ 0.43	0.606 $\pm$ 0.43	0.794 $\pm$ 0.34	0.986 $\pm$ 0.03	0.448 $\pm$ 0.38	53.947 $\pm$ 140.81	0.253 $\pm$ 0.15

Table 3: Team results for the polyp segmentation methods proposed by the participating teams. Top-two values for each metric are highlighted in bold.

Teams	GPU	Training	Inference	R time	R algo.	R overall
		(hrs)	(fps)
Task I: Polyp detection
IMED	2 $\times$ Tesla V100 32G	1	72	1	4	3
SDS_RBS	1 NVIDIA RTX3090	5	9.10	3	1	1
WürzVision	NVIDIA QUADRO RTX 8000	14	24	2	2	2
Task II: Polyp Segmentation

Table 4: Summary of the GPU configuration utilized by each participating team, training time in hours and inference time in frame per second (fps). Teams with longer post-processing are not included in ranking. Ranking for detection is based on AP mean and that for segmentation is based on DSC.

Medical Image Analysis

Medical image analysis uses computational methods to interpret clinical images, measure anatomy and disease, and support diagnosis and treatment.

See all 291 papers

DepthPolyp: Pseudo-Depth Guided Lightweight Segmentation for Real-Time Colonoscopy
May 15, 2026Zhuoyu Wu, Wenhui Ou, Lexi Zhang +5Endoscopy Image Segmentation
CyPhi AI Lab, Monash University, Malaysia Campus, Malaysia · Department of Electronic & Computer Engineering, Hong Kong University of Science & Technology, Hong Kong, P.R. China · Harbin Institute of Technology, Harbin, P.R. China +1
PolypVision: A Three-Stage Hierarchical Deep Learning Framework for Classification and Segmentation of Colorectal Polyps
Aug 11, 2026Hamidreza Bolhasani, Hamidreza Rastad, Amir Mohammad Akbari +4Medical Image Classification Polyp Segmentation
DataBioX Research, Tehran, Iran · Iran University of Science and Technology, Tehran, Iran · Tehran University of Medical Sciences, Tehran, Iran +1
ASGNet: Adaptive Spectrum Guidance Network for Automatic Polyp Segmentation
Apr 16, 2026Yanguang Sun, Hengmin Zhang, Jianjun Qian +2Medical Image Analysis Polyp Segmentation
PCA Laboratory, Key Laboratory of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China · School of Information Science and Engineering, East China University of Science and Technology, Shanghai, China

May 15, 2026J/K move · Enter open · S save

DepthPolyp: Pseudo-Depth Guided Lightweight Segmentation for Real-Time Colonoscopy

Zhuoyu Wu, Wenhui Ou, Lexi Zhang, Pei-Sze Tan+4

CyPhi AI Lab, Monash University, Malaysia Campus, Malaysia · Department of Electronic & Computer Engineering, Hong Kong University of Science & Technology, Hong Kong, P.R. China · Harbin Institute of Technology, Harbin, P.R. China+1

Endoscopy Image Segmentation Domain Generalization Polyp Segmentation Medical Image Segmentation Semantic Segmentation

Open paper AskPDF arXiv

Accurate polyp segmentation in colonoscopy is essential for early colorectal cancer detection, yet real-world clinical environments pose persistent challenges such as motion blur, specular reflections, and illumination instability. Most existing methods are optimized on clean benchmark images and suffer noticeable performance degradation when deployed in authentic surgical scenarios. We propose DepthPolyp, a lightweight and robust segmentation framework based on pseudo-depth-guided multi-task learning and efficient feature modulation. The architecture combines hierarchical Ghost factorization for compact feature generation, Interleaved Shuffle Fusion for low-cost cross-scale interaction, and Dynamic Group Gating for adaptive group-wise feature weighting. Extensive experiments demonstrate that DepthPolyp achieves strong cross-dataset generalization when trained on degraded data and evaluated on both clean and noisy target domains, consistently outperforming lightweight baselines and remaining competitive with substantially larger models. In real surgical video evaluation on PolypGen, DepthPolyp achieves better segmentation performance than models up to

20\times

larger while preserving real-time inference speed. With only 3.57M parameters and 0.86 GMACs, the proposed method runs at over 180 FPS on mobile devices, making it well suited for real-time deployment in resource-constrained clinical environments. Code and pretrained weights are available at: https://github.com/ReaganWu/DepthPolyp/

Abstract

Medical Image Analysis

Explore similar work

DepthPolyp: Pseudo-Depth Guided Lightweight Segmentation for Real-Time Colonoscopy

PolypVision: A Three-Stage Hierarchical Deep Learning Framework for Classification and Segmentation of Colorectal Polyps

ASGNet: Adaptive Spectrum Guidance Network for Automatic Polyp Segmentation