Authors: Kristian Dalsbø Hindberg, Puneet Sharma, Benedicte Schelde-Olesen, Ulrik Deding, Jakob Frederik Frøkjær Justsen, Jan-Matthias Braun, Esmaeil S. Nadimi
Organizations: Department of Physics and Technology, Machine Learning Group, UiT – The Arctic University of Norway, Tromsø, Norway · Department of Automation and Process Engineering, Machine Learning Group, UiT – The Arctic University of Norway, Tromsø, Norway · Department of Surgery, Odense University Hospital, Odense, Denmark · Department of Clinical Research, University of Southern Denmark, Odense, Denmark · Faculty of Engineering, University of Southern Denmark, Odense, Denmark · Applied AI and Data Science (AID), Maersk Mc-Kinney Moller Institute, Odense, Denmark
A key challenge in colon capsule endoscopy (CCE) analysis is accurately identifying whether multiple images depict the same polyp, which is critical to avoid over-counting and unnecessary clinical interventions. Additionally, this prevents prolonged follow-up colonoscopies where endoscopists search for duplicate polyps without precise localization. This study presents a novel contrastive learning-based approach to polyp matching using a Siamese neural network architecture. The model was trained on a dataset from the Danish CareForColon2015 trial. The dataset comprises 5,103 image pairs of confirmed same polyp and 5,103 pairs of different polyps, annotated by experienced clinicians. The proposed model employs ResNet variants (ResNet18, ResNet34, ResNet50), DenseNet121 and EfficientNet B0/B3 as backbones and utilizes contrastive learning with cosine vector distance metric for classification. Experimental results demonstrate high classification accuracies of up to 92% and AUC values close to 0.98. These results suggest that automated polyp matching has the potential to reduce polyp over-counting, improve patient risk stratification, and reduce unnecessary follow-up colonoscopies.
Figures & tables
Figure 1: Rows 1–5: Each row shows images of a single polyp. Row 6: Example images of five different polyps. The figure also shows varying degree of cleansing quality, artifacts and partially visible polyps. The polyps are marked by a green ellipse.
Number
Fraction of total
Patients in total
722
1.00
Polyps in total
1701
1.00
Polyp pairs
5103
1.00
Split 1: Cross-validation
- Polyp pairs
4338
0.85
Split 2: Final full model
Table 1: Overview of the datasets used in this study. As all polyps of a given patient will remain in the same data set or fold split, the number of polyps within a fold will vary slightly. The set of “Test pairs” of “Split 2” is the same as the “missing” 15% not used in the CV of Split 1.
Figure 2: Network architecture of the proposed Siamese neural network with ResNet backbone and contrastive loss. Each input image (256x256 pixels, resized to 224x224 before processing) is independently fed through a shared backbone, producing a 128-dimensional embedding vector. During training, the cosine distance between the two embedding vectors is used to compute the contrastive loss, which pulls same-polyp pairs together and pushes different-polyp pairs apart. During inference, the cosine distance is compared against a learned threshold, where pairs below the threshold are classified as the same polyp and pairs above as different polyps.
Parameter
Value
Parameter
Value
Backbones
ResNet 18/34/50
Image resize
256x256 to 224x224
Max epochs
100
Augmentation
On-the-fly
Batch size
64
- Random rotation
Uniform, 0 ° – 360 °
Early stopping
- Vertical/horizontal flip
p=0.5
- Patience
20
- Contrast jitter
0.2
- min loss decrease
0.001
- Brightness jitter
0.2
Table 2: Parameters used in Siamese deep neural net training. The “train/val split” for CV is within the 90% of the CV data set used in that fold. The remaining 10% of each fold is used to evaluate the test set performance of that CV split.
Metric
ResNet18
ResNet34
ResNet50
Accuracy
0.9243 ± 0.0136
0.9231 ± 0.0096
0.9167 ± 0.0085
Precision
0.9253 ± 0.0179
0.9181 ± 0.0131
0.9148 ± 0.0168
Recall
0.9234 ± 0.0148
0.9292 ± 0.0125
0.9194 ± 0.0112
Specificity
0.9252 ± 0.0191
0.9170 ± 0.0143
0.9140 ± 0.0195
F1 score
0.9242 ± 0.0135
0.9236 ± 0.0095
0.9170 ± 0.0079
AUC
0.9780 ± 0.0051
0.9775 ± 0.0056
0.9759 ± 0.0053
Table 3: Cross-validation test set performance measures for different ResNets. Mean values and ± 1 standard deviation across the 10 folds. AUC is area under the receiver operating characteristic curve.
Metric
ResNet18
ResNet34
ResNet50
Accuracy
0.9193
0.9225
0.9206
Precision
0.9237
0.9275
0.9184
Recall
0.9141
0.9167
0.9232
Specificity
0.9245
0.9284
0.9180
F1 score
0.9188
0.9221
0.9208
AUC
0.9754
0.9768
0.9804
Table 4: Test set performance measures for different ResNets. AUC is area under the receiver operating characteristic curve.
Figure 3: Training and validation loss using cosine distance metrics, and ResNet of size 18, 34 and 50 from left to right. Best model following triggering of early stopping marked in red.
Figure 4: Training and validation loss when running for 500 epochs. The other hyperparameters are as in Table 2 , except that no early stopping is used. ResNet of size 18, 34 and 50 are shown from left to right, respectively.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1: Density plot of polyp size.
Figure A.2: Count plot of different polyp sizes.
Figure A.3: Polyp size for patients with a single polyp versus those with more than one polyp.
Morphology
Number of polyps
Broad-based/sessile
1375
Pedunculated
246
Sessile w/mucusol cap
15
Flat
14
Tumor
12
Other
27
Appendix
Table A.1: Morphology of included polyps.
Seconds
Cumulative fraction
0
0.118
1
0.269
2
0.342
3
0.389
4
0.440
5
0.467
Appendix
Table A.2: Polyp transit times.
Metric
DenseNet121
EfficientNet B0
EfficientNet B3
Accuracy
0.9172 ± 0.0092
0.9017 ± 0.0130
0.9096 ± 0.0143
Precision
0.9218 ± 0.0144
0.8984 ± 0.0196
0.9094 ± 0.0193
Recall
0.9121 ± 0.0116
0.9063 ± 0.0129
0.9103 ± 0.0228
Specificity
0.9224 ± 0.0154
0.8971 ± 0.0220
0.9088 ± 0.0222
F1 score
0.9168 ± 0.0090
0.9022 ± 0.0123
0.9096 ± 0.0146
AUC
0.9737 ± 0.0042
0.9673 ± 0.0094
0.9691 ± 0.0062
Appendix
Table B.1: Cross-validation test set performance measures for different backbones. Mean values and ± 1 standard deviation across the 10 folds. AUC is area under the receiver operating characteristic curve.
Metric
DenseNet121
EfficientNet B0
EfficientNet B3
Accuracy
0.9160
0.9010
0.9128
Precision
0.9029
0.8979
0.9238
Recall
0.9323
0.9049
0.8997
Specificity
0.8997
0.8971
0.9258
F1 score
0.9174
0.9014
0.9116
AUC
0.9745
0.9708
0.9736
Appendix
Table B.2: Test set performance measures for different backbones. AUC is area under the receiver operating characteristic curve.
Figure B.1: Training and validation loss using cosine distance metrics, and DenseNet121, EfficientNet B0 and EfficientNet B3 from left to right. Best model following triggering of early stopping marked in red.
Figure B.2: Training and validation loss when running for 500 epochs. DenseNet121, EfficientNet B0 and EfficientNet B3 from left to right.
Figure C.1: Distribution of cosine vector distances of the 10 test sets of the cross-validation for all six backbones.
Figure C.2: Distribution of cosine vector distances of the final test set for all six backbones.
NIMISHES Lab, Dhaka, Bangladesh · ELITE Research Lab · Department of Computer Science and Engineering, Bangladesh University of Business and Technology (BUBT), Dhaka, Bangladesh +1
CyPhi AI Lab, Monash University, Malaysia Campus, Malaysia · Department of Electronic & Computer Engineering, Hong Kong University of Science & Technology, Hong Kong, P.R. China · Harbin Institute of Technology, Harbin, P.R. China +1