Understanding Annotation Error Propagation and Learning an Adaptive Policy for Expert Intervention in Barrett's Video Segmentation

Lokesha Rasanjalee; Jin Lin Tan; Dileepa Pitawela; Rajvinder Singh; Hsiang-Ting Chen

cs.CVFeb 25, 2026

Understanding Annotation Error Propagation and Learning an Adaptive Policy for Expert Intervention in Barrett's Video Segmentation

Authors: Lokesha Rasanjalee, Jin Lin Tan, Dileepa Pitawela, Rajvinder Singh, Hsiang-Ting Chen

Organizations: School of Computer Science and Information Technology, Adelaide University, Australia · The Australian Institute for Machine Learning (AIML), Australia · Adelaide Medical School, Faculty of Health and Medical Sciences, Adelaide University, Australia · Department of Gastroenterology and Hepatology, Lyell McEwin Hospital, SA, Australia

Abstract

Accurate annotation of endoscopic videos is essential yet time-consuming, particularly for challenging datasets such as dysplasia in Barrett's esophagus, where the affected regions are irregular and lack clear boundaries. Semi-automatic tools like Segment Anything Model 2 (SAM2) can ease this process by propagating annotations across frames, but small errors often accumulate and reduce accuracy, requiring expert review and correction. To address this, we systematically study how annotation errors propagate across different prompt types, namely masks, boxes, and points, and propose Learning-to-Re-Prompt (L2RP), a cost-aware framework that learns when and where to seek expert input. By tuning a human-cost parameter, our method balances annotation effort and segmentation accuracy. Experiments on a private Barrett's dysplasia dataset and the public SUN-SEG benchmark demonstrate improved temporal consistency and superior performance over baseline strategies.

Figures & tables

Figure 1: Example Barrett’s video showing mask propagation from the initial prompt and after an L2RP-suggested correction, with improved annotation quality.

Figure 2: Mean Dice Loss averaged across all videos at each frame, showing how segmentation error changes over time for different prompt types. Smoothed (20-frame window). Shaded regions indicate 95% confidence intervals.

Method	Barrett’s			SUN-SEG
	Mask	Box	Point	Mask	Box	Point
Initial Propagation	$0.7371\pm 0.0383$	$0.6323\pm 0.0645$	$0.5477\pm 0.0665$	$0.5466\pm 0.0225$	$0.5250\pm 0.0820$	$0.4726\pm 0.0595$
Midpoint	$0.7988\pm 0.0196$	$0.6448\pm 0.0617$	$0.5928\pm 0.0664$	$0.6820\pm 0.0190$	$0.5363\pm 0.0560$	$0.5686\pm 0.0532$
Random	$0.7899\pm 0.0241$	$0.6459\pm 0.0604$	$0.5823\pm 0.0648$	$0.6697\pm 0.0200$	$0.5503\pm 0.0560$	$0.5421\pm 0.0508$
EVA-VOS [ 1 ]	$0.8244\pm 0.0234$	$0.6537\pm 0.0565$	$0.6184\pm 0.0647$	$0.6882\pm 0.0244$	$0.5701\pm 0.0341$	$0.5838\pm 0.0498$
L2RP ( $\lambda_{\text{corr}}=0.01$ )	$\mathbf{0.8436\pm 0.0188^{***}}$	$\mathbf{0.6642\pm 0.0568^{***}}$	$\mathbf{0.6249\pm 0.0657^{***}}$	$\mathbf{0.7307\pm 0.0076^{***}}$	$\mathbf{0.5825\pm 0.0186^{***}}$	$\mathbf{0.6097\pm 0.0281^{***}}$

Table 1: Deferral strategy comparison: Dice score (mean

\pm

std) across prompt types on Barrett’s and SUN-SEG datasets. ∗∗∗ Statistically significant improvement over EVA-VOS (paired Wilcoxon signed-rank test,

p<10^{-6}

Interactive Medical Image Segmentation

Interactive medical image segmentation uses clinician prompts and iterative model feedback to refine anatomical masks and reduce manual annotation effort.

See all 48 papers

Toward Mask Annotation-Free Surgical Instrument Segmentation from Endoscopic Images Using Text-Prompted Segment Anything Model 3 (SAM3)
Aug 9, 2026Nakul Poudel, Richard Simon, Cristian A. LinteImage Segmentation Text-Guided Image Segmentation
Center for Imaging Science, Rochester Institute of Technology, Rochester, NY 14623, USA · Biomedical Engineering, Rochester Institute of Technology, Rochester, NY 14623, USA
Learning from Noisy Prompts: Saliency-Guided Prompt Distillation for Robust Segmentation with SAM
Apr 25, 2026Jingxuan Kang, Ziqi Zhang, Shaoming Zheng +9Medical Imaging Foundation Models Promptable Image Segmentation
Imperial College London · Beihang University · National Health Service +1
Active Learning for Efficient Annotation of Surgical Videos with Weak Supervision
Jul 14, 2026Manasa Dendukuri, Matjaz Jogan, Daniel A. Hashimoto +1Video Object Segmentation Human-in-the-Loop Annotation
1*GRASP Laboratory, University of Pennsylvania, 3330 Walnut Street, Philadelphia, PA, 19104, USA. · 2PCASO Laboratory, Dept. of Surgery, University of Pennsylvania, 3400 Spruce Street, Philadelphia, PA, 19104, USA. · 3Dept. of Computer and Information Science, University of Pennsylvania, 3330 Walnut Street, Philadelphia, PA, 19104, USA.

Aug 9, 2026J/K move · Enter open · S save

Toward Mask Annotation-Free Surgical Instrument Segmentation from Endoscopic Images Using Text-Prompted Segment Anything Model 3 (SAM3)

Nakul Poudel, Richard Simon, Cristian A. Linte

Center for Imaging Science, Rochester Institute of Technology, Rochester, NY 14623, USA · Biomedical Engineering, Rochester Institute of Technology, Rochester, NY 14623, USA

Image Segmentation Text-Guided Image Segmentation Instance Segmentation Medical Image Segmentation

Open paper AskPDF arXiv

Surgical instrument segmentation is a fundamental task for computer-assisted interventions, yet most existing methods rely on pixel-level annotations or manual spatial prompts, which limit scalability and automation. The recently introduced Segment Anything Model 3 (SAM3) offers a pathway to annotation-free, automatic segmentation via text-based prompting; however, the instrument name as a text prompt could not be directly used due to a large domain gap. To overcome these limitations, we propose a two-stage framework that achieves instance-level segmentation without requiring ground truth masks or manual interaction. In the first stage, we leverage a natural-language-aligned generic prompt - "tool" - to produce binary masks using SAM3's zero-shot capability. In the second stage, these masks are extended to instance-level by integrating a vision-language model (Qwen) that is fine-tuned on SAM3-generated masked regions for instrument classification. We evaluate our approach on the EndoVis 2017 and 2018 datasets. Results show that, while our two-stage approach does not reach the performance of current fully supervised methods, it significantly outperforms the direct use of SAM3 for instance-level instrument segmentation with text prompts. Overall, our findings highlight both the limitations and potential of SAM3, suggesting a promising direction toward annotation-free surgical instrument segmentation.

Abstract

Interactive Medical Image Segmentation

Explore similar work

Toward Mask Annotation-Free Surgical Instrument Segmentation from Endoscopic Images Using Text-Prompted Segment Anything Model 3 (SAM3)

Learning from Noisy Prompts: Saliency-Guided Prompt Distillation for Robust Segmentation with SAM

Active Learning for Efficient Annotation of Surgical Videos with Weak Supervision