ReGround-Surg: Reliability-Guided Anchor Grounding for Referring Surgical Video Segmentation

Jiaxin Wen; Ming Yin; Lu Liu; Zeyu Fu

cs.CVAug 25, 2026

ReGround-Surg: Reliability-Guided Anchor Grounding for Referring Surgical Video Segmentation

Authors: Jiaxin Wen, Ming Yin, Lu Liu, Zeyu Fu

Organizations: University of Exeter, Exeter, UK

Abstract

Referring surgical video segmentation requires segmenting a target instrument or tissue region across video frames according to a natural language expression. Recent Segment Anything Model 2 (SAM2) based two-stage methods (e.g., ReSurgSAM2) first ground the referred target in an initial or selected frame, then propagate the selected mask via tracking. Although effective, their performance is highly sensitive to the quality of the initial grounded mask: once an incorrect anchor is selected, subsequent tracking tends to propagate the error. This issue is especially challenging in surgical videos due to visually similar instruments, occlusion, and complex tissue-tool interactions. To address this issue, we propose ReGround-Surg, a lightweight reliability-guided anchor grounding framework to improve SAM2-based referring surgical video segmentation. It first predicts a text-conditioned spatial reliability map from the referring expression and current-frame visual features. The map is then reused in two complementary branches: a Gated Side Adapter enhances expression-relevant visual regions before text-to-vision fusion, while a Reliability-Weighted Vision-to-Text Attention module suppresses off-target visual evidence during prompt-token aggregation. Experiments on Ref-EndoVis17 and Ref-EndoVis18 show consistent improvements over state-of-the-art methods across three evaluation splits with negligible speed reduction. Code is publicly available at https://github.com/JiaxinWen1/ReGround-Surg.

Figures & tables

Figure 1: Visualisation of ReSurgSAM2 initial frame grounding and subsequent tracking failures under multi-instrument confusion, showing per-frame segmentation results for the expression “large needle driver is manipulating tool” across four conditions: ground truth, ReSurgSAM2, ReSurgSAM2 (w/o Stage II), and ReGround-Surg (Ours, w/o Stage II).

Figure 2: Overview of our proposed ReGround-Surg for Referring Surgical Video Segmentation.

Figure 3: Architecture of the proposed Text-Guided Reliability Gate, Gated Side Adapter (GSA), and Reliability-Weighted V2T Attention (RW-V2T).

Dataset	Training				Testing
Dataset	Seq.	Frame	Object	Pair	Seq.	Frame	Object	Pair
Ref-EndoVis17 (tool)	7	2100	20	4873	3	900	10	2265
Ref-EndoVis18 (tool)	11	1639	34	3787	4	596	15	1384
Ref-EndoVis18 (tissue)	11	1639	25	2995	4	596	7	807

Table 1: Dataset statistics.

	EV17 (tool)			EV18 (tool)			EV18 (tissue)
Method	$\mathcal{J\&F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$	$\mathcal{J}$	$\mathcal{F}$	FPS
ReferFormer † [ 23 ]	52.64	53.06	52.22	65.43	66.14	64.72	66.37	72.18	60.56	—
MUTR † [ 24 ]	54.09	53.80	54.38	66.92	67.71	66.13	66.14	72.37	59.91	—
RSVIS [ 20 ]	61.76	62.06	61.46	71.28	71.85	70.71	70.64	76.26	65.02	—
OnlineRefer [ 22 ]	60.35	60.92	59.78	70.17	70.93	69.41	68.83	74.51	63.15	—
RefSAM [ 10 ]	63.56	63.84	63.28	72.86	73.40	72.32	71.90	77.48	66.32	—

Table 2: Comparison with state of the art on Ref-EndoVis17 and Ref-EndoVis18.

\dagger

: offline method. Bold : best; underline : second best.

\Delta

: gain over ReSurgSAM2. FPS measured on NVIDIA A100-SXM4-80GB.

GSA	RW-V2T	$\mathcal{J\&F}$	$\mathcal{J}$	$\mathcal{F}$
		80.62	80.94	80.31
✓		82.43	82.68	82.18
	✓	81.22	81.40	81.04
✓	✓	83.71	83.83	83.60

Table 3: Module ablation on Ref-EndoVis18 (tool).

Table 7

Table 8

Method	Init. $\mathcal{J}$	Init. $\mathcal{F}$	Init. $\mathcal{J\&F}$	AnchorAcc	Final $\mathcal{J\&F}$
ReSurgSAM2 [ 11 ]	63.30	62.49	62.90	76.19	80.62
ReGround-Surg (Ours)	72.31	72.52	72.42	85.17	83.71
$\Delta$	+9.01	+10.03	+9.52	+8.98	+3.09

Table 8: Initial grounding analysis on Ref-EndoVis18 (tool). Metrics are computed on the CIFS-selected anchor frame before SAM2 tracking.

Figure 4: Initial anchor IoU vs. final video IoU on Ref-EndoVis17 and Ref-EndoVis18 (tool). Each point corresponds to one video-text pair.

Case type	ReSurgSAM2	Ours	Gain
Multiple instruments	75.33	79.48	+4.15
Same-type instruments	71.39	77.01	+5.62
Partial occlusion	75.43	79.60	+4.17

Table 9: Difficult-case analysis on Ref-EndoVis18, stratified by scene type (

\mathcal{J\&F}

Figure 5: Qualitative comparison on Ref-EndoVis18. Rows (top to bottom): ground truth; ReSurgSAM2 prediction; ours. Columns: multi-instrument confusion; partial occlusion; complete grounding failure.

Explore similar work

GroundedSurg: A Multi-Procedure Benchmark for Language-Conditioned Surgical Tool Segmentation
Mar 1, 2026Tajamul Ashraf, Abrar Ul Riyaz, Wasif Tak +4Medical Image Grounding Referring Image Segmentation
King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Gaash Research Lab, National Institute of Technology Srinagar, India · Thapar Institute of Engineering and Technology, India +1
UniSurgSAM: A Unified Promptable Model for Reliable Surgical Video Segmentation
Apr 4, 2026Haofeng Liu, Ziyue Wang, Alex Y. W. Kong +6Video Object Segmentation Surgical Video Understanding
Department of Biomedical Engineering, National University of Singapore, Singapore, Singapore · Department of Electrical and Computer Engineering, National University of Singapore, Singapore, Singapore · School of Computer Science, The University of Sheffield, Sheffield, UK +1
LD-RSVIS: A Large-Scale and Diverse Benchmark for Referring Surgical Video Instrument Segmentation
Sep 19, 2026Zan Wang, Yunhe Feng, Dong Nie +4Referring Video Object Segmentation Surgical Video Understanding
University of North Texas, Denton, USA · Meta Inc, USA

Mar 1, 2026J/K move · Enter open · S save

GroundedSurg: A Multi-Procedure Benchmark for Language-Conditioned Surgical Tool Segmentation

Tajamul Ashraf, Abrar Ul Riyaz, Wasif Tak, Tavaheed Tariq+3

King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Gaash Research Lab, National Institute of Technology Srinagar, India · Thapar Institute of Engineering and Technology, India+1

Medical Image Grounding Referring Image Segmentation Medical Image Benchmarks Medical VLMs Medical Image Segmentation

Open paper AskPDF arXiv

Clinically reliable perception of surgical scenes is essential for advancing intelligent, context-aware intraoperative assistance such as instrument handoff guidance, collision avoidance, and workflow-aware robotic support. Existing surgical tool benchmarks primarily evaluate category-level segmentation, requiring models to detect all instances of predefined instrument classes. However, real-world clinical decisions often require resolving references to a specific instrument instance based on its functional role, spatial relation, or anatomical interaction capabilities not captured by current evaluation paradigms. We introduce GroundedSurg, the first language-conditioned, instance-level surgical grounding benchmark. Each instance pairs a surgical image with a natural-language description targeting a single instrument, accompanied by structured spatial grounding annotations including bounding boxes and point-level anchors. The dataset spans ophthalmic, laparoscopic, robotic, and open procedures, encompassing diverse instrument types, imaging conditions, and operative complexities. By jointly evaluating linguistic reference resolution and pixel-level localization, GroundedSurg enables a systematic and realistic evaluation of vision-language models in clinically realistic multi-instrument scenes. Extensive experiments demonstrate substantial performance gaps across modern segmentation and VLMs, highlighting the urgent need for clinically grounded vision-language reasoning in surgical AI systems. Code and data are publicly available at https://github.com/gaash-lab/GroundedSurg

Abstract

Explore similar work

GroundedSurg: A Multi-Procedure Benchmark for Language-Conditioned Surgical Tool Segmentation

UniSurgSAM: A Unified Promptable Model for Reliable Surgical Video Segmentation

LD-RSVIS: A Large-Scale and Diverse Benchmark for Referring Surgical Video Instrument Segmentation