LD-RSVIS: A Large-Scale and Diverse Benchmark for Referring Surgical Video Instrument Segmentation

Zan Wang; Yunhe Feng; Dong Nie; Oluwatosin Oluwadare; Kewei Sha; Yan Huang; Heng Fan

cs.CVSep 19, 2026

LD-RSVIS: A Large-Scale and Diverse Benchmark for Referring Surgical Video Instrument Segmentation

Authors: Zan Wang, Yunhe Feng, Dong Nie, Oluwatosin Oluwadare, Kewei Sha, Yan Huang, Heng Fan

Organizations: University of North Texas, Denton, USA · Meta Inc, USA

Abstract

Referring surgical video instrument segmentation (RSVIS) aims at segmenting the instrument in a surgical video, given a textual description. Despite recent progress, current models are trained and assessed on relatively small-scale benchmarks, hindering the development of more general RSVIS. In addition, existing benchmarks support only the single-target expression that refers to one instrument in the video, while overlooking multi-target and no-target referring expressions, restricting the applicability of RSVIS in practical scenarios. Addressing these issues, we propose LD-RSVIS, a new benchmark aiming to facilitate more robust and general RSVIS. Specifically, LD-RSVIS consists of 3,536 surgical videos with 1.09 million frames and covers a broad set of 30 instrument classes from 25 various procedures. By including abundant videos and classes, LD-RSVIS could benefit large-scale training and evaluation of more general RSVIS methods. Besides, unlike existing datasets, LD-RSVIS offers diverse referring settings, including no-target, single-target, and multi-target expressions, which enables the development of more practical RSVIS models in real applications. In order to ensure high-quality annotations, all videos in LD-RSVIS are manually labeled with multiple rounds of inspection and refinement. To our knowledge, LD-RSVIS is the largest and most diverse benchmark for RSVIS. To analyze LD-RSVIS and to provide comparison for future research, we evaluate 12 representative methods, and the results reveal that more efforts are required for improvements. To encourage future research, we present a simple yet effective RSVIS method, dubbed Cascade-RSVIS, that first mines target-specific cues using the complementary multi-cue text information and then employs such cues and textual information for segmentation, achieving promising performance. Our benchmark and code will be released.

Figures & tables

Figure 1: Comparison of existing RSVIS benchmarks. LD-RSVIS is larger than all other benchmarks. Best viewed in color for all figures .

Figure 2: Illustration of different referring settings in RSVIS. Image (a): Single-target expression in LD-RSVIS and other benchmarks; Image (b): Multi-target expression in our LD-RSVIS; Image (c): No-target expression in LD-RSVIS. All examples are from LD-RSVIS.

								Referring Expression
Benchmark	Year	Videos	Total Frames	Instru. Classes	Avg. Vid. Length	Num. of Expressions	Num. of Masks	ST	MT	NT
EndoVis-RS ( Wang et al., 2024 )	2024	32	4.8 K	9	150	n/a	9.8 K	✓	✗	✗
Ref-EndoVis ( Liu et al., 2025 )	2025	25	5.2 K	9	209	n/a	12.3 K	✓	✗	✗
UniPVOS ( Liu et al., 2026 )	2026	130	25.0 K	n/a	195	1,367	178.8 K	✓	✗	✗
Surg-IMotion ( Wei et al., 2026 )	2026	319	21.0 K	n/a	67	718	n/a	✓	✗	✗
LD-RSVIS (Ours)	2026	3,536	1.09 M	30	308	58,666	3.85 M	✓	✓	✓

Table 1: Comparison of our LD-RSVIS and current RSVIS datasets. “NT”, “ST”, and “MT” indicate no-target, single-target, and multi-target, respectively. “n/a” denotes that data is not available. The video frame rate of all datasets is 1 frame per second (fps).

Panel of Figure 3 — Figure 3: Representative statistics of LD-RSVIS, comprising distributions of video length, referring expression length, mask area ratio, and wordcloud of referring expressions.

	Videos	Total Frames	Avg. Length	Num. of Expre.	Num. of Masks
LD-RSVIS ${}_{\text{Tra}}$	2,853	880,080	308	47,500	3,135,866
LD-RSVIS ${}_{\text{Tst}}$	683	207,282	303	11,166	711,398

Table 2: Comparison of training and test sets.

Figure 4: Overview of Cascade-RSVIS that first mines target-relevant context via TCM and then leverages such context together with the referring expression for segmentation through CGS.

	Overall			MTE			STE			NTE
	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}$ & $\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}$ & $\mathcal{F}$	$\mathcal{J}$ & $\mathcal{F}$
VIS-Net ( Wang et al., 2024 ) [T-MI’24]	20.8	18.6	19.7	19.7	15.5	17.6	24.4	22.1	23.3	14.7
ReferFormer ( Wu et al., 2022 ) [CVPR’22]	19.0	23.6	21.3	23.3	31.9	27.6	21.2	26.1	23.7	10.3
OnlineRefer ( Wu et al., 2023 ) [ICCV’23]	20.4	24.2	22.3	27.8	34.0	30.9	21.5	26.0	23.7	10.9
LoSh ( Yuan et al., 2024 ) [CVPR’24]	22.5	28.1	25.3	26.4	38.8	32.6	26.0	31.0	28.5	11.7
LMPM ( Ding et al., 2023 ) [ICCV’23]	25.5	27.3	26.4	37.4	39.0	38.2	21.0	23.8	22.4	22.7

Table 3: Comparison with existing methods on LD-RSVIS

{}_{\text{Tst}}

. “MTE”, “STE”, and “NTE” indicate evaluations on multi-target, single-target, and no-target referring expressions, respectively.

Table 8

	Num. of queries $N_{q}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$
❶	$N_{q}=1$	34.9	38.1	36.5
❷	$N_{q}=3$	36.1	38.7	37.4
❸	$N_{q}=5$	35.5	38.3	36.9
❹	$N_{q}=7$	35.3	38.1	36.7

Table 8: Ablation on

N_{q}

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Illustration of the construction pipeline of LD-RSVIS.

Figure 6: Additional annotation examples in LD-RSVIS.

Source Dataset	Task of the Source Dataset	Selected Videos	Selected Frames
EndoVis17 ( Allan et al., 2019 )	surgical scene segmentation and/or detection	18	3,000
EndoVis18 ( Allan et al., 2020 )	surgical scene segmentation and/or detection	15	2,235
Cholec80 ( Twinanda et al., 2017 )	surgical workflow understanding	318	98,620
Endoscapes2023 ( Murali et al., 2023 )	surgical scene segmentation and/or detection	181	55,840
HeiChole ( Wagner et al., 2023 )	surgical workflow understanding	170	52,460
PhaKIR ( Rueckert et al., 2026 )	surgical phase and action recognition	60	18,640

Appendix

Table 9: Composition of selected data from source datasets for constructing LD-RSVIS.

	MTE	STE	NTE	Total
LD-RSVIS ${}_{\text{Tra}}$	11,412	28,171	7,917	47,500
LD-RSVIS ${}_{\text{Tst}}$	2,732	6,573	1,861	11,166
LD-RSVIS	14,144	34,744	9,778	58,666

Appendix

Table 10: Distribution of different types of referring expression in LD-RSVIS.

	EndoVis-RS						Ref-EndoVis (tool)
	RS17			RS18			Ref17			Ref18
Methods	oIoU	mIoU	mAP	oIoU	mIoU	mAP	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$
VIS-Net ( Wang et al., 2024 ) [T-MI’24]	65.5	70.1	53.8	74.2	72.3	60.1	61.4	61.1	61.2	68.6	68.2	68.4
SurgRef ( Wei et al., 2026 ) [AAAI’26]	66.4	70.8	54.7	74.8	73.4	60.8	70.8	69.6	70.2	76.1	74.7	75.4
ReSurgSAM2 ( Liu et al., 2025 ) [MICCAI’25]	68.7	72.9	57.2	76.5	75.3	62.9	77.8	77.7	77.7	80.9	80.3	80.6
ReferMo ( Liang et al., 2026 ) [CVPR’26]	69.4	73.4	57.6	77.2	75.8	63.3	78.2	78.8	78.5	81.0	81.4	81.2

Appendix

Table 11: Comparison on existing public RSVIS benchmarks. oIoU and mIoU denote Overall IoU and Mean IoU, respectively.

Figure 7: Qualitative comparison of Cascade-RSVIS with representative methods on LD-RSVIS. Image (a) shows the results of a single target case, while image (b) shows a multi-target case.

Explore similar work

Aug 25, 2026

ReGround-Surg: Reliability-Guided Anchor Grounding for Referring Surgical Video Segmentation

Referring surgical video segmentation requires segmenting a target instrument or tissue region across video frames according to a natural language expression. Recent Segment Anything Model 2 (SAM2) based two-stage methods (e.g., ReSurgSAM2) first ground the referred target in an initial or selected frame, then propagate the selected mask via tracking. Although effective, their performance is highly sensitive to the quality of the initial grounded mask: once an incorrect anchor is selected, subsequent tracking tends to propagate the error. This issue is especially challenging in surgical videos due to visually similar instruments, occlusion, and complex tissue-tool interactions. To address this issue, we propose ReGround-Surg, a lightweight reliability-guided anchor grounding framework to improve SAM2-based referring surgical video segmentation. It first predicts a text-conditioned spatial reliability map from the referring expression and current-frame visual features. The map is then reused in two complementary branches: a Gated Side Adapter enhances expression-relevant visual regions before text-to-vision fusion, while a Reliability-Weighted Vision-to-Text Attention module suppresses off-target visual evidence during prompt-token aggregation. Experiments on Ref-EndoVis17 and Ref-EndoVis18 show consistent improvements over state-of-the-art methods across three evaluation splits with negligible speed reduction. Code is publicly available at https://github.com/JiaxinWen1/ReGround-Surg.

Jiaxin Wen, Ming Yin, Lu Liu +1

University of Exeter, Exeter, UK

Referring Video Object Segmentation Surgical Video Understanding Visual Grounding

Ask PDF

Mar 1, 2026

GroundedSurg: A Multi-Procedure Benchmark for Language-Conditioned Surgical Tool Segmentation

Clinically reliable perception of surgical scenes is essential for advancing intelligent, context-aware intraoperative assistance such as instrument handoff guidance, collision avoidance, and workflow-aware robotic support. Existing surgical tool benchmarks primarily evaluate category-level segmentation, requiring models to detect all instances of predefined instrument classes. However, real-world clinical decisions often require resolving references to a specific instrument instance based on its functional role, spatial relation, or anatomical interaction capabilities not captured by current evaluation paradigms. We introduce GroundedSurg, the first language-conditioned, instance-level surgical grounding benchmark. Each instance pairs a surgical image with a natural-language description targeting a single instrument, accompanied by structured spatial grounding annotations including bounding boxes and point-level anchors. The dataset spans ophthalmic, laparoscopic, robotic, and open procedures, encompassing diverse instrument types, imaging conditions, and operative complexities. By jointly evaluating linguistic reference resolution and pixel-level localization, GroundedSurg enables a systematic and realistic evaluation of vision-language models in clinically realistic multi-instrument scenes. Extensive experiments demonstrate substantial performance gaps across modern segmentation and VLMs, highlighting the urgent need for clinically grounded vision-language reasoning in surgical AI systems. Code and data are publicly available at https://github.com/gaash-lab/GroundedSurg

Tajamul Ashraf, Abrar Ul Riyaz, Wasif Tak +4

King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Gaash Research Lab, National Institute of Technology Srinagar, India · Thapar Institute of Engineering and Technology, India +1

Medical Image Grounding Referring Image Segmentation Medical Image Benchmarks Medical VLMs Medical Image Segmentation

Ask PDF

Apr 20, 2026

Weakly-Supervised Referring Video Object Segmentation through Text Supervision

Referring video object segmentation (RVOS) aims to segment the target instance in a video, referred by a text expression. Conventional approaches are mostly supervised learning, requiring expensive pixel-level mask annotations. To tackle it, weakly-supervised RVOS has recently been proposed to replace mask annotations with bounding boxes or points, which are however still costly and labor-intensive. In this paper, we design a novel weakly-supervised RVOS method, namely WSRVOS, to train the model with only text expressions. Given an input video and the referring expression, we first design a contrastive referring expression augmentation scheme that leverages the captioning capabilities of a multimodal large language model to generate both positive and negative expressions. We extract visual and linguistic features from the input video and generated expressions, then perform bi-directional vision-language feature selection and interaction to enable fine-grained multimodal alignment. Next, we propose an instance-aware expression classification scheme to optimize the model in distinguishing positive from negative expressions. Also, we introduce a positive-prediction fusion strategy to generate high-quality pseudo-masks, which serve as additional supervision to the model. Last, we design a temporal segment ranking constraint such that the overlaps between mask predictions of temporally neighboring frames are required to conform to specific orders. Extensive experiments on four publicly available RVOS datasets, including A2D Sentences, J-HMDB Sentences, Ref-YouTube-VOS, and Ref-DAVIS17, demonstrate the superiority of our method. Code is available at https://github.com/viscom-tongji/WSRVOS.

Miaojing Shi, Jun Huang, Zijie Yue +1

College of Electronic and Information Engineering, Tongji University

Video Object Segmentation Referring Video Object Segmentation Weakly Supervised Learning

Ask PDF