UniSurgSAM: A Unified Promptable Model for Reliable Surgical Video Segmentation
Authors: Haofeng Liu, Ziyue Wang, Alex Y. W. Kong, Guanyi Qin, Yunqiu Xu, Chang Han Low, Mingqi Gao, Lap Yan Lennon Chan, +1 more
Organizations: Department of Biomedical Engineering, National University of Singapore, Singapore, Singapore · Department of Electrical and Computer Engineering, National University of Singapore, Singapore, Singapore · School of Computer Science, The University of Sheffield, Sheffield, UK · Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China
Surgical video segmentation is fundamental to computer-assisted surgery. In practice, surgeons need to dynamically specify targets throughout extended procedures, using heterogeneous cues such as visual selections, textual expressions, or audio instructions. However, existing Promptable Video Object Segmentation (PVOS) methods are typically restricted to a single prompt modality and rely on coupled frameworks that cause optimization interference between target initialization and tracking. Moreover, these methods produce hallucinated predictions when the target is absent and suffer from accumulated mask drift without failure recovery. To address these challenges, we present UniSurgSAM, a unified PVOS model enabling reliable surgical video segmentation through visual, textual, or audio prompts. Specifically, UniSurgSAM employs a decoupled two-stage framework that independently optimizes initialization and tracking to resolve the optimization interference. Within this framework, we introduce three key designs for reliability: presence-aware decoding that models target absence to suppress hallucinations; boundary-aware long-term tracking that prevents mask drift over extended sequences; and adaptive state transition that closes the loop between stages for failure recovery. Furthermore, we establish a multi-modal and multi-granular benchmark from four public surgical datasets with precise instance-level masklets. Extensive experiments demonstrate that UniSurgSAM achieves state-of-the-art performance in real time across all prompt modalities and granularities, providing a practical foundation for computer-assisted surgery. Code and datasets will be available at https://jinlab-imvr.github.io/UniSurgSAM.
Figures & tables
Figure 1: Motivation. (A) Existing single-modal PVOS methods rely on a coupled decoder that causes optimization interference, with fragile open-loop pipelines prone to hallucinations and mask drift. (B) UniSurgSAM achieves unified PVOS through the decoupled two-stage framework for stable optimization, presence-aware decoding to suppress hallucinations, boundary-aware tracking to prevent mask drift, and adaptive state transition for closed-loop failure recovery. Textual prompt for the example: “large needle driver is manipulating tool on the right.” In (A) at time t , the model hallucinates a mask on the wrong object when the target is absent.
Figure 2: Overview of UniSurgSAM. The model adopts a decoupled two-stage framework that independently supports visual, textual, or audio prompts: Stage I performs promptable initialization from the given prompt, while Stage II conducts boundary-aware long-term tracking. For linguistic prompts, AST acts as a central controller that routes data to the detector or tracker via a selector, coordinating bidirectional switching through credible activation (Entry) and consensus-based fallback (Exit).
Figure 3: Illustration of Diversity-Driven Long-Term Memory.
Dataset
Video
Frame
Masklet (whole)
Mask (whole)
Masklet (part)
Mask (part)
Avg. Dur. (s)
Training Set
Uni-EndoVis17
7
2100
20
4873
55
11696
300
Uni-EndoVis18-I
11
1639
34
3787
78
7086
149
Uni-EndoVis18-T
11
1639
25
2995
-
-
149
Uni-RARP50
44
13043
367
42554
482
63749
296
Uni-SurgAI3.8K
36
2033
36
2033
-
-
56
Table 1: Dataset Statistics. Avg. Dur. denotes average duration
Method
Uni-EndoVis17
Uni-EndoVis18-I
Uni-RARP50
Uni-EndoVis18-T
Uni-SurgAI3.8K
FPS
J & F
J
F
J & F
J
F
J & F
J
F
J & F
J
F
J & F
J
F
SAM2 ( 2025 )
75.9
76.0
75.8
76.7
76.5
76.9
80.2
79.0
81.3
74.4
79.3
69.5
86.9
92.5
81.2
69
SAM2Long ( 2025 )
72.1
72.1
72.1
75.4
75.2
75.6
79.7
78.5
80.9
74.5
79.3
69.6
86.8
92.5
81.0
22
DAM4SAM ( 2025 )
76.2
76.1
76.2
73.1
72.8
73.3
77.5
76.3
78.6
74.5
79.4
69.5
86.8
92.5
81.1
63
SAMURAI ( 2024 )
75.3
75.3
75.3
74.5
74.2
74.8
77.9
77.9
77.8
72.9
77.5
68.3
86.6
92.4
80.7
22
MA-SAM2 ( 2025 )
75.5
75.5
75.4
78.6
78.3
78.9
82.5
81.3
83.7
74.7
79.6
69.8
86.7
92.4
81.0
65
Table 2: Quantitative comparison with Visual PVOS methods using 3-point prompts. Bold and underline denote best and second best.
Method
Setting
Uni-EndoVis17
Uni-EndoVis18-I
Uni-RARP50
Uni-EndoVis18-T
Uni-SurgAI3.8K
FPS
J & F
J
F
J & F
J
F
J & F
J
F
J & F
J
F
J & F
J
F
ReferFormer ( 2022 )
Offline
62.5
62.3
62.6
71.1
71.0
71.2
81.7
80.6
82.8
61.9
69.9
53.8
82.5
90.1
74.8
42
MUTR ( 2024 )
Offline
61.0
60.8
61.2
67.6
67.8
67.3
74.5
73.3
75.6
63.6
71.5
55.6
81.4
89.5
73.2
32
SurgRef ( 2026 )
Offline
62.3
62.0
62.5
70.3
70.4
70.2
76.2
75.3
77.1
65.3
71.5
59.1
81.6
89.3
73.9
40
RSVIS ( 2024 )
Semi-online
61.3
61.4
61.1
68.4
68.6
68.2
72.4
71.7
73.0
65.7
72.9
58.5
75.7
83.2
68.2
28
SAMWISE ( 2025 )
Semi-online
60.1
59.5
60.6
70.6
69.8
71.4
82.5
81.7
83.2
64.5
71.9
57.0
79.3
88.0
70.6
12
Table 3: Quantitative analysis of Linguistic PVOS (Textual and Audio). Bold and underline denote best and second best in Textual PVOS.
Figure 4: Qualitative comparison in Uni-EndoVis18 (left) and Uni-RARP50 (right) for Textual PVOS. FP and FN denote false positive and false negative predictions, respectively. Numbers in images denote time in seconds.
Figure 5: Qualitative comparison in Uni-EndoVis17 for Visual PVOS under three-point initialization.
Method
Uni-EndoVis17
Uni-EndoVis18-I
Uni-RARP50
J & F
J
F
J & F
J
F
J & F
J
F
ReferFormer
60.2
57.5
62.8
64.3
63.0
65.6
72.7
72.0
73.3
SAMWISE
63.7
60.1
67.2
67.6
65.4
63.2
77.9
77.5
78.3
RefSAM
66.0
64.6
67.4
69.3
67.8
66.2
74.7
74.2
75.1
ReSurgSAM2
69.1
67.1
71.0
71.7
69.5
73.8
78.7
77.4
79.9
UniSurgSAM
77.7
75.1
80.3
77.3
75.3
79.3
88.4
87.6
89.1
Table 4: Instrument part segmentation under Textual PVOS.
Decoupled Decoders
RPAD
BLT
AST
Uni- EndoVis17
Uni- EndoVis18-I
Uni- EndoVis18-T
68.8
74.5
72.2
√
71.3
76.8
73.5
√
√
74.5
78.7
74.6
√
√
√
77.4
81.5
76.7
√
√
√
78.5
82.1
75.5
√
√
√
√
82.4
85.1
79.8
Table 5: Ablation study of the main components ( J & F ).
Model Configuration
Uni- EndoVis17
Uni- EndoVis18-I
Uni- EndoVis18-T
Baseline
78.5
82.1
75.5
+ Boundary Decoding
79.7
83.5
78.2
+ Memory Integration
80.9
84.0
79.2
+ DLM (FIFO eviction)
81.6
84.5
79.4
+ DLM (full)
82.4
85.1
79.8
Table 6: Ablation on boundary-aware tracking components ( J & F ). Baseline: decoupled decoders + RPAD + AST.
Model Configuration
Uni- EndoVis17
Uni- EndoVis18-I
Uni- EndoVis18-T
Baseline
77.4
81.5
76.7
+ Credible Activation
79.8
83.2
78.1
+ Consensus Fallback
82.4
85.1
79.8
Table 7: Ablation on AST components under Textual PVOS ( J & F ).
Figure 6: Qualitative analysis of consensus-based fallback in AST. The system successfully detects tracking drift and triggers re-initialization.
Backbone
Uni-EndoVis17
Uni-EndoVis18-I
Uni-EndoVis18-T
FPS
J & F
J
F
J & F
J
F
J & F
J
F
Small
80.6
80.6
80.5
82.5
82.8
82.1
77.6
82.9
72.3
58
Base+
82.4
82.3
82.4
85.1
85.3
84.9
79.8
86.2
73.4
55
Table 8: Comparison across backbone scales under Textual PVOS.
Figure 7: Analysis of optimization interference.
Figure 8: Analysis of presence-aware decoding with negative sampling.
Referring surgical video segmentation requires segmenting a target instrument or tissue region across video frames according to a natural language expression. Recent Segment Anything Model 2 (SAM2) based two-stage methods (e.g., ReSurgSAM2) first ground the referred target in an initial or selected frame, then propagate the selected mask via tracking. Although effective, their performance is highly sensitive to the quality of the initial grounded mask: once an incorrect anchor is selected, subsequent tracking tends to propagate the error. This issue is especially challenging in surgical videos due to visually similar instruments, occlusion, and complex tissue-tool interactions. To address this issue, we propose ReGround-Surg, a lightweight reliability-guided anchor grounding framework to improve SAM2-based referring surgical video segmentation. It first predicts a text-conditioned spatial reliability map from the referring expression and current-frame visual features. The map is then reused in two complementary branches: a Gated Side Adapter enhances expression-relevant visual regions before text-to-vision fusion, while a Reliability-Weighted Vision-to-Text Attention module suppresses off-target visual evidence during prompt-token aggregation. Experiments on Ref-EndoVis17 and Ref-EndoVis18 show consistent improvements over state-of-the-art methods across three evaluation splits with negligible speed reduction. Code is publicly available at https://github.com/JiaxinWen1/ReGround-Surg.
Multimodal Large Language Models (MLLMs) have demonstrated strong image-level visual understanding and reasoning, yet their pixel-level perception across both images and videos remains limited. Foundation segmentation models such as the SAM series produce high-quality masks, but they rely on low-level visual prompts and cannot natively interpret complex conversational instructions. Existing segmentation MLLMs narrow this gap, but are usually specialized for either images or videos and rarely support both textual and visual prompts in one interface. We introduce X2SAM, a unified segmentation MLLM that extends any-segmentation capabilities from images to videos. Given conversational instructions and visual prompts, X2SAM couples an LLM with a Mask Memory module that stores guided vision features for temporally consistent video mask generation. The same formulation supports generic, open-vocabulary, referring, reasoning, grounded conversation generation, interactive, and visual grounded segmentation across image and video inputs. We further introduce the Video Visual Grounded (V-VGD) segmentation benchmark, which evaluates whether a model can segment object tracks in videos from interactive visual prompts. With a unified joint training strategy over heterogeneous image and video datasets, X2SAM delivers strong video segmentation performance, remains competitive on image segmentation benchmarks, and preserves general image and video chat ability.
Hao Wang, Limeng Qiao, Chi Zhang +4
Sun Yat-Sen University · 2 Peng Cheng Laboratory · 3 Meituan Inc
Surgical instrument segmentation is a fundamental task for computer-assisted interventions, yet most existing methods rely on pixel-level annotations or manual spatial prompts, which limit scalability and automation. The recently introduced Segment Anything Model 3 (SAM3) offers a pathway to annotation-free, automatic segmentation via text-based prompting; however, the instrument name as a text prompt could not be directly used due to a large domain gap. To overcome these limitations, we propose a two-stage framework that achieves instance-level segmentation without requiring ground truth masks or manual interaction. In the first stage, we leverage a natural-language-aligned generic prompt - "tool" - to produce binary masks using SAM3's zero-shot capability. In the second stage, these masks are extended to instance-level by integrating a vision-language model (Qwen) that is fine-tuned on SAM3-generated masked regions for instrument classification. We evaluate our approach on the EndoVis 2017 and 2018 datasets. Results show that, while our two-stage approach does not reach the performance of current fully supervised methods, it significantly outperforms the direct use of SAM3 for instance-level instrument segmentation with text prompts. Overall, our findings highlight both the limitations and potential of SAM3, suggesting a promising direction toward annotation-free surgical instrument segmentation.
Nakul Poudel, Richard Simon, Cristian A. Linte
Center for Imaging Science, Rochester Institute of Technology, Rochester, NY 14623, USA · Biomedical Engineering, Rochester Institute of Technology, Rochester, NY 14623, USA