GroundedSurg: A Multi-Procedure Benchmark for Language-Conditioned Surgical Tool Segmentation
Authors: Tajamul Ashraf, Abrar Ul Riyaz, Wasif Tak, Tavaheed Tariq, Sonia Yadav, Moloud Abdar, Janibul Bashir
Organizations: King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Gaash Research Lab, National Institute of Technology Srinagar, India · Thapar Institute of Engineering and Technology, India · The University of Queensland, Australia
Clinically reliable perception of surgical scenes is essential for advancing intelligent, context-aware intraoperative assistance such as instrument handoff guidance, collision avoidance, and workflow-aware robotic support. Existing surgical tool benchmarks primarily evaluate category-level segmentation, requiring models to detect all instances of predefined instrument classes. However, real-world clinical decisions often require resolving references to a specific instrument instance based on its functional role, spatial relation, or anatomical interaction capabilities not captured by current evaluation paradigms. We introduce GroundedSurg, the first language-conditioned, instance-level surgical grounding benchmark. Each instance pairs a surgical image with a natural-language description targeting a single instrument, accompanied by structured spatial grounding annotations including bounding boxes and point-level anchors. The dataset spans ophthalmic, laparoscopic, robotic, and open procedures, encompassing diverse instrument types, imaging conditions, and operative complexities. By jointly evaluating linguistic reference resolution and pixel-level localization, GroundedSurg enables a systematic and realistic evaluation of vision-language models in clinically realistic multi-instrument scenes. Extensive experiments demonstrate substantial performance gaps across modern segmentation and VLMs, highlighting the urgent need for clinically grounded vision-language reasoning in surgical AI systems. Code and data are publicly available at https://github.com/gaash-lab/GroundedSurg
Figures & tables
Dataset
Domain
Multi-Proc.
Language
Spatial
Disambig.
Prompt
CaDIS [ 8 ]
Cataract
✗
✗
✗
✗
✗
AutoLaparo [ 30 ]
Laparoscopic
✗
✗
✗
✗
✗
SISVE [ 32 ]
Gastrectomy
✗
✗
✓
✓
✗
EndoVis [ 26 ]
Robotic
✗
✗
✗
✗
✗
CholecSeg8k [ 10 ]
Cholecystectomy
✗
✗
✗
✗
✗
CholecInstanceSeg [ 1 ]
Cholecystectomy
✗
✗
✗
✓
✗
Table 1: Comparison of surgical tool datasets with respect to grounding capabilities. Our benchmark supports multi-procedure, language conditioning, explicit spatial grounding, instance-level disambiguation, and prompt-based evaluation.
Figure 1: Overview of GroundedSurg . (a) Existing datasets focus on category-level segmentation without language conditioning or instance-level grounding. (b) GroundedSurg introduces natural-language queries with structured spatial annotations for query-conditioned instrument localization. (c) Baseline results reveal substantial performance gaps, highlighting the challenges of grounded surgical perception.
Figure 2: GroundedSurg benchmark pipeline. Surgical images are paired with initial prompts and processed by a vision–language model to generate structured instrument descriptions. All 1071 queries are human and clinician verified for semantic correctness and ambiguity removal. Final annotations are stored in a standardized JSON schema with spatial grounding (bounding box and center point) and segmentation masks.
Table 4: Quantitative comparison of vision-language models on GroundedSurg under a unified query-conditioned segmentation protocol. Best results are shown in bold , second-best are underlined .
Table 5: Sensitivity analysis under two prompt formulations. Best results per metric are shown in bold , second-best are underlined .
Figure 3: Qualitative comparison on GroundedSurg showing that reasoning-oriented models produce more spatially precise masks than general-purpose models when projecting structured localization outputs onto a frozen segmentation backend, particularly in multi-instrument and visually cluttered scenes.
Global College, Shanghai Jiao Tong University, Shanghai, China · Institute for Medical Engineering & Science, Massachusetts Institute of Technology, Cambridge, MA, USA · Department of Biomedical Engineering and Department of Electrical and Computer Engineering, National University of Singapore, Singapore +11