SIRA: Reasoning-Aware Surgical Instrument Segmentation via Query-Anchored Alignment

Zhibo Zhang; Qijie Wang; Zengqiang Yan

cs.CVSep 18, 2026

SIRA: Reasoning-Aware Surgical Instrument Segmentation via Query-Anchored Alignment

Authors: Zhibo Zhang, Qijie Wang, Zengqiang Yan

Organizations: School of Electronic Information and Communications, Huazhong University of Science and Technology

Abstract

Surgical instrument segmentation (SIS) plays a critical role in robotic assistance and surgical workflow analysis. However, most existing SIS methods formulate segmentation as a category-driven localization problem, limiting their ability to capture procedural context and task-dependent semantics in surgical workflows. We introduce Reasoning-Aware Surgical Instrument Segmentation (RA-SIS), a task formulation that frames segmentation as query-conditioned inference under surgical context. To benchmark this setting, we construct SurgRS, a surgical reasoning segmentation dataset consisting of 41,000 image-text pairs, which aligns instance-level masks with structured query-answer supervision to enable semantic grounding at the pixel level. Based on SurgRS, we propose Surgical Instrument Reasoning and Segmentation Assistant (SIRA), a multimodal framework that disentangles target-level and query-level semantics and integrates them with visual features through query-anchored dual alignment. By aligning query semantics with spatial features and segmentation prompts, SIRA enhances semantic-visual consistency in mask prediction. Extensive experiments on SurgRS demonstrate improvements over existing reasoning-aware baselines. Code is available at https://github.com/linxir226/SIRA.

Figures & tables

Figure 1: Overall construction pipeline of SurgRS.

Code	Type	Definition
LT	Leading Tool	Identify the instrument executing the maneuver.
SE	Support Exposure	Analyze exposure or counter-traction roles.
SGT	Suture Guide/Tension	Identify instruments guiding or tensioning sutures.
PAI	Procedural Action Inference	Infer the current action from visible instruments.
NSI	Next-Step Importance	Identify instruments important for the next action.
NDL	Near/Deep Layout	Analyze spatial distribution in surgical field.

Table 1: SurgRS reasoning query taxonomy.

Figure 2: Overview of the proposed SIRA framework.

Method	Overall			Reasoning Query Type
Method	gIoU	cIoU	Dice	LT	SE	SGT	PAI	NSI	NDL	OR	PL	CR	OT
LISA	65.05	74.32	74.21	79.34	71.14	69.24	73.23	74.55	74.43	68.30	70.86	58.81	54.35
GSVA	65.30	75.08	74.39	79.14	72.20	69.52	73.03	74.52	74.89	67.21	69.64	60.68	53.94
PathMR	66.42	75.14	76.25	80.18	74.11	72.39	75.91	76.40	75.69	71.15	73.37	63.20	61.10
VRS-HQ	70.79	80.60	79.61	83.53	78.57	75.40	78.53	78.92	79.78	72.31	76.29	64.97	59.82
SIRA	72.24	81.78	80.93	86.82	80.05	76.76	79.53	80.02	81.05	74.42	76.78	65.58	61.61

Table 2: Quantitative comparison of reasoning segmentation performance on SurgRS (% for gIoU, cIoU, and Dice).

Figure 3: Exemplar qualitative results of different approaches on SurgRS.

Baseline	QVA	TQA	gIoU	cIoU	Dice
$\bullet$	$\circ$	$\circ$	70.79	80.60	79.61
$\bullet$	$\circ$	$\bullet$	71.40	81.23	80.09
$\bullet$	$\bullet$	$\circ$	71.93	81.62	80.59
$\bullet$	$\bullet$	$\bullet$	72.24	81.78	80.93

Table 3: Ablation study of QVA and TQA on SurgRS (%).

Explore similar work

GroundedSurg: A Multi-Procedure Benchmark for Language-Conditioned Surgical Tool Segmentation
Mar 1, 2026Tajamul Ashraf, Abrar Ul Riyaz, Wasif Tak +4Medical Image Grounding Referring Image Segmentation
King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Gaash Research Lab, National Institute of Technology Srinagar, India · Thapar Institute of Engineering and Technology, India +1
TEDi: Temporal Memory-Enhanced and Denoising Transformer for Surgical Instrument Segmentation
Sep 15, 2026Jiahong Yuan, Weiming Mi, Tao Zhang +1Video Object Segmentation Medical Image Segmentation
Department of Automation, Tsinghua University, Beijing, China · Surgical Planning Laboratory, Brigham and Women’s Hospital, Harvard Medical School, United States
Towards Unified Surgical Scene Understanding:Bridging Reasoning and Grounding via MLLMs
May 13, 2026Jincai Huang, Shihao Zou, Yuchen Guo +5Surgical Video Understanding Vision-Language Grounding
Southern University of Science and Technology · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Northwestern University +4

Mar 1, 2026J/K move · Enter open · S save

GroundedSurg: A Multi-Procedure Benchmark for Language-Conditioned Surgical Tool Segmentation

Tajamul Ashraf, Abrar Ul Riyaz, Wasif Tak, Tavaheed Tariq+3

King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Gaash Research Lab, National Institute of Technology Srinagar, India · Thapar Institute of Engineering and Technology, India+1

Medical Image Grounding Referring Image Segmentation Medical Image Benchmarks Medical VLMs Medical Image Segmentation

Open paper AskPDF arXiv

Clinically reliable perception of surgical scenes is essential for advancing intelligent, context-aware intraoperative assistance such as instrument handoff guidance, collision avoidance, and workflow-aware robotic support. Existing surgical tool benchmarks primarily evaluate category-level segmentation, requiring models to detect all instances of predefined instrument classes. However, real-world clinical decisions often require resolving references to a specific instrument instance based on its functional role, spatial relation, or anatomical interaction capabilities not captured by current evaluation paradigms. We introduce GroundedSurg, the first language-conditioned, instance-level surgical grounding benchmark. Each instance pairs a surgical image with a natural-language description targeting a single instrument, accompanied by structured spatial grounding annotations including bounding boxes and point-level anchors. The dataset spans ophthalmic, laparoscopic, robotic, and open procedures, encompassing diverse instrument types, imaging conditions, and operative complexities. By jointly evaluating linguistic reference resolution and pixel-level localization, GroundedSurg enables a systematic and realistic evaluation of vision-language models in clinically realistic multi-instrument scenes. Extensive experiments demonstrate substantial performance gaps across modern segmentation and VLMs, highlighting the urgent need for clinically grounded vision-language reasoning in surgical AI systems. Code and data are publicly available at https://github.com/gaash-lab/GroundedSurg

Abstract

Explore similar work

GroundedSurg: A Multi-Procedure Benchmark for Language-Conditioned Surgical Tool Segmentation

TEDi: Temporal Memory-Enhanced and Denoising Transformer for Surgical Instrument Segmentation

Towards Unified Surgical Scene Understanding:Bridging Reasoning and Grounding via MLLMs