Sounding Surgical Instrument Localisation through Audio-Visual Correspondence

Dominic Behling; Jonas Hein; Matthias Seibold

cs.SDOct 23, 2026

Sounding Surgical Instrument Localisation through Audio-Visual Correspondence

Authors: Dominic Behling, Jonas Hein, Matthias Seibold

Organizations: Research in Orthopedic Computer Science (ROCS), Balgrist University Hospital, University of Zurich, Zurich, Switzerland. · Eberhard Karls Universität Tübingen, Germany.

Abstract

Surgical scene understanding is a prerequisite for context-aware and intelligent systems in the operating room. Current methods rely mostly on visual data, while acoustic sensing has shown potential to provide complementary information on surgical activity. First approaches for the localisation of surgical activity relied on microphone arrays and acoustic beamforming. This work investigates whether the active instrument can be localized from a single microphone and a camera by learning the correspondence between the appearance and the sound of surgical instruments, without annotating its position. We adapt weakly supervised audio-visual localisation, which learns from the co-occurrence of image and sound, to the surgical setting. We record simulated surgical actions, sawing, drilling, and chiseling, and train the model on these recordings without any spatial annotation to produce a heatmap over the sounding tool. We extend the training objective by two terms: a silence term, which suppresses the output for silence and background noise, and a composite term, which trains the model to select the sounding one of two visible instruments. The model localizes the active instrument on an independent corpus recorded with a different camera and microphone, in which instrument positions were obtained by optical tracking. It separates tool activity from silence and background noise and can attribute sound correctly when two tools are present. Audio-visual correspondence learned from in-domain recordings connects the sound of a surgical action to the instrument that produces it, using a single microphone and a camera. This proof of concept introduces a novel paradigm that enables the detection and localisation of surgical activity without manual annotations.

Figures & tables

Figure 1: Schematic overview of the method. Training adds two generated samples, a pair with negative audio (silence, background noise) and a composite of cropped halves with different instruments. Cosine similarity between each cell of the visual embedding grid and the audio embedding yields the heatmap $S$ , which is upsampled at inference time.

Split	Corpus	Sequences	bounding boxes	Used for
Training	action camera	45	$\times$	training
Validation	action camera	6	$\checkmark$	validation
Two-tool test	action camera	3	$\checkmark$	test selection, silence
Localisation test	acoustic camera	21	$\checkmark$	test localisation

Table 1: Data splits. Bounding boxes are used for model selection and evaluation only.

	IoU				Distance [px]
Model	Saw	Chisel	Drill	All	Saw	Chisel	Drill	All
SLAVC Mo and Morgado (2022)	0.03	0.01	0.05	0.03	70	65	75	69
ACL-SSL Park et al. (2024)	0.04	0.02	0.02	0.03	47	28	77	37
Ours	$0.46\pm 0.04$	$0.14\pm 0.02$	$0.50\pm 0.04$	$0.36\pm 0.03$	$9\pm 1$	$23\pm 8$	$12\pm 1$	$12\pm 2$

Table 2: Localisation on the acoustic camera corpus (853 frames). IoU and median distance between the heatmap maximum and the center of the ground-truth box in px at

224\times 224

, per instrument and over all frames. Ours: mean

\pm

standard deviation over three seeds.

Panel of Figure 2 — Figure 2: Heatmaps of the proposed model on the acoustic camera corpus. Green: ground-truth box. Left: drill. Middle: chisel, where the model responds to the hand swinging the hammer as well as to the chisel (Sect. 5 ). Right: saw, where the empty hand of the assistant also receives part of the response.

Panel of Figure 3 — Figure 3: One frame of a two-tool test split under background noise (left), matched saw audio (middle), and swapped to chisel audio (right). Green: active saw, grey: idle chisel. For visualisation purposes, each heatmap is shown as the deviation from its mean, on one common colour scale.

				Localisation		Detection	Selection
$\mathcal{L}_{\mathrm{sil}}$	$\mathcal{L}_{\mathrm{area}}$	$\mathcal{L}_{\mathrm{comp}}$	Visual encoder	IoU	Dist. px	AUC	matched	swapped
✓	✓	✓	DINOv2	0.36	12	1.00	0.93	0.87
✓	✓	$\times$	DINOv2	0.35	9.4	1.00	0.67	0.46
✓	$\times$	✓	DINOv2	0.29	10.6	0.79	0.97	0.98
$\times$	✓	✓	DINOv2	0.36	9.9	0.04	0.87	0.96
$\times$	$\times$	$\times$	DINOv2	0.15	16.8	0.36	0.48	0.74

Table 3: Ablation of the loss terms and the visual encoder. The contrastive term is always used. First row: the proposed model as in Table 2 , mean over three seeds. The other rows are single runs.

Surgical Instrument Tracking

See all 17 papers

Mar 1, 2026

GroundedSurg: A Multi-Procedure Benchmark for Language-Conditioned Surgical Tool Segmentation

Clinically reliable perception of surgical scenes is essential for advancing intelligent, context-aware intraoperative assistance such as instrument handoff guidance, collision avoidance, and workflow-aware robotic support. Existing surgical tool benchmarks primarily evaluate category-level segmentation, requiring models to detect all instances of predefined instrument classes. However, real-world clinical decisions often require resolving references to a specific instrument instance based on its functional role, spatial relation, or anatomical interaction capabilities not captured by current evaluation paradigms. We introduce GroundedSurg, the first language-conditioned, instance-level surgical grounding benchmark. Each instance pairs a surgical image with a natural-language description targeting a single instrument, accompanied by structured spatial grounding annotations including bounding boxes and point-level anchors. The dataset spans ophthalmic, laparoscopic, robotic, and open procedures, encompassing diverse instrument types, imaging conditions, and operative complexities. By jointly evaluating linguistic reference resolution and pixel-level localization, GroundedSurg enables a systematic and realistic evaluation of vision-language models in clinically realistic multi-instrument scenes. Extensive experiments demonstrate substantial performance gaps across modern segmentation and VLMs, highlighting the urgent need for clinically grounded vision-language reasoning in surgical AI systems. Code and data are publicly available at https://github.com/gaash-lab/GroundedSurg

Tajamul Ashraf, Abrar Ul Riyaz, Wasif Tak +4

King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Gaash Research Lab, National Institute of Technology Srinagar, India · Thapar Institute of Engineering and Technology, India +1

Medical Image Grounding Referring Image Segmentation Medical Image Benchmarks Medical VLMs Medical Image Segmentation

Ask PDF

Oct 25, 2026

From Presence Labels to Bounding Boxes: Can Multimodal Large Language Models Scale Surgical Instrument Localization?

Spatial localization of surgical instruments through bounding-box annotations is a foundational component of surgical scene understanding, supporting detection, tracking, interaction analysis, and grounded vision-language applications. Yet existing annotations often cover selected frames, video subsets, or dataset variants rather than complete procedures. Manual extension is time-consuming and expert-dependent. Additionally, weakly supervised and detector-based pseudo-labels require task-specific training and may be imprecise. In this study, we design a scalable MLLM-based annotation framework that converts surgical frames and existing instrument-presence labels into accurate bounding boxes, and systematically analyze its reliability, cost, and design sensitivity. We instantiate the framework using six MLLMs and a multi-call configuration on laparoscopic cholecystectomy in Cholec80. Using a carefully designed stratification scheme, two annotators independently label 1,000 frames, and an expert surgeon reviews one annotation set to establish the reference. Results show that instrument-set F1 ranges from 88.2 to 92.7,whereas mIoU ranges from 0.180 to 0.735, showing that comparable recognition can conceal large localization differences. A structured multi-stage annotation pipeline achieves 0.784 mIoU, compared with 0.735 for the best single-call baseline and 0.814 human-human agreement. Using the best accuracy pipeline, we create and publicly release a spatial extension of Cholec80 comprising 285,977 bounding boxes across all 184,498 annotated frames from the 80 videos. The annotations and human references are made available on GitHub

Mohamed Hamdy, Muraam Abdel-Ghani, Fatmaelzahraa Ahmed +5

Department of Computer Science and Engineering, Qatar University, Doha, Qatar · Hamad Medical Corporation, Doha, Qatar · College of Medicine, Qatar University, Doha, Qatar

LLM-Assisted Annotation Medical Image Localization

Ask PDF

Jun 30, 2026

Dense Structural Priors for Sparse Functional Landmark Localization in Surgical Videos

Vision foundation models such as SAM 3 can provide transferable object-level structure across diverse surgical video conditions, but segmentation outputs do not explicitly encode the action-conditioned semantics that define functional surgical landmarks. Estimating instrument extent and geometry differs from localizing the tip or anchor relevant to clipping, grasping, or dissecting. We investigate vision foundation model-enabled sparse action-aware landmark localization, using zero-shot, point-prompted structural masks to provide dense instrument-level context without manual pixel-level mask annotations. We propose a lightweight refinement framework that uses SAM 3 as a structural prior. A coarse multi-frame network predicts tip and anchor prompts, generating non-oracle masks that are fused with visual and heatmap features to refine functional landmark predictions. We compare direct mask-augmented supervision, prediction-derived mask-prior refinement, and auxiliary mask supervision to examine how vision foundation model-derived structure should enter a precision-oriented localization system. Experiments on 7,867 clips from 60 surgical videos spanning YouTube, Cholec80, HeiChole, SurgVU, and CRCD evaluate the approach under heterogeneous conditions. Without manual pixel-level mask annotations for training, the proposed model achieves overall F1 scores of 72.4% for tip and 58.0% for anchor localization. Directly imposing masks on heatmap targets biases learning toward broad tool regions, whereas prediction-derived priors and auxiliary supervision provide effective intermediate structural guidance for action-dependent landmark prediction.

Chenyan Jing, Hao Ding, Lalithkumar Seenivasan +2

Johns Hopkins University, Baltimore, MD, USA

Surgical Video Understanding Medical Image Localization

Ask PDF

Abstract

Surgical Instrument Tracking

Explore similar work

GroundedSurg: A Multi-Procedure Benchmark for Language-Conditioned Surgical Tool Segmentation

From Presence Labels to Bounding Boxes: Can Multimodal Large Language Models Scale Surgical Instrument Localization?

Dense Structural Priors for Sparse Functional Landmark Localization in Surgical Videos