cs.CVMar 17, 2026

Speak, Segment, Track, Navigate: An Interactive System for Video-Guided Skull-Base Surgery

Authors: Jecia Z. Y. Mao, Francis X. Creighton, Russell H. Taylor, Manish Sahu

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We introduce a speech-guided embodied agent framework for video-guided skull base surgery that dynamically executes perception and image-guidance tasks in response to surgeon queries. The proposed system integrates natural language interaction with real-time visual perception directly on live intraoperative video streams, thereby enabling surgeons to request computational assistance without disengaging from operative tasks. Unlike conventional image-guided navigation systems that rely on external optical trackers and additional hardware setup, the framework operates purely on intraoperative video. The system begins with interactive segmentation and labeling of the surgical instrument. The segmented instrument is then used as a spatial anchor that is autonomously tracked in the video stream to support downstream workflows, including anatomical segmentation, interactive registration of preoperative 3D models, monocular video-based estimation of the surgical tool pose, and image guidance through real-time anatomical overlays. We evaluate the proposed system in video-guided skull base surgery scenarios and benchmark its tracking performance against a commercially available optical tracking system. Across three experimental trials, the hybrid vision-based method achieved a mean absolute tool-tip position error of 2.32 Plus/Minus 1.10 mm in the camera frame, with inter-frame yaw and pitch propagation discrepancies of 0.18 Plus/Minus 0.25° and 0.21 Plus/Minus 0.30°, respectively. The system completes tool segmentation and anatomy registration within approximately two minutes, substantially reducing setup complexity relative to conventional tracking workflows. These results demonstrate that speech-guided embodied agents can provide accurate spatial guidance while improving workflow integration and enabling rapid deployment of video-guided surgical systems.

Figures & tables

Fig. 1 : System overview of the embodied surgical agent. The surgeon interacts through a hands-free interface (speech-to-text) that issues high-level commands to the front end , which orchestrates live video streaming, tool/anatomy segmentation, pose tracking, and anatomy registration. Intermediate outputs (masks, pose hypotheses, and registered anatomy overlays) are persisted in a streaming memory and can be retrieved on demand to support iterative refinement and rapid task switching without disrupting the surgical workflow. The back end executes modular perception and geometry components—including promptable segmentation, temporal mask propagation, surgical navigation, and pose/registration solvers—to produce stable tool state interpretation and anatomy-aligned navigation overlays in the endoscopic view.
Fig. 2 : Speech-driven tool segmentation with streaming memory and catch-up propagation. A voice command triggers GSAM to generate candidate masks at time t0t_{0} . After the surgeon confirms a proposal, the selected mask is stored and used to seed propagation through buffered intermediate frames, producing an updated mask at the latest time tnt_{n} so online tracking can resume without interrupting tool motion.
Fig. 3 : Interactive anatomy segmentation and refinement with event-triggered prompt retrieval. A voice command initiates anatomy segmentation while the system buffers the drill-tip trajectory. When the surgeon says “Done,” the buffered trajectory is converted into spatial prompts for SAM to generate an initial mask. Refinement is then triggered by a “Click” command on regions to remove, which adds prompts for mask update. The final anatomy mask seeds CUTIE for temporal propagation. Blue arrows indicate segmentation-mode prompting and trajectory retrieval, while purple arrows indicate refinement interactions and SAM updates.
Fig. 4 : Interactive anatomy registration workflow using the embodied surgical agent. The surgeon initiates registration via a voice command (“Register the anatomy”) and sequentially identifies four anatomical landmarks in the video stream (“landmark 1–4”). The embodied surgical agent records the selected 2D landmarks in the memory module and associates them with corresponding 3D anatomical mesh points. After collecting sufficient correspondences, the system performs pose estimation to compute the transformation between the camera and anatomy frames. The registered anatomy is then overlaid onto the live surgical scene, enabling consistent spatial alignment and visualization during the procedure.
Fig. 5 : Surgical navigation with registered anatomical overlay. After registration, the segmented anatomical structures are transformed into the camera frame and rendered directly onto the live surgical view. Color-coded regions denote critical structures (e.g., facial nerve, cochlear nerve, vestibular aqueduct), providing spatial context within the operative field. The highlighted region illustrates how the system enhances intraoperative awareness by overlaying anatomy onto the exposed surgical cavity, supporting precise and anatomy-aware tool manipulation.
Fig. 6 : Stereo vs. fore-ground conditioned monocular depth for surgical tool reconstruction. Stereo depth estimates degrade under narrow baseline, leading to noisy 3D reconstructions, whereas DepthAnything v2 constrained to the segmented fore-ground produces denser, more consistent depth and a smoother tool point cloud.
Fig. 7 : Foreground-conditioned monocular depth scaling for metric tool reconstruction. From the real-time video stream, the mask tracking module enables a foreground-masked frame. The registered anatomy provides a reliable metric depth reference, enabling an affine scale alignment of DepthAnything v2 predictions to obtain a scaled depth map in millimeters for both anatomy and tool regions.
Fig. 8 : 3D tool pose initialization from mask geometry and metric depth. From the tracked tool mask and foreground-conditioned metric depth, the 2D branch extracts the principal direction dMC\mathbf{d}_{M}^{C} and apparent length ℓmaskpx\ell^{px}_{mask} , while the 3D branch back-projects the masked depth to obtain the tool point set PtoolC\mathcal{P}^{C}_{\mathrm{tool}} , a coarse PCA axis prior d~3DC\tilde{\mathbf{d}}^{C}_{3D} , and the tip position ptipC\mathbf{p}^{C}_{\mathrm{tip}} . From the CAD model, we use the mesh tip ptipmesh\mathbf{p}^{\mathrm{mesh}}_{\mathrm{tip}} , tool length LL , and canonical local axis alocal\mathbf{a}_{\mathrm{local}} . The pose solver computes the ratio r=ℓmaskpx/ℓmeshpxr=\ell^{px}_{mask}/\ell^{px}_{mesh} , refines the camera-frame axis d3DC\mathbf{d}^{C}_{3D} using the 2D projection constraint and coarse 3D prior, and aligns the mesh by (R(d3DC),tC)(R(\mathbf{d}^{C}_{3D}),\mathbf{t}^{C}) to recover TmeshCT^{C}_{\mathrm{mesh}} , which is then rendered as the tool overlay.
Fig. 9 : Experimental setup for quantitative comparison to optical tracking. A surgical microscope provides the primary operating view and streams video to a large monitor for real-time visualization. The main display shows the live video with real-time tool and anatomy segmentation overlays, while the inset view visualizes the vision-tracked tool mesh together with overlaid critical anatomical structures. A microphone captures spoken commands, while an external speaker provides audio feedback from the system. The optical tracker and the proposed vision-based pipeline operate simultaneously; both outputs are time-matched and expressed in a common anatomy coordinate frame using the same anatomy registration.
 Manual ClickingVirtual Cursor
Trial 13.43 px (0.22 mm)(0.22\ \mathrm{mm})3.63 px (0.23 mm)(0.23\ \mathrm{mm})
Trial 23.03 px (0.18 mm)(0.18\ \mathrm{mm})2.48 px (0.15 mm)(0.15\ \mathrm{mm})
Trial 32.28 px (0.15 mm)(0.15\ \mathrm{mm})11.64 px (0.74 mm)(0.74\ \mathrm{mm})
Mean2.91 px (0.18 mm)(0.18\ \mathrm{mm})5.92 px (0.37 mm)(0.37\ \mathrm{mm})
TABLE I : Registration pipeline RMSE for manual clicking and virtual cursor interaction.
   Translation error in camera frameRotation propagation discrepancy in camera frame
MethodTrialSpeed (FPS)∣Δx∣|\Delta x| (mm)∣Δy∣|\Delta y| (mm)∣Δz∣|\Delta z| (mm)∥Δp∥2\|\Delta\mathbf{p}\|_{2} (mm)∣Δyprop∣|\Delta y_{\mathrm{prop}}| (deg)∣Δpprop∣|\Delta p_{\mathrm{prop}}| (deg)Δϕ\Delta\phi (deg)
Depth AnythingTrial 18.133.59 ±\pm 0.760.80 ±\pm 0.7024.03 ±\pm 5.9924.35 ±\pm 5.940.27 ±\pm 0.3710.88 ±\pm 9.7810.91 ±\pm 9.76
Trial 28.210.42 ±\pm 0.440.60 ±\pm 0.4211.77 ±\pm 7.1711.85 ±\pm 7.110.33 ±\pm 0.4110.34 ±\pm 9.4010.37 ±\pm 9.38
Trial 38.032.79 ±\pm 0.871.20 ±\pm 0.5112.82 ±\pm 7.2013.36 ±\pm 6.910.60 ±\pm 4.858.58 ±\pm 9.138.67 ±\pm 9.37
Mean8.122.267 ±\pm 0.660.87 ±\pm 0.5416.21 ±\pm 6.7816.52 ±\pm 6.650.40 ±\pm 1.889.93 ±\pm 9.449.98 ±\pm 9.50
Video Depth AnythingTrial 16.203.73 ±\pm 0.730.78 ±\pm 0.6526.90 ±\pm 5.7427.19 ±\pm 5.720.23 ±\pm 0.3110.78 ±\pm 10.3810.81 ±\pm 10.36
TABLE II : Tool-tip translation, inter-frame rotation propagation errors, and runtime for depth-based and hybrid pose estimation methods.
Fig. 10 : Trial 1 tool-tip trajectory in the camera frame (Hybrid Approach). The xx , yy , and zz components are plotted over time (time-matched sample index) for the vision-based method (Blue) and the optical tracker (Orange).
Fig. 11 : Trial 2 tool-tip trajectory in the camera frame (Hybrid Approach).
Fig. 12 : Trial 3 tool-tip trajectory in the camera frame (Hybrid Approach).
 Segmentation + SelectionRegistrationTotal
Trial 10 min 26 sec1 min 17 sec1 min 43 sec
Trial 20 min 27 sec1 min 21 sec1 min 48 sec
Trial 30 min 25 sec1 min 27 sec1 min 52 sec
Mean ±\pm Std0 min 26 ±\pm 1 sec1 min 22 ±\pm 5 sec1 min 48 ±\pm 5 sec
TABLE III : User time required for interactive tool segmentation + selection and anatomy registration.
Fig. 13 : Tool-tip depth trajectory in the camera frame using the depth-derived formulation. The estimated tool-tip position along the camera zz axis shows substantial frame-to-frame fluctuation relative to the optical tracker reference. While the coarse trend is preserved, the trajectory contains frequent abrupt variations, reflecting limited temporal consistency when pose is inferred directly from monocular depth.
Fig. 14 : Tool-tip depth trajectory in the camera frame using Video Depth Anything. Although designed for video input, Video Depth Anything does not improve stability in this setting. The estimated depth trajectory still contains substantial fluctuations and, in some cases, even larger deviations than the frame-wise depth-derived approach.
Fig. 15 : Example of per-step roll propagation in the camera frame. Absolute inter-frame roll changes are shown for the proposed vision-based tracker and the optical tracker. The vision-based estimate remains smooth over time, while the optical tracker exhibits an isolated spike, likely caused by transient line-of-sight loss or marker occlusion. This example highlights the robustness of the proposed method to common failure modes of external tracking systems.
More in this topic

Surgical Instrument Tracking

See all 16 papers

Explore similar work

CardsList
Oct 8, 2026

Multimodal Image-based Robotic Assistance and Guidance Environment (MIRAGE): A Novel Unified Framework for Surgical Navigation and Visualization

This paper presents the computational architecture and quantitative preclinical evaluation of MIRAGE, a multimodal image-guided robotic surgery framework for real-time visualization and navigation. It integrates stereovision scene reconstruction, augmented-reality (AR) video overlay, quantified heads-up display (HUD) navigation, and intraoperative ultrasound within the da Vinci console's TilePro display (Intuitive Surgical). Additional modules include optical/electromagnetic tracker registration, stereo calibration, fiducial rigid registration with RMSE feedback, and 3D point-cloud reconstruction. Video I/O and rendering run in a C++ backend, while a Python GUI coordinates calibration, registration, reconstruction, visualization, and user interaction through an in-process zero-copy binding layer. The framework formalizes spatial transforms, module states, and synchronization of heterogeneous video and tracking streams. Preoperative calibration uses synchronized checkerboard image pairs and optimization, with estimated intrinsics, extrinsics, and rectification reused intraoperatively. MIRAGE was validated in phantom experiments across da Vinci S, Si, Xi, and SP platforms. Across clinically relevant working distances, mean 3D reprojection error was 0.6mm, with best 0.1mm RMSE at 100mm on Xi. Results support MIRAGE as a preclinical technical-validation platform for accurate, responsive, and extensible multimodal surgical guidance.

Ehsan Nasiri, Yi Zheng, Michael A. Kokko +4

Thayer School of Engineering at Dartmouth 15 Thayer Dr, Hanover, NH, USA

AR in SurgerySurgical Robotics
AskPDF
Mar 9, 2026

Extend Your Horizon: A Device-Agnostic Surgical Tool Tracking Framework with Multi-View Optimization for Augmented Reality

Surgical navigation provides real-time guidance by estimating the pose of patient anatomy and surgical instruments to visualize relevant intraoperative information. In conventional systems, instruments are typically tracked using fiducial markers and stationary optical tracking systems (OTS). Augmented reality (AR) has further enabled intuitive visualization and motivated tracking using sensors embedded in head-mounted displays (HMDs). However, most existing approaches rely on a clear line of sight, which is difficult to maintain in dynamic operating room environments due to frequent occlusions caused by equipment, surgical tools, and personnel. This work introduces a framework for tracking surgical instruments under occlusion by fusing multiple sensing modalities within a dynamic scene graph representation. The proposed approach integrates tracking systems with different accuracy levels and motion characteristics while estimating tracking reliability in real time. Experimental results demonstrate improved robustness and enhanced consistency of AR visualization in the presence of occlusions.

Jiaming Zhang, Mingxu Liu, Hongchao Shu +6

Johns Hopkins University · Johns Hopkins University, School of Medicine · University of Arkansas, Johns Hopkins University

Augmented RealitySurgical Instrument TrackingAR in SurgeryVisual Object TrackingMultimodal Sensor FusionSurgical Robotics
AskPDF
Aug 25, 2026

ReGround-Surg: Reliability-Guided Anchor Grounding for Referring Surgical Video Segmentation

Referring surgical video segmentation requires segmenting a target instrument or tissue region across video frames according to a natural language expression. Recent Segment Anything Model 2 (SAM2) based two-stage methods (e.g., ReSurgSAM2) first ground the referred target in an initial or selected frame, then propagate the selected mask via tracking. Although effective, their performance is highly sensitive to the quality of the initial grounded mask: once an incorrect anchor is selected, subsequent tracking tends to propagate the error. This issue is especially challenging in surgical videos due to visually similar instruments, occlusion, and complex tissue-tool interactions. To address this issue, we propose ReGround-Surg, a lightweight reliability-guided anchor grounding framework to improve SAM2-based referring surgical video segmentation. It first predicts a text-conditioned spatial reliability map from the referring expression and current-frame visual features. The map is then reused in two complementary branches: a Gated Side Adapter enhances expression-relevant visual regions before text-to-vision fusion, while a Reliability-Weighted Vision-to-Text Attention module suppresses off-target visual evidence during prompt-token aggregation. Experiments on Ref-EndoVis17 and Ref-EndoVis18 show consistent improvements over state-of-the-art methods across three evaluation splits with negligible speed reduction. Code is publicly available at https://github.com/JiaxinWen1/ReGround-Surg.

Jiaxin Wen, Ming Yin, Lu Liu +1

University of Exeter, Exeter, UK

Referring Video Object SegmentationSurgical Video UnderstandingVisual Grounding
AskPDF