Feasibility of Augmented Reality-Guided Robotic Ultrasound with Cone-Beam CT Integration for Spine Procedures

Tianyu Song; Felix Pabst; Feng Li; Yordanka Velikova; Miruna-Alexandra Gafencu; Yuan Bi; Ulrich Eck; Nassir Navab

cs.HCMar 23, 2026

Feasibility of Augmented Reality-Guided Robotic Ultrasound with Cone-Beam CT Integration for Spine Procedures

Authors: Tianyu Song, Felix Pabst, Feng Li, Yordanka Velikova, Miruna-Alexandra Gafencu, Yuan Bi, Ulrich Eck, Nassir Navab

Organizations: Chair for Computer Aided Medical Procedures(CAMP), Technical University of Munich(TUM), Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany

Abstract

Accurate needle placement in spine interventions is critical for effective pain management, yet it depends on reliable identification of anatomical landmarks and careful trajectory planning. Conventional imaging guidance often relies both on CT and X-ray fluoroscopy, exposing patients and staff to high dose of radiation while providing limited real-time 3D feedback. We present an optical see-through augmented reality (OST-AR)-guided robotic system for spine procedures that provides in situ visualization of spinal structures to support needle trajectory planning. We integrate a cone-beam CT (CBCT)-derived 3D spine model which is co-registered with live ultrasound, enabling users to combine global anatomical context with local, real-time imaging. We evaluated the system in a phantom user study involving two representative spine procedures: facet joint injection and lumbar puncture. Sixteen participants performed insertions under two visualization conditions: conventional screen vs. AR. Results show that AR significantly reduces execution time and across-task placement error, while also improving usability, trust, and spatial understanding and lowering cognitive workload. These findings demonstrate the feasibility of AR-guided robotic ultrasound for spine interventions, highlighting its potential to enhance accuracy, efficiency, and user experience in image-guided procedures.

Figures & tables

Fig. 1 : AR-Guided Needle Insertion. The system integrates robotic ultrasound, CBCT, and AR visualization into a unified framework, providing intuitive guidance during needle insertion for spine procedures.

Fig. 2 : System Overview. The pipeline integrates robotic CBCT, robotic ultrasound, and AR into a unified framework, enabling in situ AR visualization of spine anatomy and needle guidance. ( $\boldsymbol{\bullet}$ ) represents CBCT volume acquisition, ( $\boldsymbol{\bullet}$ ) depicts ultrasound and robot state acquisition, and ( $\boldsymbol{\bullet}$ ) indicates the visualization process.

Fig. 3 : Transformation Chain for System Registration. Note that the transformations shown with solid arrows are acquired, while the transformations with dashed arrows are derived. ( $\boldsymbol{\bullet}$ ) represents transformations in the virtual domain, ( $\boldsymbol{\bullet}$ ) corresponds to transformations in the real world, and ( $\boldsymbol{\bullet}$ ) denotes the registration between virtual and real robot.

Fig. 4 : AR Guidance Visualization. Raycasting from the needle guide onto the CT-derived spine model yields the predicted intersection point (marked in red) between the needle and the spine model.

Fig. 5 : Study Setup. Experimental configuration with robotic arm, ultrasound probe, lumbar spine phantom, and screen-based guidance interface.

Fig. 6 : Overall Across-Task Performance. Significance: $\star p<.05$ , $\star\star p<.01$ , $\star\star\star p<.001$ .

Measure	Screen	AR	$p$ value
FJ time (s)	95.25 $\pm$ 54.51	43.50 $\pm$ 19.17	0.003
FJ error (mm)	11.76 $\pm$ 6.37	9.69 $\pm$ 2.93	0.228
LP time (s)	78.25 $\pm$ 56.46	45.31 $\pm$ 24.90	0.002
LP error (mm)	13.12 $\pm$ 5.43	8.89 $\pm$ 2.71	0.015

TABLE I : Task Specific Results (mean

\pm

SD). Significance is indicated in bold.

Fig. 7 : Subjective Ratings. AR significantly increased usability, ease of use, trust, and spatial understanding, and reduced cognitive workload.

AR in Surgery

See all 14 papers

X-GuideAR: An Augmented Reality Framework to Mitigate Radiation Exposure during Fluoroscopic Guidance
Jul 12, 2026Mingxu Liu, Zixuan Liu, Ruchen Cai +5Medical Imaging Healthcare
Department of Computer Science Johns Hopkins University Baltimore, MD, 21211, USA · Department of Orthopaedic Surgery Johns Hopkins University Baltimore, MD, USA · Department of Orthopaedic Surgery Johns Hopkins School of Medicine Baltimore, MD, USA +3
Robotic Ultrasound Makes CBCT Alive
Mar 10, 2026Feng Li, Ziyuan Li, Zhongliang Jiang +2Cone-Beam CT Deformable Image Registration
Chair for Computer-Aided Medical Procedures and Augmented Reality, Technical University of Munich, Munich, Germany · Munich Center for Machine Learning, Munich, Germany · The University of Hong Kong, Hong Kong, China
Multi-Camera AR Guidance System for Surgical Instrument Handling and Assembly: Investigating Workload and Efficiency
Jun 3, 2026Shiyu Li, Julian Kreimeier, Hannah Schieber +4Object Pose Estimation AR in Surgery
Technical University of Munich, Human-Centered-Computing and Extended Reality Lab · TUM University Hospital, Orthopedics and Sports Orthopedics · Munich Institute of Robotics and Machine Intelligence (MIRMI) +2

Jul 12, 2026J/K move · Enter open · S save

X-GuideAR: An Augmented Reality Framework to Mitigate Radiation Exposure during Fluoroscopic Guidance

Mingxu Liu, Zixuan Liu, Ruchen Cai, Yu-Chen Ku+4

Department of Computer Science Johns Hopkins University Baltimore, MD, 21211, USA · Department of Orthopaedic Surgery Johns Hopkins University Baltimore, MD, USA · Department of Orthopaedic Surgery Johns Hopkins School of Medicine Baltimore, MD, USA+3

Medical Imaging Healthcare AR in Surgery

Open paper AskPDF arXiv

Achieving optimal screw placement for orthopedic surgeries requires frequent alignment checks and multiple anatomical views under X-ray -- a process known as "fluoro-hunting" that increases radiation exposure to patients and surgical teams. This work introduces X-GuideAR, an augmented reality (AR) framework for identifying optimal X-ray views, aimed at reducing radiation exposure while ensuring accurate screw placement. To exemplify the benefits of X-GuideAR, we focus on S2 alar-iliac (S2AI) screw placement. Our system provides radiation-free guidance for view acquisition and drilling by generating synthetic X-ray previews that accelerate fluoro-hunting. Once the required anatomical views are identified using these previews, a real X-ray is acquired, and the preview of the drilling trajectory is augmented onto it, facilitating precise screw placement with minimal additional radiation. A preliminary study involving eight S2AI trajectories performed by an expert spine surgeon demonstrated a 62.3% reduction in the number of X-rays. Post-procedure evaluations showed that trajectories done with X-GuideAR supported an average safe screw diameter of 12.95 mm compared to 5.9 mm under the conventional workflow, suggesting improved bony containment and potential biomechanical benefit. X-GuideAR shows great potential to reduce radiation exposure and streamline S2AI screw placement, offering a promising direction toward safer and more efficient surgeries.

Abstract

AR in Surgery

Explore similar work

X-GuideAR: An Augmented Reality Framework to Mitigate Radiation Exposure during Fluoroscopic Guidance

Robotic Ultrasound Makes CBCT Alive

Multi-Camera AR Guidance System for Surgical Instrument Handling and Assembly: Investigating Workload and Efficiency