Bridging the Markerless Tracking Validation Gap: Intraoperative X-ray Workflow Integration
Authors: Connor Daly, Salvatore Russo, Jinendra Ekanayake, Daniel S. Elson, Ferdinando Rodriguez y Baena
Organizations: The Hamlyn Centre for Robotic Surgery, Imperial College London, London, UK · Department of Neurosurgery, Imperial College Healthcare NHS Trust, London, UK · Stanford University School of Medicine, Stanford University, California, USA
Markerless intraoperative tracking of spinal anatomy via RGB-D sensors is a promising alternative to invasive bone-anchored fiducials, yet no existing clinical dataset provides independent subsurface ground truth for validating these methods. We address this gap with a clinically viable pipeline that derives per-vertebra 6-DoF pose from routine intraoperative C-arm fluoroscopy, co-registered to the RGB-D camera frame through a multimodal fiducial array detectable in both imaging modalities. Critically, the pipeline requires no additional radiation, instrumentation, or deviation from standard surgical workflows, relying solely on the fluoroscopic images already acquired during routine level checks. A calibrated transform chain links the vertebral pose recovered via multi-view 2D-3D CT-to-fluoroscopy registration to the RGB-D coordinate frame, enabling direct comparison of any markerless registration estimate against a subsurface reference. Ex vivo validation against optically tracked ovine vertebrae confirms end-to-end accuracy of 2.06 +- 0.29 mm and 3.35 +- 0.93 degrees, sufficient to serve as independent ground truth for methods with substantially larger errors. We demonstrate the pipeline's utility by evaluating two markerless registration strategies on clinical data spanning 11 vertebrae across 1,500 frames from three surgical sequences: the first such evaluation against an independent subsurface reference in a clinical setting. The proposed framework provides reusable infrastructure for benchmarking emerging markerless tracking methods on real surgical data.
Figures & tables
Figure 1: (A) Intraoperative setup with the marker inside a sterile cover, secured taut with a rubber band by the scrub nurse to maintain detectability. The overlay shows automatic pose detection and projection of the preoperative CT into the surgical scene using the transform chain in (I) ; (B, C) Automatic marker pose detection in C-arm images acquired during a fixation case, with projection errors (pixels) overlaid; (D) AP DRR images, showing CT registered to C-arm images in ImFusion; (E) Calibration phantom under fluoroscopy; (F) Marker array under fluoroscopy; (G) Marker array under fluoroscopy with ovine vertebrae; (H) Data processing overview, showing how transforms needed to project preoperative CT into the intraoperative RGB-D stream are acquired. (I) Solid lines depict our method’s full transform chain enabling projection of vertebra j’s CT into the Azure Kinect RGB-D camera frame (dashed line, TVjRGBD ).
1→2
1→3
1→4
Modality
Trans.
Rot.
Trans.
Rot.
Trans.
Rot.
(mm)
( ∘ )
(mm)
( ∘ )
(mm)
( ∘ )
RGB-D tracking
0.40
0.31
0.70
0.61
0.10
1.85
X-ray tracking (AP)
0.58
0.03
1.19
0.07
0.02
0.35
X-ray tracking (oblique)
0.46
0.18
0.96
0.50
0.66
0.38
Table 1: Independent validation of each coordinate recovery stage against optical tracker ground truth. RGB-D tracking is evaluated with the sterile probe cover in place. Transitions 1–3 represent lateral displacement within the image plane; transition 4 represents displacement along the sensor optical axis.
Vertebra
Combined TRE (mm)
Rotational Error ( ∘ )
T1
1.96
4.06
T2
2.32
3.40
T3
1.62
3.94
T4
2.32
1.98
Agg.
2.06 ± 0.29
3.35 ± 0.93
Table 2: Ex vivo registration accuracy across ovine thoracic vertebrae. Combined TRE represents the mean of both anatomical landmark measurements.
Figure 2: (A) The pose of ovine CT meshes directly tracked via the Azure Kinect in blue, compared to those tracked via fluoroscopy and projected into the Kinect camera space in yellow; (B) Per vertebra patches predicted by the network in Sec. 3.2 ; (C, D) Intraoperative pose estimates via X-ray in yellow, and through per vertebra ICP in blue.
Exposure
Per Vertebra ICP
Rigid ICP
Trans. (mm)
Rot. ( ∘ )
Trans. (mm)
Rot. ( ∘ )
5
5.15±3.65
8.21±3.74
5.70±1.59
4.61±1.55
3
3.98±1.81
9.10±3.68
4.66±0.82
3.53±0.78
3
4.02±3.10
7.86±3.63
3.38±0.58
5.36±0.30
Overall
4.52 ± 3.00
8.36 ± 3.69
4.78 ± 1.10
4.52 ± 1.00
Table 3: Comparison of [ 5 ] ’s rigid registration method across three sequences of clinical data, versus individual ICP. Translation (Trans.) errors are reported in mm and rotation (Rot.) errors in degrees. Each sequence is 500 frames. "Exposure" represents the exposed vertebra count.
Purpose: Marker-based tracking of surgical robots is occlusion-prone in cluttered operating rooms. We evaluate stereo differentiable rendering for marker-free, real-time robot pose tracking, potentially improving safety, reducing setup time, and enabling multi-robot interaction. Methods: We extend the markerless pose estimation framework roboreg to online dynamic tracking via (i) sequential optimisation that propagates pose estimates across frames with motion-adaptive hyperparameter tuning, and (ii) CUDA stream parallelisation of segmentation and optimisation, combined with CUDA-graph accelerated segmentation. We evaluate on 38 unobstructed and 5 occluded displacement sequences with static start/end ground-truth calibrations and dynamic marker-based reference tracking. Results: We achieve real-time 1080p tracking at 30 fps (up from 14 fps for vanilla roboreg), matching the camera frame rate. Accuracy reaches 1.7 cm / 0.6 deg against static ground truth and 1.2 cm mean 3D error over 27,460 frames against the marker-based reference (1.53 cm over 1,242 occluded frames). Our method outperforms FoundationPose by 11% in dynamic estimation (63% under occlusion) and 250% in static estimation, with 6x faster inference. Conclusions: Stereo differentiable rendering enables real-time, high-resolution marker-free surgical robot tracking, on par with marker-based approaches and surpassing foundation-model baselines.
Yanghe Hao, Martin Huber, Christos Bergeles +1
School of Biomedical Engineering & Imaging Sciences, King’s College London, London, United Kingdom.
Recovering the 6-DoF pose of the knee bones from a plain radiograph, given the patient's segmented pre-operative CT, turns a routine low-dose image into a quantitative measurement of joint geometry, without the added dose of a repeat CT or a fixed biplanar rig. Classic solutions align a rendered bone silhouette to image edges; recent alternatives refine pose by backpropagating an image-similarity loss through a differentiable X-ray renderer. Both operate one patient at a time and are fragile under a single view. Silhouettes are depth-ambiguous, and differentiable-rendering refinement has a narrow capture range at substantial per-iteration cost. We instead learn an amortized, subject-agnostic dense 2D-3D correspondence, supervised solely by projection geometry. One shared-weight model per bone, trained across 758 patients, registers patients unseen during training. The pose then follows in closed form from a global, initialization-free, render-free PnP+RANSAC solve. Because X-ray formation is transmissive, our correspondence target is transmission-aware rather than tied to a single surface. Though trained only to register, the representation is anatomically semantic: a simple classifier reads a landmark's anatomical region from its embedding across held-out patients, and the same features separate the knee's bones into a 2D-3D-consistent identity learned without any bone label. On a large single-institution cohort the model generalizes well to held-out patients.
In operating rooms (OR), world-scale multi-view 3D tracking supports downstream applications such as surgeon behavior recognition, where physically meaningful quantities such as distances and motion statistics must be measured in meters. However, real clinical deployments rarely satisfy the geometric prerequisites for stable multi-view fusion and tracking: camera calibration and RGB-D registration are always unreliable, leading to cross-view geometric inconsistency that produces "ghosting" during fusion and degrades 3D trajectories in a shared OR coordinate frame. To address this, we introduce Geometry OR Tracker, a two-stage pipeline that first rectifies imprecise calibration into a scaleconsistent and geometrically consistent camera setup with a single global scale via a Multi-view Metric Geometry Rectification module, and then performs Occlusion-Robust 3D Point Tracking directly in the unified OR world frame. On the MM-OR benchmark, improved geometric consistency translates into tracking gains: our rectification front-end reduces cross-view depth disagreement by more than 30× compared to raw calibration. Ablation studies further demonstrate the relationship between calibration quality and tracking accuracy, showing that improved geometric consistency yields stronger world-frame tracking.
Yihua Shao, Kang Chen, Feng Xue +6
Centre for Artificial Intelligence and Robotics Hong Kong Institute of Science and Innovation Chinese Academy of Sciences · State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS) Institute of Automation Chinese Academy of Sciences (CASIA) · Peking University +4