Depth Augmented and FE Free 3D/2D Liver Registration for Laparoscopic Liver AR
Authors: Hanyuan Zhang, Lucas He, Runlong He, Weixi Yi, Abdolrahim Kadkhodamohammadi, Danail Stoyanov, Brian R. Davidson, Evangelos B. Mazomenos, Matthew J. Clarkson
Organizations: UCL Hawkes Institute, University College London, London WC1E 6BT, UK · Unit for Lifelong Health and Ageing at UCL, University College London, London WC1E 7HB, UK · Medtronic plc., London, UK · Division of Surgery and Interventional Science, University College London, London WC1E 6BT, UK
Augmented reality (AR) guidance in laparoscopic liver surgery requires accurate registration of preoperative 3D models to intraoperative 2D video, but remains challenging due to partial visibility, specularities, and tissue deformation. Existing methods often rely on contour-based rigid initialization and finite-element (FE) models for deformable registration, increasing modeling and engineering complexity. We present a depth-augmented, FE-free 3D--2D registration pipeline that combines robust rigid initialization with patient-specific non-rigid refinement. For rigid alignment, we adapt the RefineNet module of FoundationPose to laparoscopic liver scenes by using multi-class contour maps and monocular depth for relative pose refinement. For deformable alignment, we construct a patient-specific statistical deformation model from non-rigid ICP (NICP) correspondences and optimize pose and shape parameters using a coarse-to-fine L-BFGS-B strategy. On a public clinical laparoscopic liver dataset, the proposed method achieves a mean target registration error (TRE) of 14.73,mm under a controlled manual-contour setting designed to isolate registration performance. Ablation studies show that monocular depth improves rigid initialization over contour-only inputs, while tumor-mapping analysis indicates that good surface alignment does not necessarily translate into lower target localization error. On an external dataset without ground truth, the method produces visually plausible overlays for qualitative assessment. These results suggest that depth-augmented pose refinement and FE-free statistical deformation modeling provide a promising alternative to FE-based pipelines for controlled 3D--2D liver registration in surgical AR.
Figures & tables
Figure 1: The registration workflow diagram: The first stage adopts the RefineNet component from FoundationPose by Wen et al., with the inputs replaced by the liver’s multi-class contour and depth images. The output is a relative pose update, which is used to initialize the subsequent non-rigid optimization.
Figure 2: Patient-specific statistical deformation model constructed from NICP-registered liver shapes.
Method
Annotation
P1
P2
P3
P4
Avg
Avg w/o P2
MA
N/A
15.14
35.48
30.48
16.29
24.35
20.63
Sil-B [ 1 ]
Manual
8.25
37.25
28.40
15.83
22.43
17.47
LMR [ 11 ]
Manual
17.40
53.80
17.60
17.00
26.45
17.33
NM [ 15 ]
Manual
14.82
51.43
20.15
12.95
24.83
15.87
Opt-B [ 12 ]
Manual
14.87
N/A
22.40
7.23
N/A
14.83
ADeLiR [ 7 ]
Auto
12.45
46.44
17.13
11.20
21.81
13.60
Table 1: Registration performance in TRE (mm, lower is better). Best results are in bold . MA: manual initialization. For each patient, TRE is first averaged over all available frames using the public benchmark evaluation code. For the proposed method, we additionally report mean ± std over 10 runs with different random initializations, where each run yields one case-level mean TRE.
Figure 3: Qualitative overlays on four cases from an independent laparoscopic liver dataset, comparing CMA–ES and L-BFGS-B optimization. In each group, the leftmost column shows the original intraoperative image, followed by CMA–ES and L-BFGS-B results.
Method
P1
P2
P3
P4
Mean
(a) Input modality
Contour
15.25
39.48
18.94
20.21
23.47
Contour+Mask
14.59
28.00
12.52
31.12
21.56
Contour+Depth
13.68
32.32
11.03
18.18
18.80
(b) Tumor mapping
FFD
9.70
31.54
7.65
10.08
14.74
Table 2: Ablation studies (TRE in mm, lower is better). Best in bold .
In laparoscopic liver surgery, augmented reality technology enhances intraoperative anatomical guidance by overlaying 3D liver models from preoperative CT/MRI onto laparoscopic 2D views. However, existing registration methods lack explicit modeling of reliable 2D-3D geometric correspondences supported by latent evidence, leading to limited interpretability and potentially unstable alignment in clinical scenarios. In this work, we introduce Land-Reg, a correspondence-driven deformable registration framework that explicitly learns latent-grounded 2D-3D landmark correspondences as an interpretable intermediate representation to bridge cross-modal alignment. For rigid registration, Land-Reg embraces a Cross-modal Latent Alignment module to map multi-modal features into a unified latent space. Further, an Uncertainty-enhanced Overlap Landmark Detector with similarity matching is proposed to robustly estimate explicit 2D-3D landmark correspondences. For non-rigid registration, we design a novel shape-constrained supervision strategy that anchors shape deformation to matched landmarks through reprojection consistency and incorporates local-isometric regularization to alleviate inherent 2D-3D depth ambiguity, while a rendered-mask alignment enforces global shape consistency. Experimental results on the P2ILF dataset demonstrate the superiority of our method on both rigid pose estimation and non-rigid deformation. Our code will be available at https://github.com/cuiruize/Land-Reg.
Ruize Cui, Jialun Pei, Haiqiao Wang +4
The Hong Kong Polytechnic University, Hong Kong, China · The Chinese University of Hong Kong, Hong Kong, China
Accurate 3D--2D liver registration, which aligns preoperative 3D models to partial, view-dependent intraoperative surface observations, is critical for AR-guided laparoscopic surgery but remains challenging due to severe occlusion, limited visibility, and the lack of 3D ground-truth supervision. Existing landmark-free approaches perform partial-to-complete geometric alignment, yet robust self-supervision under extreme partial visibility remains difficult. We propose Vis2Reg, a visibility-aware registration framework that explicitly constrains deformation using mask-consistent visible regions. We introduce a visibility-aware self-supervision that derives a visible-domain 3D supervision signal from intraoperative masks, enabled by differentiable point rasterization and mask-guided back-projection. This formulation improves robustness under severe occlusion while maintaining fully self-supervised learning. Vis2Reg combines a robust geometric rigid initialization module with an implicit neural deformation field for stable alignment. Vis2Reg achieves a Dice score of 92.6% and a Chamfer Distance of 1.43 mm on real intraoperative datasets, with 111 ms per-frame inference time, demonstrating both accuracy and practical efficiency.
Jiaming Feng, Xukun Zhang, Shahid Farid +1
AI in Medicine and Surgery Group, School of Computer Science, University of Leeds, LS2 9JT, Leeds, United Kingdom · Department of Diagnostic Radiology, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Pok Fu Lam, Hong Kong · Department of HPB and Transplant Surgery, St. James’s University Hospital, Leeds, United Kingdom
Registration between preoperative CT and intraoperative laparoscopic video plays a crucial role in augmented reality (AR) guidance for minimally invasive surgery. Learning-based methods have recently achieved registration errors comparable to optimization-based approaches while offering faster inference. However, many supervised methods produce coarse alignments that rely on additional optimization-based refinement, thereby increasing inference time. We present a discrete-action reinforcement learning (RL) framework that formulates CT-to-video registration as a sequential decision-making process. A shared feature encoder, warm-started from a supervised pose estimation network to provide stable geometric features and faster convergence, extracts representations from CT renderings and laparoscopic frames, while an RL policy head learns to choose rigid transformations along six degrees of freedom and to decide when to stop the iteration. Experiments on a public laparoscopic dataset demonstrated that our method achieved an average target registration error (TRE) of 15.70 mm, comparable to supervised approaches with optimization, while achieving faster convergence. The proposed RL-based formulation enables automated, efficient iterative registration without manually tuned step sizes or stopping criteria. This discrete framework provides a practical foundation for future continuous-action and deformable registration models in surgical AR applications.
Hanyuan Zhang, Lucas He, Zijie Cheng +5
UCL Hawkes Institute, University College London, London, UK · Medtronic plc., London, UK