H-2DGS: Height-Field Gaussian Splatting for Sharp and Deformable Neurosurgical Surface Reconstruction
Authors: Zhang Zichuan, Bart Brouwers, Rembrandt Bakker, Maud Landers, Geert-Jan Rutten, Maxime Chamberland
Organizations: Department of Computer Science & Mathematics, Eindhoven University of Technology, Eindhoven, The Netherlands · Department of Neurosurgery, Elisabeth-Tweesteden Hospital Tilburg, The Netherlands
Neurosurgical navigation commonly presents magnetic resonance imaging (MRI) as 2D slices, making it difficult to relate the anatomy in those slices to the cortical surface observed through the surgical microscope. We seek a common 3D visualization that combines a photorealistic reconstruction of the exposed cortex during tumor surgery with whole-brain MRI context and diffusion tractography pathways. This requires the reconstructed cortex to form a clear boundary around the brain anatomy rather than a diffuse volume that obscures internal structures. Conventional 3D Gaussian Splatting (3DGS) can reproduce observed views but, under limited surgical view coverage, may distribute opacity through a thick and geometrically ambiguous band. We introduce height-field 2D Gaussian Splatting (H-2DGS), a three-stage pipeline that concentrates a robust 3DGS reconstruction, fuses its depth into a shared truncated signed distance field (TSDF) surface, and represents that surface with curved, surface-oriented Gaussian primitives. Separate height, color, and opacity bases provide local geometry and appearance, while a neighbor-transition objective coordinates overlapping patches. The explicit surface further supports postoperative alignment: MRI-derived depth deforms the reconstruction, and tracked-color supervision with alternating optimization preserves its appearance. Across five real neurosurgical cases, H-2DGS gives low multi-view depth inconsistency, a compact 7.97MB representation, and interactive rendering at 29--42ms per frame. The resulting sharp, deformable surface integrates clearly with tractography and MRI anatomy, including a postoperative resection cavity.
Figures & tables
Figure 1: A geometrically sharp, deformable surgical surface. (a) H-2DGS reconstructs a localized photorealistic cortical layer that remains clear when combined with subcortical fiber tracts. (b) Preoperative alignment between the deformed surgical surface and MRI.
Figure 2: Ray-wise density concentration. Stage II contracts the density interval between soft 1% and 99% quantiles.
Figure 3: Multi-view TSDF fusion. Rendered depths define multi-view signed-distance observations whose fused zero level set initializes Stage III.
Figure 4: Concept of the proposed H-2DGS primitive. Each primitive is a curved local chart with independent basis fields over one tangent domain.
Figure 5: Fusion of reconstructed cortex and tractography derived from diffusion MRI. The top row shows the results of case preop 2, the second and the bottom rows show the results of case preop 3. columns are 2DGS, 3D-HGS, QGS, and H-2DGS.
Figure 6: Uncovered-view comparison over three cases and four methods. Each row corresponds to the results of one of the four case. From top to bottom rows: results of case preop 1, preop 2 and preop 3 respectively. H-2DGS remains localized, whereas baseline geometry streaks, fragments, or blurs.
Figure 7: Result of deforming the post-resection reconstruction of the case postop 1. Initial (top) and deformed (bottom) reconstruction fitted to postoperative MRI morphology.
Figure 8: Depth-supervised postoperative alignment of the case postop 1. From left to right: initial, optimized, and target postoperative-MRI depths.
Case
Method
PSNR ↑
SSIM ↑
LPIPS ↓
MVDC ↓
Size (MB) ↓
Time (ms) ↓
Preop 1
H-2DGS
23.84
0.776
0.218
4e−5
7.97
39.13
2DGS
28.54
0.901
0.109
7e−5
40.87
25.42
3D-HGS
28.00
0.887
–
1.18
32.24
152.81
QGS
31.46
0.952
–
9e−5
86.04
120.68
Preop 2
H-2DGS
23.29
0.810
0.212
2e−5
7.97
29.68
2DGS
27.35
0.906
0.109
6e−5
45.72
29.69
Table 1: Quantitative comparison on five surgical cases. MVDC is normalized by scene extent. Lower is better for LPIPS, MVDC, size, and time. Missing LPIPS values were not reported by the corresponding baseline runs.
Figure 9: Components of postoperative deformation, case postop 1. Top: appearance-window iterations recover texture after depth fitting. Bottom: stronger geometry-continuity weighting coordinates neighboring patches.
Real-time reconstruction of deformable surgical scenes is vital for advancing robotic surgery, improving intraoperative guidance, and enabling automation. Recent methods achieve dense reconstructions from da Vinci robotic surgery videos, with Gaussian Splatting (GS) offering real-time performance via graphics acceleration. However, reconstruction quality in occluded regions remains limited, and depth accuracy has not been fully assessed, as benchmarks like EndoNeRF and StereoMIS lack 3D ground truth. We propose Diff2DGS, a two-stage framework for reliable 3D reconstruction of occluded surgical scenes. First, a diffusion-based video module with temporal priors inpaints tissue occluded by instruments with high spatiotemporal consistency. Second, we adapt 2D Gaussian Splatting (2DGS) with a Learnable Deformation Model (LDM) to capture dynamic tissue deformation and anatomical geometry, and introduce adaptive depth weight to improve geometric fidelity. We further extend evaluation beyond image-quality metrics by performing quantitative depth analysis on the SCARED dataset. Diff2DGS outperforms state-of-the-art methods in both appearance and geometry, reaching 38.02 dB PSNR on EndoNeRF and 34.40 dB on StereoMIS. Our experiments also show that optimizing image quality alone does not necessarily ensure accurate 3D geometry.The code is available at https://diff2dgs.github.io/.
3D Gaussian Splatting (3DGS) has recently enabled real-time novel view synthesis with impressive quality. However, it struggles to recover accurate surfaces under limited viewpoints and due to the inherent irregularity of Gaussian primitives. The resulting geometric errors are notoriously difficult to correct manually. To address these issues, we propose Gaussian Sculpting, a fully differentiable end-to-end framework for high-quality surface reconstruction. Our key insight is to anchor Gaussians onto an evolving differentiable surface, allowing them to guide signed distance field (SDF) optimization instead of extracting the surface only during post-processing. To enable stable gradient isolation during joint optimization, we design a bi-level training strategy in which the outer loop optimizes the geometry represented by the SDF, while the inner loop updates the Gaussians with the geometry fixed. We further impose constraints on Gaussian parameters to ensure consistency with the underlying surface, thereby improving both geometric and appearance fidelity during optimization. In addition, we introduce a multi-resolution subdivision scheme based on octree-like partitioning to preserve fine details while reducing memory consumption. Experiments on object-level scenes demonstrate that our method effectively removes redundant surfaces, recovers missing structures caused by limited viewpoints, and achieves strong reconstruction quality even at relatively low resolutions.
Ke Jiaxin, Juncheng Liu, Yi Wang +4
Dalian University of Technology, China · Massey University, New Zealand · Peking University, China +1
3D Gaussian Splatting (3DGS) has revolutionized fast novel view synthesis, yet its opacity-based formulation makes surface extraction fundamentally difficult. Unlike implicit methods built on Signed Distance Fields or occupancy, 3DGS lacks a global geometric field, forcing existing approaches to resort to heuristics such as TSDF fusion of blended depth maps. Inspired by the Objects as Volumes framework, we derive a principled occupancy field for Gaussian Splatting and show how it can be used to extract highly accurate watertight meshes of complex scenes. Our key contribution is to introduce a learnable oriented normal at each Gaussian element and to define an adapted attenuation formulation, which leads to closed-form expressions for both the normal and occupancy fields at arbitrary locations in space. We further introduce a novel consistency loss and a dedicated densification strategy to enforce Gaussians to wrap the entire surface by closing geometric holes, ensuring a complete shell of oriented primitives. We modify the differentiable rasterizer to output depth as an isosurface of our continuous model, and introduce Primal Adaptive Meshing for Region-of-Interest meshing at arbitrary resolution. We additionally expose fundamental biases in standard surface evaluation protocols and propose two more rigorous alternatives. Overall, our method Gaussian Wrapping sets a new state-of-the-art on DTU and Tanks and Temples, producing complete, watertight meshes at a fraction of the size of concurrent work-recovering thin structures such as the notoriously elusive bicycle spokes.
Diego Gomez, Antoine Guédon, Nissim Maruani +2
LIX, École Polytechnique, France · Inria, Côte d’Azur, France