We present UltraVoxelGS, a 3D ultrasound reconstruction method that achieves both fast reconstruction and physically faithful rendering for the first time, bringing live spatial feedback during 2D ultrasound scanning within reach. This is made possible by a feed-forward design that predicts a complete Gaussian scene representation from posed ultrasound images directly, entirely bypassing the per-scene optimization bottleneck that has confined prior methods to offline use. Realising such feed-forward prediction in this setting is non-trivial: unlike natural images, ultrasound slices share at most one-dimensional intersections and exhibit view-dependent appearance due to acoustic propagation and attenuation, violating the dense overlap and photometric consistency assumed by existing feed-forward Gaussian methods. Hence, to achieve fast reconstruction, we introduce a feed-forward voxel-to-Gaussian pass that decouples representation size from input count by lifting all posed slices into a fixed-size voxel support and predicting Gaussian parameters directly. For faithful rendering, we propose ultrasound-aware appearance rendering strategy factorize reflection from cumulative attenuation during training and exploit appearance continuity among spatially adjacent slices for efficient Render-time Appearance Adaptation. On both real-world and simulated datasets, UltraVoxelGS achieves best PSNR/SSIM compared with per-scene optimization baselines while reducing inference from 10-20 minutes to approximately 10 seconds (over 35 times faster) and scales to sequences exceeding 1,000 frames. Code is available at https://github.com/YuebyYue/UltraVoxelGS-camera-ready.
Figures & tables
Figure 1: Clinical workflow. A sequence of posed 2D ultrasound slices are reconstructed into a 3D representation, from which novel views are rendered to guide the operator toward unseen regions.
Figure 2: Optical feed-forward GS paradigm : (a) Natural images provide dense cross-view overlap and photometric consistency. (b) Existing methods predict pixel-aligned primitives from individual views. Our ultrasound-specific feed-forward GS paradigm : (c) Ultrasound slices share minimal intersection and exhibit view-dependent intensity. (d) We lift posed slices into a unified voxel support and predict Gaussians from the shared voxel support.
Figure 3: Overview of UltraVoxelGS. (a) Posed ultrasound slices {Ti,Ii} are lifted into a shared voxel support V , which is mapped by a feed-forward Transformer ϕθ to the Gaussian scene representation G . (b) Ultrasound-aware appearance rendering uses Decoupled Reflection and Attenuation Rendering (DRA) to render each target slice as cumulative attenuation A modulating anisotropic reflection R^ for image-level supervision. At render time, Render-time Appearance Adaptation (RAA) uses a nearby observed slice Pt as an appearance anchor to refine target-view appearance.
Figure 4
Method
Type
Dataset A: Real-world
Dataset B: Simulated
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
Kernel Interpolation [ 6 ]
Interpolation
13.52
0.3952
0.5419
17.47
0.2391
0.6259
ImplicitVol [ 8 ]
Per-scene optimization
19.13
0.6128
0.5218
18.97
0.2837
0.7192
UltraNeRF [ 10 ]
Per-scene optimization
18.49
0.4056
0.5761
17.75
0.1975
0.5985
3DGS [ 12 ]
Per-scene optimization
22.55
0.6695
0.3884
18.04
0.2532
0.6702
UltraGauss [ 13 ]
Per-scene optimization
22.06
0.6220
0.4750
15.01
0.1838
0.7057
Table 1: Quantitative comparison on real-world and simulated ultrasound datasets. The best and second-best results are highlighted in bold and underline, respectively.
Figure 5: Qualitative comparison on real-world and simulated ultrasound datasets.
Figure 6: Quality–efficiency trade-off on Dataset A.
Figure 7: 3D anatomical segmentation results on Dataset B. Top: one representative example visualized from two viewpoints; Bottom: quantitative evaluation across all test cases.
Figure 8: Failure cases on Dataset B. Visual comparisons under challenging scanning conditions, including sparse coverage, acoustic shadowing, and inaccurate probe poses.
Table 10
Figure 9: Decoupled Reflection and Attenuation Rendering visualization on Dataset B. Note that the ground-truth attenuation maps are obtained from the simulation pipeline [ 32 ] and serve solely as a visual reference; no attenuation supervision is used during training. The red boxes highlight a structure that is present in the reflection component but becomes weak in the final ultrasound appearance due to cumulative attenuation.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Item
Setting
Input image resolution
176×88
Voxel grid spacing
0.00075m
Voxel patch size
8×8×8
Top- k neighbors
3
SH order
2
Gaussian parameters
(cg,Σg,ρg,βg)
Appendix
Table 4: Implementation details of UltraVoxelGS.
Method
Type
Input Setting
Optimization / Inference Setting
Kernel Interpolation
Interpolation
Same sparse input
No optimization
ImplicitVol
Per-scene optimization
Same sparse input
50k steps
UltraNeRF
Per-scene optimization
Same sparse input
10k steps
3DGS
Per-scene optimization
Same sparse input
10k steps
UltraGauss
Per-scene optimization
Same sparse input
10k steps
UltraVoxelGS w/o RAA
Feed-forward
Same sparse input
Feed-forward only
Appendix
Table 5: Baseline settings. We summarize the main baseline configurations used for comparison.
Stage
Time (s) ↓
Voxel support construction
1.12
View-interacted block extraction
1.96
Network inference
4.83
Gaussian fusion
2.13
Representation construction
10.04
Rendering per target view w/o RAA
0.00362
Appendix
Table 6: Runtime breakdown of UltraVoxelGS. We report per-scene representation construction time and average per-view rendering time on Dataset A.
Variant
Steps
PSNR ↑
SSIM ↑
LPIPS ↓
Render / View (s) ↓
w/o RAA
0
19.76
0.5770
0.5130
0.00362
RAA
10
23.00
0.7484
0.3513
0.140
RAA
20
23.95
0.8003
0.2872
0.259
RAA
50
24.29
0.8329
0.2380
0.623
RAA
100
24.19
0.8350
0.2224
1.212
Appendix
Table 7: Render-time Appearance Adaptation step ablation. We evaluate the quality–efficiency trade-off of different adaptation steps on Dataset A.
Figure 10: Visual effect of Render-time Appearance Adaptation. RAA@ K denotes applying K steps of Render-time Appearance Adaptation. Increasing the number of RAA steps progressively improves local sharpness and contrast toward the ground truth.
Strategy
Anchor
Translation distance (mm)
Rotation angle ( ∘ )
PSNR ↑
SSIM ↑
LPIPS ↓
UltraVoxelGS
Top-1
7.87±3.52
2.78±2.31
20.87
0.3138
0.6190
Translation
Top-3
14.09±5.56
5.16±3.87
19.89
0.2889
0.6919
Top-5
19.09±6.83
6.80±4.47
19.39
0.2813
0.7018
Top-10
28.86±8.27
10.28±6.12
18.76
0.2719
0.7136
Rotation
Top-3
16.98±10.25
4.11±1.95
19.91
0.2889
0.6922
Top-5
22.24±10.46
5.59±2.39
19.41
0.2812
0.7020
Appendix
Table 8: RAA refinement under translation- and rotation-ranked anchors at different Top- k ranks on Dataset B.
Per-scene optimization
RAA
Target
Complete scene
Current queried view
Updated variables
Geometry and appearance
Transient ρ,β
Global representation
Optimized per scene
Feed-forward and fixed
Scope
Entire scene
Query-relevant local support
Persistence
Stored in the scene
Discarded after rendering
Runtime
Increases with scene scale and input views
Independent of source views (623 ms / 50 steps)
Appendix
Table 9: Per-scene optimization versus RAA.
Rendering Variant
Decoupled (ρ,β)
SH Reflection
PSNR ↑
SSIM ↑
UltraGauss
✗
✗
22.06
0.6220
UltraVoxelGS-opt
✓
✓
22.55
0.6245
Appendix
Table 10: Supporting analysis of Decoupled Reflection and Attenuation Rendering. We compare UltraGauss with an optimized variant of our representation under per-scene optimization.
Variant
Consistency Loss
RAA
PSNR ↑
SSIM ↑
Block-wise only
✗
✗
19.85
0.5742
Block-wise + Consistency
✓
✗
19.76
0.5770
Appendix
Table 11: Overlap consistency ablation. We evaluate whether overlap consistency stabilizes repeated local predictions under the proposed local-to-global assembly. RAA is disabled in this ablation.
Support resolution
Method
PSNR ↑
SSIM ↑
LPIPS ↓
0.75 mm
UltraVoxelGS w/o RAA
19.76
0.5770
0.5130
0.75 mm
UltraVoxelGS
24.29
0.8329
0.2384
0.50 mm
UltraVoxelGS w/o RAA
20.14
0.5909
0.4719
0.50 mm
UltraVoxelGS
24.22
0.8200
0.2131
Appendix
Table 12: Expressive support-density analysis.
Method
Regime
PSNR ↑
SSIM ↑
LPIPS ↓
VolSplat
Feed-forward
15.70
0.4522
0.5884
UltraVoxelGS w/o RAA
FF + DRA
19.76
0.5770
0.5130
UltraVoxelGS
FF + DRA + RAA
24.29
0.8329
0.2384
Appendix
Table 13: Feed-forward comparison on Dataset A.
σt (mm)
σr ( ∘ )
Condition
PSNR ↑
SSIM ↑
LPIPS ↓
0
0
A
20.86
0.3158
0.6206
8
4
B
20.92±0.01
0.3177±0.0001
0.6199±0.0001
C
20.11±0.08
0.3017±0.0015
0.6336±0.0013
D
19.92±0.08
0.2985±0.0016
0.6384±0.0043
12
6
B
20.87±0.10
0.3165±0.0027
0.6232±0.0052
C
19.62±0.01
0.2920±0.0007
0.6423±0.0004
Appendix
Table 14: Reproducibility of controlled pose experiment on dataset B.
Figure 11: Additional qualitative comparison on dataset A. UltraVoxelGS better preserves anatomical structures, local sharpness, and cross-view consistency under sparse ultrasound observations.
Figure 12: Additional qualitative comparison on datasetB. UltraVoxelGS better preserves anatomical structures, local sharpness, and cross-view consistency under sparse ultrasound observations.
Setting
PSNR ↑
SSIM ↑
LPIPS ↓
UltraGauss (Handheld acquisition)
20.05
0.5858
0.5580
UltraVoxelGS (Robotic acquisition)
24.29
0.8329
0.2384
UltraVoxelGS (Handheld acquisition)
23.96
0.7703
0.3356
Appendix
Table 15: Subject-level average on robotic and tracked-handheld test cases.
Setting
PSNR ↑
SSIM ↑
LPIPS ↓
Progressive streaming
24.08
0.8240
0.2555
Final streaming
24.11
0.8280
0.2476
Direct reconstruction
24.29
0.8329
0.2384
Appendix
Table 16: Performance of streaming reconstruction on dataset A.
Figure 13: Qualitative comparison of reconstruction modes on Dataset B. Progressive and final streaming achieve visual quality comparable to direct reconstruction.
Novel view synthesis (NVS) in ultrasound has gained attention as a technique for generating anatomically plausible views beyond the acquired frames, offering new capabilities for training clinicians or data augmentation. However, current methods struggle with complex tissue and view-dependent acoustic effects. Physics-based NVS aims to address these limitations by including the ultrasound image formation process into the simulation. Recent approaches combine a learnable implicit scene representation with an ultrasound-specific rendering module, yet a substantial gap between simulation and reality remains. In this work, we introduce UltraG-Ray, a novel ultrasound scene representation based on a learnable 3D Gaussian field, coupled to an efficient physics-based module for B-mode synthesis. We explicitly encode ultrasound-specific parameters, such as attenuation and reflection, into a Gaussian-based spatial representation and realize image synthesis within a novel ray casting scheme. In contrast to previous methods, this approach naturally captures view-dependent attenuation effects, thereby enabling the generation of physically informed B-mode images with increased realism. We compare our method to state-of-the-art and observe consistent gains in image quality metrics (up to 15% increase on MS-SSIM), demonstrating clear improvement in terms of realism of the synthesized ultrasound images.
Felix Duelmer, Jakob Klaushofer, Magdalena Wysocki +2
Chair for Computer Aided Medical Procedures (CAMP), Technical University of Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany
Physically-based differentiable rendering enables gradient-based optimization of scene parameters by matching rendered images to measurements, but has so far mainly focused on light transport. We extend this paradigm to medical ultrasound, where image formation resembles transient rendering: echoes are binned by time-of-flight rather than projected onto an image plane. We present UltraDiff, a modular framework for differentiable ultrasound ray tracing. UltraDiff formulates ultrasound image formation as a path-space integral, gated by travel time between the transducer and tissue interfaces, and derives a Monte Carlo estimator of both the forward model and its gradients with respect to scene parameters. We demonstrate this on an inverse geometry estimation: starting from a sphere, an SDF is optimized until simulated echoes match measured ones, recovering vertebral surfaces from simulated B-mode sweeps and from a real robotic acquisition of a spine phantom. Unlike state-of-the-art ultrasound shape reconstruction methods, which rely on pre-segmented images, our approach operates unsupervised on B-mode images through analysis-by-synthesis, while achieving competitive geometric accuracy. Implemented on top of Mitsuba 3, UltraDiff brings differentiable path tracing to a new sensing modality and provides a foundation for inverse problems in acoustic imaging.
Felix Duelmer, Magdalena Wysocki, Nassir Navab +1
Technical University of Munich & Munich Center for Machine Learning
Wide Field-of-View (WFoV) reconstruction enhances 3D ultrasound imaging by providing valuable anatomical context for segmentation models and visualization. Clinical ultrasound volumes are predominantly acquired using convex probes, which generate expanding, diverging acoustic beams to maximize anatomical coverage. Stitching these sweeps together traditionally introduces significant compounding artifacts and aliasing due to depth-dependent resolution changes. Here, we introduce Ultra-Wide-NeRF, a Multivariate 3D Gaussian (MVG) NeRF-based method for WFoV ultrasound reconstruction. By explicitly modeling the complex beam geometry using distance-dependent convex volumetric sampling and anisotropic 3D Gaussians, our method inherently mitigates these compounding artifacts and provides anti-aliasing. Beyond simply reconstructing a static 3D grid, our NeRF-based approach yields a continuous neural representation of the tissue, enabling the synthesis of high-fidelity novel views from arbitrary virtual trajectories. We validate Ultra-Wide-NeRF for intracardiac echocardiography on phantom and porcine datasets, demonstrating that our method expands the spatial context important in intraoperative navigation. Code will be open-sourced upon publication.
Patris Valera, Magdalena Wysocki, Felix Duelmer +4
Chair for Computer Aided Medical Procedures (CAMP), Technical University of Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · LUMA Vision, Beech Hill Road, Dublin, Ireland