UltraG-Ray: Physics-Based Gaussian Ray Casting for Novel Ultrasound View Synthesis

Felix Duelmer; Jakob Klaushofer; Magdalena Wysocki; Nassir Navab; Mohammad Farid Azampour

cs.CVMar 30, 2026

UltraG-Ray: Physics-Based Gaussian Ray Casting for Novel Ultrasound View Synthesis

Authors: Felix Duelmer, Jakob Klaushofer, Magdalena Wysocki, Nassir Navab, Mohammad Farid Azampour

Organizations: Chair for Computer Aided Medical Procedures (CAMP), Technical University of Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany

Abstract

Novel view synthesis (NVS) in ultrasound has gained attention as a technique for generating anatomically plausible views beyond the acquired frames, offering new capabilities for training clinicians or data augmentation. However, current methods struggle with complex tissue and view-dependent acoustic effects. Physics-based NVS aims to address these limitations by including the ultrasound image formation process into the simulation. Recent approaches combine a learnable implicit scene representation with an ultrasound-specific rendering module, yet a substantial gap between simulation and reality remains. In this work, we introduce UltraG-Ray, a novel ultrasound scene representation based on a learnable 3D Gaussian field, coupled to an efficient physics-based module for B-mode synthesis. We explicitly encode ultrasound-specific parameters, such as attenuation and reflection, into a Gaussian-based spatial representation and realize image synthesis within a novel ray casting scheme. In contrast to previous methods, this approach naturally captures view-dependent attenuation effects, thereby enabling the generation of physically informed B-mode images with increased realism. We compare our method to state-of-the-art and observe consistent gains in image quality metrics (up to 15% increase on MS-SSIM), demonstrating clear improvement in terms of realism of the synthesized ultrasound images.

Figures & tables

Figure 1: Overview of UltraG-Ray pipeline: a) Progressive adaptation of the learnable 3D Gaussian distribution to match the acquired data b) Image synthesis pipeline: First Gaussians are filtered based on the respective pose, second Gaussian ray-intersection is used to create the echo and transmittance maps and finally the resulting B-Mode image c) Downstream task evaluation based on NVS, where green and orange frames denote novel views

Figure 2: Depiction of the attenuation and intensity formation process: a) Transmittance accumulation, where each Gaussian gradually reduces the remaining energy along the ray. b) Computation of pixel-wise intensity contributions from view-dependent Gaussian intensity and their distance-weighted influence. c) Final pixel intensity obtained by combining accumulated attenuation with the weighted Gaussian contributions.

Figure 3: Qualitative comparison of UltraG-Ray and baselines on the spine phantom.

Figure 4: NVS comparison on ex vivo porcine muscle. Highlighted regions (red and orange) are enlarged as cut-outs on the right side of the respective image.

Dataset	Method	PSNR $\uparrow$	MS-SSIM $\uparrow$	GMS $\uparrow$	GMSD $\downarrow$
Porcine Muscle	Re-sliced Median Comp.	24.96 $\pm$ 0.67	0.73 $\pm$ 0.03	0.87 $\pm$ 0.01	0.17 $\pm$ 0.01
	Re-sliced Max Comp.	24.18 $\pm$ 0.59	0.73 $\pm$ 0.03	0.87 $\pm$ 0.01	0.17 $\pm$ 0.01
	Ultra-NeRF	23.60 $\pm$ 0.54	0.62 $\pm$ 0.03	0.81 $\pm$ 0.02	0.23 $\pm$ 0.01
	ImplicitVol	25.16 $\pm$ 0.56	0.74 $\pm$ 0.02	0.85 $\pm$ 0.01	0.19 $\pm$ 0.01
	UltraG-Ray (w/o Trans.)	24.31 $\pm$ 0.76	0.74 $\pm$ 0.03	0.86 $\pm$ 0.01	0.18 $\pm$ 0.00
	UltraG-Ray (SH=0)	25.10 $\pm$ 0.69	0.76 $\pm$ 0.02	0.87 $\pm$ 0.01	0.17 $\pm$ 0.01

Table 1: Evaluation results for NVS on both datasets. Best values are in bold. Arrows indicate whether higher (

\uparrow

) or lower (

\downarrow

) values correspond to better performance.

Figure 5: NVS examples from both datasets for the intermediate echo and transmittance maps, the synthesized B-mode output, and the corresponding ground truth image.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Evolution of GMSD scores of novel views for the porcine muscle dataset and number of Gaussians over training steps.

Figure 7: Comparison of novel views with differing number of max. Gaussians in the porcine muscle dataset. The metrics are comparable with Table 2 .

Dataset	#Gaussians	PSNR $\uparrow$	MS-SSIM $\uparrow$	GMS $\uparrow$	GMSD $\downarrow$
Porcine Muscle	500k	25.49 $\pm$ 0.60	0.77 $\pm$ 0.02	0.88 $\pm$ 0.01	0.16 $\pm$ 0.01
	300k	25.39 $\pm$ 0.62	0.77 $\pm$ 0.02	0.87 $\pm$ 0.01	0.17 $\pm$ 0.01
	100k	25.46 $\pm$ 0.66	0.77 $\pm$ 0.02	0.87 $\pm$ 0.01	0.17 $\pm$ 0.01
	10k	25.39 $\pm$ 0.54	0.75 $\pm$ 0.01	0.85 $\pm$ 0.01	0.18 $\pm$ 0.00

Appendix

Table 2: Comparison of image quality metrics for different maximum numbers of Gaussians.

Eval. angle	PSNR $\uparrow$	MS-SSIM $\uparrow$	GMS $\uparrow$	GMSD $\downarrow$
$-3^{\circ}$	25.42 $\pm$ 0.59	0.77 $\pm$ 0.02	0.87 $\pm$ 0.01	0.16 $\pm$ 0.01
$-5^{\circ}$	23.96 $\pm$ 0.54	0.70 $\pm$ 0.04	0.86 $\pm$ 0.01	0.18 $\pm$ 0.01
$-7^{\circ}$	22.77 $\pm$ 0.54	0.63 $\pm$ 0.03	0.85 $\pm$ 0.01	0.19 $\pm$ 0.00
$-10^{\circ}$	21.59 $\pm$ 0.35	0.55 $\pm$ 0.03	0.83 $\pm$ 0.00	0.20 $\pm$ 0.00

Appendix

Table 3: Out-of-plane generalization on the porcine muscle dataset. We evaluate UltraG-Ray at increasingly out-of-plane probe tilt angles on the same training dataset (two sweeps at

0^{\circ}

and

+3^{\circ}

-3^{\circ}

represents the same values as in Table 1 , and the remaining evaluations are further tilts. Best values are in bold. Arrows indicate whether higher (

\uparrow

) or lower (

\downarrow

) values correspond to better performance.

Novel View Synthesis

Novel View Synthesis creates images of a scene from camera viewpoints not present in the input, using learned geometry and appearance models.

See all 289 papers

UltraDiff: Differentiable Ray Tracing in Ultrasound for Shape Optimization
Oct 6, 2026Felix Duelmer, Magdalena Wysocki, Nassir Navab +1Differentiable Rendering Ultrasound Imaging
Technical University of Munich & Munich Center for Machine Learning
Multivariate Gaussian NeRF for Wide Field-of-View Ultrasound Reconstruction
Apr 27, 2026Patris Valera, Magdalena Wysocki, Felix Duelmer +4Novel View Synthesis Ultrasound Imaging
Chair for Computer Aided Medical Procedures (CAMP), Technical University of Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · LUMA Vision, Beech Hill Road, Dublin, Ireland
Feature-Space Guided Diffusion for Realistic Ultrasound Image Synthesis
Jul 13, 2026Marina Domínguez, Nélida Mirabet-Herranz, Valery NaranjoDiffusion Model Guidance Diffusion Guidance
Instituto Universitario de Investigación en Tecnología Centrada en el Ser Humano, HUMAN-tech, Universitat Politècnica de València, Valencia, Spain · Artikode Intelligence S.L., Valencia, Spain

Oct 6, 2026J/K move · Enter open · S save

UltraDiff: Differentiable Ray Tracing in Ultrasound for Shape Optimization

Felix Duelmer, Magdalena Wysocki, Nassir Navab, Mohammad Farid Azampour

Technical University of Munich & Munich Center for Machine Learning

Differentiable Rendering Ultrasound Imaging Image Inverse Problems Differentiable Ray Tracing

Open paper AskPDF arXiv

Physically-based differentiable rendering enables gradient-based optimization of scene parameters by matching rendered images to measurements, but has so far mainly focused on light transport. We extend this paradigm to medical ultrasound, where image formation resembles transient rendering: echoes are binned by time-of-flight rather than projected onto an image plane. We present UltraDiff, a modular framework for differentiable ultrasound ray tracing. UltraDiff formulates ultrasound image formation as a path-space integral, gated by travel time between the transducer and tissue interfaces, and derives a Monte Carlo estimator of both the forward model and its gradients with respect to scene parameters. We demonstrate this on an inverse geometry estimation: starting from a sphere, an SDF is optimized until simulated echoes match measured ones, recovering vertebral surfaces from simulated B-mode sweeps and from a real robotic acquisition of a spine phantom. Unlike state-of-the-art ultrasound shape reconstruction methods, which rely on pre-segmented images, our approach operates unsupervised on B-mode images through analysis-by-synthesis, while achieving competitive geometric accuracy. Implemented on top of Mitsuba 3, UltraDiff brings differentiable path tracing to a new sensing modality and provides a foundation for inverse problems in acoustic imaging.

Abstract

Novel View Synthesis

Explore similar work

UltraDiff: Differentiable Ray Tracing in Ultrasound for Shape Optimization

Multivariate Gaussian NeRF for Wide Field-of-View Ultrasound Reconstruction

Feature-Space Guided Diffusion for Realistic Ultrasound Image Synthesis