CTBakeGS: training-free baking of photorealistic CT volume renderings into 3D Gaussian splats
Authors: Haruki Fukuda, Constantin Lux, Wolf Schweitzer, Tadashi Hosoya, Lars Ebert
Organizations: Department of Forensic Medicine, Gunma University Graduate School of Medicine, Maebashi, Japan · Institute of Forensic Medicine, Universität Zürich, Zürich, Switzerland · Biometrics/Forensic Imaging, Zurich Forensic Science Institute, Zurich, Switzerland
Photorealistic volume rendering of computed tomography (CT) may convey anatomy intuitively, but it needs the CT data, dedicated software and a powerful graphics processor. Three-dimensional Gaussian splatting (3DGS) displays photorealistic scenes in real time on less powerful devices, but existing medical approaches train each scene for minutes to hours. We aimed to convert a CT volume rendering directly into a 3DGS model without training. Splats were placed on an adaptive octree of millimetre-sized cells. Their colour and opacity were computed on the graphics processor with the renderer's lighting, cell opacity was converted analytically into splat opacity, and faint splats were retained stochastically above the 1/255 opacity floor of standard 3DGS rasterisers. Six public CT scans (1.5-mm voxels) were converted with three transfer functions and compared with converged references by peak signal-to-noise ratio (PSNR), structural similarity and a learned perceptual metric. Two-million-splat models reached a PSNR of 32.6 +/- 0.9 dB for bone, 30.1 +/- 1.0 dB for contrast-enhanced vessels and 32.3 +/- 1.5 dB for skin and bone. A conversion pass (usually two or three per model) took 4-30 s on a current laptop and 9-60 s on a desktop with a 4-GB graphics card. On the laptop, the models rendered at 68 frames per second at 800 x 800 pixels. Training-free conversion produced real-time 3DGS models of photorealistic CT renderings in about a minute or less per conversion pass on ordinary computers, with fidelity that we consider sufficient for interactive viewing.
Figures & tables
Fig. 1: Training-free conversion of a CT volume rendering into 3D Gaussian splats. The CT volume is shown with a chosen transfer function, which maps Hounsfield units to colour and opacity; the boxes marked Step 1–5 correspond to the steps in the text. A light volume, shared with the photorealistic renderer, stores the visibility of a sky dome in 24 directions and the transmittance towards a key light with soft shadows. The volume is divided into an adaptive octree of millimetre-sized cubic cells, split only where opacity changes, down to sub-voxel size; the finest size is adjusted to reach the requested number of splats. Each cell is sampled at 27 stratified points with the colour and lighting of the volume renderer, giving one colour with baked lighting and an opacity. Opacity mapping replaces each cell by a Gaussian with the same total opacity, fainter at the centre and wider than the cell ( s=0.55h ; top); in a profile through a row of cells, the overlapping Gaussians add up to the opacity of the cells (bottom). Faint splats, whose opacity would fall below the 1/255 floor of standard rasterisers, are retained stochastically with raised opacity. The output is a standard 3DGS file (.ply or .spz); the example shows case 2 with the contrast-enhanced vessel transfer function. CT, computed tomography; 3DGS, three-dimensional Gaussian splatting
Look, views
PSNR (dB)
SSIM
LPIPS
PSNR, real-time
1 M
2 M
3 M
2 M
2 M
photorealistic (dB)
Bone, wide
32.02 ± 1.42
32.61 ± 0.89
32.52 ± 0.86
0.980 ± 0.005
0.028 ± 0.014
44.61 ± 1.26
Bone, close-up
28.46 ± 1.87
29.63 ± 1.55
30.21 ± 1.42
0.949 ± 0.012
0.135 ± 0.041
45.89 ± 1.64
Vessels, wide
29.17 ± 1.15
30.09 ± 0.98
30.43 ± 0.87
0.960 ± 0.011
0.036 ± 0.016
42.28 ± 1.47
Vessels, close-up
25.23 ± 1.55
26.56 ± 1.62
27.29 ± 1.60
0.895 ± 0.023
0.150 ± 0.019
43.89 ± 1.51
Skin and bone, wide
31.78 ± 1.80
32.32 ± 1.51
32.19 ± 1.18
0.961 ± 0.017
0.091 ± 0.040
47.12 ± 0.79
Table 1: Image quality of the 3DGS models against converged photorealistic references
Fig. 2: Representative results: converged references, 3DGS models and their differences. (a) Bone, case 1; (b) contrast-enhanced vessels, case 6; (c) skin and bone, case 3; one wide view each, cropped to the body. Top: converged photorealistic volume rendering (reference); middle: 3DGS model with two million splats; bottom: difference from the reference (mean absolute difference over the colour channels, 0 to ≥ 50 grey levels of 255). Insets (yellow frames) show a close-up view of the same scene; dashed boxes mark the region it covers in the wide view. Errors lie mainly along the edges of dense structures for bone and vessels and are spread over the translucent layer for skin and bone. 3DGS, three-dimensional Gaussian splatting
Look
Views
Case 1
Case 2
Case 3
Case 4
Case 5
Case 6
Bone
wide
33.6/0.982/0.031
32.0/0.971/0.054
33.1/0.980/0.023
31.1/0.979/0.022
33.1/0.980/0.018
32.6/0.985/0.017
Bone
close-up
30.0/0.961/0.178
31.2/0.948/0.192
30.0/0.945/0.124
27.1/0.935/0.118
30.9/0.965/0.085
28.5/0.941/0.114
Vessels
wide
30.2/0.966/0.042
30.4/0.942/0.065
30.3/0.955/0.034
28.3/0.962/0.030
31.3/0.969/0.019
30.1/0.969/0.024
Vessels
close-up
27.4/0.925/0.156
28.7/0.911/0.174
25.9/0.861/0.163
24.0/0.877/0.147
27.4/0.903/0.120
26.1/0.893/0.138
Skin and bone
wide
32.3/0.968/0.088
29.8/0.927/0.170
32.9/0.963/0.087
32.0/0.968/0.070
34.5/0.976/0.064
32.5/0.967/0.069
Skin and bone
close-up
29.4/0.942/0.306
28.7/0.935/0.356
29.9/0.922/0.322
28.2/0.904/0.314
30.5/0.926/0.267
27.9/0.921/0.316
Table 2: Per-case image quality with two million splats, PSNR (dB) / SSIM / LPIPS
Model
1 M
2 M
3 M
Splats (million)
1.04 (0.92–1.37)
2.00 (1.89–2.15)
2.97 (2.76–3.23)
Conversion time, MacBook Pro (s)
11.0 (2.2–24.0)
14.3 (4.2–29.5)
17.1 (3.1–30.7)
Frame rate, MacBook Pro (fps)
122 (97–137)
68 (62–74)
48 (43–53)
Conversion time, Windows desktop (s)
23.1 (4.6–50.0)
29.4 (8.8–60.2)
35.1 (7.4–63.0)
Frame rate, Windows desktop (fps)
52 (36–60)
28 (22–33)
19 (16–22)
Conversion time, MacBook Air 2017 (s)
85 (22–193)
106 (30–216)
— a
Table 3: Splat counts, conversion times and frame rates
Look
Views
Case 1
Case 2
Case 3
Case 4
Case 5
Case 6
Bone
wide
33.6 / 33.4
32.0 / 31.6
33.1 / 32.5
31.2 / 30.8
33.1 / 32.9
32.6 / 32.6
Bone
close-up
30.0 / 29.7
31.2 / 30.7
30.0 / 29.2
27.1 / 26.7
30.9 / 30.0
28.5 / 28.5
Vessels
wide
30.2 / 30.1
30.4 / 29.7
30.3 / 29.9
28.3 / 27.9
31.3 / 30.9
30.1 / 29.7
Vessels
close-up
27.4 / 27.3
28.7 / 27.7
25.9 / 25.4
24.0 / 23.5
27.4 / 26.6
26.1 / 25.6
Skin and bone
wide
32.3 / 29.8
29.8 / 23.8
32.9 / 31.4
32.0 / 31.2
34.5 / 33.7
32.5 / 30.7
Skin and bone
close-up
29.4 / 27.4
28.7 / 23.7
29.9 / 28.6
28.2 / 27.2
30.5 / 29.8
27.9 / 26.5
Table 4: Per-case PSNR (dB) with two million splats, with / without faint splats
Fig. 3: Faint splats: case 2, skin and bone, two million splats. (a) Converged photorealistic volume rendering (reference); (b) 3DGS model without faint splats (ablation of Step 5); (c) 3DGS model with faint splats (CTBakeGS). Top row: one wide view, cropped to the body; middle row: close-up of the dashed box; bottom row: difference from the reference as in Fig. 2 (0 to ≥ 50 grey levels of 255). Without them, the thin outer soft-tissue layer, whose splats fall below the 8-bit opacity floor, is missing and the ribs show through (PSNR 22.5 dB in this view); with them it is restored (30.1 dB), with a fine grain visible at close range. 3DGS, three-dimensional Gaussian splatting; PSNR, peak signal-to-noise ratio
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Case
ID
Description
Matrix (voxels)
Voxels (million)
MacBook Pro (s)
Windows desktop (s)
MacBook Air 2017 (s)
1
s0289
Head CT angiography
145 × 145 × 233
4.9
5.4
11.1
46
2
s0138
Heart and thoracic aorta
221 × 221 × 215
10.5
12.7
25.4
82
3
s1340
Thorax to pelvis
275 × 275 × 431
32.6
18.5
37.9
131
4
s0970
Polytrauma
333 × 333 × 511
56.7
20.8
43.2
166
5
s1121
Pelvis-to-leg CT angiography
281 × 281 × 519
41.0
20.5
42.4
148
6
s0931
Neck-to-leg CT angiography
245 × 187 × 479
21.9
8.0
16.6
60
Appendix
Table S1: Cases (TotalSegmentator dataset v2)
Look
Views
Variant
Δ PSNR (dB)
p
Δ SSIM
p
Δ LPIPS
p
Bone
wide
1 M splats
−0.885
0.156
−0.003
0.312
+0.005
0.094
Bone
wide
3 M splats
+0.036
1.000
−0.000
0.562
−0.001
0.438
Bone
wide
No anti-aliasing compensation
−3.832
0.031
−0.010
0.031
+0.001
0.562
Bone
wide
No opacity mapping
−4.498
0.031
−0.012
0.062
+0.001
1.000
Bone
wide
Without faint splats
−0.274
0.031
−0.001
0.031
+0.001
0.031
Bone
wide
Uniform grid (1 cell/voxel)
−0.171
0.438
−0.000
0.438
+0.001
0.219
Appendix
Table S2: Paired comparisons of each variant with the full method (two million splats)
Photorealistic volumetric rendering of CT scans greatly benefits clinical workflows, yet neural approaches such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) require prohibitive per-scan optimization (hours for NeRF, about 30 minutes for 3DGS), making them impractical in clinical settings. We propose Render-FM, a feedforward model that eliminates this bottleneck by directly regressing 6D Gaussian Splatting (6DGS) parameters from a CT volume in a single 2.8-second forward pass, a 500x speedup over per-scan optimization. To bridge the domain gap between natural scene reconstruction and medical volumetric rendering, we introduce Anatomy-Guided Priming (AGP), which incorporates segmentation masks and transfer functions as structural and appearance priors, information that existing Gaussian splatting methods overlook. Built on an nnU-Net-inspired 3D U-Net trained on diverse CT scans, Render-FM predicts per-voxel 6DGS parameters and supports immediate real-time rendering. Unlike per-scan methods, it generalizes to unseen anatomies, novel transfer functions, and enables compositional organ visualization with zero additional preparation time. Optional 89-second fine-tuning further improves quality, surpassing per-scan optimized baselines. Project page: https://gaozhongpai.github.io/renderfm/.
High-fidelity 3D medical visualization supports applications such as clinical assessment and surgical planning. Sparse-view computed tomography (CT) can reduce projection requirements and associated radiation exposure, but limited observations may introduce structural artifacts and reconstruction uncertainty. Although 3D Gaussian Splatting (3DGS) provides an efficient explicit representation for volumetric rendering, existing CT methods based on standard Gaussian primitives may be sensitive to unreliable observations under sparse-view acquisition. We present TR-GS, a Gaussian-splatting framework for sparse view CT volumetric rendering. TR-GS replaces standard Gaussian primitives with projectable Student's t-distribution primitives and introduces a ray-confidence model that regulates their degrees of freedom according to local ray observability. Confidence-guided 3D wavelet regularization is further used to balance high-frequency detail preservation and noise suppression. This work is licensed under a Creative Commons Attribution 4.0 International License. Experiments on synthetic and real-world datasets show that TR-GS improves over representative baselines in most evaluated settings and remains competitive in the remaining cases. The resulting volumetric representations may support downstream medical multimedia applications, including XR-based visualization and interactive clinical rendering.
Zedong Xiao, Yiren Wang, Zhou Liu +2
Shenzhen University Shenzhen, China · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) Shenzhen, China
Gaussian Splatting (GS) has emerged as a dominating technique for image rendering and has quickly been adapted for the X-ray Computed Tomography (CT) reconstruction task. However, despite its growing popularity, the benefits of GS are typically not substantial enough to motivate a transition from well-established reconstruction algorithms. This paper addresses the most significant remaining limitations of the GS-based approach by introducing FaCT-GS, a framework for fast and flexible CT reconstruction. Enabled by an in-depth optimization of the voxelization and rasterization pipelines, our new method is significantly faster than its predecessors and scales well with projection and output volume size. Furthermore, the improved voxelization enables rapid fitting of Gaussians to pre-existing volumes, which can serve as a prior for warm-starting the reconstruction, or simply as an alternative, compressed representation. FaCT-GS is over 4X faster than the State of the Art GS CT reconstruction on standard 512x512 projections, and over 13X faster on 2k projections. Implementation and data available through: https://github.com/PaPieta/fact-gs.
Pawel Tomasz Pieta, Rasmus Juul Pedersen, Sina Borgi +3
Technical University of Denmark, Kgs. Lyngby, Denmark