X-ray, known for its ability to reveal internal structures of objects, is expected to provide richer information for 3D reconstruction than visible light. Yet, existing neural radiance fields (NeRF) algorithms overlook this important nature of X-ray, leading to their limitations in capturing structural contents of imaged objects. In this paper, we propose a framework, Structure-Aware X-ray Neural Radiodensity Fields (SAX-NeRF), for sparse-view X-ray 3D reconstruction. Firstly, we design a Line Segment-based Transformer (Lineformer) as the backbone of SAX-NeRF. Linefomer captures internal structures of objects in 3D space by modeling the dependencies within each line segment of an X-ray. Secondly, we present a Masked Local-Global (MLG) ray sampling strategy to extract contextual and geometric information in 2D projection. Plus, we collect a larger-scale dataset X3D covering wider X-ray applications. Experiments on X3D show that SAX-NeRF surpasses previous NeRF-based methods by 12.56 and 2.49 dB on novel view synthesis and CT reconstruction. Code, models, and data are released at https://github.com/caiyuanhao1998/SAX-NeRF . Our project page at https://caiyuanhao1998.github.io/project/SAX-NeRF/ shows more video and interactive results. An online Gradio demo is available on HuggingFace at https://huggingface.co/spaces/CaiYuanhao/SAX-NeRF
Figures & tables
Figure 2: Visible light vs. X-ray. Visible light imaging relies on reflection. X-ray imaging is based on penetration and attenuation.
Figure 3: Overview of our method. (a) SAX-NeRF uses (i) MLG strategy to sample an X-ray batch R . Then N point positions P on each X-ray r∈R are sampled and input into (ii) Lineformer to produce the radiodensity D . (b) Line Segment-based Attention Block (LSAB) is the basic unit of Lineformer. It captures inner structural dependencies by (c) Line Segment-based Multi-head Self-Attention (LS-MSA).
Figure 4: Comparison of ray sampling. (a) The naive strategy samples X-rays that land on scattered pixels. (b) Our MLG strategy performs pixel- and patch-level sampling on foreground regions.
Method
InTomo [ 56 ]
NeRF [ 35 ]
NeAT [ 43 ]
TensoRF [ 15 ]
NAF [ 59 ]
SAX-NeRF
Scene
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
Jaw
26.91
0.9937
34.84
0.9981
32.68
0.9911
34.06
0.9964
39.89
0.9988
42.75
0.9992
Leg
42.53
0.9976
45.92
0.9989
47.71
0.9981
41.40
0.9969
50.87
0.9988
56.86
0.9996
Box
34.65
0.9963
35.67
0.9985
36.14
0.9957
35.43
0.9977
35.98
0.9955
39.67
0.9992
Carp
24.04
0.9648
20.62
0.9467
31.26
0.9620
37.35
0.9973
29.60
0.9593
59.88
0.9999
Foot
39.48
0.9979
41.05
0.9989
38.24
0.9963
37.73
0.9929
38.35
0.9913
46.64
0.9994
Table 1: Quantitative comparisons on the novel view synthesis task. The best results are in bold and the second-best results are underlined .
Figure 5: Qualitative results of novel view synthesis on the scenes of backpack (top) and carp (bottom). Please zoom in for a better view.
Method
FDK [ 25 ]
ASD-POCS [ 46 ]
SART [ 2 ]
InTomo [ 56 ]
NeRF [ 35 ]
NeAT [ 43 ]
TensoRF [ 15 ]
NAF [ 59 ]
SAX-NeRF
Scene
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
Jaw
28.58
0.7816
33.25
0.9325
33.13
0.9301
31.95
0.9162
32.17
0.9114
32.53
0.9139
31.90
0.8971
34.14
0.9358
35.47
0.9525
Leg
28.48
0.6690
35.39
0.9826
35.30
0.9809
36.41
0.9882
39.27
0.9938
40.29
0.9902
40.70
0.9923
41.28
0.9940
43.47
0.9973
Box
24.14
0.5616
31.27
0.9226
31.20
0.9200
30.59
0.9140
33.58
0.9494
31.58
0.9298
32.17
0.9314
31.78
0.9309
35.33
0.9602
Carp
32.32
0.8177
37.63
0.9777
36.89
0.9682
32.47
0.9493
32.99
0.9529
36.85
0.9576
37.52
0.9687
37.93
0.9711
42.72
0.9902
Foot
24.53
0.6000
29.98
0.9208
30.29
0.9296
31.43
0.9127
30.03
0.9072
30.86
0.9221
30.46
0.9153
31.63
0.9363
32.25
0.9403
Table 2: Quantitative comparisons on the CT reconstruction task. The best results are in bold and the second-best results are underlined .
Figure 6: Visual results of CT reconstruction on the scenes of head, carp, box, and teapot (top to bottom). Please zoom in for a better view.
Table 3: We conduct ablation study on all scenes of X3D. Average PSNR and SSIM are reported on the NVS and CT reconstruction tasks.
NeRF and Gaussian splatting methods have been successfully applied on X-ray scenes where the views are too sparse for 3D reconstruction via classical methods. Ultra-sparse scenes with 10 or fewer views such as those with high-rate or low-dose acquisition still, however, present a significant challenge. To address this problem we present a new framework, Optimised X-ray Neural Radiance Fields (OX-NeRF), that combines cross-scene feature learning with scene-specific optimisation to reconstruct sets of related scenes. OX-NeRF employs a convolutional neural network (CNN) to identify cross-scene features while maintaining scene-specific multi-resolution hash grids of spatial features. The paired representations are fused and passed to a multilayer perceptron (MLP); the CNN, hash grids and MLP are then jointly optimised end-to-end. Benchmarking on parallel-beam and cone-beam X-ray datasets shows OX-NeRF provides significantly higher reconstruction accuracy on ultra-sparse scenes compared to existing radiance field methods.
Thomas Welsch, Min-Hsin Tu, David J. Chapman +1
Department of Engineering Science University of Oxford Oxford, United Kingdom
X-ray is widely applied for transmission imaging due to its stronger penetration than natural light. When rendering novel view X-ray projections, existing methods mainly based on NeRF suffer from long training time and slow inference speed. In this paper, we propose a 3D Gaussian splatting-based method, namely X-Gaussian, for X-ray 3D reconstruction. Firstly, we redesign a radiative Gaussian point cloud model inspired by the isotropic nature of X-ray imaging. Our model excludes the influence of view direction when learning to predict the radiation intensity of 3D points. Based on this model, we develop a Differentiable Radiative Rasterization (DRR) with CUDA implementation. Secondly, we formulate a principled Radiative Gaussian Volume Querying (RGVQ) method that defines a continuous attenuation field from the original radiative point attributes and enables direct CT volume reconstruction through CUDA-based parallel querying. Thirdly, we customize an Angle-pose Cuboid Uniform Initialization (ACUI) strategy that directly uses the parameters of the X-ray scanner to compute the camera information and then uniformly samples point positions within a cuboid enclosing the scanned object. On novel view synthesis, X-Gaussian outperforms SOTA methods by 6.5 dB while enjoying less than 15% training time and over 73x inference speed. On CT reconstruction, X-Gaussian outperforms SOTA methods by 0.40 dB and 4.34 dB on synthetic and real scenes, while achieving 4.30x faster inference of the learned CT volume. Code, models, and datasets are at https://github.com/caiyuanhao1998/X-Gaussian . More video and interactive visualizations are presented on our project page at https://caiyuanhao1998.github.io/project/X-Gaussian . We also built an online Gradio demo page on HuggingFace at https://huggingface.co/spaces/CaiYuanhao/X-Gaussian .
Yuanhao Cai, Yixun Liang, Jiahao Wang +5
Johns Hopkins University · HKUST(GZ) · Shanghai Jiao Tong University
Sparse-view X-ray 3D reconstruction is essential for reducing radiation exposure, but recovering a density field from a handful of X-ray projections is severely ill-posed. Recently, 3D Gaussian Splatting has achieved state-of-the-art performance in sparse-view reconstruction by representing the volume using explicit, optimized primitives, but it requires dozens of projected views. With fewer views, reconstruction quality degrades severely since the explicit primitives are optimized freely without any anatomical information. Anatomical structures, in contrast, share similar geometry and density across a population. Their variations are bounded within a limited range that statistical shape models can capture. This paper proposes a shape-guided Gaussian splatting framework for sparse-view X-ray 3D reconstructions. Our contribution lies in driving Gaussian positions toward anatomically valid configurations, alongside atlas-based density regularization. Our method ensures anatomically consistent reconstruction and improves PSNR by 2.83 dB over a state-of-the-art Gaussian splatting baseline with as few as 5 views. Code Available: https://github.com/polyshape-lab/ShapeGuidedGaussian