X-Splat: Unified Gaussian Splatting for Joint RGB and X-ray 3D Reconstruction

Wangze Xu; Chu Zhou; Mae Yamaguchi; Linrui Dai; I-Chao Shen; Takashi Ijiri; Imari Sato

cs.CVOct 16, 2026

X-Splat: Unified Gaussian Splatting for Joint RGB and X-ray 3D Reconstruction

Authors: Wangze Xu, Chu Zhou, Mae Yamaguchi, Linrui Dai, I-Chao Shen, Takashi Ijiri, Imari Sato

Organizations: The University of Tokyo, Japan · National Institute of Informatics, Japan · Shibaura Institute of Technology, Japan

Abstract

Jointly reconstructing RGB and X-ray provides new insights by connecting external appearance with internal structure, enabling a more complete understanding of real-world objects than single modality. Yet existing methods train separate models per modality and suffer from redundant geometry, alignment artifacts, and loose coupling that limits cross-modal insight discovery. We propose a unified representation in which all Gaussian primitives share a single set of geometric parameters, with only surface-proximal primitives additionally carrying appearance attributes. We perform cross-modal registration through clean multi-view stereo and X-ray geometric proxies instead of aligning two sets of optimized Gaussians, improving alignment stability. We then jointly optimize the unified representation via supervision from both modalities, complemented by a surface-depth consistency loss derived from X-ray geometry to guide RGB reconstruction. Experiments on the InsideOut dataset and our Flower & Insect dataset show that our framework achieves more robust cross-modal alignment, higher RGB rendering quality without compromising X-ray reconstruction, and a more compact representation than maintaining two independent geometries.

Figures & tables

Figure 1. We propose X-Splat , a unified representation where one set of Gaussian primitives carries both reflective (RGB) and absorptive (X-ray) attributes, simultaneously capturing external appearance and internal structure while enabling joint optimization. Single-modality specialists (3DGS ( Kerbl et al., 2023 ) , R 2 -Gaussian ( Zha et al., 2024 ) ) cover only one axis; the prior method InsideOut † ( Lee et al., 2025 ) covers both modalities, but with lower RGB fidelity in our reproduction (Sec. 5.1 ). Ours achieves high fidelity on both from a single model. Each Ours/InsideOut panel shows three vertical strips at the same viewpoint: pure RGB / overlay / pure X-ray.

Figure 2. Framework overview. A coarse X-ray reconstruction anchors cross-modal alignment (Sec. 4.2 ) and initializes the unified Gaussian set, in which each primitive carries shared geometry $\{\mu,R,S\}$ and an X-ray attribute $\rho$ , with RGB appearance $\{\alpha,c\}$ restricted to a surface subset (Sec. 4.1 ). The unified set is jointly optimized under three losses (right): X-ray photometric, RGB photometric, and a CT-guided depth loss (Sec. 4.3 ).

Figure 3. Geometry-based alignment. We extract surface point clouds from each modality independently (left): an X-ray surface $\mathcal{P}^{\mathrm{ct}}$ via Marching Cubes on the coarse X-ray 3DGS, and an MVS surface $\mathcal{P}^{\mathrm{mvs}}$ from multi-view RGB. PCA+ICP (right) registers the two surfaces into a shared frame.

Figure 4. Qualitative comparison of RGB novel-view synthesis. Held-out test views across both datasets; yellow boxes mark zoom-in regions.

Figure 5. CT volume slice and X-ray projection comparison. Left 4 columns: CT slices; right 4: X-ray projections. InsideOut is not shown separately, as its X-ray stage follows R 2 -Gaussian and coincides with that column.

(a) RGB Novel View Synthesis
Subject:	Matryoshka			Pharaoh			Skull			Toy Gun			Average
Metric:	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
3DGS (MVS init) ( Kerbl et al., 2023 )	29.56	0.9643	0.0393	29.64	0.9719	0.0291	31.08	0.9670	0.0475	34.54	0.9871	0.0133	31.20	0.9726	0.0323
3DGS (FDK init) ( Kerbl et al., 2023 )	29.75	0.9628	0.0408	29.51	0.9707	0.0298	30.97	0.9660	0.0501	34.55	0.9870	0.0136	31.20	0.9716	0.0336
InsideOut † ( Lee et al., 2025 )	18.13	0.8836	0.1522	21.78	0.9039	0.0927	19.57	0.9003	0.1497	25.39	0.9609	0.0481	21.22	0.9122	0.1107
Ours	30.37	0.9693	0.0354	30.89	0.9756	0.0272	32.42	0.9713	0.0433	36.68	0.9900	0.0114	32.59	0.9765	0.0293

Table 1. Per-modality reconstruction quality on InsideOut Dataset. Best in bold , second in underline .

(a) RGB Novel View Synthesis
Subject:	Tulip			Eustoma			E02			E15			E32			Average
Metric:	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
3DGS (MVS init) ( Kerbl et al., 2023 )	38.34	0.9888	0.0322	37.98	0.9883	0.0414	30.63	0.9743	0.0283	30.10	0.9749	0.0246	31.55	0.9793	0.0234	33.72	0.9811	0.0300
3DGS (FDK init) ( Kerbl et al., 2023 )	37.85	0.9874	0.0387	37.74	0.9875	0.0466	30.88	0.9741	0.0289	30.11	0.9742	0.0255	31.53	0.9788	0.0239	33.62	0.9804	0.0327
InsideOut † ( Lee et al., 2025 )	18.89	0.9148	0.1262	17.79	0.9378	0.1184	19.82	0.8809	0.1435	19.90	0.8748	0.1347	19.09	0.8534	0.1611	19.10	0.8923	0.1368
Ours	39.71	0.9917	0.0219	39.22	0.9901	0.0370	34.75	0.9839	0.0246	33.74	0.9844	0.0223	34.77	0.9851	0.0253	36.44	0.9871	0.0262

Table 2. Per-modality reconstruction quality on Flower & Insect Dataset.

Method	Input Source	CD $\downarrow$
InsideOut † Hierarchical Alignment ( Lee et al., 2025 )	Optimized Gaussian primitives	0.0978
ICP Only	Geometry-based (MVS + CT mesh)	0.1453
PCA Only		0.0250
Ours (PCA + ICP)		0.0122

Table 3. Cross-modal alignment quality on InsideOut Dataset. The Input Source column indicates whether each method operates on optimized Gaussian primitives or on clean geometric proxies (MVS surface + CT mesh).

Figure 6. Unified RGB-X-ray rendering at the same viewpoint. Each method’s cell shows overlay, RGB, X-ray, and an RGB error map at one test view.

Figure 7. Cross-modal alignment comparison across both datasets. Aligned MVS surface (orange) overlaid on X-ray surface (blue); lower CD is better.

	RGB Novel View			X-ray		Compactness
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	2D-PSNR $\uparrow$	3D-PSNR $\uparrow$	#G (k)	Size (MB)
Independent (X-ray / RGB only)	31.20	0.9716	0.0336	49.76	38.25	168.8	17.28
InsideOut † ( Lee et al., 2025 )	21.22	0.9122	0.1107	49.76	38.25	166.3	18.75
Ours (unified)	32.59	0.9765	0.0293	49.81	38.16	129.1	12.79

Table 4. Joint inside-outside reconstruction summary on InsideOut Dataset. The Independent baseline’s RGB-only model uses FDK initialization to match Ours’s init source, and its two model sizes are summed.

Configuration	Components				RGB Novel View			X-ray		Compactness
Configuration	CT depth	Surf. set	Ours align.	Shared geom.	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	2D-PSNR $\uparrow$	3D-PSNR $\uparrow$	#G (k)	Size (MB)
(a). Ours (full joint)	✓	✓	✓	✓	32.59	0.9765	0.0293	49.81	38.16	129.1	12.79
(b). w/o CT depth		✓	✓	✓	32.43	0.9757	0.0299	49.75	38.19	129.1	12.24
(c). w/o surface set			✓	✓	32.41	0.9751	0.0297	49.65	38.12	129.9	31.19
(d). w/o Ours align.				✓	31.50	0.9677	0.0382	49.65	38.16	131.6	30.12
(e). Independent					31.20	0.9716	0.0336	49.76	38.25	168.8	17.28

Table 5. Main ablation study on InsideOut Dataset. Components : CT depth , CT-depth consistency loss (Eq. 9 ); Surf. set , surface-set mechanism (Eq. 7 ); Ours align. , geometry-based alignment (Sec. 4.2 ); Shared geom. , unified Gaussian representation (Sec. 4.1 ).

Figure 8. (a) CT-depth loss ablation: removing the loss introduces visible depth artifacts. (b) Storage breakdown on Flower & Insect: Ours uses ${\sim}2.2\times$ less storage than Independent.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Scene	RGB Views (Tr / Te)	RGB Resolution	CT Volume Size
InsideOut ( Lee et al., 2025 )	Matryoshka	39 / 14	$800\times 800$	$256\times 256\times 256$
	Pharaoh	49 / 17	$800\times 800$	$256\times 256\times 256$
	Skull	36 / 13	$800\times 800$	$256\times 256\times 256$
	Toy Gun	75 / 26	$800\times 800$	$256\times 256\times 256$
Flower ( Ijiri et al., 2014 )	Eustoma	60 / 20	$800\times 800$	$310\times 211\times 230$
Flower ( Ijiri et al., 2014 )	Tulip	60 / 20	$800\times 800$	$487\times 321\times 337$

Appendix

Table 6. Per-scene dataset specifications. RGB Views (Tr/Te) lists train and test counts under our

1{:}3

split. RGB Resolution is the resolution used at training and evaluation (raw images are downsampled to a width of

800

pixels). CT Volume Size is the voxel grid at which we synthesize X-ray projections; for InsideOut it is the released volume downsampled to a uniform

256^{3}

, while for Flower and Insect we keep the native voxel grid.

Parameter	InsideOut / Flower	Insect
Source-to-origin (DSO)	$5.0$	$5.0$
Source-to-detector (DSD)	$7.0$	$7.0$
Detector resolution	$512\times 512$	$512\times 512$
Detector physical extent	$3.0\times 3.0$	$3.5\times 3.5$
Angular range	$[0^{\circ},360^{\circ})$	$[0^{\circ},360^{\circ})$
#Training / Testing views	50 / 120	50 / 120

Appendix

Table 7. Cone-beam X-ray geometry used in TIGRE. Linear quantities are given in normalized scene units (the longest extent of each object equals

2

Figure 9. Flower & Insect dataset gallery. Each row shows one scene with two RGB photographs (left) and two X-ray projections (right) at different angles. X-ray projections are visualized with the viridis colormap.

Parameter	Learning rate (init $\to$ final)
Position $\mu$	$2{\times}10^{-4}\to 2{\times}10^{-5}$
Density $\rho$	$1{\times}10^{-2}\to 1{\times}10^{-3}$
Scaling $S$	$5{\times}10^{-3}\to 5{\times}10^{-4}$
Rotation $R$	$1{\times}10^{-3}\to 1{\times}10^{-4}$
SH features $c$	$2.5{\times}10^{-3}$ (constant)
Opacity $\alpha$	$5{\times}10^{-2}$ (constant)

Appendix

Table 8. Per-attribute learning rates. Position, density, scaling, and rotation rates are linearly decayed from their initial to final values over

30{,}000

steps.

Scene	Training time (min)
Matryoshka	$21.2$
Pharaoh	$14.3$
Skull	$14.7$
Toy Gun	$14.0$
Average	$16.0$

Appendix

Table 9. Per-scene training time on the InsideOut dataset (single NVIDIA RTX A6000). Total includes Stage 1 (

3{,}000

X-ray pretraining iterations) and Stage 2 (

12{,}000

joint optimization iterations); evaluation time is excluded.

Metric	Method	Matryoshka	Pharaoh	Skull	Toy Gun	Avg.
X-ray 3D-PSNR $\uparrow$	R 2 -Gaussian (clean)	33.35	42.28	37.09	40.28	38.25
	R 2 -Gaussian (noisy)	32.92	41.92	36.39	39.90	37.78
	Ours (clean)	33.39	42.13	36.90	40.25	38.17
	Ours (noisy)	32.94	41.68	36.30	39.78	37.68
RGB PSNR $\uparrow$	Ours (clean)	30.37	30.89	32.42	36.68	32.59
RGB PSNR $\uparrow$	Ours (noisy)	30.35	30.85	32.38	36.68	32.57

Appendix

Table 10. Robustness to noisy X-ray projections on the InsideOut dataset.

Metric	Method	Matryoshka	Pharaoh	Skull	Toy Gun	Avg.
X-ray 2D-PSNR $\uparrow$	R 2 -Gaussian	42.57	51.20	44.68	45.67	46.03
X-ray 2D-PSNR $\uparrow$	Ours	42.66	51.05	44.59	45.65	45.99
X-ray 3D-PSNR $\uparrow$	R 2 -Gaussian	28.67	40.23	33.01	36.54	34.61
X-ray 3D-PSNR $\uparrow$	Ours	28.74	39.77	32.96	36.58	34.51
Surface CD $\downarrow$	R 2 -Gaussian	0.0035	0.0028	0.0046	0.0028	0.0034
Surface CD $\downarrow$	Ours	0.0035	0.0029	0.0046	0.0028	0.0034

Appendix

Table 11. X-ray reconstruction with

25

training projections (uniform over

360^{\circ}

) on the InsideOut dataset. Surface CD denotes the Chamfer distance to the ground-truth CT surface.

Metric	Method	Matryoshka	Pharaoh	Skull	Toy Gun	Avg.
X-ray 2D-PSNR $\uparrow$ (out-of-arc)	R 2 -Gaussian	41.63	46.38	41.19	40.89	42.52
X-ray 2D-PSNR $\uparrow$ (out-of-arc)	Ours	41.64	46.03	40.97	40.79	42.36
X-ray 2D-PSNR $\uparrow$ (in-arc)	R 2 -Gaussian	49.88	53.96	51.60	51.71	51.79
X-ray 2D-PSNR $\uparrow$ (in-arc)	Ours	50.51	54.78	51.23	51.59	52.03
X-ray 3D-PSNR $\uparrow$	R 2 -Gaussian	27.45	36.62	31.63	34.88	32.64
X-ray 3D-PSNR $\uparrow$	Ours	27.35	36.18	31.52	34.88	32.48

Appendix

Table 12. Limited-angle X-ray reconstruction on the InsideOut dataset. Training projections are confined to a

120^{\circ}

arc; X-ray 2D-PSNR is reported on test projections inside the arc (in-arc) and on the missing wedge (out-of-arc). The alignment is fixed to the clean full-view transformation.

Metric	Alignment	Matryoshka	Pharaoh	Skull	Toy Gun	Avg.
RGB PSNR $\uparrow$	fx (clean $T$ )	30.36	30.83	32.42	36.66	32.57
RGB PSNR $\uparrow$	al (derived $T$ )	30.43	30.84	32.37	36.63	32.57
X-ray 2D-PSNR $\uparrow$ (out-of-arc)	fx (clean $T$ )	41.64	46.03	40.97	40.79	42.36
X-ray 2D-PSNR $\uparrow$ (out-of-arc)	al (derived $T$ )	41.59	46.00	41.20	40.88	42.42
Surface CD $\downarrow$	fx (clean $T$ )	0.0038	0.0030	0.0049	0.0029	0.0036
Surface CD $\downarrow$	al (derived $T$ )	0.0038	0.0030	0.0050	0.0035	0.0038

Appendix

Table 13. Alignment robustness under limited-angle X-ray on the InsideOut dataset. Both settings use our full model, with the alignment fixed to the clean full-view transformation (fx) or re-derived by PCA+ICP from the CT mesh of the

120^{\circ}

limited-angle reconstruction (al).

Metric	Method	Tulip	Eustoma	E02	E15	E32	Avg.
PSNR $\uparrow$	3DGS (15 views)	28.10	26.81	29.73	29.13	30.18	28.79
	Ours (15 views)	33.59	33.46	32.86	32.16	32.69	32.95
	$\Delta$ (Ours $-$ 3DGS)	$+5.49$	$+6.65$	$+3.13$	$+3.03$	$+2.51$	$+4.16$
SSIM $\uparrow$	3DGS (15 views)	0.9596	0.9611	0.9644	0.9647	0.9691	0.9638
SSIM $\uparrow$	Ours (15 views)	0.9807	0.9806	0.9773	0.9775	0.9780	0.9788
LPIPS $\downarrow$	3DGS (15 views)	0.0662	0.0735	0.0323	0.0287	0.0279	0.0457

Appendix

Table 14. RGB reconstruction with

15

training views on the Flower & Insect dataset (X-ray unchanged). The 3DGS baseline uses FDK initialization, the same initialization source as Ours; the full-view gap is

+2.72

dB (Table 2 ).

Figure 10. Additional CT slice and X-ray projection results (supplemental to Fig. 5 ). Six additional scenes with the same eight-column layout. Findings match the main figure: our unified model reproduces R 2 -Gaussian’s fidelity on both modalities, while FDK shows the expected closed-form noise floor.

Method	Matryoshka	Pharaoh	Skull	Toy Gun	Avg.
3DGS (MVS init)	29.56	29.64	31.08	34.54	31.20
Separate RGB set + frozen CT prior	29.77	29.63	31.06	34.59	31.26
Ours (joint)	30.37	30.89	32.42	36.68	32.59

Appendix

Table 15. RGB PSNR of a separate RGB Gaussian set with a frozen CT depth prior on the InsideOut dataset. The 3DGS and Ours rows are from Table 1 .

Figure 11. Additional RGB novel-view synthesis scenes (supplemental to Fig. 4 ).

Figure 12. Additional unified RGB-X-ray showcase scenes (supplemental to Fig. 6 ). Six additional scenes spanning both datasets, with the same three-panel cell layout. Findings match: Ours produces consistent inside/outside rendering across all scenes; Ours + IO-align retains the unified representation but with mild surface-alignment artifacts; our InsideOut reproduction shows consistently lower RGB quality, consistent with the main-paper results.

Figure 13. Additional alignment-comparison scenes (supplemental to Fig. 7 ).

3D Reconstruction

3D reconstruction recovers a scene’s shape, appearance, and camera motion from images, video, depth, or other sensor measurements.

See all 950 papers

No Calibration, No Depth, No Problem: Cross-Sensor View Synthesis with 3D Consistency
Feb 27, 2026Cho-Ying Wu, Zixun Huang, Xinyu Huang +1Novel View Synthesis Cross-Modal Image Registration
Bosch Research North America & Bosch Center for Artificial Intelligence (BCAI)
Shape-guided Gaussian Splatting for Sparse-View X-ray 3D Reconstruction
Sep 9, 2026Pranav Poudel, Florence Dell'Aniello Picard, Nairouz Shehata +23D Gaussian Splatting 3D Medical Imaging
Polytechnique Montréal, Canada · Mila - Quebec AI Institute, Canada · CHUM - University of Montreal Hospital, Canada
StructSplat: Generalizable 3D Gaussian Splatting from Uncalibrated Sparse Views
Jun 26, 2026Jia-Chen Zhao, Beiqi Chen, Xinyang Chen +23D Gaussian Splatting 3D Reconstruction
Harbin Institute of Technology (Shenzhen) · Great Bay University · Guangzhou CloudButterfly Technology Co., Ltd.

Feb 27, 2026J/K move · Enter open · S save

No Calibration, No Depth, No Problem: Cross-Sensor View Synthesis with 3D Consistency

Cho-Ying Wu, Zixun Huang, Xinyu Huang, Liu Ren

Bosch Research North America & Bosch Center for Artificial Intelligence (BCAI)

Novel View Synthesis Cross-Modal Image Registration

Open paper AskPDF arXiv

We present the first study of cross-sensor view synthesis across different modalities. We examine a practical, fundamental, yet widely overlooked problem: getting aligned RGB-X data, where most RGB-X prior work assumes such pairs exist and focuses on modality fusion, but it empirically requires huge engineering effort in calibration. We propose a match-densify-consolidate method. First, we perform RGB-X image matching followed by guided point densification. Using the proposed confidence-aware densification and self-matching filtering, we attain better view synthesis and later consolidate them in 3D Gaussian Splatting (3DGS). Our method uses no 3D priors for X-sensor and only assumes nearly no-cost COLMAP for RGB. We aim to remove the cumbersome calibration for various RGB-X sensors and advance the popularity of cross-sensor learning by a scalable solution that breaks through the bottleneck in large-scale real-world RGB-X data collection.

Abstract

3D Reconstruction

Explore similar work

No Calibration, No Depth, No Problem: Cross-Sensor View Synthesis with 3D Consistency

Shape-guided Gaussian Splatting for Sparse-View X-ray 3D Reconstruction

StructSplat: Generalizable 3D Gaussian Splatting from Uncalibrated Sparse Views