No Calibration, No Depth, No Problem: Cross-Sensor View Synthesis with 3D Consistency

Cho-Ying Wu; Zixun Huang; Xinyu Huang; Liu Ren

cs.CVFeb 27, 2026

No Calibration, No Depth, No Problem: Cross-Sensor View Synthesis with 3D Consistency

Authors: Cho-Ying Wu, Zixun Huang, Xinyu Huang, Liu Ren

Organizations: Bosch Research North America & Bosch Center for Artificial Intelligence (BCAI)

Abstract

We present the first study of cross-sensor view synthesis across different modalities. We examine a practical, fundamental, yet widely overlooked problem: getting aligned RGB-X data, where most RGB-X prior work assumes such pairs exist and focuses on modality fusion, but it empirically requires huge engineering effort in calibration. We propose a match-densify-consolidate method. First, we perform RGB-X image matching followed by guided point densification. Using the proposed confidence-aware densification and self-matching filtering, we attain better view synthesis and later consolidate them in 3D Gaussian Splatting (3DGS). Our method uses no 3D priors for X-sensor and only assumes nearly no-cost COLMAP for RGB. We aim to remove the cumbersome calibration for various RGB-X sensors and advance the popularity of cross-sensor learning by a scalable solution that breaks through the bottleneck in large-scale real-world RGB-X data collection.

Figures & tables

Figure 2: Homography warping assumes 3D planar structures and causes visible misalignment (statue areas) when the scene contains distinct fore-/background layers.

Figure 3: Method Overview. Our approach consists of three stages. In the first stage, we perform cross-modality feature matching to establish correspondences between RGB and X-images. The matched points are sampled and accumulated onto RGB views to produce semi-dense X-images $\mathcal{X}_{m}$ along with multi-level confidence maps $C_{m}$ . In the second stage, we conduct RGB-guided densification to get dense X-images from RGB with semi-dense X as cues. Our Confidence-Aware Densification and Fusion module integrates confidence maps from the image matching stage to guide the densification to concentrate on higher-confidence points and get robust $\mathcal{X}_{d}$ In the final stage, our proposed self-matching mechanism further filters inconsistent patches, and the results are fed back into the densification stage for refinement. To further improve multi-view consistency, we train RGB-X 3DGS using COLMAP-calibrated RGB views to consolidate both modalities into a unified 3D RGB-X radiance field, which improves multi-view consistency and further enables cross-sensor view synthesis.

Figure 4: Visual Results on METU-VisTIR-Cloudy. Our results attain much clearer, sharper, and smoother surface for rendering.

Method	Icos $\uparrow$	p30 $\uparrow$	p50 $\uparrow$	p70 $\uparrow$	p90 $\uparrow$	ITM $\uparrow$	ITcos $\uparrow$
XoFTR [ 79 ]	0.62	25.13	27.49	29.31	31.48	0.69	0.39
LightGlue [ 51 ]	0.61	25.89	28.28	30.14	32.35	0.91	0.40
LoFTR [ 74 ]	0.66	29.38	32.07	33.95	36.04	0.89	0.45
MINIMA lg [ 66 ]	0.67	29.93	32.78	34.72	36.99	0.88	0.44
Ours	0.69	31.18	34.39	36.43	38.72	0.92	0.45

Table 1: Results on METU-VisTIR-Cloudy . Results are the mean of all six sequences. We compare with warping by different image matchers, all trained and rendered by 3DGS.

Figure 5: Comparison on Temporal Consistency for Image Generation. StyleBooth [ 25 ] generation for thermal images cannot guarantee temporal consistency due to inherent ambiguity, while ours densification creates more consistent multi-views. NIR is closer to the visual spectrum and thus easier to ensure consistency, but the specialized method PixNext [ 35 ] still cannot ensure the correct intensity. Compared with our strategy, image translation from the original domain still suffers from inaccurate transformations, whereas our match-densify-consolidate approach uses information from the target domain as anchors for densification and achieves better results.

Figure 6: With the aid of sharper and clearer thermal images, the RGB view synthesis quality is also slightly enhanced.

MEt3R $\downarrow$	Sc1	Sc2	Sc3	Sc4	Sc5	Sc6	Mean
StyleBooth [ 25 ]	0.202	0.331	0.357	0.285	0.289	0.319	0.297
Ours	0.141	0.123	0.143	0.258	0.158	0.202	0.171

Table 2: Comparison on Temporal Consistency . Compared with pure image generation, our match-densify-consolidate strategy obtains better multiview consistency from lower MEt3R score.

Figure 7: Visual Results on RGB-NIR-Stereo. Our view synthesis showcases better structures closer to the groundtruth (GT).

Train view	Building		Landscape		Parterre		RoadBlock		Mean
	RMSE	MAE	RMSE	MAE	RMSE	MAE	RMSE	MAE	RMSE	MAE
XoFTR [ 79 ]	2.50	1.81	1.08	0.79	2.25	1.57	1.21	0.93	1.76	1.28
LightGLUE [ 51 ]	2.48	1.79	1.10	0.81	2.28	1.57	1.26	0.97	1.78	1.29
LoFTR [ 74 ]	2.44	1.75	1.06	0.77	2.20	1.52	1.36	0.99	1.77	1.26
MINIMA [ 66 ]	2.51	1.83	1.08	0.80	2.35	1.66	1.36	1.04	1.83	1.33
Ours	2.43	1.72	1.02	0.71	2.18	1.49	1.18	0.91	1.70	1.21

Table 3: Comparison on RGBT-Scenes. We show results for train and novel view synthesis. Each scene reports RMSE

\downarrow

and MAE

\downarrow

in ∘ C. Novel views from the official split contain simpler views with lower errors than train views.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
PixNext [ 35 ]	11.283	0.441	0.452
XoFTR [ 79 ]	14.846	0.321	0.486
LightGlue [ 51 ]	16.253	0.377	0.455
LoFTR [ 74 ]	20.179	0.551	0.356
MINIMA [ 66 ]	20.392	0.568	0.360
Ours	21.152	0.581	0.344

Table 4: Results on RGB-NIR-Stereo . Image quality metrics are shown. All methods are run with 3DGS.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Ours (final)	21.152	0.581	0.344
-3DGS	21.042	0.597	0.378
-Self-matching & filtering	20.235	0.522	0.386
-DySPN confidence	19.621	0.508	0.396
-Multi-level thresholds	19.215	0.495	0.420
-Area sampling	16.454	0.408	0.467

Table 5: Ablation Study. We ablate each component in turn from the pipeline and report the average scores on RGB-NIR-Stereo.

Method	09-28-16-48-17/1	09-28-16-48-17/2	09-28-17-06-04/1	09-28-17-06-04/2	10-26-15-53-03
PixNext [ 35 ]	11.19	11.99	12.81	12.22	8.65
XoFTR [ 79 ]	14.08	14.01	14.72	16.57	16.02
LightGlue [ 51 ]	14.03	14.10	14.61	16.67	16.07
LoFTR [ 74 ]	19.58	17.15	18.54	19.19	16.02
MINIMA [ 66 ]	21.16	18.80	19.67	21.76	19.70
Ours	21.47	20.27	20.24	22.67	20.56

Table 6: Comparison without 3DGS for all methods . PSNR

\uparrow

on each sequence is shown on RGB-NIR-Stereo.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
XoFTR [ 79 ]	13.552	0.168	0.575
LoFTR [ 74 ]	13.462	0.164	0.615
LightGlue [ 51 ]	13.084	0.161	0.650
MINIMA [ 66 ]	14.849	0.229	0.377
Ours	17.102	0.302	0.339

Table 7: Comparison DDHR-HK SAR . We show image quality metrics against groundtruth.

Novel View Synthesis

Novel View Synthesis creates images of a scene from camera viewpoints not present in the input, using learned geometry and appearance models.

See all 289 papers

X-Splat: Unified Gaussian Splatting for Joint RGB and X-ray 3D Reconstruction
Oct 16, 2026Wangze Xu, Chu Zhou, Mae Yamaguchi +43D Reconstruction 3D Gaussian Splatting
The University of Tokyo, Japan · National Institute of Informatics, Japan · Shibaura Institute of Technology, Japan
AVSplat: Dense-View Feed-Forward 3D Gaussian Splatting with Assist-View Preconditioning
Sep 5, 2026Muyu Xu, Fangneng Zhan, Yu Wei +23D Gaussian Splatting Novel View Synthesis
Nanyang Technological University, Singapore 639798, Singapore · Harvard University, MA 02138, USA
XCalib Depth-Guided Geometric Optimization for Dense Thermal-Visible Video Registration
Sep 18, 2026Aurelien Godet, Gabriel Jobert, Mauro Dalla MuraCross-Modal Image Registration
Univ. Grenoble Alpes, CNRS, Grenoble INP–UGA, GIPSA-lab, 38000 Grenoble, France · Lynred, 364 Avenue de Valence, 38113 Veurey-Voroize, France · Institut Universitaire de France (IUF), France

Oct 16, 2026J/K move · Enter open · S save

X-Splat: Unified Gaussian Splatting for Joint RGB and X-ray 3D Reconstruction

Wangze Xu, Chu Zhou, Mae Yamaguchi, Linrui Dai+3

The University of Tokyo, Japan · National Institute of Informatics, Japan · Shibaura Institute of Technology, Japan

3D Reconstruction 3D Gaussian Splatting Cross-Modal Image Registration

Open paper AskPDF arXiv

Jointly reconstructing RGB and X-ray provides new insights by connecting external appearance with internal structure, enabling a more complete understanding of real-world objects than single modality. Yet existing methods train separate models per modality and suffer from redundant geometry, alignment artifacts, and loose coupling that limits cross-modal insight discovery. We propose a unified representation in which all Gaussian primitives share a single set of geometric parameters, with only surface-proximal primitives additionally carrying appearance attributes. We perform cross-modal registration through clean multi-view stereo and X-ray geometric proxies instead of aligning two sets of optimized Gaussians, improving alignment stability. We then jointly optimize the unified representation via supervision from both modalities, complemented by a surface-depth consistency loss derived from X-ray geometry to guide RGB reconstruction. Experiments on the InsideOut dataset and our Flower & Insect dataset show that our framework achieves more robust cross-modal alignment, higher RGB rendering quality without compromising X-ray reconstruction, and a more compact representation than maintaining two independent geometries.

Abstract

Novel View Synthesis

Explore similar work

X-Splat: Unified Gaussian Splatting for Joint RGB and X-ray 3D Reconstruction

AVSplat: Dense-View Feed-Forward 3D Gaussian Splatting with Assist-View Preconditioning

XCalib Depth-Guided Geometric Optimization for Dense Thermal-Visible Video Registration