Transforming Omnidirectional RGB-LiDAR data into 3D Gaussian Splatting

Semin Bae; Hansol Lim; Jongseong Brad Choi

cs.CVMar 6, 2026

Transforming Omnidirectional RGB-LiDAR data into 3D Gaussian Splatting

Authors: Semin Bae, Hansol Lim, Jongseong Brad Choi

Organizations: Department of Computer Science, State University of New York, Stony Brook, NY 11794, USA · Department of Mechanical Engineering, State University of New York, Stony Brook, NY 11794, USA

Abstract

The demand for large-scale digital twins is rapidly growing in robotics and autonomous driving. However, constructing these environments with 3D Gaussian Splatting (3DGS) usually requires expensive, purpose-built data collection. Meanwhile, deployed platforms routinely collect extensive omnidirectional RGB and LiDAR logs, but a significant portion of these sensor data is directly discarded or strictly underutilized due to transmission constraints and the lack of scalable reuse pipeline. In this paper, we present an omnidirectional RGB-LiDAR reuse pipeline that transforms these archived logs into robust initialization assets for 3DGS. Direct conversion of such raw logs introduces practical bottlenecks: inherent non-linear distortion leads to unreliable Structure-from-Motion (SfM) tracking, and dense, unorganized LiDAR clouds cause computational overhead during 3DGS optimization. To overcome these challenges, our pipeline strategically integrates an ERP-to-cubemap conversion module for deterministic spatial anchoring, alongside PRISM-a color stratified downsampling strategy. By bridging these multi-modal inputs via Fast Point Feature Histograms (FPFH) based global registration and Iterative Closest Point (ICP), our pipeline successfully repurposes a considerable fraction of discarded data into usable SfM geometry. Furthermore, our LiDAR-reinforced initialization consistently enhances the final 3DGS rendering fidelity in structurally complex scenes compared to vision-only baselines. Ultimately, this work provides a deterministic workflow for creating simulation-grade digital twins from standard archived sensor logs.

Figures & tables

Fig. 1: Example of ERP-to-cubemap conversion used in our pipeline. One omnidirectional panorama is projected into six rectilinear cubemap faces for robust feature matching in SfM.

Fig. 2: Overview of the proposed omnidirectional RGB-LiDAR reuse pipeline. ERP images are converted to cubemaps for robust SfM anchoring, LiDAR odometry maps are colorized and PRISM-downsampled, and ICP-based registration fuses both modalities into a final 3DGS-ready asset.

Sequence	ERP	LiDAR	Keyframes	SfM imgs	KF reuse	SfM rec.
Dormitory 1	280	280	103	509	36.8%	82.4%
College of Engineering	279	279	143	716	51.3%	83.4%
College of Physical Edu	479	479	170	907	35.5%	88.9%

TABLE I: Pipeline summary with reuse-efficiency proxies for three AirLab-360 sequences.

Sequence	Metric	Raw data	SfM	$n=1$	$n=5$	$n=10$	$n=20$
Dormitory 1	# points	2,058,126	82,724	145,437	426,720	628,446	878,976
Dormitory 1	Size (MB)	53.00	1.18	3.75	10.99	16.18	22.63
College of Engineering	# points	2,275,569	88,903	146,667	442,041	660,160	942,330
College of Engineering	Size (MB)	58.59	1.27	3.78	11.38	17.00	24.26
College of Physical Edu	# points	3,336,973	248,964	209,707	600,370	883,179	1,252,454
College of Physical Edu	Size (MB)	85.92	3.56	5.40	15.46	22.74	32.25

TABLE II: Point-cloud data summary used for initialization diagnostics.

$n$	Dormitory 1			College of Engineering			College of Physical Edu
$n$	Global $\uparrow$	ICP $\uparrow$	RMSE $\downarrow$	Global $\uparrow$	ICP $\uparrow$	RMSE $\downarrow$	Global $\uparrow$	ICP $\uparrow$	RMSE $\downarrow$
1	0.9586	0.9874	0.3179	0.9944	0.9940	0.2640	0.9999	1.0000	0.1999
5	0.9505	0.9868	0.3171	0.9926	0.9948	0.2923	0.9922	0.9974	0.2200
10	0.8904	0.9867	0.3180	0.9127	0.9970	0.2076	0.9999	0.9999	0.2118
20	0.9275	0.9870	0.3185	0.9978	0.9985	0.2177	0.9997	1.0000	0.2138
30	0.9869	0.9876	0.2812	0.9941	0.9988	0.2004	0.9998	0.9983	0.2679

TABLE III: Alignment diagnostics grouped by

n

(updated SLAM-colorized sweep).

$n$	Dormitory 1		College of Engineering		College of Physical Edu
$n$	Points	Reduction	Points	Reduction	Points	Reduction
1	145,437	0.9293	146,667	0.9355	209,707	0.9372
5	426,720	0.7927	442,041	0.8057	600,370	0.8201
10	628,446	0.6947	660,160	0.7099	883,179	0.7353
20	878,976	0.5729	942,330	0.5859	1,252,454	0.6247
30	1,042,608	0.4934	1,130,716	0.5031	1,502,772	0.5497

TABLE IV: PRISM output size and reduction ratio grouped by

n

Sequence	Variant	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Train sec	Points	Model MB
Dormitory 1	Vanilla	27.4389	0.7817	0.3541	1030	1,205,789	1338
Dormitory 1	Ours ( $n=5$ )	27.5551	0.7816	0.3536	1256	1,207,071	1326
Dormitory 1	Ours ( $n=50$ )	27.7505	0.7854	0.3486	1611	1,542,370	1404
Dormitory 1	Ours ( $n=100$ )	27.8043	0.7864	0.3473	1717	1,692,739	1439
College of Engineering	Vanilla	27.6077	0.7773	0.3538	626	1,054,880	1149
College of Engineering	Ours ( $n=5$ )	27.7042	0.7747	0.3597	774	1,033,338	1141

TABLE V: Absolute 3DGS metrics at 30k iterations.

Sequence	Variant	$\Delta$ PSNR	$\Delta$ SSIM	$\Delta$ LPIPS
Dormitory 1	$n=5$	+0.1162	-0.0001	-0.0005
Dormitory 1	$n=50$	+0.3116	+0.0036	-0.0056
Dormitory 1	$n=100$	+0.3654	+0.0047	-0.0068
College of Engineering	$n=5$	+0.0966	-0.0026	+0.0059
College of Engineering	$n=50$	+0.2887	+0.0020	-0.0012
College of Engineering	$n=100$	+0.3028	+0.0025	-0.0022

TABLE VI: Delta metrics (LiDAR-initialized variant - vanilla) at 30k iterations.

Fig. 3: Qualitative comparison across the three sequences. Columns show Vanilla, Ours ( $n=50$ ), and GT.

3D Gaussian Splatting

3D Gaussian Splatting represents scenes as optimizable 3D Gaussian primitives for fast, photorealistic rendering from new viewpoints.

See all 1,019 papers

Jun 25, 2026

Structured-Li-GS: Structured 3D Gaussians Splatting with LiDAR Incorporation and Spatial Constraints

In this study, we develop a Structured framework for Gaussian Splatting (3DGS) with LiDAR integration (Structured-Li-GS). It is a lightweight Gaussian Splatting pipeline that leverages LiDAR-inertial-visual SLAM. Structured-Li-GS achieves high-quality 3D reconstructions with fewer Gaussians by training on accurate, dense, colorized point clouds. Gaussian primitives are anchored using sub-sampled point clouds, and their ellipsoidal parameters are initialized from local surface geometry. Our training strategy integrates a comprehensive set of loss terms, including photometric, flattening, offset, depth, and normal losses, guided by the dense point cloud, enabling accurate reconstruction without Gaussian densification. This approach produces up-to-scale, high-fidelity results with a moderate model size. For experimental validation, we develop a custom hardware-synchronized LiDAR-camera handheld scanner. Experiments on both benchmark datasets and our real-world in-house dataset demonstrate that Structured-Li-GS surpasses state-of-the-art methods while using fewer Gaussians.

Huaiyuan Weng, Huibin Li, Chul Min Yeum

Department of Civil and Environmental Engineering, University of Waterloo, 200 University Ave W, Waterloo, N2L 3G1, Ontario, Canada

3D Gaussian Splatting 3D Reconstruction

Ask PDF

Mar 13, 2026

LR-SGS: Robust LiDAR-Reflectance-Guided Salient Gaussian Splatting for Self-Driving Scene Reconstruction

Recent 3D Gaussian Splatting (3DGS) methods have demonstrated the feasibility of self-driving scene reconstruction and novel view synthesis. However, most existing methods either rely solely on cameras or use LiDAR only for Gaussian initialization or depth supervision, while the rich scene information contained in point clouds, such as reflectance, and the complementarity between LiDAR and RGB have not been fully exploited, leading to degradation in challenging self-driving scenes, such as those with high ego-motion and complex lighting. To address these issues, we propose a robust and efficient LiDAR-reflectance-guided Salient Gaussian Splatting method (LR-SGS) for self-driving scenes, which introduces a structure-aware Salient Gaussian representation, initialized from geometric and reflectance feature points extracted from LiDAR and refined through a salient transform and improved density control to capture edge and planar structures. Furthermore, we calibrate LiDAR intensity into reflectance and attach it to each Gaussian as a lighting-invariant material channel, jointly aligned with RGB to enforce boundary consistency. Extensive experiments on the Waymo Open Dataset demonstrate that LR-SGS achieves superior reconstruction performance with fewer Gaussians and shorter training time. In particular, on Complex Lighting scenes, our method surpasses OmniRe by 1.18 dB PSNR.

ZY Chen, F Zhu, H Zhu +4

3D Gaussian Splatting Autonomous Driving Camera-LiDAR Fusion 3D Scene Understanding

Ask PDF

Aug 6, 2026

G $^2$ ARD-GS: Geometry-Guided Anchor-Regularized Gaussian Splatting Distillation

Dense colored LiDAR maps provide accurate city-scale geometry, but lifting them into 3D Gaussian Splatting (3DGS) retains millions of primitives, making the resulting models costly to store, transmit, render, and adapt. Aggressive primitive reduction alleviates this burden, but can remove the local surface support needed for stable novel-view synthesis and downstream geometric use. We introduce G

^2

ARD-GS, a geometry-guided distillation method that converts a dense Gaussian prior instantiated either as a training-free point-cloud lift or a trained GS model into a compact, reusable representation. G

^2

ARD-GS progressively consolidates the prior into surface-aware representatives, then recovers appearance on the resulting fixed topology under construction-time anchor constraints, with no primitives added or removed during recovery. Under limited supervision, geometry-aware view selection allocates the available view budget. On MatrixCity, G

^2

ARD-GS achieves the best PSNR, SSIM, and LPIPS across matched

5\times

30\times

compression budgets, outperforming PUP by

3.2

6.8

,dB in PSNR. When reused as frozen geometry, the compact model improves off-trajectory appearance adaptation by

3.7

4.9

,dB over PUP 3D-GS and preserves image-to-model registration accuracy on Cambridge KingsCollege at

30\times

compression. Project page: https://patrick1159.github.io/gardGS-page/.

Puyuan Zhang, Jianming Huang, Wenkai Ye +1

Shanghai Jiao Tong University, Shanghai, China · Jishu Technology Co., Ltd., China

3DGS Compression 3D Gaussian Splatting Novel View Synthesis

Ask PDF

Abstract

3D Gaussian Splatting

Explore similar work

Structured-Li-GS: Structured 3D Gaussians Splatting with LiDAR Incorporation and Spatial Constraints

LR-SGS: Robust LiDAR-Reflectance-Guided Salient Gaussian Splatting for Self-Driving Scene Reconstruction

G2^22ARD-GS: Geometry-Guided Anchor-Regularized Gaussian Splatting Distillation

G $^2$ ARD-GS: Geometry-Guided Anchor-Regularized Gaussian Splatting Distillation