Transforming Omnidirectional RGB-LiDAR data into 3D Gaussian Splatting
Organizations: Department of Computer Science, State University of New York, Stony Brook, NY 11794, USA · Department of Mechanical Engineering, State University of New York, Stony Brook, NY 11794, USA
Abstract
The demand for large-scale digital twins is rapidly growing in robotics and autonomous driving. However, constructing these environments with 3D Gaussian Splatting (3DGS) usually requires expensive, purpose-built data collection. Meanwhile, deployed platforms routinely collect extensive omnidirectional RGB and LiDAR logs, but a significant portion of these sensor data is directly discarded or strictly underutilized due to transmission constraints and the lack of scalable reuse pipeline. In this paper, we present an omnidirectional RGB-LiDAR reuse pipeline that transforms these archived logs into robust initialization assets for 3DGS. Direct conversion of such raw logs introduces practical bottlenecks: inherent non-linear distortion leads to unreliable Structure-from-Motion (SfM) tracking, and dense, unorganized LiDAR clouds cause computational overhead during 3DGS optimization. To overcome these challenges, our pipeline strategically integrates an ERP-to-cubemap conversion module for deterministic spatial anchoring, alongside PRISM-a color stratified downsampling strategy. By bridging these multi-modal inputs via Fast Point Feature Histograms (FPFH) based global registration and Iterative Closest Point (ICP), our pipeline successfully repurposes a considerable fraction of discarded data into usable SfM geometry. Furthermore, our LiDAR-reinforced initialization consistently enhances the final 3DGS rendering fidelity in structurally complex scenes compared to vision-only baselines. Ultimately, this work provides a deterministic workflow for creating simulation-grade digital twins from standard archived sensor logs.
Figures & tables
| Sequence | ERP | LiDAR | Keyframes | SfM imgs | KF reuse | SfM rec. |
|---|---|---|---|---|---|---|
| Dormitory 1 | 280 | 280 | 103 | 509 | 36.8% | 82.4% |
| College of Engineering | 279 | 279 | 143 | 716 | 51.3% | 83.4% |
| College of Physical Edu | 479 | 479 | 170 | 907 | 35.5% | 88.9% |
| Sequence | Metric | Raw data | SfM | ||||
|---|---|---|---|---|---|---|---|
| Dormitory 1 | # points | 2,058,126 | 82,724 | 145,437 | 426,720 | 628,446 | 878,976 |
| Size (MB) | 53.00 | 1.18 | 3.75 | 10.99 | 16.18 | 22.63 | |
| College of Engineering | # points | 2,275,569 | 88,903 | 146,667 | 442,041 | 660,160 | 942,330 |
| Size (MB) | 58.59 | 1.27 | 3.78 | 11.38 | 17.00 | 24.26 | |
| College of Physical Edu | # points | 3,336,973 | 248,964 | 209,707 | 600,370 | 883,179 | 1,252,454 |
| Size (MB) | 85.92 | 3.56 | 5.40 | 15.46 | 22.74 | 32.25 |
| Dormitory 1 | College of Engineering | College of Physical Edu | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Global | ICP | RMSE | Global | ICP | RMSE | Global | ICP | RMSE | |
| 1 | 0.9586 | 0.9874 | 0.3179 | 0.9944 | 0.9940 | 0.2640 | 0.9999 | 1.0000 | 0.1999 |
| 5 | 0.9505 | 0.9868 | 0.3171 | 0.9926 | 0.9948 | 0.2923 | 0.9922 | 0.9974 | 0.2200 |
| 10 | 0.8904 | 0.9867 | 0.3180 | 0.9127 | 0.9970 | 0.2076 | 0.9999 | 0.9999 | 0.2118 |
| 20 | 0.9275 | 0.9870 | 0.3185 | 0.9978 | 0.9985 | 0.2177 | 0.9997 | 1.0000 | 0.2138 |
| 30 | 0.9869 | 0.9876 | 0.2812 | 0.9941 | 0.9988 | 0.2004 | 0.9998 | 0.9983 | 0.2679 |
| Dormitory 1 | College of Engineering | College of Physical Edu | ||||
|---|---|---|---|---|---|---|
| Points | Reduction | Points | Reduction | Points | Reduction | |
| 1 | 145,437 | 0.9293 | 146,667 | 0.9355 | 209,707 | 0.9372 |
| 5 | 426,720 | 0.7927 | 442,041 | 0.8057 | 600,370 | 0.8201 |
| 10 | 628,446 | 0.6947 | 660,160 | 0.7099 | 883,179 | 0.7353 |
| 20 | 878,976 | 0.5729 | 942,330 | 0.5859 | 1,252,454 | 0.6247 |
| 30 | 1,042,608 | 0.4934 | 1,130,716 | 0.5031 | 1,502,772 | 0.5497 |
| Sequence | Variant | PSNR | SSIM | LPIPS | Train sec | Points | Model MB |
|---|---|---|---|---|---|---|---|
| Dormitory 1 | Vanilla | 27.4389 | 0.7817 | 0.3541 | 1030 | 1,205,789 | 1338 |
| Dormitory 1 | Ours ( ) | 27.5551 | 0.7816 | 0.3536 | 1256 | 1,207,071 | 1326 |
| Dormitory 1 | Ours ( ) | 27.7505 | 0.7854 | 0.3486 | 1611 | 1,542,370 | 1404 |
| Dormitory 1 | Ours ( ) | 27.8043 | 0.7864 | 0.3473 | 1717 | 1,692,739 | 1439 |
| College of Engineering | Vanilla | 27.6077 | 0.7773 | 0.3538 | 626 | 1,054,880 | 1149 |
| College of Engineering | Ours ( ) | 27.7042 | 0.7747 | 0.3597 | 774 | 1,033,338 | 1141 |
| Sequence | Variant | PSNR | SSIM | LPIPS |
|---|---|---|---|---|
| Dormitory 1 | +0.1162 | -0.0001 | -0.0005 | |
| Dormitory 1 | +0.3116 | +0.0036 | -0.0056 | |
| Dormitory 1 | +0.3654 | +0.0047 | -0.0068 | |
| College of Engineering | +0.0966 | -0.0026 | +0.0059 | |
| College of Engineering | +0.2887 | +0.0020 | -0.0012 | |
| College of Engineering | +0.3028 | +0.0025 | -0.0022 |
3D Gaussian Splatting
3D Gaussian Splatting represents scenes as optimizable 3D Gaussian primitives for fast, photorealistic rendering from new viewpoints.
See all 1,019 papers