Open-vocabulary panoptic reconstruction is essential for advanced robotics perception and simulation. However, existing methods based on 3D Gaussian Splatting (3DGS) often struggle to simultaneously achieve geometric accuracy, coherent panoptic understanding, and real-time inference frequency in large-scale scenes. In this paper, we propose a comprehensive framework that integrates geometric reinforcement, end-to-end panoptic learning, and efficient rendering. First, to ensure physical realism in large-scale environments, we leverage LiDAR data to construct plane-constrained multimodal Gaussian Mixture Models (GMMs) and employ 2D Gaussian surfels as the map representation, enabling high-precision surface alignment and continuous geometric supervision. Building upon this, to overcome the error accumulation and cumbersome cross-frame association inherent in traditional multi-stage panoptic segmentation pipelines, we design a query-guided end-to-end learning architecture. By utilizing a local cross-attention mechanism within the view frustum, the system lifts 2D mask features directly into 3D space, achieving globally consistent panoptic understanding. Finally, addressing the computational bottlenecks caused by high-dimensional semantic features, we introduce Precise Tile Intersection and a Top-K Hard Selection strategy to optimize the rendering pipeline. Experimental results demonstrate that our system achieves superior geometric and panoptic reconstruction quality in large-scale scenes while maintaining an inference rate exceeding 40 FPS, meeting the real-time requirements of robotic control loops.
Figures & tables
Figure 1: Ψ -Map provides a high-performance bridge from multi-modal sensor inputs to robotic applications. By fusing LiDAR-RGB-D data with Vision-Language Foundation Models, our framework achieves real-time, 3D-consistent panoptic reconstruction. This representation serves as a powerful engine for high-fidelity scene editing and virtual asset generation, ultimately empowering complex downstream skills such as generative 3D object completion and robust robotic navigation in wild environments.
Figure 2: Overview of the Ψ -Map Framework. The Input consists of multi-view RGB-D frames. During Geometric Reinforcement, the point cloud is modeled as a plane-constrained SOGMM to provide continuous structural supervision for the 2D Gaussian Surfels. In the Panoptic Learning stage, 2D mask features are lifted into 3D via a query-guided end-to-end architecture, where instance tokens are refined through local cross-attention within the view frustum to ensure global identity consistency. For Inference, our highly optimized rendering pipeline employs Precise Tile Intersection and Top- K Hard Selection. By leveraging the geometric sparsity and surface-aligned nature of 2DGS, Ψ -Map significantly reduces the computational overhead of high-dimensional feature accumulation, enabling real-time, high-fidelity multi-modal rendering.
Figure 3: The instance branch utilizes a cross-attention mechanism between 3D Gaussian-modulated query tokens and SOGMM-reinforced 2DGS scene fields.
Figure 4: Comparison of the quality of semantic segmentation and panoptic segmentation of different methods on ScanNet V2 and ScanNet++.
Method
Acc. ↓
Comp. ↓
C-L1 ↓
Recall ↑
Prec. ↑
F1 ↑
2DGS [ 15 ]
13.98
18.60
16.28
71.65
80.85
75.82
Gaussian Surfels [ 31 ]
14.25
35.95
25.10
70.47
48.33
57.29
PGSR [ 16 ]
13.52
19.63
16.58
72.66
73.67
73.08
RaDe-GS [ 32 ]
13.28
17.13
15.22
73.42
78.59
75.88
LIV-GaussMap [ 33 ]
14.17
15.48
14.80
72.54
80.33
76.18
GOF [ 34 ]
14.27
15.37
14.83
70.55
80.20
75.04
Table 1: Geometric quality comparison on self-collected datasets.
Figure 5: Qualitative rendering results on the Replica dataset, demonstrating Ψ -Map’s ability to produce high-fidelity geometry, semantic, and panoptic maps.
Method
ScanNet-V2
ScanNet++
PQ ↑
SQ ↑
RQ ↑
mIoU ↑
mAcc ↑
mCov ↑
mW-Cov ↑
PQ ↑
SQ ↑
RQ ↑
mIoU ↑
mAcc ↑
mCov ↑
mW-Cov ↑
NeRF-based Lifting Methods
Panoptic Lifting [ 36 ]
57.86
61.96
85.31
67.91
78.59
45.88
59.93
71.14
77.48
88.14
81.34
89.67
56.17
68.51
Contrastive Lift [ 37 ]
37.35
41.91
57.60
64.77
75.80
13.21
23.26
47.58
57.23
65.81
81.09
89.30
27.39
36.51
PVLFF [ 38 ]
30.11
51.71
44.43
55.41
63.96
45.75
48.41
52.24
66.86
65.56
62.53
70.31
67.95
75.47
PanopticRecon [ 39 ]
63.70
64.81
81.17
68.62
80.87
66.58
77.84
68.29
77.01
85.05
77.75
87.08
51.34
62.79
Table 2: Quantitative comparison of panoptic segmentation performance on ScanNet-V2 and ScanNet++ datasets.
Config
Time (ms) ↓
FPS ↑
RN-Total ↓
RN / Tile ↓
PQ ↑
Baseline
46
22
6.1M
1230
74.75
w/ Precise Tile
40
25
3.9M
800
74.73
w/ Top- K
27
37
1.2M
610
74.17
Full Method
22
45
1.2M
580
74.12
Table 3: Study on Precise Tile Intersection and Top- K Hard Selection.
Base Model
PQ ↑
SQ ↑
RQ ↑
mIoU ↑
mAcc ↑
mCov ↑
mW-Cov ↑
3DGS [ 41 ]
73.83
74.08
99.33
75.08
83.68
72.20
76.43
2DGS [ 15 ]
73.30
73.45
99.93
73.75
82.58
71.28
75.98
Ours
74.75
74.75
100.0
74.95
83.70
73.18
79.63
Table 4: Ablation study on different geometric representations on ScanNet-V2.
Method
CD-S ↓
F-Score-S ↑
CD-O ↓
F-Score-O ↑
IoU-B ↑
Col-O ↓
Col-S ↓
AdaPoinTr
0.018
0.841
0.062
0.532
0.597
0.329
0.023
InstPIFu
0.128
0.493
0.089
0.470
0.207
0.510
0.159
Gen3DSR
0.070
0.655
0.103
0.430
0.398
0.291
0.093
Ours
0.016
0.903
0.036
0.710
0.666
0.203
0.015
Table 5: Quantitative comparisons on Scan2CAD dataset
Figure 6: Quantitative comparisons on Scan2CAD dataset.
Figure 7: Comparison of navigation performance before and after fine-tuning with 3D-consistent labels.
Scene
SR-Before
SR-After
Laboratory Room
20%
100%
Private House
25%
50%
Table 6: Impact of fine-tuning with 3D-consistent labels on navigation success rate (SR).