OnlinePG: Online Open-Vocabulary Panoptic Mapping with 3D Gaussian Splatting

Hongjia Zhai; Qi Zhang; Xiaokun Pan; Xiyu Zhang; Yitong Dong; Huaqi Zhang; Dan Xu; Guofeng Zhang

cs.CVMar 19, 2026

OnlinePG: Online Open-Vocabulary Panoptic Mapping with 3D Gaussian Splatting

Authors: Hongjia Zhai, Qi Zhang, Xiaokun Pan, Xiyu Zhang, Yitong Dong, Huaqi Zhang, Dan Xu, Guofeng Zhang

Organizations: State Key Lab of CAD & CG, Zhejiang University · VIVO BlueImage Lab · HKUST

Abstract

Open-vocabulary scene understanding with online panoptic mapping is essential for embodied applications to perceive and interact with environments. However, existing methods are predominantly offline or lack instance-level understanding, limiting their applicability to real-world robotic tasks. In this paper, we propose OnlinePG, a novel and effective system that integrates geometric reconstruction and open-vocabulary perception using 3D Gaussian Splatting in an online setting. Technically, to achieve online panoptic mapping, we employ an efficient local-to-global paradigm with a sliding window. To build local consistency map, we construct a 3D segment clustering graph that jointly leverages geometric and semantic cues, fusing inconsistent segments within sliding window into complete instances. Subsequently, to update the global map, we construct explicit grids with spatial attributes for the local 3D Gaussian map and fuse them into the global map via robust bidirectional bipartite 3D Gaussian instance matching. Finally, we utilize the fused VLM features inside the 3D spatial attribute grids to achieve open-vocabulary scene understanding. Extensive experiments on widely used datasets demonstrate that our method achieves better performance among online approaches, while maintaining real-time efficiency.

Figures & tables

Figure 2: Overview of Our Approach . Our system performs online open-vocabulary panoptic mapping from RGB-D streams using a local-to-global paradigm. (a) Maintaining a sliding window $\mathcal{W}$ to build locally consistent instance and fuse them globally. (b) Using noisy 2D masks and features from [ 21 , 17 ] to initialize 3D Gaussian segments, which are clustered via multi-cues to form consistent 3D instances. (c) Bidirectional bipartite matching incrementally fuses local instances and spatial attributes into the global map.

Methods	Online	Pano.	ScanNetV2 [ 5 ]				Replica [ 40 ]
Methods	Online	Pano.	mIoU $\uparrow$	mAcc. $\uparrow$	PRQ (T) $\uparrow$	PRQ (S) $\uparrow$	mIoU $\uparrow$	mAcc. $\uparrow$	PRQ (T) $\uparrow$	PRQ (S) $\uparrow$
LangSplat [ 33 ] ∗	✗	✗	29.47	45.29	22.57	28.44	4.82	10.03	8.29	1.28
OpenGaussian [ 50 ] ∗	✗	✗	24.89	37.35	22.87	19.71	–	–	–	–
OpenScene [ 31 ] ∗	✗	✗	47.63	69.74	43.53	40.43	49.03	62.89	33.04	11.84
InstanceGaussian [ 19 ]	✗	✓	34.14	54.95	39.04	27.41	–	–	–	–
PanoGS [ 57 ]	✗	✓	50.72	70.20	33.84	36.22	54.98	67.35	43.04	30.60

Table 1: 3D Semantic and Panoptic Segmentation Results on ScanNetV2 and Replica Datasets.

*

indicates the baseline results are taken from [ 57 ] which use the 3D instance obtained from supervised approach [ 44 ] for panoptic comparison (their panoptic results are marked in gray ). We mark the top-3 segmentation results among all approaches in red , green , and blue , respectively.

Figure 3: Qualitative 3D Semantic Segmentation Comparison of ScanNetV2 Dataset . Our approach outperforms recent online approaches, O2V-Mapping [ 42 ] and OnlineAnySeg [ 41 ] , by a large margin. Compared with the offline SOTA PanoGS [ 57 ] , our approach still exists a small performance gap.

Figure 4: Qualitative Results of Open-Vocabulary Query . We use different colors to distinguish different instances found in the query. Compared to OnlineAnySeg [ 41 ] , our approach shows advanced text query capabilities for long-tail and multiple instances scenario.

Figure 5: Left: ablation studies of using different segments clustering cues for local map construction. Right: ablation studies of using different feature grid resolutions. The results are evaluated on ScanNetV2 dataset.

Settings	PRQ (T)	PRQ (S)
#1 NN Match	24.67	22.98
#2 Only Forward Match $\mathcal{M}_{l\to g}$	35.83	38.40
#3 Only Backward Match $\mathcal{M}_{g\to l}$	33.71	42.72
#4 Ours Full Bidirectional Match	37.97	41.81

Table 2: Ablation studies of different matching strategies for global map fusion on ScanNetV2 dataset.

Settings	mIoU	PRQ (T)	PRQ (S)
#1 w/o. Segment Clustering	48.48	32.25	30.68
#2 w/o. Feature Grid $\mathcal{F}$	30.40	26.71	24.92
#3 Ours Full	48.48	37.97	41.81

Table 3: Ablation studies of our system components. The results are evaluated on ScanNetV2 dataset.

Figure 6: Qualitative comparison of using segment clustering.

Figure 8: Runtime Performance of OnlinePG . We divided the system’s time consumption into four parts and statistically analyzed the distribution of the execution time of each part across ten scenarios used by ScanNet. The black dashed line represents the average time.

Figure 9: Visualization Results of 3D Language Features . For better visualization, we perform principal components analysis (PCA) on the high-dimensional language features.

Figure 10: Qualitative Rendering Results . We show the rendered RGB, Depth, and language feature of different scenes.

Figure 11: Qualitative 3D Instance Segmentation Comparison . We show the 3D instance segmentation results from several scenes from ScanNetV2 [ 5 ] and Replica [ 40 ] datasets.

Method	Metrics	Room 0	Room 1	Room 2	Office 0	Office 1	Office 2	Office 3	Office 4
O2V-Mapping [ 42 ]	mIoU	29.25	39.58	32.53	18.78	17.54	20.75	12.59	23.24
	mAcc.	38.00	52.60	43.22	27.56	20.25	30.20	17.95	38.47
	FPS	3.08	3.05	3.08	3.05	3.10	3.13	3.18	3.28
OnlineAnySeg [ 41 ]	mIoU	46.76	45.13	51.30	37.54	25.38	32.91	25.45	48.23
	mAcc.	62.41	67.50	66.84	50.47	31.24	46.99	42.46	59.45
	PRQ (T)	30.26	51.65	26.59	45.34	22.10	24.90	16.32	27.17

Table 4: 3D Semantic and Panoptic Segmentation Results on Replica Datasets . We demonstrate the 3D segmentation performance and FPS data of the online method in different scenes of Replica [ 40 ] .

Method	Metrics	0000	0062	0070	0097	0140	0200	0347	0400	0590	0645
O2V-Mapping [ 42 ]	mIoU	35.80	46.09	41.21	26.29	24.25	39.08	27.54	39.74	33.18	28.27
	mAcc.	59.92	72.72	55.59	55.73	34.15	63.91	47.12	58.33	49.85	52.22
	FPS	3.38	3.43	3.43	3.53	3.40	3.50	3.43	3.38	3.43	3.38
OnlineAnySeg [ 41 ]	mIoU	32.64	31.13	23.26	39.08	12.62	39.32	32.97	33.93	34.66	33.26
	mAcc.	50.61	60.73	31.92	69.62	31.86	52.43	54.51	56.42	55.89	58.02
	PRQ (T)	42.32	33.47	36.29	71.30	53.32	33.76	38.43	25.97	38.95	56.06

Table 5: 3D Semantic and Panoptic Segmentation Results on ScanNetV2 Datasets . We demonstrate the 3D segmentation performance and FPS data of the online method in different scenes of ScanNetV2 [ 5 ] .

Explore similar work

Ψ-Map: Panoptic Surface Integrated Mapping Enables Real2Sim Transfer
Apr 13, 2026Xuan Yu, Yuxuan Xie, Changjian Jiang +43D Reconstruction Robotic Perception
Department of Control Science and Engineering, Zhejiang University, Hangzhou, Zhejiang, P. R. China
OVI-MAP:Open-Vocabulary Instance-Semantic Mapping
Mar 27, 2026Zilong Deng, Federico Tombari, Marc Pollefeys +2Semantic Mapping Open-Vocabulary 3D Segmentation
ETH Zurich · University of Zurich · Google +2
OGScene3D: Incremental Open-Vocabulary 3D Gaussian Scene Graph Mapping for Scene Understanding
Mar 17, 2026Siting Zhu, Ziyun Lu, Guangming Wang +53D Scene Understanding 3D SGG
Shanghai Jiao Tong University, China · University of Cambridge, England · University of Technology Nuremberg, Germany +2

Apr 13, 2026J/K move · Enter open · S save

Ψ-Map: Panoptic Surface Integrated Mapping Enables Real2Sim Transfer

Xuan Yu, Yuxuan Xie, Changjian Jiang, Shichao Zhai+3

Department of Control Science and Engineering, Zhejiang University, Hangzhou, Zhejiang, P. R. China

3D Reconstruction Robotic Perception 3D Gaussian Splatting Panoptic Segmentation

Open paper AskPDF arXiv

Open-vocabulary panoptic reconstruction is essential for advanced robotics perception and simulation. However, existing methods based on 3D Gaussian Splatting (3DGS) often struggle to simultaneously achieve geometric accuracy, coherent panoptic understanding, and real-time inference frequency in large-scale scenes. In this paper, we propose a comprehensive framework that integrates geometric reinforcement, end-to-end panoptic learning, and efficient rendering. First, to ensure physical realism in large-scale environments, we leverage LiDAR data to construct plane-constrained multimodal Gaussian Mixture Models (GMMs) and employ 2D Gaussian surfels as the map representation, enabling high-precision surface alignment and continuous geometric supervision. Building upon this, to overcome the error accumulation and cumbersome cross-frame association inherent in traditional multi-stage panoptic segmentation pipelines, we design a query-guided end-to-end learning architecture. By utilizing a local cross-attention mechanism within the view frustum, the system lifts 2D mask features directly into 3D space, achieving globally consistent panoptic understanding. Finally, addressing the computational bottlenecks caused by high-dimensional semantic features, we introduce Precise Tile Intersection and a Top-K Hard Selection strategy to optimize the rendering pipeline. Experimental results demonstrate that our system achieves superior geometric and panoptic reconstruction quality in large-scale scenes while maintaining an inference rate exceeding 40 FPS, meeting the real-time requirements of robotic control loops.

Abstract

Explore similar work

Ψ-Map: Panoptic Surface Integrated Mapping Enables Real2Sim Transfer

OVI-MAP:Open-Vocabulary Instance-Semantic Mapping

OGScene3D: Incremental Open-Vocabulary 3D Gaussian Scene Graph Mapping for Scene Understanding