PanoGraphGS: Unifying Panoptic Segmentation and Hierarchical 3D Scene Graphs with Gaussian Splatting

Jinghan Gao; Siyu Wang; Dongyu Feng; Yiyuan Pan; Rong Xiong

cs.CVOct 20, 2026

PanoGraphGS: Unifying Panoptic Segmentation and Hierarchical 3D Scene Graphs with Gaussian Splatting

Authors: Jinghan Gao, Siyu Wang, Dongyu Feng, Yiyuan Pan, Rong Xiong

Organizations: Zhejiang Humanoid Robot Innovation Center, Ningbo, China.

Abstract

3D scene graphs support language-guided navigation by organizing physical entities and their containment and spatial relations into hierarchical maps. Building such structured maps requires a scene representation that consistently distinguishes semantic regions and object instances across viewpoints. To this end, we present PanoGraphGS, a unified framework that integrates panoptic 3D Gaussian representation learning with hierarchical 3D scene graph construction. The framework learns semantic and instance features on shared Gaussian geometry. Prototype-based semantic learning and association-based instance learning produce consistent panoptic predictions across viewpoints. The learned representation supports novel-view panoptic segmentation and direct Gaussian-level panoptic decoding. A semantic-aware scene graph module further recovers a floor-room-stuff-object hierarchy with containment and spatial relations. We further release NaviGS, a real-world dataset for panoramic scene understanding. The dataset provides 3DGS, mesh, RGB, semantic, and depth data. For 3D scene graph identification, PanoGraphGS outperforms the strongest evaluated baseline in object-node accuracy by more than 50% points on average across Replica and NaviGS. PanoGraphGS also achieves state-of-the-art performance among the evaluated methods in 2D panoptic segmentation. These results demonstrate the effectiveness of PanoGraphGS in both hierarchical 3D scene understanding and 2D panoptic segmentation. The project website is available at: https://github.com/gaojh0333/PanoGraphGS.

Figures & tables

Fig. 1: Results of PanoGraphGS. PanoGraphGS is the first method to construct floor–room–stuff–object scene graphs for language-guided robotic navigation, improving object accuracy while reducing stuff interference. It also enables cross-frame consistent panoptic segmentation with persistent instance identities.

Fig. 2: Overview of PanoGraphGS. From multi-view RGB inputs and SAM3-generated masks [ 6 ] to a shared panoptic Gaussian representation, panoptic predictions, and hierarchical 3D scene graph construction. The center illustrates joint semantic and instance learning on shared Gaussian geometry.

Fig. 3: Hierarchical 3D scene graph construction from panoptic Gaussians.

Fig. 4: 3D scene graph visualization on NaviGS and Replica.

Method	Replica			NaviGS
Method	Floor Acc. (%)	Room Acc. (%)	Object Acc. (%)	Floor Acc. (%)	Room Acc. (%)	Object Acc. (%)
KeySG [ 20 ]	100.00	22.50	17.27	90.00	35.00	5.25
HOV-SG [ 5 ]	100.00	87.50	21.72	90.00	66.67	14.57
OP3DSG [ 21 ]	–	–	35.91	–	–	16.97
BBQ [ 22 ]	–	–	33.08	–	–	26.25
ConceptGraphs [ 23 ]	–	–	39.32	–	–	30.50

TABLE I: Hierarchical scene graph comparison on Replica and NaviGS datasets.

Model	ScanNet++		Replica		NaviGS
Model	mIoU	mBIoU	mIoU	mBIoU	mIoU	mBIoU
Gaussian Grouping [ 14 ]	15.67	13.03	23.94	16.98	56.91	35.86
ObjectGS [ 28 ]	8.91	6.41	12.22	5.39	43.77	26.69
Trace3D [ 29 ]	30.22	27.54	54.90	48.50	67.51	44.75
Unified-Lift [ 8 ]	38.10	35.92	63.91	59.69	78.48	57.84
PanopticRecon++ [ 30 ]	31.93	29.31	54.32	43.51	49.36	25.58

TABLE II: Comparison of PanoGraphGS with state-of-the-art algorithms on Scannet++, Replica and NaviGS benchmarks Panoptic segmentation. The best and second-best results are indicated in bold and underlined, respectively.

Fig. 5: Panoptic segmentation visualization on NaviGS and Replica dataset.

	Confidence Weighted	Instance Association	Semantic Prototypes	Object Acc. (%)	2D mIoU	2D mBIoU
EXP0	$\times$	$\times$	$\times$	53.20	59.34	48.19
EXP1	$\checkmark$	$\times$	$\checkmark$	55.31	63.84	54.10
EXP2	$\checkmark$	$\checkmark$	$\times$	80.42	70.23	61.00
EXP3	$\times$	$\checkmark$	$\checkmark$	86.33	68.97	59.53
EXP4	$\checkmark$	$\checkmark$	$\checkmark$	89.90	76.73	70.51

TABLE III: Ablation study on Replica.

3D SGG

3D scene graph generation (3D SGG) represents objects, parts, and their spatial or functional relations as structured models of 3D environments.

See all 65 papers

PointGauss: Point Cloud-Guided Multi-Object Segmentation for Gaussian Splatting
Aug 1, 2025Wentao Sun, Yiping Chen, John S. Zelek +13D Gaussian Splatting 3DGS Segmentation
University of Waterloo, Waterloo, ON N2L 3G1, Canada · Sun Yat-Sen University, Zhuhai 519082, China
ReLaGS: Relational Language Gaussian Splatting
Mar 18, 2026Yaxu Xie, Abdalla Arafa, Alireza Javanmardi +5Language-Embedded 3DGS 3D SGG
German Research Center for Artificial Intelligence (DFKI) · RPTU University Kaiserslautern-Landau · University of Modena and Reggio Emilia
TreeGaussian: Tree-Guided Cascaded Contrastive Learning for Hierarchical Consistent 3D Gaussian Scene Segmentation and Understanding
Mar 31, 2026Jingbin You, Zehao Li, Hao Jiang +83D Semantic Segmentation 3D Scene Understanding
Institute of Computing Technology, Chinese Academy of Sciences, ICT · University of Chinese Academy of Sciences, UCAS · Beihang University, BUAA

Aug 1, 2025J/K move · Enter open · S save

PointGauss: Point Cloud-Guided Multi-Object Segmentation for Gaussian Splatting

Wentao Sun, Yiping Chen, John S. Zelek, Jonathan Li

University of Waterloo, Waterloo, ON N2L 3G1, Canada · Sun Yat-Sen University, Zhuhai 519082, China

3D Gaussian Splatting 3DGS Segmentation 3D Semantic Segmentation Remote Sensing 3D Instance Segmentation

Open paper AskPDF arXiv

While 3D Gaussian Splatting (3DGS) has established new standards for high-fidelity 3D scene modeling, interpreting massive, unstructured Gaussian primitives into meaningful geospatial entities remains a critical challenge for remote sensing and urban applications. Existing 2D-to-3D distillation methods suffer from projection ambiguities, geometric inconsistencies, and computational bottlenecks when applied to complex, large-scale topographies. Furthermore, current benchmarks lack natively aligned 2D-3D ground truth tailored for splatting-based representations. To address these limitations, we propose PointGauss, a unified 3D-native framework for robust semantic parsing and instance segmentation. By treating Gaussian primitives as unstructured point sets, PointGauss leverages Point Transformer V3 (PTv3) to extract scale-invariant geometric features directly from Gaussian attributes, effectively resolving projection ambiguities. To ensure scalability in building-scale scenes, we introduce an adaptive region-of-interest cropping strategy and an instance-aware distance-constrained rasterization pipeline for pixel-level, view-consistent projection. Additionally, we present SplatSeg-360, the first rigorous cross-scale benchmark tailored for 3DGS, comprising 32 complex scenes with full 360

^\circ

coverage and over 6,300 natively aligned 2D-3D masks. Extensive experiments demonstrate that PointGauss operates in real-time and achieves state-of-the-art performance. Notably, it attains approximately 90% 3D-mIoU in large-scale building scenarios and roughly 80% 2D-mIoU in view-consistent 2D instance segmentation, outperforming baseline methods by 16%. (Code)

Abstract

3D SGG

Explore similar work

PointGauss: Point Cloud-Guided Multi-Object Segmentation for Gaussian Splatting

ReLaGS: Relational Language Gaussian Splatting

TreeGaussian: Tree-Guided Cascaded Contrastive Learning for Hierarchical Consistent 3D Gaussian Scene Segmentation and Understanding