SemGS: Feed-Forward Semantic 3D Gaussian Splatting from Sparse Views for Generalizable Scene Understanding

Sheng Ye; Zhen-Hui Dong; Ruoyu Fan; Tian Lv; Yong-Jin Liu

cs.CVMar 3, 2026

SemGS: Feed-Forward Semantic 3D Gaussian Splatting from Sparse Views for Generalizable Scene Understanding

Authors: Sheng Ye, Zhen-Hui Dong, Ruoyu Fan, Tian Lv, Yong-Jin Liu

Organizations: Department of Computer Science, Tsinghua University, Beijing, China

Abstract

Semantic understanding of 3D scenes is essential for robots to operate effectively and safely in complex environments. Existing methods for semantic scene reconstruction and semantic-aware novel view synthesis often rely on dense multi-view inputs and require scene-specific optimization, limiting their practicality and scalability in real-world applications. To address these challenges, we propose SemGS, a feed-forward framework for reconstructing generalizable semantic fields from sparse image inputs. SemGS uses a dual-branch architecture to extract color and semantic features, where the two branches share shallow CNN layers, allowing semantic reasoning to leverage textural and structural cues in color appearance. We also incorporate a camera-aware attention mechanism into the feature extractor to explicitly model geometric relationships between camera viewpoints. The extracted features are decoded into dual-Gaussians that share geometric consistency while preserving branch-specific attributes, and further rasterized to synthesize semantic maps under novel viewpoints. Additionally, we introduce a regional smoothness loss to enhance semantic coherence. Experiments show that SemGS achieves state-of-the-art performance on benchmark datasets, while providing rapid inference and strong generalization capabilities across diverse synthetic and real-world scenarios.

Figures & tables

Fig. 1: We propose SemGS , a novel framework for generalizable semantic 3DGS. Given sparse-view images of an unseen scene, SemGS can rapidly infer semantic maps under novel viewpoints in a single feed-forward pass.

Fig. 2: Overall pipeline of our proposed SemGS . Given sparse-view RGB images as inputs, SemGS leverages a dual-branch architecture (Sec. III-A ) to extract both color and semantic features. These features are used to regress multi-view depth maps (Sec. III-B ) and are subsequently decoded into per-pixel dual-Gaussians (Sec. III-C ). The resulting dual-Gaussians share geometric attributes while maintaining branch-specific attributes, enabling efficient rasterization for synthesizing both novel RGB views and semantic maps.

Methods	2 Input Views ( $N=2$ )				3 Input Views ( $N=3$ )				4 Input Views ( $N=4$ )
	mIoU $\uparrow$	acc. $\uparrow$	class acc. $\uparrow$	FPS $\uparrow$	mIoU $\uparrow$	acc. $\uparrow$	class acc. $\uparrow$	FPS $\uparrow$	mIoU $\uparrow$	acc. $\uparrow$	class acc. $\uparrow$	FPS $\uparrow$
S-Ray [ 31 ]	0.538	0.772	0.619	0.52	0.563	0.778	0.632	0.41	0.604	0.802	0.673	0.34
GSNeRF [ 32 ]	0.529	0.751	0.616	0.25	0.550	0.752	0.648	0.23	0.587	0.796	0.671	0.19
SemGS (Ours)	0.754	0.912	0.803	8.49	0.757	0.908	0.811	7.35	0.765	0.919	0.815	6.10

TABLE I: Quantitative comparisons on ScanNet dataset (averaged over 10 scenes). The best results are highlighted in bold.

Methods	2 Input Views ( $N=2$ )				3 Input Views ( $N=3$ )				4 Input Views ( $N=4$ )
	mIoU $\uparrow$	acc. $\uparrow$	class acc. $\uparrow$	FPS $\uparrow$	mIoU $\uparrow$	acc. $\uparrow$	class acc. $\uparrow$	FPS $\uparrow$	mIoU $\uparrow$	acc. $\uparrow$	class acc. $\uparrow$	FPS $\uparrow$
S-Ray [ 31 ]	0.411	0.775	0.502	0.60	0.441	0.808	0.523	0.46	0.467	0.804	0.558	0.38
GSNeRF [ 32 ]	0.421	0.760	0.503	0.29	0.450	0.794	0.538	0.26	0.479	0.813	0.562	0.21
SemGS (Ours)	0.625	0.903	0.689	9.24	0.671	0.924	0.734	7.68	0.676	0.927	0.739	6.29

TABLE II: Quantitative comparisons on ScanNet++ dataset (averaged over 10 scenes). The best results are highlighted in bold.

Fig. 3: Swin Transformer with camera-aware attention. Different colors denote attention windows, and the window shifting mechanism enables model to capture cross-window connections. Camera poses are injected into the attention process to enhance the 3D reasoning ability.

Fig. 4: Qualitative comparisons on ScanNet and ScanNet++ datasets. We demonstrate novel-view semantic rendering results of different methods. Compared to existing works, our method produces sharper object boundaries, fewer misclassified regions, and more spatially consistent segmentations.

Fig. 5: Generalization ability on unseen domains. Different models trained on ScanNet are directly evaluated on Replica synthetic scenes (left) and real-world robot-captured scenes (right). Our SemGS generates more accurate and complete semantic maps than prior works.

Sh-Layer	Sw-Trans	C-Inj	Sm-Loss	mIoU $\uparrow$	acc. $\uparrow$	class acc. $\uparrow$
✗	✗	✗	✗	0.681	0.856	0.751
✓	✗	✗	✗	0.718	0.873	0.776
✓	✓	✗	✗	0.744	0.898	0.795
✓	✓	✓	✗	0.761	0.910	0.812
✓	✓	✓	✓	0.765	0.919	0.815

TABLE III: Ablation study on ScanNet dataset. The full model, with all components combined, achieves the best performance.

Fig. 6: Visualization of the effect of regional smoothness loss ( Sm-Loss ). The proposed loss suppresses noise and enhances semantic coherence.

Explore similar work

InstanceSplat: Instance-Aware Feed-Forward 3D Gaussian Splatting for Scene Understanding
Aug 7, 2026Minchao Jiang, Xiaoxuan Ma, Shunyu Jia +33D Gaussian Splatting Open-Vocabulary 3D Segmentation
1Shanghai Jiao Tong University · 2Eastern Institute of Technology, Ningbo · 3Carnegie Mellon University +2
SparseLGS: Sparse View Language Embedded Gaussian Splatting
Dec 3, 2024Jun Hu, Zhang Chen, Zhong Li +23D Gaussian Splatting Language-Embedded 3DGS
School of Mathematical Science, University of Science and Technology of China · Meta · Apple Inc. +1
UniSem: Generalizable Semantic 3D Reconstruction from Sparse Unposed Images
Mar 18, 2026Guibiao Liao, Qian Ren, Kaimin Liao +43D Reconstruction Sparse-View 3D Reconstruction
Moore Threads AI

Aug 7, 2026J/K move · Enter open · S save

InstanceSplat: Instance-Aware Feed-Forward 3D Gaussian Splatting for Scene Understanding

Minchao Jiang, Xiaoxuan Ma, Shunyu Jia, Haoru Wang+2

1Shanghai Jiao Tong University · 2Eastern Institute of Technology, Ningbo · 3Carnegie Mellon University+2

3D Gaussian Splatting Open-Vocabulary 3D Segmentation Feed-Forward 3D Reconstruction 3D Instance Segmentation 3D Scene Understanding

Open paper AskPDF arXiv

Feed-forward 3D Gaussian Splatting (3DGS) enables efficient and generalizable 3D reconstruction, but current feed-forward 3DGS methods for scene understanding remain largely category-oriented. In contrast, instance-aware 3DGS methods typically rely on per-scene optimization and often decouple reconstruction from instance and semantic learning, limiting reciprocal interactions among them. We present InstanceSplat, a unified feed-forward 3DGS framework for generalizable 3D reconstruction and instance-aware scene understanding from pose-free multi-view images. In a single forward pass, InstanceSplat constructs an instance-aware Gaussian representation that jointly encodes appearance, geometry, instance identity, and language-aligned semantics. Shared 3D Gaussians ground instance identities across views, producing renderable and cross-view-consistent instance features. To allow reconstruction and scene understanding to benefit from each other, we further design an instance-centric learning strategy that connects reconstruction, instance learning, and semantic learning through shared instance structure. Specifically, instance cues guide reconstruction, language-aligned semantics strengthen the discrimination of confusing same-category instances, and instance regions aggregate semantic evidence into coherent object-level predictions. Experiments on novel-view synthesis, instance segmentation, and open-vocabulary semantic understanding under varying input-view settings and on an unseen dataset demonstrate state-of-the-art performance, practical efficiency, and strong generalization.

Abstract

Explore similar work

InstanceSplat: Instance-Aware Feed-Forward 3D Gaussian Splatting for Scene Understanding

SparseLGS: Sparse View Language Embedded Gaussian Splatting

UniSem: Generalizable Semantic 3D Reconstruction from Sparse Unposed Images