GroupForward: Building Referable 3D Scenes via Instance-Grouped Feed-Forward Gaussian Splatting

Qijian Tian; Zimeng Wu; Xuhong Wang; Lizhuang Ma; Xin Tan

cs.CVAug 18, 2026

GroupForward: Building Referable 3D Scenes via Instance-Grouped Feed-Forward Gaussian Splatting

Authors: Qijian Tian, Zimeng Wu, Xuhong Wang, Lizhuang Ma, Xin Tan

Organizations: School of Computer Science, Shanghai Jiao Tong University, Shanghai 200240, China · School of Computer Science and Engineering, Beihang University, Beijing 100191, China · Shanghai Artificial Intelligence Laboratory, Shanghai 200232, China · School of Computer Science and Technology, East China Normal University, Shanghai 200241, China

Abstract

Simultaneously reconstructing and understanding 3D environments is essential for embodied agents. Toward this goal, feed-forward semantic 3D Gaussian Splatting (3DGS) efficiently constructs semantic scene representations from sparse multi-view observations. However, existing methods lack explicit instance discrimination and mainly support category- or phrase-based semantic queries. To this end, we propose GroupForward, an instance-grouped feed-forward Gaussian splatting model that reconstructs geometry, appearance, instance structure, and semantics from sparse, unposed, and uncalibrated multi-view images. Unlike existing methods that attach high-dimensional semantic features to each Gaussian, GroupForward learns compact instance embeddings that group Gaussians into cross-view consistent 3D instances, reformulating feed-forward semantic 3DGS from per-Gaussian semantic feature rendering to instance-level semantic aggregation and propagation. Building on these instance groups, we further propose a Referential Scene Reasoning Framework (RSRF) for complex 3D referring segmentation. RSRF constructs an instance-grouped 3D scene graph and retrieves candidate instances for a given referring expression. A vision-language model then reasons over structured instance evidence and multi-view observations to identify the referred instance among the candidates. RSRF thereby extends language interaction from simple semantic querying to complex referential scene reasoning. Experiments on semantic reconstruction and referential reasoning demonstrate the effectiveness of our instance-grouped reconstruction and reasoning framework.

Figures & tables

Fig. 1: The proposed instance-grouped formulation for feed-forward semantic 3D Gaussian Splatting builds referable scenes with low-dimensional instance embeddings rather than attaching high-dimensional semantic features, enabling complex referential reasoning beyond category- or phrase-level queries.

Fig. 2: Overview of GroupForward and the Referential Scene Reasoning Framework (RSRF). Given sparse, unposed, and uncalibrated multi-view images, GroupForward reconstructs cross-view consistent 3D instance groups with compact instance embeddings and aggregates open-vocabulary semantics at the instance level. RSRF further organizes the instance groups into a 3D scene graph and integrates structured graph evidence with multi-view observations for VLM-based complex referential reasoning and 3D referring segmentation.

Method	Venue & Year	Source View				Target View
Method	Venue & Year	mIoU $\uparrow$	Acc. $\uparrow$	rel $\downarrow$	$\tau\uparrow$	mIoU $\uparrow$	Acc. $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
LSeg [ 35 ]	ICLR 2022	0.4701	0.7891	–	–	0.4819	0.7927	–	–	–
NeRF-DEF [ 23 ]	NeurIPS 2022	0.4540	0.7173	27.68	9.61	0.4037	0.6755	19.86	0.6650	0.3629
Feature 3DGS [ 3 ]	CVPR 2024	0.4453	0.7276	12.95	21.07	0.4223	0.7174	24.49	0.8132	0.2293
PixelSplat [ 17 ]	CVPR 2024	–	–	–	–	–	–	24.89	0.8392	0.1641
LSM [ 7 ]	NeurIPS 2024	0.5034	0.7740	3.38	67.77	0.5078	0.7686	24.39	0.8072	0.2506

TABLE I: Quantitative comparison on ScanNet with two input views, following the benchmark established by LSM [ 7 ] and Uni3R [ 8 ] .

Method	4 Views					6 Views
Method	mIoU $\uparrow$	Acc. $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	mIoU $\uparrow$	Acc. $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
LSeg [ 35 ]	0.5002	0.7982	–	–	–	0.5381	0.8077	–	–	–
IGGT [ 24 ]	0.5247	0.7911	–	–	–	0.5799	0.8131	–	–	–
AnySplat [ 21 ]	–	–	21.82	0.8204	0.1415	–	–	22.28	0.8093	0.1509
Feature 3DGS [ 3 ]	0.5192	0.7250	23.78	0.7779	0.2661	0.5461	0.7399	24.36	0.7905	0.2356
LSM [ 7 ]	0.4949	0.8002	22.85	0.8066	0.1971	0.5323	0.8076	20.80	0.7986	0.2056

TABLE II: Quantitative comparison on ScanNet with varying numbers of input views for novel-view semantic segmentation and synthesis.

Method	mIoU $\uparrow$	Acc. $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
LSeg [ 35 ]	0.4690	0.8410	–	–	–
LSM [ 7 ]	0.4341	0.8233	16.73	0.6539	0.2814
Uni3R [ 8 ]	0.4754	0.8351	23.03	0.7880	0.1276
Ours	0.5633	0.8769	27.26	0.8790	0.1226

TABLE III: Zero-shot generalization from ScanNet to Replica. Results are averaged over 2, 4, 6, and 8 input views.

Method	Language Understanding	3D Output	mIoU $\uparrow$
Uni3R [ 8 ]	Phrase / Category	$\checkmark$	0.2538
Sa2VA [ 40 ]	Referential	$\times$	0.5012
Ours	Referential	$\checkmark$	0.6496

TABLE IV: Comparison of referential scene reasoning capabilities with novel-view referring segmentation performance.

Semantic Model	mIoU $\uparrow$	Acc. $\uparrow$
LSeg [ 35 ] (default)	0.5905	0.8294
OpenSeg [ 54 ]	0.4536	0.6292
CLIP [ 34 ]	0.1630	0.3759

TABLE V: Comparison of different semantic models for GroupForward.

Fig. 3: Qualitative comparisons of novel-view open-vocabulary segmentation on ScanNet. Our GroupForward produces more complete semantic regions with cleaner object boundaries and fewer cross-category inconsistencies than existing methods.

Fig. 4: Qualitative comparison of novel-view synthesis on ScanNet. Red boxes highlight representative rendering artifacts in competing methods, while our GroupForward better preserves scene structure and fine-grained visual details.

Fig. 5: Qualitative comparisons of referential scene reasoning. Our method correctly grounds target instances with accurate novel-view segmentation and corresponding 3D outputs, while competing methods fail to ground the target instances or lack native 3D outputs.

#	Method	mIoU $\uparrow$	Acc. $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
1	Ours (default)	0.5989	0.8302	26.23	0.8948	0.1230
Semantic ablation
2	w/o $\mathcal{L}_{\mathrm{render}}$	0.5772	0.8131	26.17	0.8924	0.1237
3	w/o instance embedding	0.5479	0.8096	26.19	0.8929	0.1234
Rendering ablation
4	w/o self-rendering	0.5950	0.8297	26.11	0.8917	0.1267

TABLE VI: Ablation studies of GroupForward on ScanNet. Starting from the default full model, we remove or replace key components to assess the contributions of individual design choices and the effectiveness of pseudo-label supervision.

Explore similar work

InstanceSplat: Instance-Aware Feed-Forward 3D Gaussian Splatting for Scene Understanding
Aug 7, 2026Minchao Jiang, Xiaoxuan Ma, Shunyu Jia +33D Gaussian Splatting Open-Vocabulary 3D Segmentation
1Shanghai Jiao Tong University · 2Eastern Institute of Technology, Ningbo · 3Carnegie Mellon University +2
Scenes as Objects, Not Primitives: Instance-Structured 3D Tokenization from Unposed Views
Jun 28, 2026Mijin Yoo, In Cho, Subin Jeon +3Multi-View 3D Reconstruction Object-Centric Representation Learning
1Yonsei University · 2Seoul National University
Open-Vocabulary and Referring Segmentation for 3D Gaussians Using 2D Detectors
Jun 29, 2026Jameel Hassan, Yasiru Ranasinghe, Vishal Patel3D Gaussian Splatting Open-Vocabulary 3D Segmentation
Johns Hopkins University

Aug 7, 2026, cs.CVJ/K move · Enter open · S save

InstanceSplat: Instance-Aware Feed-Forward 3D Gaussian Splatting for Scene Understanding

Minchao Jiang, Xiaoxuan Ma, Shunyu Jia, Haoru Wang+2

1Shanghai Jiao Tong University · 2Eastern Institute of Technology, Ningbo · 3Carnegie Mellon University+2

3D Gaussian Splatting Open-Vocabulary 3D Segmentation Feed-Forward 3D Reconstruction 3D Instance Segmentation 3D Scene Understanding

Open paper AskPDF arXiv

Feed-forward 3D Gaussian Splatting (3DGS) enables efficient and generalizable 3D reconstruction, but current feed-forward 3DGS methods for scene understanding remain largely category-oriented. In contrast, instance-aware 3DGS methods typically rely on per-scene optimization and often decouple reconstruction from instance and semantic learning, limiting reciprocal interactions among them. We present InstanceSplat, a unified feed-forward 3DGS framework for generalizable 3D reconstruction and instance-aware scene understanding from pose-free multi-view images. In a single forward pass, InstanceSplat constructs an instance-aware Gaussian representation that jointly encodes appearance, geometry, instance identity, and language-aligned semantics. Shared 3D Gaussians ground instance identities across views, producing renderable and cross-view-consistent instance features. To allow reconstruction and scene understanding to benefit from each other, we further design an instance-centric learning strategy that connects reconstruction, instance learning, and semantic learning through shared instance structure. Specifically, instance cues guide reconstruction, language-aligned semantics strengthen the discrimination of confusing same-category instances, and instance regions aggregate semantic evidence into coherent object-level predictions. Experiments on novel-view synthesis, instance segmentation, and open-vocabulary semantic understanding under varying input-view settings and on an unseen dataset demonstrate state-of-the-art performance, practical efficiency, and strong generalization.

Abstract

Explore similar work

InstanceSplat: Instance-Aware Feed-Forward 3D Gaussian Splatting for Scene Understanding

Scenes as Objects, Not Primitives: Instance-Structured 3D Tokenization from Unposed Views

Open-Vocabulary and Referring Segmentation for 3D Gaussians Using 2D Detectors