Scene-Agnostic Object-Centric Representation Learning for 3D Gaussian Splatting

Tsuheng Hsu; Guiyu Liu; Juho Kannala; Janne Heikkilä

cs.CVApr 10, 2026

Scene-Agnostic Object-Centric Representation Learning for 3D Gaussian Splatting

Authors: Tsuheng Hsu, Guiyu Liu, Juho Kannala, Janne Heikkilä

Organizations: Aalto University · University of Oulu

Abstract

Recent works on 3D scene understanding leverage 2D masks from visual foundation models (VFMs) to supervise radiance fields, enabling instance-level 3D segmentation. However, the supervision signals from foundation models are not fundamentally object-centric and often require additional mask pre/post-processing or specialized training and loss design to resolve mask identity conflicts across views. The learned identity of the 3D scene is scene-dependent, limiting generalizability across scenes. Therefore, we propose a dataset-level, object-centric supervision scheme to learn object representations in 3D Gaussian Splatting (3DGS). Building on a pre-trained slot attention-based Global Object Centric Learning (GOCL) module, we learn a scene-agnostic object codebook that provides consistent, identity-anchored representations across views and scenes. By coupling the codebook with the module's unsupervised object masks, we can directly supervise the identity features of 3D Gaussians without additional mask pre-/post-processing or explicit multi-view alignment. The learned scene-agnostic codebook enables object supervision and identification without per-scene fine-tuning or retraining. Our method thus introduces unsupervised object-centric learning (OCL) into 3DGS, yielding more structured representations and better generalization for downstream tasks such as robotic interaction, scene understanding, and cross-scene generalization.

Figures & tables

Figure 1 : Overview of our proposed pipeline. (b) GOLD learns a global, scene-agnostic codebook storing a total of $C$ object intrinsic features while preserving the underlying functionality of slot attention.During scene optimization (a), for each view, we render Gaussian identity features and feed the input image into the trained GOLD module to obtain $K$ object masks and mask-to-codebook probabilities. We assign a codebook feature to each mask, broadcast it to image resolution, and blend the masks to construct dense supervision for Gaussian identity features.

Figure 2 : Qualitative comparison of rendered Gaussian feature masks on the OCTA dataset (left) and GSO dataset (right), where each object ID is visualized with a distinct color generated through a deterministic mapping. VFM-based supervision lacks object-level consistency, as highlighted in the images, where different surfaces of the cube or the rim of the vessel are often identified as separate instances.

Dataset	Method	FG-ARI $\uparrow$	ARI-A $\uparrow$	FG-AMI $\uparrow$	AMI-A $\uparrow$	mIoU $\uparrow$
OCTA	Gaussian Grouping [ 41 ]	0.700 ± 0.146	0.149 ± 0.042	0.751 ± 0.142	0.411 ± 0.076	0.697 ± 0.074
	Unified Lift [ 47 ]	0.474 ± 0.106	0.104 ± 0.027	0.567 ± 0.113	0.309 ± 0.060	0.500 ± 0.062
	ObjectGS [ 46 ]	0.586 ± 0.158	0.126 ± 0.057	0.642 ± 0.141	0.343 ± 0.087	0.466 ± 0.139
	Ours	0.870 ± 0.061	0.760 ± 0.032	0.868 ± 0.042	0.677 ± 0.030	0.593 ± 0.079
GSO	Gaussian Grouping [ 41 ]	0.549 ± 0.211	0.554 ± 0.275	0.651 ± 0.232	0.557 ± 0.180	0.594 ± 0.084
	Unified Lift [ 47 ]	0.292 ± 0.110	0.197 ± 0.103	0.478 ± 0.174	0.278 ± 0.083	0.396 ± 0.049

Table 1 : Quantitative comparison of our method and prior works on the OCTA and GSO datasets, evaluated across test scenes. Best results are in bold , second best are underlined.

Panel of Figure 3 — Figure 3 : Visualization of cross-scene identity on the OCTA dataset (a) and GSO dataset (b). Our method achieves cross-scene identification using the learned global codebook: as highlighted, the same object is assigned a consistent ID across different scenes, whereas VFM supervised methods remain scene-dependent.

Dataset	Method	FG-ARI $\uparrow$	ARI-A $\uparrow$	FG-AMI $\uparrow$	AMI-A $\uparrow$	mIoU $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FPS $\uparrow$
GSO	uOCF [ 26 ]	0.635 ± 0.281	0.404 ± 0.097	0.665 ± 0.264	0.356 ± 0.082	0.508 ± 0.110	26.458 ± 3.337	0.835 ± 0.053	0.191 ± 0.073	6.2 ± 1.1
GSO	Ours	0.846 ± 0.208	0.644 ± 0.147	0.845 ± 0.204	0.558 ± 0.130	0.508 ± 0.130	30.007 ± 4.354	0.910 ± 0.041	0.246 ± 0.100	106.1 ± 54.3

Table 2 : Quantitative comparison to NeRF-based methods on the GSO dataset on OCL, reconstruction metrics, and render speed.

Figure 4 : Qualitative comparison of rendered Gaussian feature masks using each prior method’s original supervision masks versus SAM2-based segmented/tracked mask supervision.

Figure 5 : Qualitative comparison of reconstruction and rendered masks for our method and uOCF on the GSO dataset. As highlighted, when uOCF fails to discover an object, the reconstructed scene may miss that object, whereas our reconstruction remains intact even when object discovery is imperfect.

Dataset	Method	FG-ARI $\uparrow$	ARI-A $\uparrow$	FG-AMI $\uparrow$	AMI-A $\uparrow$	mIoU $\uparrow$
OCTA	Gaussian Grouping [ 41 ] + SAM2 mask and tracking	0.839 ± 0.166	0.322 ± 0.162	0.826 ± 0.152	0.520 ± 0.117	0.805 ± 0.099
	Unified Lift [ 47 ] + SAM2 mask	0.563 ± 0.116	0.249 ± 0.071	0.602 ± 0.116	0.390 ± 0.073	0.530 ± 0.088
	ObjectGS [ 46 ] + SAM2 mask and tracking	0.777 ± 0.177	0.209 ± 0.090	0.775 ± 0.159	0.385 ± 0.090	0.489 ± 0.154
	Ours	0.870 ± 0.061	0.760 ± 0.032	0.868 ± 0.042	0.677 ± 0.030	0.593 ± 0.079
GSO	Gaussian Grouping [ 41 ] + SAM2 mask and tracking	0.826 ± 0.211	0.786 ± 0.177	0.831 ± 0.207	0.744 ± 0.126	0.790 ± 0.083
	Unified Lift [ 47 ] + SAM2 mask	0.519 ± 0.180	0.605 ± 0.102	0.596 ± 0.190	0.500 ± 0.067	0.532 ± 0.080

Table 3 : Quantitative comparison of our proposed pipeline and prior methods on the OCTA and GSO datasets, where the prior methods’ original supervision masks are replaced with SAM2-generated masks.

Figure 6 : Object discovery capability of our method (red boxes). Although SAM2 masks and tracking improve object-centric supervision, they fail to generalize to newly appearing objects that are absent in the first view without manual re-prompting.

Explore similar work

Ilov3Splat: Instance-Level Open-Vocabulary 3D Scene Understanding in Gaussian Splatting
May 6, 2026Binh Long Nguyen, Kien Nguyen, Sridha Sridharan +23D Scene Understanding Gaussian Splatting
School of Electrical Engineering and Robotics, Queensland University of Technology (QUT), Brisbane, QLD 4000, Australia · CSIRO Robotics, CSIRO, Brisbane, QLD 4069, Australia
OP2GS: Object-Aware 3D Gaussian Splatting with Dual-Opacity Primitives
May 19, 2026Guiyu Liu, Niklas Vaara, Janne Mustaniemi +23D Gaussian Opacity
Center for Machine Vision and Signal Analysis, University of Oulu, Finland · Aalto University, Finland
Sparse Code Uplifting for Efficient 3D Language Gaussian Splatting
May 13, 2026Lovre Antonio Budimir, Yushi Guan, Steve Ryhner +23D Gaussian 3D Scene Understanding
Faculty of Electrical Engineering and Computing, University of Zagreb · Department of Computer Science, University of Toronto · Vector Institute

May 6, 2026, cs.CVJ/K move · Enter open · S save

Ilov3Splat: Instance-Level Open-Vocabulary 3D Scene Understanding in Gaussian Splatting

Binh Long Nguyen, Kien Nguyen, Sridha Sridharan, Clinton Fookes+1

School of Electrical Engineering and Robotics, Queensland University of Technology (QUT), Brisbane, QLD 4000, Australia · CSIRO Robotics, CSIRO, Brisbane, QLD 4069, Australia

3D Scene Understanding Gaussian Splatting Frozen Contrastive Language-Image Pre-Training Visual Encoder

Open paper AskPDF arXiv

We introduce Ilov3Splat, a novel framework for instance-level open-vocabulary 3D scene understanding built on 3D Gaussian Splatting (3D-GS). Most prior work depends on 2D rendering-based matching or point-level semantic association, which undermines cross-view consistency, lacks coherent instance-level reasoning, and limits precision in downstream 3D tasks. To address these limitations, our method jointly optimizes scene geometry and semantic representations by augmenting Gaussian splats with view-consistent feature fields. Specifically, we leverage multi-resolution hash embedding to efficiently encode language-aligned CLIP features, enabling dense and coherent language grounding in 3D space. We further train an instance feature field using contrastive loss over SAM masks, supporting fine-grained object distinction across views. At inference time, CLIP-encoded queries are matched against the learned features, followed by two-stage 3D clustering to retrieve relevant Gaussian groups. This enables our framework to identify arbitrary objects in 3D scenes based on natural language descriptions, without requiring category supervision or manual annotations. Experiments on standard benchmarks demonstrate that Ilov3Splat outperforms prior open-vocabulary 3D-GS methods in both object selection and instance segmentation, offering a flexible and accurate solution for language-driven 3D scene understanding. Project page: https://csiro-robotics.github.io/Ilov3Splat.

Abstract

Explore similar work

Ilov3Splat: Instance-Level Open-Vocabulary 3D Scene Understanding in Gaussian Splatting

OP2GS: Object-Aware 3D Gaussian Splatting with Dual-Opacity Primitives

Sparse Code Uplifting for Efficient 3D Language Gaussian Splatting