Organizations: Department of Computer and Data Science, Case Western Reserve University, Cleveland, OH 44106 USA. · Department of Artificial Intelligence, Westlake University, Hangzhou, Zhejiang 310024, P.R. China.
Open-vocabulary 3D scene understanding aims to segment and retrieve arbitrary objects in 3D scenes using natural language queries. Most existing methods typically follow a splat-then-segment paradigm, where 3D Gaussian Splatting is first reconstructed and then augmented with labels or language features for subsequent segmentation. However, this pipeline often suffers from view inconsistency, ambiguous object boundaries, and weak object-level correspondence, especially in complex static and dynamic scenes. Building on our prior Segment-then-Splat framework, we present RefineSplat, a reconstruction-guided extension that retains its object-centric paradigm: it first tracks objects in 2D and then reconstructs each object with a dedicated set of 3D Gaussians, preserving explicit object-Gaussian correspondence for unified open-vocabulary segmentation and retrieval. To improve robustness, we further close the loop between 2D tracking and 3D reconstruction. Since off-the-shelf video trackers can produce unreliable masks that propagate errors into object reconstruction, we introduce a reconstruction-guided feedback mechanism that refines object tracking using rendered masks and multi-view consistency. Moreover, instead of assigning each object a single language embedding, we propose a viewpoint-aware embedding assignment strategy, where each object maintains multiple view-clustered embeddings to capture view-dependent semantic cues. We also introduce Blender-OVS, a synthetic benchmark of static and dynamic sequences with exact object meshes and point clouds, enabling direct evaluation of extracted geometry. Experiments on static and dynamic benchmarks show improved rendered-view segmentation and lower object-level Chamfer distances, with competitive scene reconstruction quality.
Figures & tables
Fig. 1: (a) Independent per-view segmentation produces object masks without persistent identities across views. (b) Video tracking propagates identities, but tracking failures can become direct supervision for the reconstructed object groups. (c) RefineSplat closes the loop: track-level filtering uses an intermediate object-centric reconstruction to reject unreliable tracks before final reconstruction, while observation-level filtering selects suitable mask observations for object supervision during the later stage of final optimization. Mask colors denote object identities; the reconstructed scene is visualized with object IDs on the left and RGB appearance on the right.
Fig. 2: Overview of the reconstruction-guided object-centric pipeline . SAM2 propagates class-agnostic object masks through the calibrated input sequence. Multi-view votes assign persistent object identities to the initial Gaussians, and object-specific rendering losses preserve these identities during densification and reconstruction. RefineSplat first optimizes an intermediate model, renders occlusion-aware object-membership maps, and removes tracks whose disagreement with the tracked masks is frequent and severe across valid views (track-level filtering). The retained tracks are then reinitialized for final reconstruction, during which observation-level filtering selects suitable mask observations for object-level supervision in the later optimization stage.
Fig. 3: Viewpoint-aware embedding assignment. Object crops from similar camera-pose clusters are encoded by CLIP and averaged within each cluster. The resulting cluster embeddings are attached to the same object-specific Gaussian group, preserving viewpoint-specific cues that a single global average can dilute.
Fig. 4: Examples from Blender-OVS. Static scenes (top) are rendered from multiple calibrated viewpoints; dynamic scenes (bottom) additionally vary object pose and visibility over time. Each sequence includes dense per-view instance masks (2D-level GT) and exact object geometry in world coordinates (3D-level GT), shown with consistent instance colors. The benchmark contains five static and five dynamic scenes.
Dataset
Static Objects
Dynamic Objects
3D Instance GT
Exact 2D-3D Alignment
Paired Language
3DOVS [ 41 ]
✓
×
×
×
✓
LERF [ 13 ]
✓
×
×
×
✓
Mip-NeRF360 [ 14 ]
✓
×
×
×
✓
ScanNet++ [ 42 ]
✓
×
✓
×
✓
HyperNeRF [ 15 ]
×
✓
×
×
✓
Neu3D [ 43 ]
×
✓
×
×
✓
TABLE I: Comparison of existing datasets for open-vocabulary segmentation.
Dataset
Method
PSNR ↑
SSIM ↑
LPIPS ↓
mIoU ↑
mBIoU ↑
LERF [ 13 ]
LEGaussians [ 2 ]
24.775
0.874
0.198
26.465
20.768
LangSplat [ 3 ]
24.002
0.856
0.218
47.463
41.373
Gaussian-Grouping [ 5 ]
23.861
0.862
0.211
29.360
26.690
OpenGaussian [ 33 ]
22.642
0.855
0.213
43.778
38.380
ObjectGS [ 9 ]
23.074
0.856
0.203
24.203
22.815
Segment-then-Splat [ 6 ]
25.227
0.882
0.196
52.098
48.978
TABLE II: Quantitative comparisons on 2D pixel-level open-vocabulary segmentation, on static scenes (Mip-NeRF360, LERF) and dynamic scenes (HyperNeRF). Best results are in bold , and second-best results are underlined .
Dataset
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Chamfer Distance ↓
ScanNet++ [ 42 ]
Gaussian-Grouping [ 5 ]
22.412
0.789
0.463
2.128
OpenGaussian [ 33 ]
23.153
0.801
0.426
1.605
ObjectGS [ 9 ]
23.213
0.821
0.403
3.498
Segment-then-Splat [ 6 ]
23.062
0.799
0.348
1.766
Ours
23.278
0.800
0.344
1.284
Blender-OVS (static)
Gaussian-Grouping [ 5 ]
37.836
0.969
0.148
3.232
TABLE III: Quantitative comparisons on 3D instance-level open-vocabulary segmentation on ScanNet++ and Blender-OVS. For each dataset, best results are in bold , and second-best results are underlined .
Fig. 5: Qualitative results for 2D pixel-level Open-vocabulary Segmentation . Predicted segmentation is overlaid in green. The top block contains static LERF and Mip-NeRF360 queries, and the bottom block contains two timestamps each for HyperNeRF queries. RefineSplat more consistently suppresses neighboring objects and follows thin structures than the compared outputs in these examples.
Fig. 6: Qualitative results for 3D instance-level Open-vocabulary Segmentation on Blender-OVS . Each row shows the source scene, ground-truth object, and language-guided object extractions. The dynamic block (bottom) shows two timestamps for each query.
Module
Metric
SAM2
Track Filtering
Observation Filtering
PSNR ↑
SSIM ↑
LPIPS ↓
mIoU ↑
mBIoU ↑
✓
-
-
24.932
0.879
0.194
54.672
48.935
✓
✓
-
25.087
0.876
0.203
61.704
56.182
✓
✓
✓
25.491
0.886
0.191
62.845
57.298
TABLE IV: Ablation study of reconstruction-guided feedback modules.
Query Strategy
mIoU ↑
mBIoU ↑
Global Multi-view Average
58.94
53.10
Independent Per-view
56.60
51.24
View Clusters (K-means)
59.48
53.86
View Clusters (Hierarchical)
62.85
57.30
TABLE V: Ablation study of embedding assignment strategies.
We introduce Ilov3Splat, a novel framework for instance-level open-vocabulary 3D scene understanding built on 3D Gaussian Splatting (3D-GS). Most prior work depends on 2D rendering-based matching or point-level semantic association, which undermines cross-view consistency, lacks coherent instance-level reasoning, and limits precision in downstream 3D tasks. To address these limitations, our method jointly optimizes scene geometry and semantic representations by augmenting Gaussian splats with view-consistent feature fields. Specifically, we leverage multi-resolution hash embedding to efficiently encode language-aligned CLIP features, enabling dense and coherent language grounding in 3D space. We further train an instance feature field using contrastive loss over SAM masks, supporting fine-grained object distinction across views. At inference time, CLIP-encoded queries are matched against the learned features, followed by two-stage 3D clustering to retrieve relevant Gaussian groups. This enables our framework to identify arbitrary objects in 3D scenes based on natural language descriptions, without requiring category supervision or manual annotations. Experiments on standard benchmarks demonstrate that Ilov3Splat outperforms prior open-vocabulary 3D-GS methods in both object selection and instance segmentation, offering a flexible and accurate solution for language-driven 3D scene understanding. Project page: https://csiro-robotics.github.io/Ilov3Splat.
Binh Long Nguyen, Kien Nguyen, Sridha Sridharan +2
School of Electrical Engineering and Robotics, Queensland University of Technology (QUT), Brisbane, QLD 4000, Australia · CSIRO Robotics, CSIRO, Brisbane, QLD 4069, Australia
Open-vocabulary 3D scene understanding enables object localization and segmentation from free-form text queries without a fixed category vocabulary. Many recent methods build on 3D Gaussian Splatting and consolidate multi-view observations, such as masked crops from individual views, into language features or compact object descriptors before the query is known. However, observations of the same object vary across viewpoints and are not equally informative: some reveal cues relevant to a particular query, whereas others provide incomplete or misleading evidence. Pre-query consolidation can therefore suppress cues on which a later query depends. We introduce EviSplat, which preserves individual observation features as evidence for later text queries. EviSplat retains individual observation features within class-agnostic 3D instances that represent objects, object parts, or background regions. It also learns, for each Gaussian, a distribution describing which visual appearances its observations support. Given a text query, EviSplat scores each instance using its most relevant observations. It then computes a score for each Gaussian by combining instance-level relevance with locally supported evidence, weighted by how often and how unambiguously that Gaussian was observed. Different queries can thus draw on different visual cues from the same preserved evidence. Experiments across diverse datasets and evaluation protocols demonstrate state-of-the-art performance, supporting the benefit of preserving multi-view evidence until query time and aggregating it according to the query.
Sungho Moon, Kota Shimomura, Junwoo Park +4
DGIST, Daegu, Republic of Korea · Chubu University, Japan · Elith Inc., Japan +2
Feed-forward Gaussian splatting models have demonstrated remarkable effectiveness in reconstructing three-dimensional (3D) objects from a few two-dimensional (2D) images, even if they are unseen. As existing methods typically predict Gaussian primitives uniformly across the 3D space, most primitives are placed in non-object regions. This may hinder the representation of fine object details. This paper proposes a Voxel-Selective Gaussian Splatting model (VS-Splat), a new end-to-endfeed-forward Gaussian splatting framework that predicts many primitives only within selected voxels that are likely to belong to an object, without 3D structural supervision. To achieve this, we propose a new learnable voxel selection approach that identifies object-centric voxels only with 2D rendering supervision. Our sparse-view rendering experiments with three benchmark datasets show that proposed VS-Splat outperforms several state-of-the-art methods. We further demonstrate its effectiveness as a backbone for an existing densification method and show that anoptional extension improves its robustness to inaccurate camera pose estimates.
Yunsu Jeong, Hyuk Heo, Youngsang Kwak +2
Department of Electrical and Computer Engineering (ECE), Sungkykwan University (SKKU), Suwon 16419, South Korea · Department of Display Convergence Engineering (DCE), SKKU, Suwon 16419, South Korea · AiM Future, Seoul 06804, South Korea +1