Organizations: Department of Computer and Data Science, Case Western Reserve University, Cleveland, OH 44106 USA. · Department of Artificial Intelligence, Westlake University, Hangzhou, Zhejiang 310024, P.R. China.
Open-vocabulary 3D scene understanding aims to segment and retrieve arbitrary objects in 3D scenes using natural language queries. Most existing methods typically follow a splat-then-segment paradigm, where 3D Gaussian Splatting is first reconstructed and then augmented with labels or language features for subsequent segmentation. However, this pipeline often suffers from view inconsistency, ambiguous object boundaries, and weak object-level correspondence, especially in complex static and dynamic scenes. Building on our prior Segment-then-Splat framework, we present RefineSplat, a reconstruction-guided extension that retains its object-centric paradigm: it first tracks objects in 2D and then reconstructs each object with a dedicated set of 3D Gaussians, preserving explicit object-Gaussian correspondence for unified open-vocabulary segmentation and retrieval. To improve robustness, we further close the loop between 2D tracking and 3D reconstruction. Since off-the-shelf video trackers can produce unreliable masks that propagate errors into object reconstruction, we introduce a reconstruction-guided feedback mechanism that refines object tracking using rendered masks and multi-view consistency. Moreover, instead of assigning each object a single language embedding, we propose a viewpoint-aware embedding assignment strategy, where each object maintains multiple view-clustered embeddings to capture view-dependent semantic cues. We also introduce Blender-OVS, a synthetic benchmark of static and dynamic sequences with exact object meshes and point clouds, enabling direct evaluation of extracted geometry. Experiments on static and dynamic benchmarks show improved rendered-view segmentation and lower object-level Chamfer distances, with competitive scene reconstruction quality.
Figures & tables
Fig. 1: (a) Independent per-view segmentation produces object masks without persistent identities across views. (b) Video tracking propagates identities, but tracking failures can become direct supervision for the reconstructed object groups. (c) RefineSplat closes the loop: track-level filtering uses an intermediate object-centric reconstruction to reject unreliable tracks before final reconstruction, while observation-level filtering selects suitable mask observations for object supervision during the later stage of final optimization. Mask colors denote object identities; the reconstructed scene is visualized with object IDs on the left and RGB appearance on the right.
Fig. 2: Overview of the reconstruction-guided object-centric pipeline . SAM2 propagates class-agnostic object masks through the calibrated input sequence. Multi-view votes assign persistent object identities to the initial Gaussians, and object-specific rendering losses preserve these identities during densification and reconstruction. RefineSplat first optimizes an intermediate model, renders occlusion-aware object-membership maps, and removes tracks whose disagreement with the tracked masks is frequent and severe across valid views (track-level filtering). The retained tracks are then reinitialized for final reconstruction, during which observation-level filtering selects suitable mask observations for object-level supervision in the later optimization stage.
Fig. 3: Viewpoint-aware embedding assignment. Object crops from similar camera-pose clusters are encoded by CLIP and averaged within each cluster. The resulting cluster embeddings are attached to the same object-specific Gaussian group, preserving viewpoint-specific cues that a single global average can dilute.
Fig. 4: Examples from Blender-OVS. Static scenes (top) are rendered from multiple calibrated viewpoints; dynamic scenes (bottom) additionally vary object pose and visibility over time. Each sequence includes dense per-view instance masks (2D-level GT) and exact object geometry in world coordinates (3D-level GT), shown with consistent instance colors. The benchmark contains five static and five dynamic scenes.
Dataset
Static Objects
Dynamic Objects
3D Instance GT
Exact 2D-3D Alignment
Paired Language
3DOVS [ 41 ]
✓
×
×
×
✓
LERF [ 13 ]
✓
×
×
×
✓
Mip-NeRF360 [ 14 ]
✓
×
×
×
✓
ScanNet++ [ 42 ]
✓
×
✓
×
✓
HyperNeRF [ 15 ]
×
✓
×
×
✓
Neu3D [ 43 ]
×
✓
×
×
✓
TABLE I: Comparison of existing datasets for open-vocabulary segmentation.
Dataset
Method
PSNR ↑
SSIM ↑
LPIPS ↓
mIoU ↑
mBIoU ↑
LERF [ 13 ]
LEGaussians [ 2 ]
24.775
0.874
0.198
26.465
20.768
LangSplat [ 3 ]
24.002
0.856
0.218
47.463
41.373
Gaussian-Grouping [ 5 ]
23.861
0.862
0.211
29.360
26.690
OpenGaussian [ 33 ]
22.642
0.855
0.213
43.778
38.380
ObjectGS [ 9 ]
23.074
0.856
0.203
24.203
22.815
Segment-then-Splat [ 6 ]
25.227
0.882
0.196
52.098
48.978
TABLE II: Quantitative comparisons on 2D pixel-level open-vocabulary segmentation, on static scenes (Mip-NeRF360, LERF) and dynamic scenes (HyperNeRF). Best results are in bold , and second-best results are underlined .
Dataset
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Chamfer Distance ↓
ScanNet++ [ 42 ]
Gaussian-Grouping [ 5 ]
22.412
0.789
0.463
2.128
OpenGaussian [ 33 ]
23.153
0.801
0.426
1.605
ObjectGS [ 9 ]
23.213
0.821
0.403
3.498
Segment-then-Splat [ 6 ]
23.062
0.799
0.348
1.766
Ours
23.278
0.800
0.344
1.284
Blender-OVS (static)
Gaussian-Grouping [ 5 ]
37.836
0.969
0.148
3.232
TABLE III: Quantitative comparisons on 3D instance-level open-vocabulary segmentation on ScanNet++ and Blender-OVS. For each dataset, best results are in bold , and second-best results are underlined .
Fig. 5: Qualitative results for 2D pixel-level Open-vocabulary Segmentation . Predicted segmentation is overlaid in green. The top block contains static LERF and Mip-NeRF360 queries, and the bottom block contains two timestamps each for HyperNeRF queries. RefineSplat more consistently suppresses neighboring objects and follows thin structures than the compared outputs in these examples.
Fig. 6: Qualitative results for 3D instance-level Open-vocabulary Segmentation on Blender-OVS . Each row shows the source scene, ground-truth object, and language-guided object extractions. The dynamic block (bottom) shows two timestamps for each query.
Module
Metric
SAM2
Track Filtering
Observation Filtering
PSNR ↑
SSIM ↑
LPIPS ↓
mIoU ↑
mBIoU ↑
✓
-
-
24.932
0.879
0.194
54.672
48.935
✓
✓
-
25.087
0.876
0.203
61.704
56.182
✓
✓
✓
25.491
0.886
0.191
62.845
57.298
TABLE IV: Ablation study of reconstruction-guided feedback modules.
Query Strategy
mIoU ↑
mBIoU ↑
Global Multi-view Average
58.94
53.10
Independent Per-view
56.60
51.24
View Clusters (K-means)
59.48
53.86
View Clusters (Hierarchical)
62.85
57.30
TABLE V: Ablation study of embedding assignment strategies.
School of Electrical Engineering and Robotics, Queensland University of Technology (QUT), Brisbane, QLD 4000, Australia · CSIRO Robotics, CSIRO, Brisbane, QLD 4069, Australia
Department of Electrical and Computer Engineering (ECE), Sungkykwan University (SKKU), Suwon 16419, South Korea · Department of Display Convergence Engineering (DCE), SKKU, Suwon 16419, South Korea · AiM Future, Seoul 06804, South Korea +1