ClusterSplat: Semantic Cluster Selection for 3D Visual Grounding in Gaussian Splatting

Jaesung Lee; Hyeontaek Hwang; Gabriel Manalu; Daeyoung Kim

cs.CVOct 27, 2026

ClusterSplat: Semantic Cluster Selection for 3D Visual Grounding in Gaussian Splatting

Authors: Jaesung Lee, Hyeontaek Hwang, Gabriel Manalu, Daeyoung Kim

Organizations: School of Computing, KAIST

Abstract

We study 3D visual grounding in 3D Gaussian Splatting (3DGS), where a referring expression should identify a target object as both a rendered 2D mask and a set of explicit 3D Gaussians. Prior methods typically extract 3D targets by applying heuristic thresholding or top-ratio filtering to dense primitive-level relevance scores. However, these scores often vary substantially across scenes and prompts, making such heuristics sensitive and leading to unstable targeting. ClusterSplat addresses this by reframing 3D target selection as cluster-level selection over scene-adaptive candidates. The method first learns instance-aware Gaussian features, forms feature-aware seed voxels, and merges neighboring seed voxels when a description-length criterion decreases, producing a scene-specific set of cluster candidates. Given a referring expression, a query-conditioned cluster scorer ranks the cluster candidates with a lightweight MLP, and the selected cluster directly defines the explicit 3D target while its cluster score is rendered for the 2D mask. On ScanRefer, ClusterSplat achieves the best rendered 2D segmentation and explicit 3D target metrics among the compared 3DGS baselines. It also obtains the best average Ref-LERF scores, supporting the same trend under fine-grained referring descriptions. On ScanNet v2 class name queries, ClusterSplat achieves the strongest rendered 2D results and the highest 3D matching accuracy, while maintaining competitive 3D IoU. Our project page is available at https://jaesung8.github.io/clustersplat/ .

Figures & tables

Figure 1: Primitive-level filtering versus cluster-level selection. Given a referring expression, primitive score fields can be unstable, and top-ratio filtering may scatter the 3D support across the target and nearby distractors. ClusterSplat instead performs scene-adaptive clustering followed by query-conditioned cluster scoring to select one coherent 3D target.

Figure 2: ClusterSplat overview. The pipeline first performs Instance-Aware Gaussian Feature Learning and Scene-Adaptive Clustering : it learns instance-aware Gaussian features, forms feature-aware seed voxels, and merges them into scene-adaptive cluster candidates. It then performs Query-Conditioned Cluster Scoring : the referring expression conditions the cluster tokens, and the MLP scorer ranks the candidates to select the Top-1 cluster as the explicit 3D target. Finally, Cluster-Level Mask Rendering assigns the selected cluster scores to Gaussians and renders a 2D mask supervised by the mask loss.

Methods	ScanRefer				ScanNet v2 class name
	2D		3D		2D		3D
	mIoU $\uparrow$	[email protected] $\uparrow$	mIoU $\uparrow$	[email protected] $\uparrow$	mIoU $\uparrow$	[email protected] $\uparrow$	mIoU $\uparrow$	mAcc $\uparrow$
LangSplat [ 5 ]	11.00	13.41	5.81	4.95	27.67	40.82	2.01	8.13
OpenGaussian [ 6 ]	18.48	5.53	1.06	0.10	19.55	26.11	31.05	46.39
Dr.Splat [ 10 ]	19.01	27.76	9.48	11.17	19.94	32.64	30.67	46.12
ReferSplat [ 12 ]	31.30	45.07	4.05	0.10	28.70	40.12	6.35	17.22

Table 1: ScanRefer and ScanNet v2 quantitative results on the shared evaluation scenes. All scores are percentages. Bold and underlined values indicate the best and second-best distinct scores in each column.

Figure 3: Qualitative examples on ScanRefer. Each row shows a referring expression with rendered masks and top-down 3D selections. ClusterSplat localizes the referred object more coherently and reduces distractor support compared with score-discretized baselines.

Method	Purity $\uparrow$	Coverage@1 $\uparrow$	Oracle Top-1 IoU $\uparrow$	[email protected] $\downarrow$
Fixed KMeans ( $K=128$ )	0.7825	0.5649	0.4017	5.95
Fixed KMeans ( $K=256$ )	0.8094	0.4840	0.3734	8.90
Fixed KMeans ( $K=512$ )	0.8269	0.3935	0.3200	14.15
Fixed KMeans + pair merge	0.7436	0.6693	0.4615	4.71
Scene-adaptive clustering (ours)	0.7635	0.6933	0.4798	3.86
Scene-adaptive clustering + xyz cost	0.7745	0.6720	0.4682	3.85

Table 2: Cluster candidate quality before Stage 2 on ScanRefer. Purity, Coverage@1, and Oracle Top-1 IoU are fractions in

[0,1]

; [email protected] is a cluster count. Metric definitions are in Appendix B.3 . Bold values indicate the best score in each column.

Construction	Top-1	Top-2	Top-3	Top-4	Top-5
Fixed KMeans ( $K=128$ )	18.31	17.66	16.42	15.81	14.47
Fixed KMeans ( $K=256$ )	15.44	16.34	16.27	16.32	16.21
Fixed KMeans ( $K=512$ )	10.97	13.29	14.56	14.60	14.62
Fixed KMeans + pair merge	21.23	19.66	18.67	17.47	16.46
Scene-adaptive clustering	23.33	19.78	17.42	16.09	14.96
Scene-adaptive + xyz cost	21.63	19.64	17.41	16.09	14.56

Table 3: Effect of cluster construction and cumulative Top-

k

selection on ScanRefer 3D mIoU (%). Within each construction, Top-

k

predictions combine the

k

highest-scoring clusters from the same trained model. Bold indicates the highest value in the table.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Illustration of how we prompt GPT-4.1 to obtain five diverse pseudo-captions for a training-frame instance mask. The specific view shows two tables, and the mask/bounding-box cue enables GPT-4.1 to generate captions that are descriptive and localized to the selected instance.

Figure 5: Sensitivity of primitive-level target extraction to the top-ratio hyperparameter. We evaluate the 3D targeting performance (mIoU) of OpenGaussian [ 6 ] and Dr.Splat [ 10 ] across 10 ScanNet v2 scenes by sweeping the selection ratio

\rho\in[0.01,0.10]

. The optimal ratio is highly scene-dependent, and the performance often drops sharply away from the peak, indicating that primitive-level score discretization requires careful calibration for robust open-vocabulary 3D targeting. This motivates cluster-level Top-1 (argmax) selection, which relies on relative cluster ranking rather than absolute score thresholds.

Encoder	2D		3D
Encoder	mIoU $\uparrow$	[email protected] $\uparrow$	mIoU $\uparrow$	[email protected] $\uparrow$
BERT	31.47	43.48	14.62	25.40
CLIP	39.72	54.35	23.33	43.38

Appendix

Table 4: Frozen-encoder comparison on ScanRefer. Bold values indicate the best score in each column.

Variant	Pos.	Cluster SA	Cross-attn	Token scorer	2D		3D
Variant	Pos.	Cluster SA	Cross-attn	Token scorer	mIoU $\uparrow$	[email protected] $\uparrow$	mIoU $\uparrow$	[email protected] $\uparrow$	Hit@1 $\uparrow$
Pooled scorer baseline	-	-	-	-	37.93	51.59	22.20	41.29	43.52
Token scorer baseline	-	-	-	✓	37.86	51.66	23.17	41.96	42.29
+ Cross-attention	-	-	✓	✓	37.77	52.18	21.69	40.75	42.67
+ Cluster self-attention	-	✓	✓	✓	37.81	51.69	22.55	41.34	44.05
Full scorer	✓	✓	✓	✓	39.72	54.35	23.33	43.38	46.03

Appendix

Table 5: Module ablation on ScanRefer with

d_{p}=64

for all variants. Bold values indicate the best score in each column.

$d_{p}$	2D		3D
$d_{p}$	mIoU $\uparrow$	[email protected] $\uparrow$	mIoU $\uparrow$	[email protected] $\uparrow$	mAcc $\uparrow$	Hit@1 $\uparrow$
32	38.75	52.59	22.06	40.92	33.66	43.88
64	39.72	54.35	23.33	43.38	35.86	46.03
128	40.41	55.00	21.42	40.32	32.09	41.71
256	39.77	54.78	20.52	36.69	30.30	39.42

Appendix

Table 6: Prototype dimension ablation on ScanRefer with the full scorer. Bold values indicate the best score in each column.

Caption generator	2D		3D
Caption generator	mIoU $\uparrow$	[email protected] $\uparrow$	mIoU $\uparrow$	[email protected] $\uparrow$
GPT-4.1	39.72	54.35	23.33	43.38
InternVL3.5-8B	38.60	52.82	21.23	39.00
Qwen3-VL-8B	34.22	46.63	20.01	36.82

Appendix

Table 7: Caption-source comparison on ScanRefer. Bold values indicate the best score in each column.

Methods	mIoU $\uparrow$					[email protected] $\uparrow$
Methods	ramen	figurines	teatime	kitchen	Avg.	ramen	figurines	teatime	kitchen	Avg.
LangSplat [ 5 ]	8.87	21.78	3.04	11.39	11.27	7.89	38.10	3.39	13.79	15.79
OpenGaussian [ 6 ]	3.96	24.42	8.49	10.43	11.83	2.63	39.68	10.17	20.69	18.29
Dr.Splat [ 10 ]	6.21	7.25	2.90	6.76	5.78	9.21	9.52	0.00	6.90	6.41
ReferSplat [ 12 ]	28.81	14.14	35.54	23.44	25.48	39.47	17.46	55.93	31.03	35.97
ReferSplat (w/ our loss)	33.06	16.82	36.26	20.21	26.59	38.16	25.40	54.24	27.59	36.35

Appendix

Table 8: Ref-LERF quantitative results in percentages. Bold and underlined values indicate the best and second-best distinct scores in each column; ties receive the same marking. ClusterSplat improves the average referring segmentation scores across the evaluated scenes.

Method	Offline Preparation	Online Targeting	Rendering	ScanRefer Accuracy
Method	Time (min/scene) $\downarrow$	Time (ms/query) $\downarrow$	FPS $\uparrow$	2D mIoU $\uparrow$	3D mIoU $\uparrow$
LangSplat	21.6	20.45	0.28	11.00	5.81
OpenGaussian	25.8	43.74	13.10	18.48	1.06
Dr.Splat	2.0	13.20	68.00	19.01	9.48
ReferSplat	113.7	119.71	1.13	31.30	4.05
ClusterSplat (Ours)	125.3	19.55	51.12	39.72	23.33

Appendix

Table 9: Offline preparation costs and online efficiency on an NVIDIA RTX A6000. Offline costs cover method-specific stages, excluding RGB reconstruction and cache preparation. Accuracy is in percentages. Bold and underlined values indicate the best and second-best distinct scores in each column.

Figure 6: Ref-LERF qualitative results. Each row shows a fine-grained referring description with attributes, surface cues, or spatial relations. ClusterSplat produces compact masks that better follow the referred object across the shown examples.

Figure 7: ScanRefer qualitative results on the first five evaluation scenes. Each row shows the ground-truth mask, rendered mask predictions, and top-down 3D target selections for a referring expression. The examples illustrate compact, object-aligned support with reduced distractor responses.

Figure 8: ScanRefer qualitative results on the remaining five evaluation scenes. The examples cover attribute-based and spatial referring expressions, showing consistent 2D mask concentration and top-down 3D target localization across the evaluation scenes.

Figure 9: Top-down ScanNet v2 19-class visualization across all 10 evaluation scenes. The first two columns compare annotated ScanNet v2 point-cloud labels with Gaussian pseudo labels, followed by class-level predictions from 3DGS baselines and ClusterSplat.

3D Visual Grounding

Three-dimensional visual grounding links language or visual queries to specific objects, parts, regions, and spatial relations in 3D scenes.

See all 142 papers

ZeroSplat: Generalized Referring Segmentation in 3D Gaussian Splatting
Jul 21, 2026Jiayu Ding, Meilu Song, Xiaoyi Zhang +33D Gaussian Splatting Open-Vocabulary 3D Segmentation
Peking University · North China Electric Power University · China University of Geosciences
Open-Vocabulary and Referring Segmentation for 3D Gaussians Using 2D Detectors
Jun 29, 2026Jameel Hassan, Yasiru Ranasinghe, Vishal Patel3D Gaussian Splatting Open-Vocabulary 3D Segmentation
Johns Hopkins University
Beyond Similarity Matching: Structured Reasoning for Open-Vocabulary Referring Segmentation in 3DGS
Aug 17, 2026Yizhao Wang, Xinfa Wang, Jingbo Wang +53D Referring Segmentation Referring Expression Segmentation
School of Computer Science, Henan Institute of Science and Technology

Jul 21, 2026J/K move · Enter open · S save

ZeroSplat: Generalized Referring Segmentation in 3D Gaussian Splatting

Jiayu Ding, Meilu Song, Xiaoyi Zhang, Hongbo Jin+2

Peking University · North China Electric Power University · China University of Geosciences

3D Gaussian Splatting Open-Vocabulary 3D Segmentation 3D Referring Segmentation 3DGS Segmentation

Open paper AskPDF arXiv

Recent advancements in 3D Gaussian Splatting (3DGS) have enabled language-guided scene understanding. However, existing Referring 3D Gaussian Splatting (R3DGS) methods are fundamentally restricted to single-target queries. To reflect the ambiguity of real-world instructions, we introduce the Generalized Referring 3D Gaussian Splatting Segmentation (GR3DGS) task, which requires dynamically segmenting an arbitrary number of targets (0, 1, or

N

). To facilitate comprehensive evaluation of this new task, we construct two new benchmarks: GR-LERF and GR-ScanNet. Crucially, existing R3DGS paradigms exhibit fundamental technical bottlenecks that severely limit their performance on the GR3DGS task: they lack intrinsic 3D point-level understanding by operating merely on 2D rendered pixels, and they incur prohibitive computational overhead by requiring per-scene optimization to embed heavy semantic features. To dismantle these bottlenecks, we propose ZeroSplat, a novel training-free and zero-feature framework. ZeroSplat lifts 2D Vision-Language Model (VLM) priors into 3D space through robust multi-view geometric constraints. This strategy enables intrinsic point-level understanding without incurring any additional feature storage. Extensive experiments demonstrate that ZeroSplat significantly outperforms state-of-the-art methods across generalized and single-target scenarios while maintaining exceptional efficiency. Project Page: https://inkmind-ai.github.io/ZeroSplat

Abstract

3D Visual Grounding

Explore similar work

ZeroSplat: Generalized Referring Segmentation in 3D Gaussian Splatting

Open-Vocabulary and Referring Segmentation for 3D Gaussians Using 2D Detectors

Beyond Similarity Matching: Structured Reasoning for Open-Vocabulary Referring Segmentation in 3DGS