SplatScout: Query-Driven Part-Level Referring Segmentation in Gaussian Scenes

Yihan Zang; Da Li; Shinkyu Park; Ivan Viola; Dominik Engel

cs.CVOct 27, 2026

SplatScout: Query-Driven Part-Level Referring Segmentation in Gaussian Scenes

Authors: Yihan Zang, Da Li, Shinkyu Park, Ivan Viola, Dominik Engel

Organizations: CEMSE Division, King Abdullah University of Science and Technology

Abstract

Existing referring segmentation methods on Gaussian scenes primarily target whole objects, relying on segmenter-generated candidates from which a vision-language model (VLM) selects. Such predetermined fixed candidates may omit fine-grained parts needed as targets or cues for identifying objects. We introduce SplatScout, a query-driven, closed-loop approach that places perception under VLM control. The model actively selects views, directs segmentation, and inspects rendered 3D results to refine its predictions. For part-level evaluation, we introduce PartRef-ScanNet++, comprising 261 manually verified queries across 12 ScanNet++ scenes with mesh-derived ground truth. On PartRef-ScanNet++, SplatScout achieves 0.538 mIoU, substantially outperforming existing methods. Ablations demonstrate the benefits of iterative refinement, while comparisons across VLM backbones highlight the importance of effective tool-feedback utilization. SplatScout also achieves higher scores on four existing referring benchmarks. Code and the PartRef-ScanNet++ benchmark are available at https://github.com/kaust-violab/SplatScout.

Figures & tables

Figure 1: Overview of SplatScout. Top: Existing methods restrict the VLM to selecting from fixed segmentation candidates, which may omit the referred target. Bottom: SplatScout enables query-driven, closed-loop perception. The VLM coordinates the Navigator, Segmenter, and Lifter to explore views, extract target regions, and construct 3D-lifted Gaussian instances. Visual feedback allows the VLM to inspect and refine intermediate results without fixed semantic candidates.

Figure 2: Examples of closed-loop grounding. Top: For “the lid of the trash bin beside the room door,” render-back reveals that the initial lift $e_{1}$ includes the bag. The VLM re-segments within parent $s_{1}$ and accepts the corrected lift $e_{2}$ . Bottom: For “the padded ear cups of the black headphones on the desk,” the VLM identifies and segments both ear cups within the parent headphones, then lifts and returns both instances $e_{1}$ and $e_{2}$ .

Candidates fixed before the query arrives
	mIoU $\uparrow$				[email protected] $\uparrow$				[email protected] $\uparrow$
Method	Obj	Part	Multi	All	Obj	Part	Multi	All	Obj	Part	Multi	All
CLIP retrieval ( Radford et al., 2021 )	0.138	0.169	0.137	0.156	0.21	0.27	0.24	0.25	0.16	0.16	0.10	0.15
Search3D ( Takmaz et al., 2025 )	0.161	0.137	0.122	0.143	0.18	0.21	0.28	0.21	0.16	0.09	0.07	0.11
GaussDet ( Hassan et al., 2026 )	0.293	0.237	0.230	0.253	0.49	0.44	0.38	0.45	0.28	0.19	0.10	0.21
OpenGround ( Huang et al., 2026 )	0.195	0.156	0.229	0.176	0.237	0.219	0.379	0.242	0.224	0.152	0.241	0.184

Table 1: Results on PartRef-ScanNet++. Groups: Obj (76), Part (151: part as target, discriminator or anchor), Multi (29). Every method uses its original model and candidate source. GaussDet uses Qwen2.5-VL-7B, OpenGround uses GLM-4.5V, and ZeroSplat uses Qwen3-VL-30B.

ReferSplat	29.2
GaussDet	45.9
ZeroSplat	32.7
CausalSplat	36.1
SplatScout	61.1

Table 2: Results on external benchmarks (mIoU, %). LERF uses 2D test-view masks, while ScanNet uses 3D vertex labels. See Appendix D for protocols.

Method	Qwen3-VL-30B	Qwen3.5-35B	Qwen3.5-397B	$\Delta$	GT intervention
ZeroSplat	0.090	0.083	0.088	$-0.002$	0.293
+ GT masks	0.062	0.057	0.060	$-0.002$	–
OpenGround	0.161	0.173	0.180	$+0.019$	0.435
+ GT table	0.349	0.360	0.424	$+0.075$	–
SplatScout	0.128	0.353	0.450	$\mathbf{+0.322}$	–

Table 3: Effect of VLM choice and perception strategy on PartRef-ScanNet++ (mIoU).

\Delta

denotes the change from Qwen3-VL-30B to Qwen3.5-397B. GT masks and GT table replace the respective methods’ segmentation masks and candidate table. The final column reports GT-guided prompting for ZeroSplat and oracle selection for OpenGround. See Appendix E .

Setting	Obj	Part	Multi	All	Wrong
Interaction and feedback ablations
One look, one cut	0.224	0.199	0.215	0.209	154
No visual exploration	0.344	0.301	0.348	0.319	114
No segmentation feedback	0.433	0.403	0.407	0.412	68
No lifting feedback	0.471	0.429	0.420	0.440	62
Full SplatScout	0.469	0.453	0.385	0.450	55

Table 4: Ablation and oracle results on PartRef-ScanNet++ using Qwen3.5-397B. Wrong counts queries with IoU below 0.1. Details in App. F .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Navigator
Action	Arguments	Feedback
explore	$n$	Unseen frames selected by additional coverage
nearby	$f$	Neighboring frames
	$f$ , box	Full-resolution crop and frames observing the selected region
	$s$ or $e$	Alternative views of a group or instance
Segmenter

Appendix

Table 5: Tool interfaces of SplatScout, organized by tool set. Optional arguments are shown in brackets.

Benchmark	Scenes	Queries	Object	Part	Multi	Ground truth
			Targets
Ref-LERF [ He et al., 2025 ]	4	59	✓	–	–	2D masks
GR-LERF [ Ding et al., 2026c ]	4	111	✓	–	✓	2D masks
GR-ScanNet [ Ding et al., 2026c ]	10	83	✓	–	✓	Point labels
Causal-LERF [ Ding et al., 2026b ]	4	158	✓	–	–	2D masks
Causal-ScanNet [ Ding et al., 2026b ]	10	73	✓	–	–	Point labels

Appendix

Table 6: Comparison with existing referring benchmarks.

Method	[email protected]	[email protected]
OpenGround	10.0	6.3
SplatScout	33.9	22.5

Appendix

Table 7: Evaluation on OpenTarget using 300 sampled queries from the 12 shared scenes. Two blocked queries are excluded from evaluation. Both methods use Qwen3.5-397B.

Figure 3: Distribution of the 261 queries in PartRef-ScanNet++ across five referring roles.

Method	Waldo	Ramen	Teatime	Figurines	Mean
ZeroSplat (published)	21.3	30.4	41.4	37.8	32.7
ZeroSplat (reproduced)	24.8	33.6	26.0	29.0	28.4
GaussDet (published)	33.3	52.3	39.2	58.7	45.9
GaussDet (reproduced)	36.2	46.7	40.5	37.6	40.2

Appendix

Table 8: Published and reproduced results on Ref-LERF (mIoU, %). The reproduced implementations are used in our comparisons on PartRef-ScanNet++.

Setting	Seg./q	Steps/q	Not found
Full SplatScout	2.5	11.0	15
No visual exploration	4.6	13.0	62
No segmentation feedback	1.9	13.4	26
No lifting feedback	2.5	11.1	17

Appendix

Table 9: Tool-use behavior under feedback ablations. Calls and tokens are averaged per query.

3D Referring Segmentation

See all 15 papers

ZeroSplat: Generalized Referring Segmentation in 3D Gaussian Splatting
Jul 21, 2026Jiayu Ding, Meilu Song, Xiaoyi Zhang +33D Gaussian Splatting Open-Vocabulary 3D Segmentation
Peking University · North China Electric Power University · China University of Geosciences
Beyond Similarity Matching: Structured Reasoning for Open-Vocabulary Referring Segmentation in 3DGS
Aug 17, 2026Yizhao Wang, Xinfa Wang, Jingbo Wang +53D Referring Segmentation Referring Expression Segmentation
School of Computer Science, Henan Institute of Science and Technology
TrackRef3D: Multi-View Consistent Track-then-Label for Open-World Referring Segmentation in 3D Gaussian Splatting
May 26, 2026Yuyang Tan, Renhe Zhang, Hang Zhang +2Multi-View Consistency 3D Gaussian Splatting
East China Normal University, Shanghai, China · Shanghai AI Laboratory · University of Electronic Science and Technology of China, Chengdu, China

Jul 21, 2026J/K move · Enter open · S save

ZeroSplat: Generalized Referring Segmentation in 3D Gaussian Splatting

Jiayu Ding, Meilu Song, Xiaoyi Zhang, Hongbo Jin+2

Peking University · North China Electric Power University · China University of Geosciences

3D Gaussian Splatting Open-Vocabulary 3D Segmentation 3D Referring Segmentation 3DGS Segmentation

Open paper AskPDF arXiv

Recent advancements in 3D Gaussian Splatting (3DGS) have enabled language-guided scene understanding. However, existing Referring 3D Gaussian Splatting (R3DGS) methods are fundamentally restricted to single-target queries. To reflect the ambiguity of real-world instructions, we introduce the Generalized Referring 3D Gaussian Splatting Segmentation (GR3DGS) task, which requires dynamically segmenting an arbitrary number of targets (0, 1, or

N

). To facilitate comprehensive evaluation of this new task, we construct two new benchmarks: GR-LERF and GR-ScanNet. Crucially, existing R3DGS paradigms exhibit fundamental technical bottlenecks that severely limit their performance on the GR3DGS task: they lack intrinsic 3D point-level understanding by operating merely on 2D rendered pixels, and they incur prohibitive computational overhead by requiring per-scene optimization to embed heavy semantic features. To dismantle these bottlenecks, we propose ZeroSplat, a novel training-free and zero-feature framework. ZeroSplat lifts 2D Vision-Language Model (VLM) priors into 3D space through robust multi-view geometric constraints. This strategy enables intrinsic point-level understanding without incurring any additional feature storage. Extensive experiments demonstrate that ZeroSplat significantly outperforms state-of-the-art methods across generalized and single-target scenarios while maintaining exceptional efficiency. Project Page: https://inkmind-ai.github.io/ZeroSplat

Abstract

3D Referring Segmentation

Explore similar work

ZeroSplat: Generalized Referring Segmentation in 3D Gaussian Splatting

Beyond Similarity Matching: Structured Reasoning for Open-Vocabulary Referring Segmentation in 3DGS

TrackRef3D: Multi-View Consistent Track-then-Label for Open-World Referring Segmentation in 3D Gaussian Splatting