Open-vocabulary referring segmentation in 3D Gaussian Splatting (3DGS) requires a neural model to select Gaussian primitives according to free-form language expressions. Existing 3DGS-based methods usually rely on global text-region similarity, which is weak for queries involving attributes, reference objects, spatial relations, and fine-grained parts. This often causes target-reference confusion, granularity mismatch, part-whole leakage, and relation violations. We propose QAGaussian, a query-adaptive neural reasoning framework for language-guided Gaussian primitive selection. QAGaussian first learns query-conditioned multi-scale Gaussian slots as differentiable candidates whose receptive fields are shaped by the input expression. It then builds a relation-aware slot graph with language-conditioned edge weighting to propagate target-reference, attribute, part-whole, and contextual evidence. A granularity-adaptive router softly combines region-level, object-level, part-level, attribute-aware, and relation-aware mask branches, followed by relation-constrained refinement for spatial, part-whole, attribute, and geometric consistency. QAGaussian is pretrained only on Mosaic3D-5.6M for Gaussian-text alignment and evaluated on independent benchmarks without target-dataset fine-tuning. It achieves 47.2 Avg. mIoU and 63.2 Avg. F1, outperforming the strongest 3DGS referring baseline by 2.7 mIoU points and 2.9 F1 points. It also improves Part-mIoU from 38.6 to 43.4, Rel-mIoU from 44.4 to 50.8, and reduces target-reference confusion from 10.8 to 7.4. These results demonstrate that query-conditioned slot learning, relation-aware graph reasoning, and adaptive routing provide an effective neural modeling strategy for open-vocabulary referring segmentation in 3DGS. The code is available at https://github.com/zqeslwyz/QAGaussian.
Figures & tables
Figure 1: Overall framework of QAGaussian. Given a 3D Gaussian Splatting scene and a free-form referring expression, the model generates query-conditioned multi-scale Gaussian slots, performs relation-aware slot reasoning, adaptively fuses multiple mask prediction branches, and refines the result to obtain the final Gaussian-level referring segmentation mask.
Item
Setting
Pretraining dataset
Mosaic3D-5.6M
Hidden dimension D
256
Gaussian tokenization scales
3
Slots per scale (K1,K2,K3)
96 / 64 / 32
Total slots K
192
Graph neighbors per slot
16
Table 1: Experimental settings and main hyperparameters.
Table 3: Average metrics on representative independent benchmarks
Figure 2: Qualitative comparison with CAGS and ReferSplat on representative 3DGS scenes. Each row shows the input scene, ground-truth mask, and segmentation results of different methods under the same referring expression. The colored query components are automatically identified by the language parser: red denotes the target object, blue denotes the spatial relation, brown denotes the reference object, and green denotes the attribute. QAGaussian produces cleaner Gaussian-level masks and better suppresses reference objects and distractors.
Figure 3: Visualization under different query types within the same complex kitchen 3DGS scene. The input scene is fixed, while the referring expression changes across object-level, attribute-aware, part-level, relation-dependent, and compositional queries. Each row shows Gaussian-level segmentation results from three views. The colored query components are automatically identified by the language parser: red denotes the target object, blue denotes the spatial relation, brown denotes the reference object, and green denotes the attribute.
Method
Part-mIoU
Rel-mIoU
TR-conf. ↓
Precision
Recall
F1
G-SAM + Fusion ( Ren et al., 2024b )
29.2
35.0
17.2
44.6
41.5
43.0
LISA + Fusion ( Lai et al., 2024a )
31.8
33.4
18.9
48.6
45.9
47.2
BUTD-DETR ( Jain et al., 2022a )
27.3
40.6
15.9
43.8
46.2
45.0
Multi3DRefer ( Zhang et al., 2023a )
29.2
41.8
15.6
45.4
48.0
46.7
Open3DIS ( Nguyen et al., 2024a )
33.5
39.6
14.8
49.2
50.4
49.8
LangSplat ( Qin et al., 2024a )
31.6
37.8
15.5
48.0
49.6
48.8
Table 4: Fine-grained diagnostic metrics on part-level queries from PartNet, PartNet-Mobility, and ShapeNetPart, and relation-dependent queries from ScanRefer, ReferIt3D, Multi3DRefer, and ReferSplat-derived splits. TR-conf. denotes the target-reference confusion rate.
Table 5: Effect of pretraining data on cross-dataset generalization. All settings are evaluated on independent benchmarks without target benchmark fine-tuning.
Figure 4: Performance retention under different query perturbations.
Table 6: Ablation study of key QAGaussian components on the independent evaluation benchmarks. TR-conf. denotes the target-reference confusion rate, where lower values are better.
Figure 5: Qualitative ablation example on a relation-dependent region query. The query components are automatically extracted by the language parser and visualized with different colors. The full QAGaussian model accurately segments the referred glass door, while removing query-conditioned slots, relation graph reasoning, or adaptive routing leads to coarse masks, activation of similar glass regions, or leakage to surrounding facade areas.
Method
Preprocess time
Query time
Memory
Avg. mIoU
Small
Medium
Large
Small
Medium
Large
Small
Medium
Large
OpenGaussian ( Wu et al., 2024a )
6.2m
8.6m
13.8m
0.28s
0.43s
0.68s
5.8GB
7.8GB
10.6GB
40.3
CAGS ( Sun et al., 2025a )
6.8m
9.4m
15.1m
0.36s
0.61s
0.92s
6.3GB
8.6GB
11.8GB
44.0
ReferSplat ( He et al., 2025a )
7.1m
9.8m
15.8m
0.40s
0.66s
1.05s
6.8GB
9.1GB
12.4GB
44.5
QAGaussian
7.4m
10.3m
16.6m
0.48s
0.82s
1.28s
7.5GB
9.8GB
13.6GB
47.2
Table 7: Efficiency comparison on Gaussian scenes of different scales. Preprocess and query time are reported for small, medium, and large scenes.
Recent advances in 3D Gaussian Splatting have enabled open-vocabulary and referring segmentation by distilling semantic knowledge from 2D foundation models into 3D representations. However, existing referring fields embed language features in a globally view-invariant space, making them fundamentally unable to resolve observer-centric spatial relations (e.g., "to the left of") that depend on camera pose. We propose DSSR-3D, an inference-time framework for view-dependent referring segmentation on continuous 3D Gaussian fields, formalized as two interfaces - pose-invariant semantic localization and pose-conditioned spatial reasoning - such that any pair of functions satisfying these constraints yields a valid instantiation, requiring no retraining of the underlying semantic field and no reliance on discrete geometric proxies such as bounding boxes. We instantiate the two interfaces with a temperature-sharpened softmax localization mechanism and a projection-based directional scoring function, fused via a lightweight, training-free step, and show they transfer zero-shot to structurally distinct semantic fields without adaptation. We further propose ViewRef-GS, a benchmark isolating view-dependent segmentation on 3D Gaussian fields, evaluated jointly with an augmented Ref-LERF to provide a comprehensive testbed for viewpoint-dependent spatial grounding. Experiments show consistent gains over existing 3DGS-based referring methods, with no additional training beyond the base semantic field
3D Gaussian Splatting (3DGS) has emerged at the forefront of 3D scene reconstruction. Extending 3DGS with language-driven, open-vocabulary understanding has gained significant attention for real-world applications such as embodied AI. Recent methods achieve this by learning an instance feature attribute and assigning semantics by distilling high-dimensional Contrastive Language-Image Pretraining (CLIP) features directly into the scene representation. However, the instance grouping mechanisms of these methods either require a predefined number of instances or suffer from noise in their bottom-up grouping strategies. Furthermore, the reliance on CLIP restricts semantic understanding to simple noun phrases, preventing complex spatial reasoning and referential expression grounding. We present GaussDet, a method that circumvents the need for dense CLIP features by leveraging discrete, open-vocabulary 2D object detectors with referring expression capabilities. We learn instance features for individual Gaussians to decompose the scene into 3D instance groups. By rendering these groups and aggregating semantic votes from multi-view 2D detections, we generate a robust View-Aggregated Semantic Label Distribution (VASD) for each 3D instance. This view-aggregation strategy acts as a strong regularizer, attenuating spurious labels caused by low-quality instance grouping. Our approach enables a straightforward, zero-shot extension from simple language queries to complex referential grounding. Extensive evaluations across two key tasks -- open-vocabulary segmentation (LeRF-OVS, ScanNet) and referring expression grounding (Ref-LeRF) -- demonstrate that GaussDet achieves consistent improvements over existing methods. Most notably, we achieve a substantial 16.7% mIoU improvement in referential grounding within a strict zero-shot setting.
Open-vocabulary 3D Gaussian segmentation is challenging because it requires language understanding for diverse queries and accurate separation of Gaussians along object boundaries. Prior approaches either embed language knowledge into individual Gaussians to improve query responsiveness or optimize per-Gaussian instance features to encode object identity. However, these strategies may produce noisy Gaussian segmentations or rely on cost-inefficient per-scene optimization. We propose PairGS, a framework that reframes Gaussian segmentation as modeling pairwise relations between Gaussians. 3D Gaussian representations provide rich signals for relation estimation, such as view contribution weights and multi-view mask evidence. By leveraging these cues, PairGS explicitly constructs a relation graph for segmentation without a heavy optimization process. PairGS first proposes sparse edge candidates using low-dimensional descriptors, computes precise pairwise affinities only on those candidates, and builds a hierarchical cluster tree for multi-granular querying. It achieves state-of-the-art results on open-vocabulary 3D Gaussian segmentation benchmarks, while the fast variant is 50x faster than optimization-based instance-feature approaches.