Examining existing benchmarks for grounding in 3D VLMs reveals a pervasive "semantic leakage" in the evaluation and prompting protocols: a heavy reliance on bounding boxes as spatial prompts or representation primitives. Because bounding boxes are highly informative of object categories, they introduce a shortcut for the models to neglect visual information during 3D VLM training. We demonstrate this with a "blind" LLM that, prompted only with the box coordinates, can perform surprisingly well. To further analyze this shortcut, we propose point-based grounding prompting - querying models with a single 3D point instead of a box. Under this protocol, the accuracy of the blind LLM drops from 36.4% to 18.0% on ScanNet's Nr3D, and similarly degrades for 3D VLMs (e.g. LL3DA). Building on this insight, we introduce P3-VLM, a novel tri-pathway 3D VLM that supports point-based prompts and achieves 52.1% state-of-the-art accuracy on location-grounded captioning under this setting. To achieve this, we redesign the architecture while keeping it detector-free, scene-centric, and suitable for 3D GS inputs. P3-VLM utilizes three complementary pathways (location-, task-, and context-aware), enabling strong reasoning about scenes without any bounding box dependencies. The key benefit of point-based learning is a consistent improvement across settings: P3-VLM achieves state-of-the-art performance under point-based grounding while also improving robustness, OOD generalization, and performance on ungrounded scene-centric 3D VQA benchmarks, reflecting its reduced reliance on bounding-box shortcuts and resulting stronger visual-spatial reasoning.
Figures & tables
Figure 2: Performance, robustness, and generalization. Our P 3 -VLM consistently outperforms VLM baselines LL3DA [ 8 ] and GaussianVLM [ 19 ] (GSVLM). Notably, P 3 -VLM on points (p) offers better robustness and generalization than box-based methods (b), despite less spatial information.
Model
Class Acc. (%)
Blind LLM (box coords only)
36.4
Box encoder + LLM
44.2
Full OD-VLM (LEO)
45.1
Table 1: Object-centric VLMs do not utilize their 3D encoder. A blind LLM operating on bounding box coordinates alone, and an LLM with only box coordinate embeddings, perform close to the full OD-VLM on Nr3D.
Dataset
Model
Box
Point
Acc
Sim
Acc
Sim
Nr3D
Blind LLM
36.4
-
18.0
-
LL3DA
47.7
48.1
33.1
39.5
GaussianVLM
46.2
48.2
29.3
35.5
Scan Refer
Blind LLM
38.1
-
18.4
-
LL3DA
48.02
55.9
29.2
34.9
Table 2: Object captioning performance on Nr3D and ScanRefer with different prompting strategies. The blind LLM’s sharp drop confirms that bounding box geometry was the dominant signal. Also the 3D VLM baselines (LL3DA, GaussianVLM) trained on boxes drop considerably even after fine-tuning on point prompts.
Figure 3: P 3 -VLM architecture overview. Three complementary pathways process the voxelized 3D scene: the location-aware pathway extracts a single token at the queried point; the task-aware pathway produces prompt-guided scene tokens by first refining the task prompt tokens and subsequently utilizing them for visual refinement of the sparsified features; the context-aware pathway distills 8 global scene summary tokens via unconditional cross-attention. Notably, the context and task pathways operate on the sparsified features for efficiency gains.
Nr3D
ScanRefer
Model
Acc
Sim
Acc
Sim
LL3DA (b)
47.7
48.1
48.0
55.9
GaussianVLM (b)
46.2
48.2
50.4
59.1
GaussianVLM Q (b)
47.1
47.8
51.7
59.3
P 3 -VLM (p)
52.1
46.0
56.1
54.2
P 3 -VLM (b)
52.4
46.3
58.4
52.5
Table 3: Object captioning performance on Nr3D and ScanRefer. We report classification accuracy (Acc) and caption similarity (Sim). P 3 -VLM consistently outperforms baselines in Acc, showing superior visual grounding across point (p) and box (b) prompts. GaussianVLM Q : [ 19 ] with Qwen3-VL-4B LLM, retrained.
Figure 4: Qualitative results on ScanQA and Nr3D. (a) P 3 -VLM can correctly answer scan questions, sometimes even with answers not captured by the benchmark ground truth. (b) Through the combination of task, context, and location pathway, on Nr3D, P 3 -VLM identifies the correct object class, surrounding context, and spatial relationships. The performance remains robust even under perturbations (offset factor x=0.9 ).
Table 9
Nr3D
ScanQA
SQA3D
Acc
Sim
EM-R
M
R
EM-R
B4
C
M
R
w/o location pathway
49.3
43.9
30.4
24.2
34.1
45.6
0
114.6
23.4
44.3
w/o task pathway
50.2
46.0
30.1
24.2
19.2
47.9
22.5
119.0
24.1
46.2
w/o context pathway
50.2
45.2
22.7
14.7
23.3
49.4
34.8
123.2
25.2
47.8
Matched baseline
39.8
41.9
31.3
21.7
33.9
49.6
20.2
124.2
25.3
48.1
Full model
52.1
46.0
31.9
22.9
34.8
51.8
30.58
130.4
26.59
50.32
Table 9: Design ablation studies on P 3 -VLM (p). Top: Each of the pathways is removed independently. Bottom: A matched baseline that isolates visual processing innovations by employing a multi-pathway variant with minimal structural complexity – utilizing only nearest-voxel features at the query point and global context tokens.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Validation accuracy of the blind LLM during fine-tuning under bounding-box and point-based prompting. Performance saturates between epochs 6–8, with overfitting emerging afterwards. This curve motivates the 6-epoch fine-tuning budget used for all VLM experiments.
Model
Sim
M
R
Class Acc
Blind LLM
34.8
17.9
21.3
36.4
LL3DA
48.1
5.8
9.9
47.7
GaussianVLM
48.2
20.8
48.2
46.2
P 3 -VLM
46.0
18.6
16.4
52.1
Appendix
Table 10: Different evaluation metrics on captioning on the Nr3D dataset. It is evident that high n-gram based metrics do not signal correct object identification.
Generated caption
Extracted class
GT class
ScanRefer
sos this is a white kitchen cabinet . it is to the left of the oven . eos
kitchen cabinet
kitchen cabinet
ScanRefer
sos there is a rectangular soap dispenser . it is on the wall above the sink . eos
soap dispenser
soap dispenser
Nr3D
sos the stool is in the corner of the room . it is to the right of the whiteboard . eos
stool
stool
Nr3D
sos the plant is on the floor . it is to the right of the door . eos
plant
plant
Appendix
Table 11: Examples of LLM-based class extraction on ScanRefer and Nr3D captions generated by LL3DA 3D VLM.
Figure 6: Mean IoU per object class from ScanRefer and Nr3D annotations, validation split.
Figure 7: P 3 -VLM on Nr3D benchmark
Nr3D
ScanRefer
Model
Train
Eval
Acc
Sim
Acc
Sim
LL3DA
box
box
47.7
48.1
48.0
55.9
GaussianVLM
box
box
46.2
48.2
50.4
59.1
P 3 -VLM (b)
box
box
52.4
46.3
58.4
52.5
P 3 -VLM (p)
c
c
52.1
46.0
56.1
54.2
P 3 -VLM (p)
c
sp
50.7
45.4
56.9
54.1
Appendix
Table 12: Center- vs. surface-point grounding on Nr3D and ScanRefer. c: object center, sp: random surface point. Training on surface points substantially improves class accuracy.
Figure 8: Qualitative results on situated question answering (SQA3D benchmark) and out-of-distribution generalization (GaussianVLM-OOD benchmark). Our model achieves better performance on both benchmarks, showing visual understanding capabilities both on in-domain ScanNet scenes, and on out-of-domain ScanNet++ scenes.
Hyperparameter
Value
Optimization
Optimizer
AdamW
Learning rate
3×10−5
(β1,β2)
(0.9,0.999)
Weight decay
0.05
LR schedule
Linear warmup + cosine decay
Appendix
Table 13: Training hyperparameters for P 3 -VLM.
Nr3D
ScanQA
SQA3D
Acc
Sim
EM-R
M
R
EM-R
B4
C
M
R
Location Sparsifier Design
Processed + MLP
42.3
47.1
-
-
-
-
-
-
-
-
Raw + FiLM
48.5
46.4
-
-
-
-
-
-
-
-
Sparsifier Design: Positional Encoding
None
47.9
43.7
29.3
18.8
31.1
46.6
28.7
114.5
23.4
44.4
Appendix
Table 14: Location pathway and Sparsifier design ablation studies. We ablate the point-based variant, P 3 -VLM (p). Loc. Design : raw vs. processed features and FiLM vs. MLP extraction. Pos. Enc. : positional encoding strategy in visual sparsification step.
Nr3D
ScanRefer
ScanQA
SQA3D
Encoder
Acc
Sim
Acc
Sim
EM-R
M
R
EM-R
B4
C
M
R
SceneSplat (3D GS)
52.1
46.0
56.1
54.2
31.9
22.6
35.3
51.8
30.6
130.4
26.6
50.3
Sonata (PC)
58.1
47.3
63.3
54.8
30.5
22.8
33.9
48.3
16.8
126.9
23.8
48.8
Appendix
Table 15: P 3 -VLM (p) with a 3D GS (SceneSplat) vs. point cloud (Sonata) encoder on grounded captioning (Nr3D, ScanRefer) and VQA (ScanQA, SQA3D).
Dataset
License / Terms of Use
ScanNet [ 15 ]
ScanNet Terms of Use 1 1 1 Available at https://kaldir.vc.in.tum.de/scannet/ScanNet_TOS.pdf (last accessed: 03/05/2026).
ScanRefer [ 6 ]
Creative Commons BY-NC-SA 3.0
ScanQA [ 3 ]
Apache 2.0
SQA3D [ 33 ]
Apache 2.0
Nr3D [ 1 ]
MIT
ScanNet++ [ 50 ]
ScanNet++ Terms of Use 2 2 2 Available at https://kaldir.vc.in.tum.de/scannetpp/static/scannetpp-terms-of-use.pdf (last accessed: 03/05/2026).
Appendix
Table 16: Licenses and terms of use for the used datasets.
Vision Language Models (VLMs) enable a unified model to solve various vision tasks through prompting. They have shown promising performance in semantic understanding. However, 3D understanding still largely relies on expert vision models with complex task-specific designs. The key argument this work wants to make is that VLMs are native 3D learners. Our in-depth large scale study shows that 1) focal length unification, 2) text-based pixel reference and 3) data mixture and scaling, are all you need for effective 3D learning. Model architecture changes, large models, heavy data augmentations, and complex losses including the regression formulation, many of which form the foundation of expert vision models, are actually not necessary conditions. As a result, we propose VLM3, a scalable method with the simplest design that enables standard VLMs to master diverse 3D tasks. VLM3 not only advances the VLM depth estimation accuracy by a large margin (0.84 -> 0.9), but also enables diverse 3D tasks such as pixel correspondence, camera pose estimation and object-level 3D understanding, matching expert vision model accuracy while maintaining standard architectures and text-based training. We believe VLM3 opens up a new paradigm for simple and scalable 3D learning.
Recent zero-shot 3D visual grounding methods leverage vision-language models (VLMs) to localize objects in 3D scenes from natural language queries. However, these methods typically rely on heuristic rules to select which camera views are provided to the VLM, often prioritizing object visibility rather than grounding relevance. We present IVSGround, a framework that learns Influential View Selection for VLM-based 3D visual grounding. Instead of using fixed heuristics, a lightweight view selector is trained to identify views that provide discriminative evidence for grounding. To obtain supervision signals, we generate training signals using feedback from a reasoning VLM through a two-stage rejection sampling process. During inference, the learned selector predicts query-conditioned influential views for each candidate object, which are then evaluated by a frozen reasoning VLM through comparative grounding. Experiments on ScanRefer and NR3D show that IVSGround consistently improves grounding accuracy over existing zero-shot pipelines, demonstrating that selecting where to look is crucial for effective 3D visual grounding. Project page: https://ivsground.github.io/
Tsung-Chih Chiang, Hsuan-Kung Yang, Jou-Min Liu +4
National Tsing Hua University, Hsinchu, Taiwan · Woven by Toyota, Inc., Tokyo, Japan · National Taiwan University, Taipei, Taiwan
We present GR3D, a spatial vision language model equipped with three complementary grounding capabilities--explicit 2D grounding, implicit 2D grounding, and monocular 3D grounding--within a single framework. GR3D introduces an implicit grounding mechanism that identifies entity mentions during generation and inserts the corresponding region tokens into the text stream, allowing the model to reference visual evidence on the fly when producing spatial chain-of-thought responses. In parallel, a region-prompted monocular 3D grounding design predicts 3D bounding boxes in the camera view from grounded region queries, supported by intrinsic-aware normalization and dense geometric supervision. Together, these grounding capabilities enable GR3D to decompose complex spatial understanding problems into grounded 2D perception followed by 3D inference. GR3D achieves consistent improvements across grounded and non-grounded spatial benchmarks, demonstrating grounding as an effective inductive bias for strengthening spatial understanding in VLMs. These grounding capabilities collectively enhance general spatial understanding beyond the grounding task itself.