P$^3$-VLM: A Point-based Alternative for Grounded 3D Vision-Language Models

Anna-Maria Halacheva; Jan-Nico Zaech; Sombit Dey; Luc Van Gool; Danda Pani Paudel

cs.CVOct 20, 2026

P $^3$ -VLM: A Point-based Alternative for Grounded 3D Vision-Language Models

Authors: Anna-Maria Halacheva, Jan-Nico Zaech, Sombit Dey, Luc Van Gool, Danda Pani Paudel

Organizations: INSAIT, Sofia University “St. Kliment Ohridski”

Abstract

Examining existing benchmarks for grounding in 3D VLMs reveals a pervasive "semantic leakage" in the evaluation and prompting protocols: a heavy reliance on bounding boxes as spatial prompts or representation primitives. Because bounding boxes are highly informative of object categories, they introduce a shortcut for the models to neglect visual information during 3D VLM training. We demonstrate this with a "blind" LLM that, prompted only with the box coordinates, can perform surprisingly well. To further analyze this shortcut, we propose point-based grounding prompting - querying models with a single 3D point instead of a box. Under this protocol, the accuracy of the blind LLM drops from 36.4% to 18.0% on ScanNet's Nr3D, and similarly degrades for 3D VLMs (e.g. LL3DA). Building on this insight, we introduce P $^3$ -VLM, a novel tri-pathway 3D VLM that supports point-based prompts and achieves 52.1% state-of-the-art accuracy on location-grounded captioning under this setting. To achieve this, we redesign the architecture while keeping it detector-free, scene-centric, and suitable for 3D GS inputs. P $^3$ -VLM utilizes three complementary pathways (location-, task-, and context-aware), enabling strong reasoning about scenes without any bounding box dependencies. The key benefit of point-based learning is a consistent improvement across settings: P $^3$ -VLM achieves state-of-the-art performance under point-based grounding while also improving robustness, OOD generalization, and performance on ungrounded scene-centric 3D VQA benchmarks, reflecting its reduced reliance on bounding-box shortcuts and resulting stronger visual-spatial reasoning.

Figures & tables

Figure 2: Performance, robustness, and generalization. Our P 3 -VLM consistently outperforms VLM baselines LL3DA [ 8 ] and GaussianVLM [ 19 ] (GSVLM). Notably, P 3 -VLM on points (p) offers better robustness and generalization than box-based methods (b), despite less spatial information.

Model	Class Acc. (%)
Blind LLM (box coords only)	36.4
Box encoder + LLM	44.2
Full OD-VLM (LEO)	45.1

Table 1: Object-centric VLMs do not utilize their 3D encoder. A blind LLM operating on bounding box coordinates alone, and an LLM with only box coordinate embeddings, perform close to the full OD-VLM on Nr3D.

Dataset	Model	Box		Point
		Acc	Sim	Acc	Sim
Nr3D	Blind LLM	36.4	-	18.0	-
	LL3DA	47.7	48.1	33.1	39.5
	GaussianVLM	46.2	48.2	29.3	35.5
Scan Refer	Blind LLM	38.1	-	18.4	-
	LL3DA	48.02	55.9	29.2	34.9

Table 2: Object captioning performance on Nr3D and ScanRefer with different prompting strategies. The blind LLM’s sharp drop confirms that bounding box geometry was the dominant signal. Also the 3D VLM baselines (LL3DA, GaussianVLM) trained on boxes drop considerably even after fine-tuning on point prompts.

Figure 3: P 3 -VLM architecture overview. Three complementary pathways process the voxelized 3D scene: the location-aware pathway extracts a single token at the queried point; the task-aware pathway produces prompt-guided scene tokens by first refining the task prompt tokens and subsequently utilizing them for visual refinement of the sparsified features; the context-aware pathway distills 8 global scene summary tokens via unconditional cross-attention. Notably, the context and task pathways operate on the sparsified features for efficiency gains.

	Nr3D		ScanRefer
Model	Acc	Sim	Acc	Sim
LL3DA (b)	47.7	48.1	48.0	55.9
GaussianVLM (b)	46.2	48.2	50.4	59.1
GaussianVLM Q (b)	47.1	47.8	51.7	59.3
P 3 -VLM (p)	52.1	46.0	56.1	54.2
P 3 -VLM (b)	52.4	46.3	58.4	52.5

Table 3: Object captioning performance on Nr3D and ScanRefer. We report classification accuracy (Acc) and caption similarity (Sim). P 3 -VLM consistently outperforms baselines in Acc, showing superior visual grounding across point (p) and box (b) prompts. GaussianVLM Q : [ 19 ] with Qwen3-VL-4B LLM, retrained.

Table 6

ScanQA
Model	EM-R	M	R
ScanQA [ 3 ]	/	13.14	33.33
3D-VLP [ 26 ]	/	13.53	34.51
VoteNet+MCAN [ 51 ]	/	11.4	29.8
LSceneLLM [ 56 ]	/	17.95	/
LL3DA [ 8 ]	30.3	22.8	34.7

Table 6: ScanQA Results. GaussianVLM Q : [ 19 ] with Qwen3-VL-4B LLM, retrained.

Figure 4: Qualitative results on ScanQA and Nr3D. (a) P 3 -VLM can correctly answer scan questions, sometimes even with answers not captured by the benchmark ground truth. (b) Through the combination of task, context, and location pathway, on Nr3D, P 3 -VLM identifies the correct object class, surrounding context, and spatial relationships. The performance remains robust even under perturbations (offset factor $x=0.9$ ).

Table 9

	Nr3D		ScanQA			SQA3D
	Acc	Sim	EM-R	M	R	EM-R	B4	C	M	R
w/o location pathway	49.3	43.9	30.4	24.2	34.1	45.6	0	114.6	23.4	44.3
w/o task pathway	50.2	46.0	30.1	24.2	19.2	47.9	22.5	119.0	24.1	46.2
w/o context pathway	50.2	45.2	22.7	14.7	23.3	49.4	34.8	123.2	25.2	47.8
Matched baseline	39.8	41.9	31.3	21.7	33.9	49.6	20.2	124.2	25.3	48.1
Full model	52.1	46.0	31.9	22.9	34.8	51.8	30.58	130.4	26.59	50.32

Table 9: Design ablation studies on P 3 -VLM (p). Top: Each of the pathways is removed independently. Bottom: A matched baseline that isolates visual processing innovations by employing a multi-pathway variant with minimal structural complexity – utilizing only nearest-voxel features at the query point and global context tokens.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Validation accuracy of the blind LLM during fine-tuning under bounding-box and point-based prompting. Performance saturates between epochs 6–8, with overfitting emerging afterwards. This curve motivates the 6-epoch fine-tuning budget used for all VLM experiments.

Model	Sim	M	R	Class Acc
Blind LLM	34.8	17.9	21.3	36.4
LL3DA	48.1	5.8	9.9	47.7
GaussianVLM	48.2	20.8	48.2	46.2
P 3 -VLM	46.0	18.6	16.4	52.1

Appendix

Table 10: Different evaluation metrics on captioning on the Nr3D dataset. It is evident that high n-gram based metrics do not signal correct object identification.

	Generated caption	Extracted class	GT class
ScanRefer	sos this is a white kitchen cabinet . it is to the left of the oven . eos	kitchen cabinet	kitchen cabinet
ScanRefer	sos there is a rectangular soap dispenser . it is on the wall above the sink . eos	soap dispenser	soap dispenser
Nr3D	sos the stool is in the corner of the room . it is to the right of the whiteboard . eos	stool	stool
Nr3D	sos the plant is on the floor . it is to the right of the door . eos	plant	plant

Appendix

Table 11: Examples of LLM-based class extraction on ScanRefer and Nr3D captions generated by LL3DA 3D VLM.

Figure 6: Mean IoU per object class from ScanRefer and Nr3D annotations, validation split.

Figure 7: P 3 -VLM on Nr3D benchmark

			Nr3D		ScanRefer
Model	Train	Eval	Acc	Sim	Acc	Sim
LL3DA	box	box	47.7	48.1	48.0	55.9
GaussianVLM	box	box	46.2	48.2	50.4	59.1
P 3 -VLM (b)	box	box	52.4	46.3	58.4	52.5
P 3 -VLM (p)	c	c	52.1	46.0	56.1	54.2
P 3 -VLM (p)	c	sp	50.7	45.4	56.9	54.1

Appendix

Table 12: Center- vs. surface-point grounding on Nr3D and ScanRefer. c: object center, sp: random surface point. Training on surface points substantially improves class accuracy.

Figure 8: Qualitative results on situated question answering (SQA3D benchmark) and out-of-distribution generalization (GaussianVLM-OOD benchmark). Our model achieves better performance on both benchmarks, showing visual understanding capabilities both on in-domain ScanNet scenes, and on out-of-domain ScanNet++ scenes.

Hyperparameter	Value
Optimization
Optimizer	AdamW
Learning rate	$3\times 10^{-5}$
$(\beta_{1},\beta_{2})$	$(0.9,\ 0.999)$
Weight decay	$0.05$
LR schedule	Linear warmup + cosine decay

Appendix

Table 13: Training hyperparameters for P 3 -VLM.

	Nr3D		ScanQA			SQA3D
	Acc	Sim	EM-R	M	R	EM-R	B4	C	M	R
Location Sparsifier Design
Processed + MLP	42.3	47.1	-	-	-	-	-	-	-	-
Raw + FiLM	48.5	46.4	-	-	-	-	-	-	-	-
Sparsifier Design: Positional Encoding
None	47.9	43.7	29.3	18.8	31.1	46.6	28.7	114.5	23.4	44.4

Appendix

Table 14: Location pathway and Sparsifier design ablation studies. We ablate the point-based variant, P 3 -VLM (p). Loc. Design : raw vs. processed features and FiLM vs. MLP extraction. Pos. Enc. : positional encoding strategy in visual sparsification step.

	Nr3D		ScanRefer		ScanQA			SQA3D
Encoder	Acc	Sim	Acc	Sim	EM-R	M	R	EM-R	B4	C	M	R
SceneSplat (3D GS)	52.1	46.0	56.1	54.2	31.9	22.6	35.3	51.8	30.6	130.4	26.6	50.3
Sonata (PC)	58.1	47.3	63.3	54.8	30.5	22.8	33.9	48.3	16.8	126.9	23.8	48.8

Appendix

Table 15: P 3 -VLM (p) with a 3D GS (SceneSplat) vs. point cloud (Sonata) encoder on grounded captioning (Nr3D, ScanRefer) and VQA (ScanQA, SQA3D).

Dataset	License / Terms of Use
ScanNet [ 15 ]	ScanNet Terms of Use 1 1 1 Available at https://kaldir.vc.in.tum.de/scannet/ScanNet_TOS.pdf (last accessed: 03/05/2026).
ScanRefer [ 6 ]	Creative Commons BY-NC-SA 3.0
ScanQA [ 3 ]	Apache 2.0
SQA3D [ 33 ]	Apache 2.0
Nr3D [ 1 ]	MIT
ScanNet++ [ 50 ]	ScanNet++ Terms of Use 2 2 2 Available at https://kaldir.vc.in.tum.de/scannetpp/static/scannetpp-terms-of-use.pdf (last accessed: 03/05/2026).

Appendix

Table 16: Licenses and terms of use for the used datasets.

3D VLMs

Three-dimensional vision-language models connect language with spatially grounded visual data for 3D understanding, reasoning, and action.

See all 213 papers

VLM3: Vision Language Models Are Native 3D Learners
May 28, 2026Zhipeng Cai, Zhuang Liu, Yunyang Xiong +3Camera Pose Estimation Monocular Depth Estimation
Meta · Princeton University
Where to Look Matters: Learning Influential Views for VLM-based 3D Visual Grounding
Sep 4, 2026Tsung-Chih Chiang, Hsuan-Kung Yang, Jou-Min Liu +43D Visual Grounding 3D VLMs
National Tsing Hua University, Hsinchu, Taiwan · Woven by Toyota, Inc., Tokyo, Japan · National Taiwan University, Taipei, Taiwan
Grounded 3D-Aware Spatial Vision-Language Modeling
May 28, 2026An-Chieh Cheng, Yang Fu, Yatai Ji +12Visually Grounded Reasoning 3D Spatial Reasoning
UCSD · NVIDIA · MIT

May 28, 2026J/K move · Enter open · S save

VLM3: Vision Language Models Are Native 3D Learners

Zhipeng Cai, Zhuang Liu, Yunyang Xiong, Zechun Liu+2

Meta · Princeton University

Camera Pose Estimation Monocular Depth Estimation 3D Scene Understanding 3D VLMs

Open paper AskPDF arXiv

Vision Language Models (VLMs) enable a unified model to solve various vision tasks through prompting. They have shown promising performance in semantic understanding. However, 3D understanding still largely relies on expert vision models with complex task-specific designs. The key argument this work wants to make is that VLMs are native 3D learners. Our in-depth large scale study shows that 1) focal length unification, 2) text-based pixel reference and 3) data mixture and scaling, are all you need for effective 3D learning. Model architecture changes, large models, heavy data augmentations, and complex losses including the regression formulation, many of which form the foundation of expert vision models, are actually not necessary conditions. As a result, we propose VLM3, a scalable method with the simplest design that enables standard VLMs to master diverse 3D tasks. VLM3 not only advances the VLM depth estimation accuracy by a large margin (0.84 -> 0.9), but also enables diverse 3D tasks such as pixel correspondence, camera pose estimation and object-level 3D understanding, matching expert vision model accuracy while maintaining standard architectures and text-based training. We believe VLM3 opens up a new paradigm for simple and scalable 3D learning.

Abstract

3D VLMs

Explore similar work

VLM3: Vision Language Models Are Native 3D Learners

Where to Look Matters: Learning Influential Views for VLM-based 3D Visual Grounding

Grounded 3D-Aware Spatial Vision-Language Modeling