Toward Comprehensive 3D Grounding: Orientation Grounding through Vision-Language Models

Tuo Liang; Disheng Liu; Nengbo Wang; Vipin Chaudhary; Yu Yin

cs.CVSep 27, 2026

Toward Comprehensive 3D Grounding: Orientation Grounding through Vision-Language Models

Authors: Tuo Liang, Disheng Liu, Nengbo Wang, Vipin Chaudhary, Yu Yin

Organizations: Case Western Reserve University

Abstract

Grounding is a core capability of spatial vision-language models, yet most existing work focuses only on where a referred object is. Many 3D tasks also require knowing how it is oriented. Although existing 3D VLMs may predict oriented boxes, box pose does not explicitly capture object-centric orientation or symmetry-induced ambiguities. We introduce orientation grounding, a referring grounding task that predicts an object's 6D orientation and axial symmetry from a language or box query in single-view or multi-view scenes. To support this task, we construct ReferOri, with 331K multi-view and 387K single-view orientation-grounding queries obtained through scalable reconstruction, consistency checking, and human verification. We further present OG-VLM, which adapts a 3D VLM with structured box/orientation outputs, sign and symmetry tokens, and geometry-aware auxiliary losses. Across single-view and multi-view benchmarks, OG-VLM substantially outperforms orientation-aware VLM baselines and surpasses object-level orientation foundation models on scene-level referring benchmarks, showing that explicit orientation grounding is a distinct and learnable capability beyond localization. Downstream results validate its benefit for orientation-related spatial reasoning.

Figures & tables

Figure 1: Orientation grounding extends localization grounding from predicting where a referred object is to estimating how it is oriented. ReferOri provides single-view and multi-view annotations with object boxes, 6D orientation, and axial symmetry labels, enabling OG-VLM to ground both object position and object-centric direction.

Figure 2: Overview of ReferOri dataset structure.

Figure 3: Overview of OG-VLM. Given multi-view frames, a single frame, or a 3D point cloud together with a text or box query, the VLM backbone generates structured grounding outputs with separate <bbox> and <ori> spans. Lightweight heads attached to the corresponding tokens provide localization, orientation, and symmetry supervision through geometry-aware losses.

Models	Multi-View Scene						Single-View Scene
Models	ARKitScenes		ScanRefer ∗		Multi3DRefer ∗		Ori-COCO	RefCOCO ∗		RefCOCO+ ∗		RefCOCOg ∗
	Acc15 $\uparrow$	Acc30 $\uparrow$	Acc15 $\uparrow$	Acc30 $\uparrow$	Acc15 $\uparrow$	Acc30 $\uparrow$	Acc $\uparrow$	Acc15 $\uparrow$	Acc30 $\uparrow$	Acc15 $\uparrow$	Acc30 $\uparrow$	Acc15 $\uparrow$	Acc30 $\uparrow$
Spatial Reasoner	–	–	–	–	–	–	14.9	4.5	14.5	4.5	14.0	3.9	13.1
VG-LLM	10.1	17.4	17.8	26.2	21.5	33.5	–	–	–	–	–	–	–
Ours (only data)	43.1	45.4	63.1	66.6	68.1	70.6	60.1	31.1	60.9	31.1	60.8	28.8	57.8
Ours (+special token)	48.2	50.5	65.2	68.0	70.5	73.1	62.2	31.6	63.3	31.7	61.5	29.7	59.1

Table 1: Results on multi- and single-view scenes. Qualitative results appear in Fig. 5 in the appendix.

Orientation foundation models
Models	ARKitScenes		ScanRefer ∗		Multi3DRefer ∗
	Acc@15	Acc@30	Acc@15	Acc@30	Acc@15	Acc@30
Orient Anything	18.3	35.8	17.4	34.5	16.9	33.1
Orient Anything V2	19.8	43.3	15.8	37.4	16.9	35.6
Ours (+geometric loss)	50.5	51.6	66.1	70.1	71.5	74.2

Table 2: Comparison with orientation foundation models.

Dataset	Ablation	X ( $\downarrow$ )	Y ( $\downarrow$ )	Z ( $\downarrow$ )
RefCOCO+ ∗	only data	37.9	40.8	17.8
	special token	35.3	38.3	17.6
	geometric loss	35.1	38.0	17.3
ScanRefer ∗	only data	38.4	38.6	0.1
	special token	38.1	38.2	0.1
	geometric loss	37.1	37.2	0.1

Table 3: Axis-wise mean orientation error (in degrees). X, Y, and Z denote the right, front, and up axes, respectively.

Model	Setting	Localization		Orientation
Model	Setting	Acc 25%	Acc 50%	Mean Err. ( ∘ )	Acc@15	Acc@30
only data	w depth	68.7	36.3	29.6	31.1	60.8
only data	w/o depth	1.2	0.0	34.3	24.7	52.3
special token	w depth	68.2	35.6	29.5	30.9	61.5
special token	w/o depth	0.8	0.0	35.0	23.6	52.1
geometric loss	w depth	67.8	35.9	30.5	29.0	58.7

Table 4: Effect of depth on RefCOCO+ ∗ . For localization, we report the accuracy at IoU thresholds of 25% and 50%. For orientation grounding, we report the accuracy of predictions whose orientation error falls within

15^{\circ}

and

30^{\circ}

, as well as the mean orientation error.

Dataset	Model	Symmetric		Non-symmetric
		Acc@15	Acc@30	Acc@15	Acc@30
RefCOCO+ ∗	only data	24.41	53.76	34.41	66.56
	special token	26.76	53.05	34.44	67.36
	geometric loss	28.87	58.69	35.31	68.79
ScanRefer ∗	only data	64.23	66.37	59.54	64.31
	special token	65.97	68.40	58.37	63.69

Table 5: Effect of object symmetry on orientation grounding.

	ScanRefer ∗		Multi3DRefer ∗		RefCOCO ∗		RefCOCO+ ∗		RefCOCOg ∗
	Acc@15	Acc@30	Acc@15	Acc@30	Acc@15	Acc@30	Acc@15	Acc@30	Acc@15	Acc@30
w/o symmetry-adjusted	63.4	66.9	64.3	71.8	31.9	63.8	32.1	61.1	30.1	55.4
w/ symmetry-adjusted	66.1	70.1	71.5	74.2	32.0	63.9	32.2	62.1	30.2	60.5
Joint w/ IoU $\geq$ 0.25	45.2	47.8	43.0	45.7	25.8	50.2	24.5	47.7	21.5	39.1

Table 6: Comparison of three evaluation settings for OG-VLM. w/o symmetry-adjusted directly computes orientation accuracy without symmetry handling; w/ symmetry-adjusted accounts for symmetry-equivalent orientations when computing Acc@15 and Acc@30; and Joint additionally requires the predicted box to satisfy a localization IoU threshold of 0.25.

Panel of Figure 4 — Figure 4: Results on three Orientation subsets of 3DSRBench ( In front of , On the left , and Viewpoint ). Avg. denotes the aggregate performance.

Model	EM@1 $\uparrow$	EM@1-Strict $\uparrow$
GS-Reasoner	29.02	18.05
Ours	32.64	28.18

Table 7: Downstream situated spatial reasoning results on MSQA.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Acc
Orient Anything V1 ( Wang et al., 2024b )	72.4
Orient Anything V2 ( Wang et al., 2026 )	86.4
SAM3D ( Chen et al., 2025a )	96.3

Appendix

Table 8: Validation of the annotation module on Ori-COCO (Acc).

Dataset	Agreement	Fleiss’ $\kappa$
ScanNet	96.00%	0.479
RefCOCO	96.27%	0.409
RefCOCO+	98.79%	0.436
RefCOCOg	98.68%	0.493
Overall	97.45%	0.454

Appendix

Table 9: Independent annotation audit: agreement between two evaluators.

Rotation	X Err.	Y Err.	Z Err.	Mean Err.	Acc@15	Acc@30
$90^{\circ}$	69.0	51.3	83.9	60.2	0.05	0.20
$180^{\circ}$	40.1	44.5	18.7	42.3	16.71	37.32
$270^{\circ}$	68.4	52.1	83.6	60.3	0.05	0.25

Appendix

Table 10: Sensitivity of OG-VLM to image-plane rotation of the input view. Axis-wise and overall mean orientation errors are in degrees; Acc@15/Acc@30 are reported in %.

Model	Settings	Acc@15	Acc@30
Ours (only data)	multi-view	68.08	70.64
	single-view	58.43	61.46
	language	20.00	40.00
Ours (+special token)	multi-view	70.50	73.06
	single-view	56.88	60.33
	language	8.04	10.80

Appendix

Table 11: Results on Multi3DRefer ∗ under different settings.

Model	ScanRefer	Multi3DRefer	RefCOCO	RefCOCO+	RefCOCOg
	Acc 25%	Acc 25%	Acc 25%	Acc 25%	Acc 25%
Ours (only data)	55.68	55.43	71.54	68.91	62.37
Ours (+special token)	58.95	57.38	71.84	69.26	62.42
Ours (+geometric loss)	61.37	60.08	72.67	70.31	63.38

Appendix

Table 12: Localization grounding results at Acc25% on ScanRefer, Multi3DRefer, RefCOCO, RefCOCO+, and RefCOCOg.

Figure 5: Qualitative orientation-grounding results on RefCOCO+ and ScanNet. For RefCOCO+, gray arrows denote ground truth, while orange and red arrows denote predictions from Ours and SpatialReasoner, respectively. For ScanNet, solid axes denote ground truth and dashed axes denote predictions from Ours and VG-LLM, respectively.

Explore similar work

GuideGround: VLM-guided Semantic Understanding and Viewpoint-aware Reasoning for 3D Visual Grounding
Aug 1, 2026Yiwen Wang, Yuyang Deng, Yihao Long +1VLM Reasoning 3D Visual Grounding
Xi’an Jiaotong University
Grounded 3D-Aware Spatial Vision-Language Modeling
May 28, 2026An-Chieh Cheng, Yang Fu, Yatai Ji +123D Spatial Reasoning Visually Grounded Reasoning
UCSD · NVIDIA · MIT
Where to Look Matters: Learning Influential Views for VLM-based 3D Visual Grounding
Sep 4, 2026Tsung-Chih Chiang, Hsuan-Kung Yang, Jou-Min Liu +43D Visual Grounding 3D VLMs
National Tsing Hua University, Hsinchu, Taiwan · Woven by Toyota, Inc., Tokyo, Japan · National Taiwan University, Taipei, Taiwan

Aug 1, 2026, cs.CVJ/K move · Enter open · S save

GuideGround: VLM-guided Semantic Understanding and Viewpoint-aware Reasoning for 3D Visual Grounding

Yiwen Wang, Yuyang Deng, Yihao Long, Xi Zhao

Xi’an Jiaotong University

VLM Reasoning 3D Visual Grounding

Open paper AskPDF arXiv

3D visual grounding aims to localize the target object in a 3D scene from a natural language query, requiring both fine-grained semantic understanding and viewpoint-dependent spatial reasoning. Existing methods typically formulate semantic understanding as an auxiliary closed-set object classification task and rely on multi-view feature aggregation for viewpoint reasoning, limiting semantic generalization and weakening viewpoint-specific evidence. We observe that vision-language models naturally provide complementary capabilities through open-vocabulary semantic understanding and global scene perception. Based on this insight, we propose GuideGround, a VLM-guided framework that complements rather than replaces task-specific grounding models by leveraging VLMs for semantic enhancement and viewpoint-specific hypothesis verification. Specifically, we replace auxiliary closed-set object classification with VLM-generated object semantic descriptions to enhance semantic understanding. Meanwhile, instead of directly aggregating multi-view representations, we preserve viewpoint-specific grounding hypotheses through per-view grounding and explicitly verify them using VLMs across candidate viewpoints. Extensive experiments on the ReferIt3D benchmark demonstrate that GuideGround consistently outperforms previous state-of-the-art methods. Comprehensive ablation studies further confirm the effectiveness of both the proposed semantic understanding and viewpoint reasoning strategies.

Abstract

Explore similar work

GuideGround: VLM-guided Semantic Understanding and Viewpoint-aware Reasoning for 3D Visual Grounding

Grounded 3D-Aware Spatial Vision-Language Modeling

Where to Look Matters: Learning Influential Views for VLM-based 3D Visual Grounding