Direction-aware 3D Large Multimodal Models

Quan Liu; Weihao Xuan; Junjue Wang; Naoto Yokoya; Ling Shao; Shijian Lu

cs.CVFeb 22, 2026

Direction-aware 3D Large Multimodal Models

Authors: Quan Liu, Weihao Xuan, Junjue Wang, Naoto Yokoya, Ling Shao, Shijian Lu

Organizations: Nanyang Technological University · The University of Tokyo · RIKEN AIP · UCAS-Terminus AI Lab, University of Chinese Academy of Sciences

Abstract

3D large multimodal models (3D LMMs) rely heavily on ego poses for enabling directional question-answering and spatial reasoning. However, most existing point cloud benchmarks contain rich directional queries but lack the corresponding ego poses, making them inherently ill-posed in 3D large multimodal modelling. In this work, we redefine a new and rigorous paradigm that enables direction-aware 3D LMMs by identifying and supplementing ego poses into point cloud benchmarks and transforming the corresponding point cloud data according to the identified ego poses. We enable direction-aware 3D LMMs with two novel designs. The first is PoseRecover, a fully automatic pose recovery pipeline that matches questions with ego poses from RGB-D video extrinsics via object-frustum intersection and visibility check with Z-buffers. The second is PoseAlign that transforms the point cloud data to be aligned with the identified ego poses instead of either injecting ego poses into textual prompts or introducing pose-encoded features in the projection layers. Extensive experiments show that our designs yield consistent improvements across multiple 3D LMM backbones such as LL3DA, LL3DA-SONATA, Chat-Scene, and 3D-LLAVA, improving ScanRefer mIoU by 30.0% and Scan2Cap LLM-as-judge accuracy by 11.7%. In addition, our approach is simple, generic, and training-efficient, requiring only instruction tuning while establishing a strong baseline for direction-aware 3D-LMMs.

Figures & tables

Figure 1: Ego pose is critical in spatial reasoning and understanding. (a) Direction-agnostic 3D LMMs are struggling to reason spatial directions due to the absence of ego-pose information. (b) Incorporating ego pose resolves directional ambiguity, enabling consistent and robust spatial reasoning.

Figure 2: The offline data generation pipeline for PoseRecover . (a) Object annotations and camera poses are obtained from ScanNet-v2 [ 12 ] . Camera poses and objects are downsampled for visibility. Zoom in for details. (b) PoseRecover exhaustively calculates the intersection rates between objects and camera frustums. (c) Visibility of the intersection is further validated with a z-buffer. (d) These intersection rates are saved and later sampled during training or inference to supplement ego poses to models.

Figure 3: Three viable designs for PoseAlign . We explore three mutually exclusive designs to incorporate ego poses into the vanilla model in (a): 1) PoseAlign-Transform that shifts point clouds to the ego reference frame in (b); 2) PoseAlign-Embed that encodes ego poses into point cloud features in (c); 3) PoseAlign-Prompt that integrates ego poses into the text prompt in (d). The projection layer and the LoRA [ 20 ] weights of the LLM are trained with instruction-tuning.

Figure 4: Effect of the pose clipping. The KDE [ 47 ] of maximum yaw difference among pose candidates rapidly concentrates around zero with increasing clip ratio in ScanQA. Higher clip ratio reduces data variety but boosts pose stability.

Model	Modality	ScanRef (val)		Multi3DRef (val)		ScanQA (val)					Scan2Cap (val)
Model	Modality	[email protected]	mIoU	[email protected]	mIoU	C	B-4	M	R	L-A	[email protected]	[email protected]	[email protected]	[email protected]	L-A
Specialist Models:
ScanQA [ 2 ]	PC	-	-	-	-	64.9	10.1	13.1	33.3	-	-	-	-	-	-
3D-VLP [ 27 ]	PC	-	-	-	-	67.0	11.2	13.5	34.5	-	54.9	32.3	24.8	51.5	-
3D-VisTA [ 64 ]	PC	-	-	-	-	69.6	10.4	13.9	45.7	-	61.6	34.1	26.8	55.0	-
Scan2Cap [ 10 ]	PC	-	-	-	-	-	-	-	-	-	39.1	23.3	22.0	44.8	-

Table 1: Cross-dataset performance comparison on multiple 3D vision-language tasks. ‘PC’ and ‘I’ represent point cloud and image modalities, respectively. Major metrics are highlighted with gray background. Performance on PoseRecover benchmark may differ from those in the original papers due to retraining with lower batch sizes. Baselines in PoseRecover benchmark are comparable with all methods because they do not use pose information, while our modifications are comparable within the benchmark due to additional pose input.

Design	ScanRef	Multi3DRef	ScanQA	Scan2Cap
Design	mIoU $\uparrow$	mIoU $\uparrow$	L-A $\uparrow$	L-A $\uparrow$
Baseline	42.6	48.1	45.7	28.1
Baseline PoseAlign-T(Top)	37.5	41.5	43.0	23.5
Random Pose	39.0	44.3	44.7	25.8
PoseAlign-T(Clip X=0.3)	55.4	54.3	47.3	31.4
PoseAlign-T(Top)	68.5	60.2	47.5	29.7

Table 2: Ablation experiment on 3D-LLAVA . ‘Baseline PoseAlign-T’ is the performance of the baseline model on PoseAlign-T data, where the input point cloud is transformed to the camera location following Equation 8 . ‘Random Pose’ uses random camera poses instead of those found by PoseRecover.

	ScanRef	Multi3DRef	ScanQA	Scan2Cap
$X$	mIoU $\uparrow$	mIoU $\uparrow$	L-A $\uparrow$	L-A $\uparrow$
0.0	55.1	54.2	47.3	30.1
0.1	55.2	54.3	46.6	30.8
0.2	54.8	54.3	47.3	31.2
0.3	55.4	54.3	47.3	31.4
0.4	55.4	54.3	46.9	30.9

Table 3: Parameter tuning experiment for Clip Ratio X of PoseAlign-Transform on 3D-LLAVA.

Model	Evaluated on	ScanQA ( $46.7\%$ Direction-critical)					Scan2Cap ( $89.7\%$ Direction-critical)
Model	Evaluated on	C $\uparrow$	B-4 $\uparrow$	M $\uparrow$	R $\uparrow$	L-A $\uparrow$	C $\uparrow$	B-4 $\uparrow$	M $\uparrow$	R $\uparrow$	L-A $\uparrow$
3D-LLAVA	Full Val set	95.4	16.3	18.9	44.6	45.7	77.4	36.4	26.93	57.4	28.1
	Direction-critical subset	91.6	18.0	17.8	41.0	37.8	76.8	36.8	26.94	57.4	28.0
	Complementary subset	97.6	11.9	20.1	47.7	52.4	82.7	32.6	26.88	56.9	29.6
3D-LLAVA+ PoseAlign-T	Full Val set	99.8 (+4.4)	17.3 (+1)	19.7 (+0.8)	46.5 (+1.9)	47.3 (+1.6)	76.1 (-1.3)	37.1 (+0.7)	27.11 (+0.18)	57.6 (+0.2)	31.4 (+3.3)
	Direction-critical subset	96.1 (+4.5)	19.6 (+1.6)	18.7 (+0.9)	43.1 (+2.1)	40.3 (+2.5)	75.9 (-0.9)	37.5 (+0.7)	27.12 (+0.18)	57.7 (+0.3)	31.3 (+3.3)

Table 4: Performance on direction-critical question subset.

Figure 5: Qualitative results of direction-critical questions for 3D-LLAVA baseline (top row) and PoseAlign-Transform (bottom row). The XYZ axes of the world coordinate frame are colored with red , green , and blue , respectively. The baseline paradigm uses default world coordinates of ScanNet-v2, which are non-informative. Instead, the PoseAlign paradigm aligns the coordinate frame to the recovered ego pose, providing an anchor for robust spatial reasoning. Red text highlights wrong answers and green text highlights correct answers.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

		ScanRefer	Multi3DRef	ScanQA	Scan2Cap
Model	Noise Bound	mIoU $\uparrow$	mIoU $\uparrow$	B-4 $\uparrow$	B-4 $\uparrow$
3D-LLAVA	/	42.6	48.1	16.3	36.4
3D-LLAVA+ PoseAlign-T (Top)	0°, 0m	68.5	60.2	17.3	37.1
	15°, 0.2m	62.5	57.7	17.1	36.9
	30°, 0.5m	53.2	55.1	17.0	36.6

Appendix

Table 5: Robustness to added uniform pose error.

Figure 6: Distribution of objects in world (L) and PoseAlign ego (R) coordinates on X-Y plane. While clustered in front view, objects still follow non-trivial distribution.

Design	ScanQA	SQA3D	Scan2Cap
Design	L-A $\uparrow$	L-A $\uparrow$	L-A $\uparrow$
LL3DA	35.2	-	23.3
LL3DA+PoseAlign-T	35.4	-	-
LL3DA-S	36.3	-	-
LL3DA-S+PoseAlign-T	36.5	-	-
Chat-Scene	44.1	56.9	38.6

Appendix

Table 6: Comparison between baselines and respective PoseAlign variants using GPT-5-mini for LLM-as-judge.

$X$	ScanQA (val)					SQA3D (test)			Scan2Cap (val)
$X$	C	B-4	M	R	L-A	EM	EM-R	L-A	[email protected]	[email protected]	[email protected]	[email protected]	L-A
None	85.6	15.6	17.8	40.4	43.9	52.8	56.1	56.9	74.0	34.5	26.8	56.4	26.6
0.0	83.8	14.2	17.4	39.7	43.1	53.4	56.2	57.2	37.2	27.9	23.4	52.0	9.76
0.1	86.7	14.9	17.9	40.8	44.3	52.6	55.7	56.6	74.1	34.7	26.9	56.5	27.0
0.2	87.2	15.1	18.1	41.1	44.7	53.3	56.1	57.1	75.5	35.0	27.1	56.7	26.9
0.3	87.5	15.5	17.9	40.9	44.8	53.3	56.2	57.1	74.9	34.7	26.9	56.5	26.9

Appendix

Table 7: Full parameter tuning experiment for Clip Ratio X of PoseAlign-Embed on Chat-Scene.

$X$	ScanQA (val)					SQA3D (test)			Scan2Cap (val)
$X$	C	B-4	M	R	L-A	EM	EM-R	L-A	[email protected]	[email protected]	[email protected]	[email protected]	L-A
None	95.4	16.3	18.9	44.6	45.7	54.9	57.3	57.9	77.4	36.4	26.9	57.4	28.1
0.0	98.7	16.3	19.3	46.2	47.3	55.1	57.6	56.1	75.5	36.6	26.9	57.2	30.1
0.1	99.4	17.4	19.4	46.1	46.6	52.6	55.0	55.6	76.6	37.0	27.1	57.5	30.8
0.2	99.8	17.3	19.5	46.5	47.3	53.4	56.2	56.6	74.7	36.9	27.0	57.5	31.2
0.3	99.8	17.3	19.7	46.5	47.3	54.2	56.7	57.4	76.1	37.1	27.1	57.6	31.4

Appendix

Table 8: Full parameter tuning experiment for Clip Ratio X of PoseAlign-Transform on 3D-LLAVA.

3D VLMs

Three-dimensional vision-language models connect language with spatially grounded visual data for 3D understanding, reasoning, and action.

See all 213 papers

Point Cloud as a Foreign Language for Multi-modal Large Language Model
Mar 10, 2026Sneha Paul, Zachary Patterson, Nizar Bouguila3D VLMs Point Cloud Learning
Concordia University, Canada
PointLLM-R: Enhancing 3D Point Cloud Reasoning via Chain-of-Thought
May 21, 2026Chaoqi Chen, Qile Xu, Wenjun Zhou +1CoT Reasoning Point Cloud Understanding
Visual Computing Research Center (VCC), College of Computer Science and Software Engineering (CSSE), Shenzhen University, China · VCC, CSSE, Shenzhen University, China
3D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models
Jun 18, 2026Jintang Xue, Xinyu Wang, Yixing Wu +23D Representation Learning Multimodal Foundation Models
University of Southern California Los Angeles, California, USA · Ohio State University Columbus, Ohio, USA

Mar 10, 2026J/K move · Enter open · S save

Point Cloud as a Foreign Language for Multi-modal Large Language Model

Sneha Paul, Zachary Patterson, Nizar Bouguila

Concordia University, Canada

3D VLMs Point Cloud Learning 3D VQA

Open paper AskPDF arXiv

Multi-modal large language models (MLLMs) have shown remarkable progress in integrating visual and linguistic understanding. Recent efforts have extended these capabilities to 3D understanding through encoder-based architectures that rely on pre-trained 3D encoders to extract geometric features. However, such approaches suffer from semantic misalignment between geometric and linguistic spaces, resolution sensitivity, and substantial computational overhead. In this work, we present SAGE, the first end-to-end 3D MLLM that directly processes raw point clouds without relying on a pre-trained 3D encoder. Our approach introduces a lightweight 3D tokenizer that combines geometric sampling and neighbourhood aggregation with vector quantization to convert point clouds into discrete tokens--treating 3D data as a foreign language that naturally extends the LLM's vocabulary. Furthermore, to enhance the model's reasoning capability on complex 3D tasks, we propose a preference optimization training strategy with a semantic alignment-based reward, specifically designed for open-ended 3D question answering where responses are descriptive. Extensive experiments across diverse 3D understanding benchmarks demonstrate that our end-to-end approach outperforms existing encoder-based methods while offering significant advantages in computational efficiency, generalization across LLM backbones, and robustness to input resolution variations. Code is available at: github.com/snehaputul/SAGE3D.

Abstract

3D VLMs

Explore similar work

Point Cloud as a Foreign Language for Multi-modal Large Language Model

PointLLM-R: Enhancing 3D Point Cloud Reasoning via Chain-of-Thought

3D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models