Scalable Object Relation Encoding for Better 3D Spatial Reasoning in Large Language Models
Authors: Shengli Zhou, Minghang Zheng, Feng Zheng, Yang Liu
Organizations: Department of Computer Science and Engineering, Southern University of Science and Technology · Wangxuan Institute of Computer Technology, Peking University · State Key Laboratory of General Artificial Intelligence, Peking University
Spatial reasoning focuses on locating target objects based on spatial relations in 3D scenes, which plays a crucial role in developing intelligent embodied agents. Due to the limited availability of 3D scene-language paired data, it is challenging to train models with strong reasoning ability from scratch. Previous approaches have attempted to inject 3D scene representations into the input space of Large Language Models (LLMs) and leverage the pretrained comprehension and reasoning abilities for spatial reasoning. However, models encoding absolute positions struggle to extract spatial relations from prematurely fused features, while methods explicitly encoding all spatial relations (which is quadratic in the number of objects) as input tokens suffer from poor scalability. To address these limitations, we propose QuatRoPE, a novel positional embedding method with an input length that is linear to the number of objects, and explicitly calculates pairwise spatial relations through the dot product in attention layers. QuatRoPE's holistic vector encoding of 3D coordinates guarantees a high degree of spatial consistency, maintaining fidelity to the scene's geometric integrity. Additionally, we introduce the Isolated Gated RoPE Extension (IGRE), which effectively limits QuatRoPE's influence to object-related tokens, thereby minimizing interference with the LLM's existing positional embeddings and maintaining the LLM's original capabilities. Extensive experiments demonstrate the effectiveness of our approaches. The code and data are available at https://github.com/oceanflowlab/QuatRoPE.
Figures & tables
Figure 1: (a) In QuatRoPE, we embed the absolute 3D position of each object to the corresponding token, thus limiting the input length linear to object count. (b) By leveraging a dedicated rotation scheme, when tokens with 3D position embedding interact in the attention layer of the LLM, the absolute coordinates are transformed into pairwise relative positions, empowering spatial reasoning. (c) In previous methods, as the positions are decoupled into individual coordinates, when the coordinate of some axis is close (as marked in green), the attention score is incorrectly inflated. (d) QuatRoPE encodes positions as holistic vectors, correctly representing spatial relations.
Table 1: Results for the comparative experiments, the best scores obtained by using ground-truth or predicted segmentation are underlined, and the overall best scores are in bold. By applying QuatRoPE, our models have achieved consistent gains under all metrics.
Model
LLM
Acc @ 0.25
Gain
Acc @ 0.5
Gain
Chat-Scene [ 14 ]
Llama-3.2-1B-Instruct
22.92
–
22.92
–
Chat-Scene + QuatRoPE (Ours)
Llama-3.2-1B-Instruct
27.38
4.46 (19.48%)
27.38
4.46 (19.48%)
3DGraphLLM [ 34 ]
Llama-3.2-1B-Instruct
25.89
–
25.60
–
3DGraphLLM + QuatRoPE (Ours)
Llama-3.2-1B-Instruct
29.76
3.87 (14.94%)
29.76
4.17 (16.28%)
3DGraphLLM [ 34 ]
Llama-3-8B-Instruct
37.50
–
36.90
–
3DGraphLLM + QuatRoPE (Ours)
Llama-3-8B-Instruct
41.96
4.46 (11.90%)
41.96
5.06 (13.71%)
Table 2: Results on our spatial reasoning benchmark. Our model achieves significant and consistent gains across various settings.
RoPE Composition Approach
ScanRefer
SQA3D
Acc @ 0.25
Acc @ 0.5
Multi @ 0.25
Multi @ 0.5
EM @ 1
Baseline: Chat-Scene [ 14 ]
None
50.72
50.33
42.69
42.29
50.72
Trans-Additive
53.12
52.79
45.48
45.14
52.96
IGRE (Ours)
55.44
55.00
47.81
47.36
53.14
Baseline: 3DGraphLLM [ 34 ]
Table 3: Results for ablation study on different composition approaches, the best scores of each baseline are marked in bold.
Explicit Positional Encoding Approach
ScanRefer
SQA3D
Acc @ 0.25
Acc @ 0.5
Multi @ 0.25
Multi @ 0.5
EM @ 1
Baseline: Chat-Scene [ 14 ]
None
50.72
50.33
42.69
42.29
50.72
Raw Coordinates
52.26
52.01
44.41
44.17
51.40
M-RoPE
54.30
53.92
46.44
46.10
51.55
QuatRoPE (Ours)
55.44
55.00
47.81
47.36
53.14
Table 4: Results for ablation study on different RoPE methods, the best scores of each baseline are marked in bold.
Figure 2: An illustration of ASR’s construction pipeline.
δ
3DGraphLLM-1B
+ QuatRoPE
Gain
1 (All)
93.72
94.65
0.93
0.5
92.28
94.21
1.93
0.3
91.47
94.31
2.84
0.2
93.21
96.30
3.09
0.1
92.39
96.74
4.35
0.05
84.62
92.31
7.69
Table 5: Verification of advantage in holistic encoding.
Figure 3: Qualitative results on the ScanRefer dataset. Target objects are correctly grounded by QuatRoPE ( green ), whereas the baseline Chat-Scene produces incorrect predictions ( red ).
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Base Vector
ScanRefer
SQA3D
Multi3dRef
Acc @ 0.25
Acc @ 0.5
EM @ 1
F1 @ 0.25
F1 @ 0.5
Chat-Scene [ 14 ]
Fixed
55.44
55.00
53.14
58.09
57.72
Learnable
54.47
54.14
52.84
57.96
57.74
3DGraphLLM [ 34 ]
Fixed
58.30
58.15
53.20
60.70
60.52
Learnable
56.89
56.64
52.68
60.67
60.51
Appendix
Table 6: Comparison between fixed and learnable base vectors for rotation.
Frequency
ScanRefer
SQA3D
Multi3dRef
Acc @ 0.25
Acc @ 0.5
EM @ 1
F1 @ 0.25
F1 @ 0.5
0.3 (Default)
58.30
58.15
60.70
60.52
53.20
0.1 (Small)
54.55
54.39
58.02
57.90
51.99
1.0 (Large)
53.41
53.14
56.28
55.99
52.18
Appendix
Table 7: Comparison between different frequencies.
3D spatial reasoning studies how AI systems infer geometry, object relations, viewpoints, and motion to understand and act in three-dimensional scenes.
Recent 3D Large-Language Models (3D-LLMs) claim to understand 3D worlds, especially spatial relationships among objects. Yet, we find that simply fine-tuning a language model on text-only question-answer pairs can perform comparably or even surpass these methods on the SQA3D benchmark without using any 3D input. This indicates that the SQA3D benchmark may not be able to detect if the model exploits textual shortcuts rather than engages in 3D-aware reasoning. To address this issue, we introduce Real-3DQA, a more rigorous evaluation benchmark that filters out easy-to-guess questions and introduces a structured taxonomy to assess various aspects of 3D reasoning. Experiments on Real-3DQA confirm that existing 3D-LLMs struggle with spatial relationships once simple cues are removed. We further propose a 3D-reweighted training objective that guides model to rely more on 3D visual clues, substantially enhancing 3D-LLMs performance in spatial reasoning tasks. Our findings underscore the need for robust benchmarks and tailored training strategies to advance genuine 3D vision-language understanding. Project page: https://real-3dqa.github.io/.
Xianzheng Ma, Tao Sun, Shuai Chen +7
VGG, University of Oxford · AVL, University of Oxford · Stanford University +2
While Multimodal Large Language Models (MLLMs) have achieved remarkable success in 2D visual understanding, their ability to reason about 3D space remains limited. To address this gap, we introduce geometrically referenced 3D scene representations (GR3D). Given a set of input images, GR3D annotates objects in the images with unique IDs and encodes their 3D geometric attributes as textual references indexed by these IDs. This representation enables MLLMs to interpret 3D cues using their advanced language-based skills in mathematical reasoning, while concurrently analyzing 2D visual features in a tightly coupled way. We present a simple yet effective approach based on GR3D, which requires no additional training and is readily applicable to different MLLMs. Implemented in a zero-shot setting, our approach yields substantial improvements on challenging spatial reasoning benchmarks, boosting GPT-5 performance by 9% on VSI-Bench and 12% on MindCube. Qualitative studies further demonstrate that GR3D empowers MLLMs to perform complex spatial reasoning with highly sparse input views.
Although multimodal large language models (MLLMs) have achieved remarkable progress, understanding 3D spatial relationships from 2D images remains a critical challenge. Existing methods primarily rely on symbolic text tokens, which inherently lack the fidelity to represent continuous geometric information. While recent methods use latent representations to enhance reasoning, relying on a single latent type cannot adapt to the diversity of spatial tasks, leading to misalignment in complex geometric scenarios. To address these limitations, we propose GeoAnchor, an interleaved text-latent reasoning framework. GeoAnchor decomposes 3D spatial information into three complementary components: position latents for object grounding, direction latents for relational orientation, and geometry latents for scene structure. These components are recombined in a structured space to construct local evidence while capturing global context, enabling dynamic and interpretable reasoning. Furthermore, we introduce a collaborative training strategy that guides the model from local spatial perception to comprehensive 3D understanding. Extensive experiments on diverse and complex 3D reasoning tasks demonstrate that GeoAnchor outperforms the state of the art, validating its effectiveness and generalization capabilities.
Hao Li, Han Fang, Zixin Pan +8
Shanghai Jiao Tong University · Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) · The Hong Kong University of Science and Technology (Guangzhou) +1