Scalable Object Relation Encoding for Better 3D Spatial Reasoning in Large Language Models
Authors: Shengli Zhou, Minghang Zheng, Feng Zheng, Yang Liu
Organizations: Department of Computer Science and Engineering, Southern University of Science and Technology · Wangxuan Institute of Computer Technology, Peking University · State Key Laboratory of General Artificial Intelligence, Peking University
Spatial reasoning focuses on locating target objects based on spatial relations in 3D scenes, which plays a crucial role in developing intelligent embodied agents. Due to the limited availability of 3D scene-language paired data, it is challenging to train models with strong reasoning ability from scratch. Previous approaches have attempted to inject 3D scene representations into the input space of Large Language Models (LLMs) and leverage the pretrained comprehension and reasoning abilities for spatial reasoning. However, models encoding absolute positions struggle to extract spatial relations from prematurely fused features, while methods explicitly encoding all spatial relations (which is quadratic in the number of objects) as input tokens suffer from poor scalability. To address these limitations, we propose QuatRoPE, a novel positional embedding method with an input length that is linear to the number of objects, and explicitly calculates pairwise spatial relations through the dot product in attention layers. QuatRoPE's holistic vector encoding of 3D coordinates guarantees a high degree of spatial consistency, maintaining fidelity to the scene's geometric integrity. Additionally, we introduce the Isolated Gated RoPE Extension (IGRE), which effectively limits QuatRoPE's influence to object-related tokens, thereby minimizing interference with the LLM's existing positional embeddings and maintaining the LLM's original capabilities. Extensive experiments demonstrate the effectiveness of our approaches. The code and data are available at https://github.com/oceanflowlab/QuatRoPE.
Figures & tables
Figure 1: (a) In QuatRoPE, we embed the absolute 3D position of each object to the corresponding token, thus limiting the input length linear to object count. (b) By leveraging a dedicated rotation scheme, when tokens with 3D position embedding interact in the attention layer of the LLM, the absolute coordinates are transformed into pairwise relative positions, empowering spatial reasoning. (c) In previous methods, as the positions are decoupled into individual coordinates, when the coordinate of some axis is close (as marked in green), the attention score is incorrectly inflated. (d) QuatRoPE encodes positions as holistic vectors, correctly representing spatial relations.
Table 1: Results for the comparative experiments, the best scores obtained by using ground-truth or predicted segmentation are underlined, and the overall best scores are in bold. By applying QuatRoPE, our models have achieved consistent gains under all metrics.
Model
LLM
Acc @ 0.25
Gain
Acc @ 0.5
Gain
Chat-Scene [ 14 ]
Llama-3.2-1B-Instruct
22.92
–
22.92
–
Chat-Scene + QuatRoPE (Ours)
Llama-3.2-1B-Instruct
27.38
4.46 (19.48%)
27.38
4.46 (19.48%)
3DGraphLLM [ 34 ]
Llama-3.2-1B-Instruct
25.89
–
25.60
–
3DGraphLLM + QuatRoPE (Ours)
Llama-3.2-1B-Instruct
29.76
3.87 (14.94%)
29.76
4.17 (16.28%)
3DGraphLLM [ 34 ]
Llama-3-8B-Instruct
37.50
–
36.90
–
3DGraphLLM + QuatRoPE (Ours)
Llama-3-8B-Instruct
41.96
4.46 (11.90%)
41.96
5.06 (13.71%)
Table 2: Results on our spatial reasoning benchmark. Our model achieves significant and consistent gains across various settings.
RoPE Composition Approach
ScanRefer
SQA3D
Acc @ 0.25
Acc @ 0.5
Multi @ 0.25
Multi @ 0.5
EM @ 1
Baseline: Chat-Scene [ 14 ]
None
50.72
50.33
42.69
42.29
50.72
Trans-Additive
53.12
52.79
45.48
45.14
52.96
IGRE (Ours)
55.44
55.00
47.81
47.36
53.14
Baseline: 3DGraphLLM [ 34 ]
Table 3: Results for ablation study on different composition approaches, the best scores of each baseline are marked in bold.
Explicit Positional Encoding Approach
ScanRefer
SQA3D
Acc @ 0.25
Acc @ 0.5
Multi @ 0.25
Multi @ 0.5
EM @ 1
Baseline: Chat-Scene [ 14 ]
None
50.72
50.33
42.69
42.29
50.72
Raw Coordinates
52.26
52.01
44.41
44.17
51.40
M-RoPE
54.30
53.92
46.44
46.10
51.55
QuatRoPE (Ours)
55.44
55.00
47.81
47.36
53.14
Table 4: Results for ablation study on different RoPE methods, the best scores of each baseline are marked in bold.
Figure 2: An illustration of ASR’s construction pipeline.
δ
3DGraphLLM-1B
+ QuatRoPE
Gain
1 (All)
93.72
94.65
0.93
0.5
92.28
94.21
1.93
0.3
91.47
94.31
2.84
0.2
93.21
96.30
3.09
0.1
92.39
96.74
4.35
0.05
84.62
92.31
7.69
Table 5: Verification of advantage in holistic encoding.
Figure 3: Qualitative results on the ScanRefer dataset. Target objects are correctly grounded by QuatRoPE ( green ), whereas the baseline Chat-Scene produces incorrect predictions ( red ).
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Base Vector
ScanRefer
SQA3D
Multi3dRef
Acc @ 0.25
Acc @ 0.5
EM @ 1
F1 @ 0.25
F1 @ 0.5
Chat-Scene [ 14 ]
Fixed
55.44
55.00
53.14
58.09
57.72
Learnable
54.47
54.14
52.84
57.96
57.74
3DGraphLLM [ 34 ]
Fixed
58.30
58.15
53.20
60.70
60.52
Learnable
56.89
56.64
52.68
60.67
60.51
Appendix
Table 6: Comparison between fixed and learnable base vectors for rotation.
Frequency
ScanRefer
SQA3D
Multi3dRef
Acc @ 0.25
Acc @ 0.5
EM @ 1
F1 @ 0.25
F1 @ 0.5
0.3 (Default)
58.30
58.15
60.70
60.52
53.20
0.1 (Small)
54.55
54.39
58.02
57.90
51.99
1.0 (Large)
53.41
53.14
56.28
55.99
52.18
Appendix
Table 7: Comparison between different frequencies.
3D spatial reasoning studies how AI systems infer geometry, object relations, viewpoints, and motion to understand and act in three-dimensional scenes.
Shanghai Jiao Tong University · Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) · The Hong Kong University of Science and Technology (Guangzhou) +1