Organizations: University of Macau · Shanghai Jiaotong University · University of Science and Technology of China · Hefei University of Technology · National University of Singapore
3D Large Vision-Language Models (3D LVLMs) built upon Large Language Models (LLMs) have achieved remarkable progress across various multimodal tasks. However, their inherited position-dependent modeling mechanism, Rotary Position Embedding (RoPE), remains suboptimal for 3D multimodal understanding. The vanilla RoPE formulation fails to preserve essential three-dimensional spatial structures when encoding 3D tokens, and its relative distance computation overlooks angular dependencies, hindering the model's ability to capture directional variations in visual representations. To overcome these limitations, we introduce Spherical Coordinate-based Positional Embedding (SoPE). Our method maps point-cloud token indices into a 3D spherical coordinate space, enabling unified modeling of spatial locations and directional angles. This formulation preserves the inherent geometric structure of point-cloud data, enhances spatial awareness, and yields more consistent and expressive geometric representations for multimodal learning. In addition, we introduce a multi-scale frequency mixing strategy to fuse feature information across different frequency domains. Experimental results on multiple 3D scene benchmarks validate the effectiveness of our approach, while real-world deployment experiments further demonstrate its strong generalization capability.
Figures & tables
Figure 1: SoPE enhances 3D spatial representation via geometry-aware positional encoding. Conventional positional encoding methods such as (a) RoPE and (b) RoPE-3D exhibit spatial perception bias: cross-modal attention concentrates on a few hotspots and fails to capture global context, resulting in less discriminative spatial features. In contrast, the proposed (c) SoPE produces more balanced and global attention patterns while amplifying spatial feature saliency, indicating a deeper understanding of the 3D environment.
Figure 2: Comparison of cross-modal attention patterns with RoPE, RoPE-3D, and SoPE . (a) With RoPE, attention is predominantly intra-modal, failing to establish effective connections between the text and the point cloud. (b) RoPE-3D yields marginal improvements in cross-modal attention but remains limited. (c) The proposed SoPE successfully establishes stronger cross-modal attention links and substantially enhances information flow between the two modalities.
Figure 3: SoPE: Spherical Coordinate-Based Positional Embedding for Multimodal 3D Scene Understanding. The figure illustrates the overall SpatialSoPE framework, where SoPE is integrated into SpatialLM. By transforming coordinates from Cartesian to spherical, SpatialSoPE mitigates spatial directional bias and enables more robust 3D object understanding and localization.
Methods
F1 - Structured3D
IoU 2D @0.25 ↑
IoU 2D @0.5 ↑
RoomFormer [ 74 ]
70.4
67.2
SceneScript [ 2 ]
83.1
80.8
SpatialLM (ft. Structured3D) [ 45 ]
32.8
17.9
SpatialSoPE (ft. Structured3D)
33.5
29.5
SpatialLM (ft. SpatialLM Dataset → Structured3D)
86.5
84.6
Table 1: Integration of SoPE enables state-of-the-art layout estimation . Our results indicate that the SpatialSoPE module, when fine‑tuned (ft), is the main factor enabling us to outperform prior methods such as RoomFormer and SceneScript.
Methods
F1 - ARKitScenes
IoU 3D @0.25 ↑
IoU 3D @0.50 ↑
VoteNet [ 14 ]
53.9
45.4
H3DNet [ 78 ]
55.7
46.3
NeRF-Det [ 64 ]
60.3
34.7
UniDet3D [ 35 ]
62.8
48.3
SpatialLM [ 45 ]
63.9
60.7
Table 2: 3D object detection results on the ARKitScenes dataset . SpatialSoPE consistently outperforms SpatialLM and other 3D LVLM baselines.
Methods
F1 - SpatialLM Dataset
F1 - Structured3D
IoU 3D @0.25 ↑
IoU 3D @0.50 ↑
IoU 3D @0.25 ↑
IoU 3D @0.50 ↑
SpatialLM [ 45 ]
69.7
62.0
31.8
17.6
SpatialSoPE
71.4
63.4
33.3
17.9
Table 3: Comparison between SpatialSoPE and SpatialLM on the SpatialLM Dataset and Structured3D . SpatialSoPE consistently outperforms the SpatialLM baseline across all metrics on both datasets.
Figure 4: Effectiveness of the SoPE module demonstrated through component analysis. The figure compares various components added to the SpatialLM, highlighting the superior contribution of SoPE to model precision. Results are shown for (a) the ARKitScenes and (b) the SpatialLM datasets.
Figure 5: Visualization of improvements in 3D object detection with SpatialSoPE . Compared to SpatialLM, our method demonstrates significantly improved localization and sizing of 3D bounding boxes across various scenes.
Methods
F1 - ARKitScenes
F1 - SpatialLM Dataset
IoU 3D @0.25 ↑
IoU 3D @0.50 ↑
IoU 3D @0.25 ↑
IoU 3D @0.50 ↑
SpatialLM [ 45 ]
63.9
60.7
69.7
62.0
+MCA [ 80 ]
63.7
60.2
70.1
61.6
+CCA [ 63 ]
64.1
60.5
69.8
62.5
+RoPE-3D
64.2
61.4
69.7
62.4
SpatialSoPE
66.1
63.2
71.4
63.4
Table 4: Analysis of Component Contributions . We demonstrate the impact of each key module by comparing the full SpatialSoPE model against the baseline and its intermediate variants on the ARKitScenes and SpatialLM datasets.
Methods
t:r:θ:ϕ
F1 - ARKitScenes
IoU 3D @0.25 ↑
IoU 3D @0.50 ↑
Angular-Biased
(8:6:9:9)
65.5
62.7
Uniform
(1:1:1:1)
63.0
59.0
Temporal-Biased
(5:1:1:1)
65.0
62.7
SpatialSoPE
(24:2:3:3)
66.1
63.2
Table 5: Ablation study for our proposed SpatialSoPE method . We validate our parameter choices by comparing against several variants on the ARKitScenes dataset.
Method
Multi-scale
F1 - ARKitScenes
F1 - SpatialLM Dataset
IoU 3D @0.25 ↑
IoU 3D @0.50 ↑
IoU 3D @0.25 ↑
IoU 3D @0.50 ↑
SpatialLM+RoPE-3D
×
64.2
61.7
69.4
62.3
✓
64.8
62.1
70.3
62.9
SpatialSoPE
×
65.4
61.4
71.0
62.5
✓
66.1
63.2
71.4
63.4
Table 6: Effect of Multi-scale Frequency Mixing Strategy for RoPE-3D and SpatialSoPE . We compare RoPE-3D and SpatialSoPE with and without the proposed Multi-scale Frequency Mixing Strategy, reporting F1 scores on ARKitScenes and the SpatialLM dataset.
Figure 6: The figure illustrates the real-world experimental environment and the physical robot used in our study. Equipped with SpatialSoPE, the robot is capable of understanding the entire scene and executing exploration and object-transport tasks according to human instructions.
Figure 7: Detection visualization and task environments of SpatialSoPE in real-world reconstructed scenes.
Department of Computer Science and Engineering, Southern University of Science and Technology · Wangxuan Institute of Computer Technology, Peking University · State Key Laboratory of General Artificial Intelligence, Peking University
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, China · YUKUN Intelligent World, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, China