cs.CLMar 6, 2026
SaveDo 3D Large Language Models Really Understand 3D Spatial Relationships?
Organizations: VGG, University of Oxford · AVL, University of Oxford · Stanford University · Google DeepMind ·
Abstract
Recent 3D Large-Language Models (3D-LLMs) claim to understand 3D worlds, especially spatial relationships among objects. Yet, we find that simply fine-tuning a language model on text-only question-answer pairs can perform comparably or even surpass these methods on the SQA3D benchmark without using any 3D input. This indicates that the SQA3D benchmark may not be able to detect if the model exploits textual shortcuts rather than engages in 3D-aware reasoning. To address this issue, we introduce Real-3DQA, a more rigorous evaluation benchmark that filters out easy-to-guess questions and introduces a structured taxonomy to assess various aspects of 3D reasoning. Experiments on Real-3DQA confirm that existing 3D-LLMs struggle with spatial relationships once simple cues are removed. We further propose a 3D-reweighted training objective that guides model to rely more on 3D visual clues, substantially enhancing 3D-LLMs performance in spatial reasoning tasks. Our findings underscore the need for robust benchmarks and tailored training strategies to advance genuine 3D vision-language understanding. Project page: https://real-3dqa.github.io/.
Figures & tables
Figure 1: Our Real-3DQA Benchmark and Solution. We demonstrate the answer differences between 3D-LLM (LEO) and its blind-finetuned version on various questions, with correct answers highlighted in green frames and incorrect ones in red frames. We discover that some questions can be answered correctly regardless of whether 3D information is used, which we consider to be 3D-independent questions. By filtering out these 3D-independent questions and introducing a new viewpoint rotation score, the original models’ performance drops significantly on Real-3DQA. Finally, using our proposed 3D-aware Reweighted Finetuning strategy, performance improves again.
Figure 2: Our finding : A language model fine-tuned only on text QA pairs without any 3D inputs ( Blind Finetuned ) can match or even surpass state-of-the-art 3D-LLMs ( Original ) on multiple 3D-QA benchmarks. This exposes a critical weakness in current benchmark design and calls into question their ability to assess genuine 3D reasoning despite linguistic shortcuts.
| Dataset | Situated | Text collection | Debiased | Robustness evaluation | Quality check | Question categories |
| ScanQA | ✗ | Human | ✗ | ✗ | ✗ | 0 |
| 3D-QA | ✗ | Human | ✗ | ✗ | ✗ | 4 |
| SQA3D | ✓ | Human | ✓ | ✗ | ✓ | 5 |
| MSQA | ✓ | LLM | ✗ | ✗ | ✓ | 6 |
| ScanReQA | ✗ | Human | ✗ | ✓ | ✓ | 0 |
| Spartun3D | ✓ | LLM | ✗ | ✗ | ✓ | 3 |
Table 1: Comparison of 3D Question Answering Benchmarks. Our Real-3DQA benchmark features situated questions, uses LLM-assisted text collection with human verification, applies debiasing techniques, evaluates robustness across different viewpoints, and provides diverse question categories to assess true 3D spatial reasoning ability.
Figure 3: Overview of Real-3DQA Construction Process. Real-3DQA provides a fair and rigorous evaluation framework for 3D spatial reasoning in 3D-LLMs. The construction process begins with Filtering 3D-independent Questions, which removes questions that can be correctly answered by both the 3D-LLM model and its text-only counterpart, as well as those answerable by the GPT model without 3D input. The remaining high-quality questions are then augmented using GPT, generating spatially consistent variations through viewpoint rotations while preserving the underlying 3D relationships. Finally, expert reviews eliminate redundancy and invalid data, ensuring the highest dataset quality.
Figure 4: Viewpoint Rotation Augmentation. Real-3DQA generates viewpoint-augmented SQA instances to enforce spatial reasoning. The left panel shows the original room layout, where an agent asks “What is on my right?” with the correct answer “white board.” The right panels illustrate the SQA examples of the original viewpoint and rotated viewpoint (90°, 180°, 270°), where the agent’s perspective shifts and the correct answers dynamically adjust while preserving spatial consistence .
| 3D-LLMs | Venue | SQA3D | Real-3DQA | ||
|---|---|---|---|---|---|
| EM | EM_R | EM | EM_R | ||
| 3D-LLM | NeurIPS 23 | 47.8 | 49.6 | 7.5 | 10.4 |
| Chat-3D v2 | Arxiv 24 | 45.0 | 48.1 | 3.4 | 9.7 |
| LEO | ICML 24 | 49.4 | 52.2 | 14.3 | 19.1 |
| Chat-Scene | NeurIPS 24 | 54.4 | 57.2 | 17.0 | 22.1 |
| GPT4Scene | ICLR 26 | 60.6 | 63.3 | 33.1 | 36.9 |
Table 2: Performance comparison on SQA3D and our Real-3DQA . We see a significant performance drop in our new benchmark.
| 3D-LLMs | Correct times | VRS | Question Types | ||||||
|---|---|---|---|---|---|---|---|---|---|
| one | two | three | four | % | distance | direction | counting | existence | |
| 3D-LLM | 33.2 | 4.1 | 1.1 | 0.1 | 9.6 | 5.5 | 24.6 | 24.2 | 44.2 |
| Chat-3D v2 | 23.2 | 2.7 | 0.5 | 0.0 | 6.6 | 4.3 | 14.6 | 16.7 | 21.2 |
| LEO | 46.9 | 8.1 | 1.6 | 0.4 | 14.3 | 11.8 | 23.2 | 20.0 | 36.5 |
| Chat-Scene | 43.3 | 7.1 | 1.2 | 0.1 | 12.9 | 10.1 | 23.7 | 16.7 | 40.4 |
| GPT4Scene | 55.5 | 14.3 | 2.5 | 0.5 | 18.2 | 15.8 | 25.2 | 31.7 | 36.5 |
Table 3: Rotation Robustness Comparison on Real-3DQA. The table shows the performance of different 3D-LLMs (using refined exact match metric) when tested with varying numbers of correct rotations. All models demonstrate a clear performance degradation as the required number of correct rotations increases, highlighting the challenge of rotation robustness in 3D understanding.
Figure 5: 3D tokens attention score after 3DR-FT.
Figure 6: Why 3DR-FT reduces SQA3D performance? For Chat-Scene, 591 questions flip from correct to wrong after 3DR-FT; 441 of these come from the Filtered Set (green box with diagonal hatching). Because SQA3D mixes 3D-dependent questions (Real-3DQA) with 3D-independent ones (Filtered Set), emphasizing 3D evidence via 3DR-FT can hurt the latter set, lowering the overall SQA3D score.
| Training Strategy | LEO | LEO | Chat-Scene | |||
|---|---|---|---|---|---|---|
| ScanQA | Real-ScanQA | SQA3D | Real-3DQA | SQA3D | Real-3DQA | |
| Supervised FT | 32.3 | 6.1 | 52.2 | 19.1 | 57.2 | 22.1 |
| Blind FT | 33.0 | 5.9 | 50.6 | 13.6 | 51.4 | 14.4 |
| 3D-reweighted FT | 31.3 | 13.9 | 48.2 | 29.3 | 48.9 | 33.9 |
Table 4: Ablation Study on Training Strategies. Columns group results by model and dataset: LEO on ScanQA/Real-ScanQA (left) and SQA3D/Real-3DQA (center), and Chat-Scene on SQA3D/Real-3DQA (right). 3D-reweighted fine-tuning (3DR-FT) delivers consistent gains across both datasets and models , with the largest improvements on the 3D-dependent sets—Real-3DQA and Real-ScanQA—while Supervised FT remains strongest on SQA3D.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| 3D-LLMs | EM | EM_R | measurement | color | number | spatial relation | shape | state | object | visibility | navigation | reasoning | other |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Refined EM on SQA3D | |||||||||||||
| 3D-LLM | 47.8 | 49.6 | 69.2 | 45.7 | 52.4 | 35.7 | 59.2 | 64.7 | 42.4 | 63.5 | 41.5 | 40.0 | 39.5 |
| Chat-3D v2 | 45.0 | 48.1 | 36.1 | 57.0 | 56.0 | 43.2 | 67.8 | 71.5 | 60.4 | 71.1 | 45.7 | 50.0 | 88.4 |
| LEO | 49.6 | 52.2 | 73.1 | 58.1 | 52.8 | 40.8 | 59.9 | 69.9 | 52.8 | 66.4 | 41.5 | 35.0 | 76.7 |
| Chat-Scene | 54.7 | 57.4 | 80.8 | 58.1 | 57.2 | 46.8 | 66.5 | 76.0 | 59.2 | 70.0 | 48.1 | 40.0 | 81.4 |
| GPT4Scene | 60.6 | 63.3 | 76.9 | 70.8 | 65.8 | 52.6 | 73.7 | 83.7 | 68.8 | 73.3 | 43.3 | 52.5 | 79.1 |
Table 5: Model ability across different question types. We compare model performance on both the original SQA3D benchmark and our more challenging Real-3DQA. On Real-3DQA, we find most 3D-LLMs have consistent deficiencies in spatial relationships, shape recognition and reasoning tasks. We report refined exact match for detailed question types.
Figure 7: Comparison of 3D-LLMs capabilities across different question types on Real-3DQA. The radar chart visualizes the performance of five models (LEO, Chat-Scene, 3D-LLM, Chat-3D v2, GPT4Scene) on ten question categories. Each axis represents a specific question type, and the distance from the center indicates the model’s performance in that category. The area covered by each model’s polygon reflects its overall capability across all question types. The chart highlights that GPT4Scene demonstrates superior performance in all categories.
Figure 8: Rotation robustness comparison. All models show a consistently increasing decline in EM Refined as the viewpoint changes from one to four, underscoring the models’ difficulty in maintaining consistency across multiple viewpoints.
| Situation | Question | 3D | EM (%) | EM_R (%) |
|---|---|---|---|---|
| ✓ | ✓ | ✓ | 49.4 | 52.2 |
| ✓ | ✓ | shuffled | 44.2 | 46.7 |
| ✓ | ✓ | ✗ | 32.4 | 43.3 |
| ✗ | ✓ | ✓ | 49.3 | 51.7 |
| ✓ | ✗ | ✓ | 0.2 | 10.4 |
| ✗ | ✓ | ✗ | 18.6 | 30.1 |
Table 6: Where does the 3D prior come from in LEO Huang et al. (2024b) on SQA3D? We conduct this ablation during only the inference time, using checkpoints provided by the authors. Results show that 3D information and questions are crucial for performance, while situation descriptions have minimal impact. We report both exact match (EM) and refined exact match (EM_Refined) metrics.
| Filter Type | Original Test | GPT | Model Comparison | ||
|---|---|---|---|---|---|
| LEO | Chat-Scene | 3D-LLM | |||
| Filtered | - | 112 | 1197 | 459 | 232 |
| Remaining | 3519 | 3407 | 2176 | 1717 | 1485 |
Table 7: Statistics of filtered questions for each filtering step. The table reports the number of questions filtered and those remaining for the original test set, GPT-based filtering, and model-based comparisons filtering across LEO, Chat-Scene, and 3D-LLM.
Figure 9: Prompt Template for Rotation-Augmented Question Generation. We design this template to guide GPT-4o-mini in creating perspective-rotated variants of the original questions. The template instructs the model to preserve the original question’s intent while altering the spatial reference frame according to specified viewpoint changes.
Figure 10: Example of a view-specific question that cannot be rotated. This question demonstrates a case where rotation is invalid because it references objects (window frames) that only exist in the current viewpoint (to my right). When the perspective is rotated, these referenced objects are no longer visible or identifiable, rendering the question meaningless in the new viewpoint.
More in this topic
3D Spatial Reasoning
3D spatial reasoning studies how AI systems infer geometry, object relations, viewpoints, and motion to understand and act in three-dimensional scenes.
See all 201 papers