cs.CVApr 8, 2026
SaveEnhancing MLLM Spatial Understanding via Active 3D Scene Exploration for Multi-Perspective Reasoning
Organizations: Tsinghua University · Nanjing University · Tencent
Abstract
Although Multimodal Large Language Models have achieved remarkable progress, they still struggle with complex 3D spatial reasoning due to the reliance on 2D visual priors. Existing approaches typically mitigate this limitation either through computationally expensive post-training procedures on limited 3D datasets or through rigid tool-calling mechanisms that lack explicit geometric understanding and viewpoint flexibility. To address these challenges, we propose a training-free framework that introduces a Visual Chain-of-Thought mechanism grounded in explicit 3D reconstruction. The proposed pipeline first reconstructs a high-fidelity 3D mesh from a single image using MLLM-guided keyword extraction and mask generation at multiple granularities. Subsequently, the framework leverages an external knowledge base to iteratively compute optimal camera extrinsic parameters and synthesize novel views, thereby emulating human perspective-taking. Extensive experiments demonstrate that the proposed approach significantly enhances spatial comprehension. Specifically, the framework outperforms specialized spatial models and general-purpose MLLMs, including GPT-5.2 and Gemini-2.5-Flash, on major benchmarks such as 3DSRBench and Rel3D.
Figures & tables
Figure 1: Effectiveness of the proposed Visual CoT method driven by 3D reconstruction. Vanilla MLLMs (left) often fail to answer complex spatial questions correctly, including height comparisons, location assessments, and facing orientations, due to perspective ambiguities and reliance on 2D visual priors derived from a single static image. In contrast, the proposed approach (right) reconstructs the underlying 3D scene and applies perspective transformations to synthesize novel and geometrically unambiguous viewpoints ( e.g . , head-on, top-down, or specific relative views). By dynamically transitioning to these informative perspectives, the proposed method resolves spatial ambiguities and enables accurate 3D spatial reasoning.
Figure 2: Architecture of the proposed end-to-end framework for spatial comprehension, comprising the 3D reconstruction and perspective transformation modules. The reconstruction phase employs MLLMs to extract multi-granularity keywords, guiding mask generation and IoU deduplication to construct a three-dimensional mesh. The perspective transformation phase integrates an external knowledge base for optimal viewpoint selection, computes the extrinsic parameters of the camera within a unified world coordinate system, and executes an iterative view synthesis loop with a re-selection mechanism to generate the final reasoning response.
Figure 3: Distribution of sub-task types across different benchmarks (3DSR, Rel3D, and SpatialScore). The horizontal axis indicates the relative proportion of each task category, whereas the absolute number of samples ( ) for each benchmark is annotated on the right. A significant bias towards the Orientation task is observable in the SpatialScore dataset. The abbreviations for the sub-tasks denote the following: Hei. (Height), Loc. (Location), Ori. (Orientation), MO. (Multi-Object), Rea. (Reach), Con. (Containment), Pos. (Position), Dis. (Distance), and Obs. (Observer).
| 3DSR | SpatialScore | Rel3D | ||||||||||||
| Method | Hei. | Loc. | Ori. | MO. | Avg. | Ori. | Rea. | Avg. | Con. | Pos. | Dis. | Obs. | Ori. | Avg. |
| Open-source Models | ||||||||||||||
| LLaVa-1.6-Mistral-7B [ 34 ] | 51.1 | 57.5 | 40.2 | 46.3 | 49.5 | 60.7 | 68.4 | 60.8 | 57.4 | 41.3 | 57.8 | 34.5 | 34.3 | 45.3 |
| Cambrian-1-8B [ 49 ] | 54.9 | 64.2 | 43.4 | 52.3 | 54.9 | 62.6 | 57.9 | 62.6 | 53.7 | 34.6 | 41.9 | 34.8 | 35.2 | 41.0 |
| LLaVA-OneVision-1.5-8B [ 1 ] | 52.9 | 60.5 | 43.2 | 47.4 | 51.7 | 48.5 | 10.5 | 48.2 | 62.1 | 49.2 | 46.7 | 68.7 | 51.4 | 56.3 |
| InternVL3.5-8B [ 50 ] | 59.7 | 72.7 | 48.4 | 51.2 | 58.9 | 72.2 | 47.4 | 72.1 | 62.4 | 47.6 | 27.4 | 62.5 | 43.4 | 52.5 |
Table 1: Performance comparison of various models on the 3DSR, SpatialScore, and Rel3D benchmarks, categorized by model type. The symbol denotes models that exclusively process video inputs; during evaluation, static images are formatted as single-frame videos for these baselines. denotes a state-of-the-art model with 100B active parameters (total estimated larger), trained on 30T tokens with an estimated – FLOPs. Abbreviations follow the definitions provided in Fig. 3 . The best and second-best results for each metric are indicated in bold and underlined text, respectively. All results denote accuracy (%).
| 3DSR | SpatialScore | Rel3D | ||||||||||||
| Method | Hei. | Loc. | Ori. | MO. | Avg. | Ori. | Rea. | Avg. | Con. | Pos. | Dis. | Obs. | Ori. | Avg. |
| Qwen3-VL-8B | 56.7 | 72.4 | 49.3 | 54.1 | 60.4 | 73.6 | 57.9 | 73.5 | 67.6 | 53.0 | 32.2 | 72.9 | 51.8 | 59.0 |
| Ours | 68.6 | 84.8 | 54.9 | 67.1 | 70.7 | 77.7 | 68.4 | 77.6 | 72.0 | 63.0 | 45.5 | 76.1 | 65.7 | 66.4 |
| w/o View Reselect | 62.8 | 80.7 | 52.4 | 63.3 | 66.8 | 74.6 | 73.4 | 74.6 | 66.3 | 53.6 | 33.1 | 68.1 | 53.8 | 58.4 |
| w/o Knowledge Base | 54.1 | 77.4 | 51.7 | 53.4 | 61.2 | 77.3 | 63.2 | 77.2 | 69.4 | 58.0 | 41.3 | 75.8 | 50.8 | 62.1 |
| w/o Coordinate Layout | 59.9 | 73.7 | 51.2 | 56.2 | 61.5 | 75.5 | 63.2 | 75.5 | 67.1 | 54.4 | 35.5 | 68.3 | 52.2 | 58.9 |
Table 2: Performance of the base model, the proposed method, and the corresponding variants on the 3DSR , SpatialScore , and Rel3D benchmarks, with the View Reselect , Knowledge Base , and Coordinate Layout modules individually ablated. The best and second-best results for each metric are indicated in bold and underlined , respectively. All results represent accuracy (%). The abbreviation w/o denotes “without”.
Figure 5: The reasoning pipeline of the proposed method on two representative examples from the dataset. Each box presents the output information from the MLLM alongside the final answer to the question.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| No. | Task | Strategy / Guideline |
| 1 | Hei. | When comparing heights, select a position where all objects are visible, view them at eye level, and compare which is the tallest. |
| 2 | M.O. 1 | To determine which side an object is on (front, back, left, right), first identify the central object in question, then switch to a viewpoint behind it, i.e., transform into the coordinate system of that object. |
| 3 | M.O. 2 | To judge whether objects are parallel, select a viewpoint that clearly shows the long edges of the objects, and check if the two long edges are parallel. |
| 4 | Loc. 1 | When the question relates to the camera, the camera position should be at least the same as in the original image. |
| 5 | Loc. 2 | To judge whether an object is directly above another, use a top-down view and project the objects onto the horizontal plane; an occlusion relationship indicates that one is directly above the other. |
| 6 | Loc. 3 | To determine whether two objects are close, choose the viewpoint that reveals their maximum distance, then check if this maximum distance exceeds the diameter of the objects; if so, the objects may be considered close. |
Table 3: Knowledge Base for spatial and relational reasoning. Here, Hei. denotes Height_Higher , M.O. 1 denotes multi_object_viewpoint_towards_objec , M.O. 2 denotes multi_object_parallel , Loc. 1 denotes location_closer_to_camera , Loc. 2 denotes location_above , Loc. 3 denotes location_next_to , and Task. denotes Task Generalization.
| Benchmark | Subtask | Proportion | Specific Question Types / Task Composition |
| 3DSR | Loc. | 33.34% | location_next_to , location_closer_to_camera , location_above |
| (N=2,625) | MO. | 33.33% | multi_object_same_direction , multi_object_viewpoint_towards_object , multi_object_facing , multi_object_closer_to , multi_object_parallel |
| Ori. | 20.00% | orientation_in_front_of , orientation_viewpoint , orientation_on_the_left | |
| Hei. | 13.33% | Height_Higher | |
| SpatialScore | Ori. | 99.24% | 3D Positional Relation , Orientation |
| (N=2,530) | Rea. | 0.76% | Object Properties , Reachability |
Table 4: Detailed configurations of the question types and task compositions for the 3DSRBench, SpatialScore, and Rel3D benchmarks. The abbreviations follow the definitions provided in Fig. 3 .
More in this topic
3D Spatial Reasoning
3D spatial reasoning studies how AI systems infer geometry, object relations, viewpoints, and motion to understand and act in three-dimensional scenes.
See all 197 papers