Boosting MLLM Spatial Reasoning with Geometrically Referenced 3D Scene Representations

Jiangye Yuan; Gowri Kumar; Baoyuan Wang

cs.CVMar 9, 2026

Boosting MLLM Spatial Reasoning with Geometrically Referenced 3D Scene Representations

Authors: Jiangye Yuan, Gowri Kumar, Baoyuan Wang

Organizations: Zillow Group

Abstract

While Multimodal Large Language Models (MLLMs) have achieved remarkable success in 2D visual understanding, their ability to reason about 3D space remains limited. To address this gap, we introduce geometrically referenced 3D scene representations (GR3D). Given a set of input images, GR3D annotates objects in the images with unique IDs and encodes their 3D geometric attributes as textual references indexed by these IDs. This representation enables MLLMs to interpret 3D cues using their advanced language-based skills in mathematical reasoning, while concurrently analyzing 2D visual features in a tightly coupled way. We present a simple yet effective approach based on GR3D, which requires no additional training and is readily applicable to different MLLMs. Implemented in a zero-shot setting, our approach yields substantial improvements on challenging spatial reasoning benchmarks, boosting GPT-5 performance by 9% on VSI-Bench and 12% on MindCube. Qualitative studies further demonstrate that GR3D empowers MLLMs to perform complex spatial reasoning with highly sparse input views.

Figures & tables

Figure 1: An overview of GR3D framework. Given a collection of images, our method reconstructs 3D scenes, extracts object-level geometric attributes, and transforms them into a GR3D representation, which consists of annotated images and textual references of geometric attributes. Such paired text and images are provided to a MLLM to perform spatial reasoning tasks.

Figure 2: Object annotation with occlusion check. Left: initial object annotation through projection. Middle: depth map from 3D reconstruction. Right: object annotation after depth-based occlusion check.

Figure 3: Prompt template used in evaluations.

Method Name	Obj. Count	Abs. Dist.	Obj. Size	Room Size	Rel. Dist.	Rel. Dir.	Route Plan	Appr. Order	Avg.
InternVL2-8B [ 6 ]	31.3	29.0	48.9	44.2	38.0	33.4	28.9	46.4	37.5
VG LLM-8B [ 43 ]	67.9	37.7	58.6	62.0	46.6	40.7	32.4	59.2	50.7
Gemini-1.5 Pro [ 32 ]	56.2	30.9	64.1	43.6	51.3	46.3	36.0	34.6	45.4
GPT-4o-2024-08-06 [ 14 ]	46.2	5.3	43.8	38.2	37.0	41.3	31.5	28.5	34.0
GPT-5-2025-08-07 [ 14 ]	39.7	28.8	68.9	50.7	42.6	42.4	50.0	69.3	50.3
GR3D + InternVL2-8B	33.9	29.3	46.2	63.7	37.1	40.3	30.1	53.7	42.5

Table 1: VSI-Bench evaluation results. GR3D combined with GPT-5 achieves state-of-the-art performance. The best result for each task among all models is indicated in bold .

Method Name	Rotation	Among	Around	Overall
LLaVA-Onevision-7B [ 18 ]	36.5	48.4	44.1	47.4
DeepSeek-VL2-Small [ 22 ]	37.0	50.4	26.9	47.6
GPT-5-2025-08-07 [ 14 ]	87.5	39.3	66.4	55.0
GR3D + GPT-5-2025-08-07	88.5	55.7	76.0	66.8

Table 2: MindCube evaluation results. Rotation, among, and around represent subsets with different camera placements. GR3D combined with GPT-5 achieves the highest accuracy. The best result for each subset across all models is indicated in bold .

Method	Rel. Dir.	Route Plan	Avg.
GR3D	69.5	58.7	59.8
No object annotation	51.3	52.0	50.3
VGGT reconstruction	62.2	56.7	58.3
Camera parameters	42.6	45.4	48.8
Scene descriptions	51.9	51.0	53.4

Table 3: Ablation study. We evaluate 1) removing object annotations in images, 2) replacing

\pi^{3}

with VGGT, 3) providing camera parameters to implicitly associate images with object geometric information, and 4) converting images into scene descriptions with object ID citations. For brevity, we show the scores of two configurational tasks and the overall average.

Figure 4: Example results of sparse view spatial reasoning. The part with incorrect reasoning is highlighted in red .

3D Spatial Reasoning

3D spatial reasoning studies how AI systems infer geometry, object relations, viewpoints, and motion to understand and act in three-dimensional scenes.

See all 198 papers

Enhancing MLLM Spatial Understanding via Active 3D Scene Exploration for Multi-Perspective Reasoning
Apr 8, 2026Jiahua Chen, Qihong Tang, Weinong Wang +13D Spatial Reasoning VLM Reasoning
Tsinghua University · Nanjing University · Tencent
Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering
Sep 29, 2026Jaewoo Jung, Hyeonseo Yu, Honggyu An +103D Spatial Reasoning 3D Scene Representation
KAIST AI · ETH Zürich · Google +2
What Makes a Good Visual Representation for Spatial Understanding in MLLMs?
Oct 13, 2026Xiaohu Huang, Yumin Guo, Kai HanMultimodal Spatial Reasoning Multimodal Large Language Models
The University of Hong Kong

Apr 8, 2026J/K move · Enter open · S save

Enhancing MLLM Spatial Understanding via Active 3D Scene Exploration for Multi-Perspective Reasoning

Jiahua Chen, Qihong Tang, Weinong Wang, Qi Fan

Tsinghua University · Nanjing University · Tencent

3D Spatial Reasoning VLM Reasoning Visual CoT

Open paper AskPDF arXiv

Although Multimodal Large Language Models have achieved remarkable progress, they still struggle with complex 3D spatial reasoning due to the reliance on 2D visual priors. Existing approaches typically mitigate this limitation either through computationally expensive post-training procedures on limited 3D datasets or through rigid tool-calling mechanisms that lack explicit geometric understanding and viewpoint flexibility. To address these challenges, we propose a training-free framework that introduces a Visual Chain-of-Thought mechanism grounded in explicit 3D reconstruction. The proposed pipeline first reconstructs a high-fidelity 3D mesh from a single image using MLLM-guided keyword extraction and mask generation at multiple granularities. Subsequently, the framework leverages an external knowledge base to iteratively compute optimal camera extrinsic parameters and synthesize novel views, thereby emulating human perspective-taking. Extensive experiments demonstrate that the proposed approach significantly enhances spatial comprehension. Specifically, the framework outperforms specialized spatial models and general-purpose MLLMs, including GPT-5.2 and Gemini-2.5-Flash, on major benchmarks such as 3DSRBench and Rel3D.

Abstract

3D Spatial Reasoning

Explore similar work

Enhancing MLLM Spatial Understanding via Active 3D Scene Exploration for Multi-Perspective Reasoning

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering

What Makes a Good Visual Representation for Spatial Understanding in MLLMs?