Spatial perception and reasoning from visual observations require recovering geometric structure, establishing correspondences, and understanding spatial relations. Existing approaches typically address these capabilities separately using task-specific architectures or external geometric modules, limiting knowledge transfer among complementary representations of the same physical scene. We introduce SPARGen, a unified multimodal framework that casts 3D reconstruction, dense correspondence, and spatial reasoning as instruction-conditioned generation tasks. SPARGen serializes compact structured and linguistic outputs as token sequences while generating dense geometric fields in image-aligned forms, enabling spatial supervision to jointly shape shared representations within a native multimodal generative model. Experiments across benchmarks for 3D reconstruction, correspondence, and spatial reasoning show that SPARGen achieves competitive performance across heterogeneous spatial tasks within a single native multimodal generative framework.
Figures & tables
Figure 2: Overview of SPARGen. Given a sequence of RGB images and a language instruction, SPARGen represents spatial outputs as sequences or image-aligned fields. These two output formats are generated through the native autoregressive and rectified-flow pathways of a shared MoT backbone. The dashed box denotes the training-only encoding of target fields.
Model
Depth Estimation
Point Map Estimation
Camera Pose Estimation
Sintel
NYU-v2
7Scenes
ETH3D
Co3Dv2
AbsRel ↓
δ1↑
AbsRel ↓
δ1↑
Acc. ↓
Comp. ↓
Acc. ↓
Comp. ↓
RRA@30 ↑
RTA@30 ↑
AUC@30 ↑
Specialized Geometry Models
FLARE ( Zhang et al. 2025b )
0.409
0.438
0.164
0.740
0.035
0.043
0.470
0.667
97.08
94.31
77.99
DUSt3R ( Wang et al. 2024b )
0.362
0.557
0.134
0.833
0.026
0.035
0.360
0.401
98.34
94.09
79.33
VGGT ( Wang et al. 2025a )
0.265
0.676
0.065
0.938
0.022
0.032
0.311
0.372
98.79
96.89
89.78
Table 1: Comparison across depth estimation, point map estimation, and camera pose estimation benchmarks. Bold indicates the best performance within each model group.
SPARGen Model
MindCube
OmniSpatial
OST
SPAR
Avg.
Rotation
Among
Around
Avg.
SI
PT
Avg.
A. State
A. Info
AO.
Avg.
Low
Medium
High
Proprietary Models
GPT-4o ( Hurst et al. 2024 )
41.52
34.50
41.67
46.80
42.39
48.67
39.04
51.42
40.93
68.38
39.25
37.88
35.36
28.40
43.27
Claude Sonnet 4.6 ( Anthropic 2026 )
44.19
42.50
43.00
48.40
53.89
67.00
46.88
42.18
35.84
63.30
29.35
37.55
36.63
35.33
39.20
Open-Source Models
Qwen2.5-VL-7B ( Bai et al. 2025 )
36.00
37.50
32.33
43.60
45.99
54.67
41.35
40.08
38.48
50.20
31.27
34.84
26.36
36.09
41.97
Table 2: Results on spatial-reasoning benchmarks. Bold and underlined values indicate the best and second-best results among non-proprietary models.
Table 4
Figure 3: Qualitative results of SPARGen on point map reconstruction, optical flow estimation, and depth estimation. For optical flow, we show the predictions and error maps before and after refinement.
Precise spatial understanding from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs), as their visual representations are predominantly semantic and lack explicit geometric grounding. While existing approaches augment visual tokens with geometric cues from visual geometry models, their MLLM is still required to implicitly infer the underlying 3D structure of the scene from these augmented tokens, limiting its spatial reasoning capability. To address this issue, we introduce Cog3DMap, a framework that recurrently constructs an explicit 3D memory from multi-view images, where each token is grounded in 3D space and possesses both semantic and geometric information. By feeding these tokens into the MLLM, our framework enables direct reasoning over a spatially structured 3D map, achieving state-of-the-art performance on various spatial reasoning benchmarks. Code will be made publicly available.
Spatial intelligence remains a persistent challenge for Multimodal Large Language Models (MLLMs), as it requires coherent spatial scene representations beyond basic object recognition. Existing methods typically build such representations through textual reasoning or 3D reconstruction. However, they often falter during multi-step reasoning, particularly when required to dynamically re-anchor evidence to the specific camera-, object-, or direction-centric reference frames demanded by complex queries. To address this, we propose OmniView-Space, a framework designed to maintain spatial consistency through multimodal egocentric evidence. Our approach consists of three core components: (1) Multi-Perspective Spatial Mapping (MPSM), which re-anchors reconstructed geometry into a query-aligned visual cognitive map and a textual spatial graph; (2) Tool-Guided Egocentric Reasoning, an interleaved policy trained to actively select the ego anchor required by the query and request the corresponding MPSM evidence; and (3) Cognitive-Map Distillation, which uses MPSM-generated trajectories and ego-frame rewards to train the model to reason with self-generated cognitive maps. Experiments on single- and multi-image spatial reasoning benchmarks show that OmniView-Space achieves state-of-the-art performance. Furthermore, the distilled model maintains this performance while reducing reliance on external geometry pipelines.
Xudong Li, Mengdan Zhang, Peixian Chen +7
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China · Tencent Youtu Lab · Beijing Institute of Technology
Reasoning about the 3D world from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs). While modern MLLMs handle single-image inputs effectively, they struggle to integrate evidence across viewpoints into a coherent 3D understanding. A growing body of work attempts to close this gap by injecting 3D awareness into MLLMs, either by boosting fine-grained pixel-level cross-view correspondence or by fusing features from 3D geometry foundation models, yet a substantial gap to human reasoning persists. In this work, we revisit human spatial reasoning, which suggests that rather than relying on fine-grained geometry cues, humans roughly identify common objects across views, infer the relative geometry between viewpoints, and assemble a coarse 3D layout of the scene. Inspired by this process, we introduce Imagine3D-LLM, an MLLM that learns to assemble a similar compact 3D representation of the scene and conditions its answer on this representation. Concretely, we append a small set of learnable summary tokens after the image tokens, decode them into a compact 3D Gaussian Splatting representation supervised by a photometric reconstruction loss, and train jointly with the standard next-token prediction objective. Notably, although only the summary tokens receive direct reconstruction supervision, this objective also induces stronger cross-frame correspondence within the LLM's underlying image features, suggesting that learning to reconstruct propagates 3D-aware signals throughout the model. As a result, Imagine3D-LLM consistently outperforms prior approaches across multiple spatial reasoning and 3D understanding benchmarks, suggesting that imagining the scene can be more effective than being told its pixel-wise geometry.