SPARGen: Unifying Spatial Perception and Reasoning through Native Multimodal Generation

Jinsheng Quan; Jianhua Li; Siyi Xie; Xuanke Shi; Kewang Deng; Zukai Chen; Feifei Shao; Lei Yang

cs.CVAug 14, 2026

SPARGen: Unifying Spatial Perception and Reasoning through Native Multimodal Generation

Authors: Jinsheng Quan, Jianhua Li, Siyi Xie, Xuanke Shi, Kewang Deng, Zukai Chen, Feifei Shao, Lei Yang,

Organizations: Zhejiang University · SenseTime Research · Peking University

Abstract

Spatial perception and reasoning from visual observations require recovering geometric structure, establishing correspondences, and understanding spatial relations. Existing approaches typically address these capabilities separately using task-specific architectures or external geometric modules, limiting knowledge transfer among complementary representations of the same physical scene. We introduce SPARGen, a unified multimodal framework that casts 3D reconstruction, dense correspondence, and spatial reasoning as instruction-conditioned generation tasks. SPARGen serializes compact structured and linguistic outputs as token sequences while generating dense geometric fields in image-aligned forms, enabling spatial supervision to jointly shape shared representations within a native multimodal generative model. Experiments across benchmarks for 3D reconstruction, correspondence, and spatial reasoning show that SPARGen achieves competitive performance across heterogeneous spatial tasks within a single native multimodal generative framework.

Figures & tables

Figure 2: Overview of SPARGen. Given a sequence of RGB images and a language instruction, SPARGen represents spatial outputs as sequences or image-aligned fields. These two output formats are generated through the native autoregressive and rectified-flow pathways of a shared MoT backbone. The dashed box denotes the training-only encoding of target fields.

Specialized Geometry Models
Model	Depth Estimation				Point Map Estimation				Camera Pose Estimation
	Sintel		NYU-v2		7Scenes		ETH3D		Co3Dv2
	AbsRel $\downarrow$	$\delta_{1}\uparrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$	Acc. $\downarrow$	Comp. $\downarrow$	Acc. $\downarrow$	Comp. $\downarrow$	RRA@30 $\uparrow$	RTA@30 $\uparrow$	AUC@30 $\uparrow$
FLARE ( Zhang et al. 2025b )	0.409	0.438	0.164	0.740	0.035	0.043	0.470	0.667	97.08	94.31	77.99
DUSt3R ( Wang et al. 2024b )	0.362	0.557	0.134	0.833	0.026	0.035	0.360	0.401	98.34	94.09	79.33
VGGT ( Wang et al. 2025a )	0.265	0.676	0.065	0.938	0.022	0.032	0.311	0.372	98.79	96.89	89.78

Table 1: Comparison across depth estimation, point map estimation, and camera pose estimation benchmarks. Bold indicates the best performance within each model group.

Proprietary Models
SPARGen Model	MindCube				OmniSpatial			OST				SPAR
SPARGen Model	Avg.	Rotation	Among	Around	Avg.	SI	PT	Avg.	A. State	A. Info	AO.	Avg.	Low	Medium	High
GPT-4o ( Hurst et al. 2024 )	41.52	34.50	41.67	46.80	42.39	48.67	39.04	51.42	40.93	68.38	39.25	37.88	35.36	28.40	43.27
Claude Sonnet 4.6 ( Anthropic 2026 )	44.19	42.50	43.00	48.40	53.89	67.00	46.88	42.18	35.84	63.30	29.35	37.55	36.63	35.33	39.20
Open-Source Models
Qwen2.5-VL-7B ( Bai et al. 2025 )	36.00	37.50	32.33	43.60	45.99	54.67	41.35	40.08	38.48	50.20	31.27	34.84	26.36	36.09	41.97

Table 2: Results on spatial-reasoning benchmarks. Bold and underlined values indicate the best and second-best results among non-proprietary models.

Table 4

Figure 3: Qualitative results of SPARGen on point map reconstruction, optical flow estimation, and depth estimation. For optical flow, we show the predictions and error maps before and after refinement.

Multimodal Spatial Reasoning

See all 42 papers

Cog3DMap: Multi-View Vision-Language Reasoning with 3D Cognitive Maps
Mar 24, 2026Chanyoung Gwak, Yoonwoo Jeong, Byungwoo Jeon +3Multimodal Spatial Reasoning 3D Spatial Reasoning
POSTECH · KAIST · RLWRLD
OmniView-Space: Reinforcing Spatial Reasoning via Multi-Perspective Spatial Mapping
Jul 1, 2026Xudong Li, Mengdan Zhang, Peixian Chen +7Visual Spatial Reasoning 3D Spatial Reasoning
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China · Tencent Youtu Lab · Beijing Institute of Technology
Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering
Sep 29, 2026Jaewoo Jung, Hyeonseo Yu, Honggyu An +103D Spatial Reasoning 3D Scene Representation
KAIST AI · ETH Zürich · Google +2

Mar 24, 2026J/K move · Enter open · S save

Cog3DMap: Multi-View Vision-Language Reasoning with 3D Cognitive Maps

Chanyoung Gwak, Yoonwoo Jeong, Byungwoo Jeon, Hyunseok Lee+2

POSTECH · KAIST · RLWRLD

Multimodal Spatial Reasoning 3D Spatial Reasoning VLM Reasoning 3D VLMs 3D Scene Representation

Open paper AskPDF arXiv

Precise spatial understanding from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs), as their visual representations are predominantly semantic and lack explicit geometric grounding. While existing approaches augment visual tokens with geometric cues from visual geometry models, their MLLM is still required to implicitly infer the underlying 3D structure of the scene from these augmented tokens, limiting its spatial reasoning capability. To address this issue, we introduce Cog3DMap, a framework that recurrently constructs an explicit 3D memory from multi-view images, where each token is grounded in 3D space and possesses both semantic and geometric information. By feeding these tokens into the MLLM, our framework enables direct reasoning over a spatially structured 3D map, achieving state-of-the-art performance on various spatial reasoning benchmarks. Code will be made publicly available.

Abstract

Multimodal Spatial Reasoning

Explore similar work

Cog3DMap: Multi-View Vision-Language Reasoning with 3D Cognitive Maps

OmniView-Space: Reinforcing Spatial Reasoning via Multi-Perspective Spatial Mapping

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering