cs.CVOct 27, 2026

Zhengrong-Spatial-Aware 1.0: Advancing 3D Multimodal Intelligence

Authors: Yu Zhang, Xiaogang Wang, Yuhang Cheng, Jingrui Li, Wenbo Chen

Organizations: Beijing Freedo Technology Co., Ltd. · Southwest University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

This report presents the implementation and validation of two end-to-end multimodal adaptation pipelines for 3D-aware visual question answering on a custom spatial dataset. The first pipeline fine-tunes HiSpatial-3B, which augments RGB observations with metric-scale 3D point maps; the second adapts Qwen3-VL through memory-efficient QLoRA-based supervised fine-tuning. The completed engineering stack covers dataset conversion, image-annotation alignment, monocular geometry preprocessing, instruction construction, mixed-domain training, checkpoint management, batch inference, and automated evaluation. On the target spatial benchmark, HiSpatial adaptation increases overall accuracy from 1.73% before fine-tuning to 79.89% at 3,600 steps. On a separately collected styled split, accuracy improves from 14.9% to 55.1%, indicating transfer beyond the original training set. The Qwen3-VL pipeline is likewise implemented end-to-end for both 8B- and 32B-scale models using mixed-domain QLoRA adaptation. The resulting accuracy improves from 20.74% to 45.27% for Qwen3-VL-8B and from 30.79% to 47.60% for Qwen3-VL-32B. Together, these results establish two complementary and reproducible adaptation pipelines for geometry-aware and general-purpose multimodal spatial reasoning. These results establish a reproducible engineering foundation for subsequent work on metric 3D reasoning, spatial grounding, and domain-adapted multimodal systems, while also identifying training-data composition as a central factor in preserving general multimodal capability during specialization.

Figures & tables

Fig. 1: Reference model architectures used in this project. The HiSpatial architecture integrates RGB appearance with a metric-scale point map for geometry-aware spatial reasoning, while Qwen3-VL employs a vision encoder, vision–language merger, and Qwen3 decoder with DeepStack-based multi-level visual feature injection [ 1 , 2 ] .
CapabilityRepresentative questions
Countingnumber of buildings in a scene
Metric estimationbuilding height; pairwise center distance
Extremum reasoningtallest/lowest building; largest/smallest footprint
Orderingfootprint ranking; left-to-right and top-to-bottom order
Relational reasoningfarthest pair; relative spatial direction
TABLE I: Target spatial VQA tasks in the provided dataset.
Fig. 2: Example visual input from the provided spatial dataset. The engineering pipeline can consume original RGB images as well as styled or region-highlighted variants.
Fig. 3: Implemented fine-tuning and evaluation pipeline. The two model-specific branches share the same task supervision and evaluation interface but differ in geometric preprocessing and adaptation strategy.
Checkpoint / modelOverall accuracy (%)
Released HiSpatial before adaptation1.73
Fine-tuned, 1,200 steps64.69
Fine-tuned, 2,400 steps73.90
Fine-tuned, 3,600 steps79.89
TABLE II: Recorded HiSpatial accuracy on the original target dataset during fine-tuning.
ModelBefore FTAfter FT
Qwen3-VL-8B20.74%45.27%
Qwen3-VL-32B30.79%47.60%
TABLE III: Qwen3-VL accuracy before and after mixed-domain QLoRA fine-tuning.
ComponentStatus
Dataset JSONL conversion and image alignmentComplete
Bounding-box / styled-image input supportComplete
MoGe point-map preprocessingComplete
HiSpatial WebDataset constructionComplete
HiSpatial mixed-domain full fine-tuningComplete
HiSpatial checkpoint inference/evaluationComplete
TABLE IV: Current engineering status.
More in this topic

3D VQA

Three-dimensional visual question answering uses spatial and geometric evidence to answer questions about scenes, objects, and volumetric images.

See all 51 papers

Explore similar work

CardsList
Aug 5, 2026

Trace, Verify, and Correct: A Training-Free Framework for Spatial Reasoning in Multimodal LLMs

Although Multimodal Large Language Models (MLLMs) have made substantial progress, their spatial reasoning may still produce intermediate judgments inconsistent with the input image, allowing errors to propagate through the reasoning chain and affect the final answer. Existing methods mainly improve spatial reasoning through training or additional spatial information, without considering whether the reasoning process itself is faithful to the model input. Our study shows that unfaithful reasoning chains significantly reduce final-answer accuracy. To address this issue, we propose a modular and training-free framework for spatial reasoning verification and correction. The framework constructs a Spatial Evidence Graph (SEG), which associates atomic spatial evidence extracted from Chain-of-Thought reasoning with visual entities, spatial relations, source steps, and visual evidence. Spatial Evidence Reliability Assessment (SERA) evaluates the reliability of visual evidence based on object existence, localization, and geometric measurements. The framework then identifies the earliest spatial evidence unit contradicted by reliable visual evidence and guides the original MLLM to revise the subsequent reasoning and final answer. Across 15 model-dataset settings, our method achieves an average accuracy of 68.94%, outperforming the compared baselines by 8.55 percentage points on average. Our code will be open-sourced.

Yang Yang, Jiawei Chen, Tairan Chen +1

Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University · Zhongguancun Academy · Stevens Institute of Technology

Multimodal GroundingVisual Spatial ReasoningMultimodal Large Language ModelsMultimodal CoT ReasoningSpatial Reasoning in Language ModelsMultimodal Spatial Reasoning
AskPDF
Jul 7, 2026

SpaR3D-MoE: Adaptive 3D Spatial Reasoning from Sparse Views Meets Geometry-Inductive Mixture-of-Experts

Recent Multimodal Large Language Models (MLLMs) struggle to bridge the representational gap between 2D semantic understanding and 3D spatial geometry. Existing 3D-aware models either rely on costly 3D-specific data or utilize RGB-only inputs with heuristic sampling and monolithic, shallow fusion, which respectively disrupt essential spatiotemporal connectivity and induce modality contention across diverse spatial tasks. To overcome these bottlenecks, we introduce SpaR3D-MoE, an end-to-end framework that enables adaptive spatial reasoning by equipping MLLMs with geometry-aware capabilities from only sparse RGB inputs. First, we propose an adaptive spatiotemporal manifold sampling mechanism that constructs a geometry-aware spatiotemporal graph to extract informative keyframes, effectively mitigating sequence redundancy while preserving the scene's topological connectivity. Second, we introduce the heterogeneous geometry-inductive Mixture-of-Experts driven by an instruction-pose aware router, which adaptively routes multimodal tokens to specialized experts, resolving the cross-modal contention inherent in monolithic fusion. Extensive experiments on VSI-Bench, ScanQA, and SQA3D demonstrate that our method achieves state-of-the-art performance. Notably, SpaR3D-MoE achieves the highest average score of 63.5 on VSI-Bench, outperforming the strongest baseline by 7.8 absolute points, alongside relative improvements of 35.4% and 51.4% in Route Plan and Relative Direction tasks, respectively.

Haida Feng, Hao Wei, Haolin Wang +3

State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, China · YUKUN Intelligent World, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, China

3D Spatial ReasoningMixture of Experts3D VLMs
AskPDF
Mar 26, 2026

HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Models

Achieving human-like spatial intelligence for vision-language models (VLMs) requires inferring 3D structures from 2D observations, recognizing object properties and relations in 3D space, and performing high-level spatial reasoning. In this paper, we propose a principled hierarchical framework that decomposes the learning of 3D spatial understanding in VLMs into four progressively complex levels, from geometric perception to abstract spatial reasoning. Guided by this framework, we construct an automated pipeline that processes approximately 5M images with over 45M objects to generate 3D spatial VQA pairs across diverse tasks and scenes for VLM supervised fine-tuning. We also develop an RGB-D VLM incorporating metric-scale point maps as auxiliary inputs to further enhance spatial understanding. Extensive experiments demonstrate that our approach achieves state-of-the-art performance on multiple spatial understanding and reasoning benchmarks, surpassing specialized spatial models and large proprietary systems such as Gemini-2.5-pro and GPT-5. Moreover, our analysis reveals clear dependencies among hierarchical task levels, offering new insights into how multi-level task design facilitates the emergence of 3D spatial intelligence.

Huizhi Liang, Yichao Shen, Yu Deng +5

Tsinghua University · Microsoft Research Asia · Xi’an Jiaotong University +1

3D Spatial Reasoning3D VLMs3D VQA
AskPDF