SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Models
Authors: Yuechen Xie, Xiaoyan Zhang, Yicheng Shan, Hao Zhu, Rui Tang, Rong Wei, Mingli Song, Yuanyu Wan, Jie Song
Organizations: Zhejiang University · The University of Sydney · ManyCore · State Key Laboratory of Blockchain and Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security
Vision-Language Models (VLMs) have been increasingly applied in real-world scenarios due to their outstanding understanding and reasoning capabilities. Although VLMs have already demonstrated impressive capabilities in common visual question answering and logical reasoning, they still lack the ability to make reasonable decisions in complex real-world environments. We define this ability as spatial logical reasoning, which not only requires understanding the spatial relationships among objects in complex scenes, but also the logical dependencies between steps in multi-step tasks. To bridge this gap, we introduce Spatial Logical Question Answering (SpatiaLQA), a benchmark designed to evaluate the spatial logical reasoning capabilities of VLMs. SpatiaLQA consists of 9,605 question answer pairs derived from 241 real-world indoor scenes. We conduct extensive experiments on 41 mainstream VLMs, and the results show that even the most advanced models still struggle with spatial logical reasoning. To address this issue, we propose a method called recursive scene graph assisted reasoning, which leverages visual foundation models to progressively decompose complex scenes into task-relevant scene graphs, thereby enhancing the spatial logical reasoning ability of VLMs, outperforming all previous methods. Code and dataset are available at https://github.com/xieyc99/SpatiaLQA.
Figures & tables
Figure 1: Common VQA [ 21 ] typically involve recognizing visual content and factual knowledge, while common logical reasoning [ 71 ] focuses on abstract, symbolic problem-solving. Spatial logical reasoning, in contrast, requires integrating both spatial understanding and multi-step logical reasoning to accomplish tasks in real-world scenes.
Benchmarks
Modality
SU
LR
Real Scene
Answer Type
Multi-step
Precondition
LLM/VLM Scoring
Size
CLEVR [ 24 ]
I
✓
✗
✗
Open
✗
✗
✗
853.6K
GQA [ 21 ]
I
✓
✗
✓
Open
✗
✗
✗
> 1M
MMBench [ 36 ]
I
✓
✗
✓
MC
✗
✗
✓
3.2K
Spatial-MM [ 50 ]
I
✓
✗
✓
MC
✗
✗
✗
2.3K
SpatialRGPT-Bench [ 14 ]
I
✓
✗
✓
MC
✗
✗
✗
1.5K
Open3DVQA [ 72 ]
I
✓
✗
✗
Open
✗
✗
✓
9.0K
Table 1: Comparison between SpatiaLQA and other benchmarks ( VQA , logical reasoning and EQA ). ‘SU’ and ‘LR’ denote spatial understanding and long-range reasoning, respectively. ‘I’, ‘P’ and ‘V’ denote image, point cloud and video, respectively. ‘Open’ and ‘MC’ stand for open-vocabulary and multiple-choice respectively. ‘Multi-step’ specifies whether the answers involve multiple steps. ‘Precondition’ indicates whether each step in the answer is annotated with its preconditions (i.e., which steps must be completed beforehand).
Figure 2: Prompt template and examples of several indoor scenes.
Figure 3: The distributions of answer step counts, scene categories, and partial object categories in SpatiaLQA. The x-axes of the three plots represent the number of answer steps, indoor scene categories, and object categories, while the y-axes indicate the number of samples.
Figure 4: The data collection pipeline for SpatiaLQA. Note that although the graph expansion augmentation in the figure is applied only to the data from subgraph extraction augmentation, we actually also applied graph expansion augmentation to the manually annotated data.
Figure 5: The matching process between the predicted and annotated steps. We first use GPT-4o to match the predicted steps and annotated steps in pairs based on the image (allowing one-to-many matches), resulting in a matching matrix. Then, we apply the Hungarian algorithm to filter the matching matrix, removing redundant matches to achieve the maximum one-to-one matches.
Size
Rc
Pc
Fc
Rp
Pp
Fp
human
-
97.6
97.6
97.6
92.3
92.7
92.5
BLIP2-OPT [ 27 ]
4B
24.6
99.9
39.5
0.0
0.0
-
BLIP2-Flan-T5-xl [ 27 ]
4B
24.0
74.3
36.3
0.1
0.2
0.1
BLIP2-Flan-T5-xxl [ 27 ]
12B
41.3
67.7
51.3
0.0
0.2
0.0
LLaVA1.5-7B [ 34 ]
7B
40.3
39.8
40.0
6.4
9.0
7.5
LLaVA1.5-13B [ 34 ]
13B
39.1
49.4
42.5
9.1
19.3
12.3
Table 2: The evaluation results of 41 VLMs. ‘Ins’ indicates that the model is an ‘instruction-tuned’ version. Recall and precision are used as reference metrics and are marked in gray . The best and second-best F1 scores (excluding human results) are marked in red and blue , respectively, and we use a dashed line to separate open-source VLMs (above) and proprietary VLMs (below).
Figure 6: Evaluation results of human and different scoring VLMs. The x-axis represents the eight representative VLMs being evaluated. Each point with the same marker shape denotes the F1 scores obtained by the same scoring VLM or by human evaluators. The solid lines indicate the F1 scores for content, while the dashed lines represent the F1 scores for preconditions.
Scoring VLMs
ρc
ρp
s
Δ
Qwen3-VL-8B-Ins
0.96
0.89
0.98
13.4
InternVL3.5-14B-Ins
0.96
0.78
0.99
14.9
Qwen-VL-Max
0.97
0.86
0.99
3.5
GPT-4o
0.99
0.96
0.99
3.0
Table 3: Comparison between scoring VLM evaluation results and human evaluation results. ρc ( ρp ) represents the Pearson correlation coefficient between the content (precondition) F1 scores obtained by VLMs and those obtained by humans. s ( Δ ) denotes the cosine similarity (mean absolute error) between the F1 scores obtained by VLMs and those obtained by humans.
Figure 7: Dimension-wise analysis. Each dot represents the F1 score of a VLM under specific conditions, including different numbers of answer steps, annotation methods, and scene categories. ‘Human’, ‘Subgraph’, and ‘Graph Expansion’ correspond to data from ‘manual annotations’, ‘subgraph extraction augmentation’, and ‘graph expansion augmentation’, respectively.
Figure 8: The overview of RSGAR. Scene graph generation and question answering are performed by the same VLM.
Table 4: (a) The results of various methods. Bold and underlined scores denote the best and second-best results. ‘+ depth’, ‘+ seg’, and ‘+ depth&seg’ represent vanilla reasoning enhanced with depth maps, segmentation maps, and both, respectively. (b) RSGAR’s Fc and Fp across answer step counts. The values in parentheses represent the changes relative to vanilla reasoning.
Table 5: Ablation studies on GPT-4o. The default settings are indicated in italics . ‘seg’ and ‘depth’ denote the segmentation map and depth map, respectively.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Method
τ
Fc
Fp
Vanilla Reasoning
27.4h
67.4
25.1
+ depth
27.9h
64.1
19.0
+ seg
29.4h
50.3
14.7
+ seg&depth
30.1h
50.5
14.5
PhysAgent
31.5h
64.7
22.2
CoT
27.5h
67.6
27.0
Appendix
Table A1: Efficiency analysis. τ is the time taken by each method to perform one validation on SpatiaLQA.
Vision-Language Models (VLMs) perform well on commonsense reasoning tasks but struggle with visual spatial reasoning. Most existing solutions introduce extra 3D prior inputs or external spatial encoders, which increase complexity and degrade the underlying VLMs' general-purpose capabilities after spatial fine-tuning. To this end, we propose a parameter-efficient Spatio-vision Language Models (SpatioLM), that enhances spatial intelligence without extra 3D prior inputs or third-party spatial encoders. Concretely, we design a plug-and-play and non-invasive spatio-vision module that elicits the spatial knowledge inherent in VLMs. Furthermore, we innovatively leverage pseudo depth and camera information as supervision to guide the model in learning physically coherent representations. Extensive experiments show that SpatioLM achieves significant improvements in diverse tasks, including spatial perception and understanding while effectively limiting the degradation of general capabilities. Notably, the model achieves an impressive score of 71.6 on the VSI-Bench (the first model to surpass 70). In addition, it attains competitive performance when transferred to embodied manipulation tasks. Code is available at \faGithub~spatio-lm.
Jing Wu, Jianhua Wu, Jiayi Guan +5
Xiaomi EV, Beijing, China · College of Automotive and Energy Engineering, Tongji University, Shanghai, China · Independent Researcher
Spatial reasoning is foundational for Vision-Language Models (VLMs), particularly when deployed as Vision-Language-Action (VLA) agents in physical environments. However, existing benchmarks predominantly focus on elementary, single-hop relations, neglecting the multi-hop compositional reasoning and precise visual grounding essential for real-world scenarios. To address this, we introduce MultihopSpatial, offering three key contributions: (1) A comprehensive benchmark designed for multi-hop and compositional spatial reasoning, featuring 1- to 3-hop complex queries across diverse spatial perspectives. (2) Acc@50IoU, a complementary metric that simultaneously evaluates reasoning and visual grounding by requiring both answer selection and precise bounding box prediction - capabilities vital for robust VLA deployment. (3) MultihopSpatial-Train, a dedicated large-scale training corpus to foster spatial intelligence. Extensive evaluation of 37 state-of-the-art VLMs yields eight key insights, revealing that compositional spatial reasoning remains a formidable challenge. Finally, we demonstrate that reinforcement learning post-training on our corpus enhances both intrinsic VLM spatial reasoning and downstream embodied manipulation performance.
Youngwan Lee, Soojin Jang, Yoorhim Cho +3
Electronics and Telecommunications Research Institute, South Korea · Korea Advanced Institute of Science and Technology, South Korea · Sungkyunkwan University, South Korea +1
Vision-language models (VLMs) have advanced rapidly, yet they still struggle with basic spatial reasoning. Despite strong performance on general benchmarks, modern VLMs remain brittle at understanding 2D spatial relationships such as relative position, layout, and counting. We argue that this failure is not merely a data problem, but is closely tied to dominant design choices in current VLM pipelines: reliance on CLIP-style image encoders and the flattening of images into 1D token sequences with 1D positional encoding. We present a controlled diagnostic study within the LLaVA framework to isolate how these choices affect spatial grounding. We evaluate frontier models and LLaVA variants on a suite of spatial benchmarks, comparing CLIP-based encoders against alternatives trained with denser or generative objectives, as well as variants augmented with 2D positional encoding. Our results show consistent spatial performance gaps across models, and indicate that encoder objectives and positional structure shape spatial behavior, but do not fully resolve it.