Spatial Reasoning is Not a Free Lunch: A Controlled Study on LLaVA
Authors: Nahid Alam, Leema Krishna Murali, Siddhant Bharadwaj, Patrick Liu, Timothy Chung, Drishti Sharma, Akshata A., Kranthi Kiran, Wesley Tam, Bala Krishna S Vegesna
Organizations: Cohere Labs Community · Eisai Inc. · Indian Institute of Science, Bangalore · UIUC · Imperial College London · EleutherAI · Georgia Institute of Technology
Vision-language models (VLMs) have advanced rapidly, yet they still struggle with basic spatial reasoning. Despite strong performance on general benchmarks, modern VLMs remain brittle at understanding 2D spatial relationships such as relative position, layout, and counting. We argue that this failure is not merely a data problem, but is closely tied to dominant design choices in current VLM pipelines: reliance on CLIP-style image encoders and the flattening of images into 1D token sequences with 1D positional encoding. We present a controlled diagnostic study within the LLaVA framework to isolate how these choices affect spatial grounding. We evaluate frontier models and LLaVA variants on a suite of spatial benchmarks, comparing CLIP-based encoders against alternatives trained with denser or generative objectives, as well as variants augmented with 2D positional encoding. Our results show consistent spatial performance gaps across models, and indicate that encoder objectives and positional structure shape spatial behavior, but do not fully resolve it.
Figures & tables
Figure 1: Our experimental approach with LLaVA Framework Liu et al. (2023c) that compares the performance of different image encoders and 2D-RoPE variants.
Models
MMVP
CV-Bench 2D Overall
TallyQA
GQA Overall
VSR
Top- ViewRS
Count- BenchQA
LLaVA-NeXT
0.667
0.606
0.733
63.786
63.994
0.409
0.515
LLaVA-OneVision
0.767
0.730
0.797
62.140
77.741
0.414
0.823
Qwen2.5-VL
0.770
0.754
0.800
60.391
89.116
0.456
0.891
SmolVLM2
0.687
0.577
0.729
50.574
71.277
0.416
0.692
Gemma3-4b-it
0.708
0.659
0.525
31.277
55.074
0.334
0.713
PaliGemma
0.667
0.624
0.794
62.570
65.139
0.322
0.674
Table 1: Comparison of frontier models and LLaVA variants across spatial understanding benchmarks. Values underlined indicate the best-performing frontier model; values in bold indicate the best-performing LLaVA variant.
Figure 3: Object localization in LLaVA-SigLIP2 vs. LLaVA-AIMv2. AIMv2 yields tighter and more spatially aligned localizations, while SigLIP2 often produces imprecise or misaligned boxes.