Vision-language models (VLMs) have shown promising spatial reasoning capabilities from static visual inputs, where the evidence needed to answer a question is available in the provided views. However, in cluttered environments, answer-relevant evidence may be occluded rather than absent: an object may lie beneath a pile, be covered by another object, or have identifying information facing away from the camera. Answering such questions requires physical interaction to reveal the hidden evidence. We formalize this setting as Manipulation-Grounded Visual Question Answering (MG-VQA), where an agent answers a question about an initially cluttered scene by using manipulation as an intermediate evidence-gathering operation. We introduce MG-VQA-Bench, comprising 600 human-verified questions across four spatial reasoning tasks, and evaluate it in a cluttered tabletop interactive environment. Eight VLMs are evaluated under three levels of environment access: Direct (image only), Perception (pointing, segmentation, and scene graphs), and Manipulation (perception, grasping, and pushing). Across all models, Direct and Perception achieve average success rates of 36.2% and 37.6%, respectively, only slightly above an image-blind chance baseline of 32.7%. In contrast, Manipulation raises average success to 56.8% (40.8-83.0% across models). Stronger tool-calling VLMs, GPT6Astra (83.0%) and Gemini3.8Flash (69.3%), search persistently and re-ground after unsuccessful interactions, while weaker models often answer without gathering sufficient evidence or stop prematurely after failed actions. Our results highlight the need for VLM agents that use manipulation for persistent, physically grounded evidence gathering and recovery. Code and benchmark is available here: https://github.com/vineet2104/MG-VQA
Figures & tables
Figure 1: MG-VQA : a simulation environment and benchmark for manipulation-grounded spatial reasoning. Questions in MG-VQA -Bench run on a high-fidelity tabletop simulator (left) with a UR5 + Robotiq platform to physically interact with the scene. MG-VQA -Bench comprises 4 spatial reasoning tasks (right).
Figure 2: Examples where answer options are constructed to confound static question-answering.
Mode
Count
Find
Beneath
Compare
Avg.
Chance
22.0
50.0
26.0
32.7
32.7
Gemma 4 31B IT
Direct
24.0
38.0
33.3
35.3
32.7
Perception
22.7
46.7
28.7
43.3
35.3
Manipulation
32.0
50.7
52.7
42.0
44.3
Kimi K3
Table 1: Success rates (%) on 600 human-verified questions (150 questions per task type). Avg. is the macro-average over the four tasks; bold marks the best result per column, underline the second best.
Figure 3: Manipulation and evidence gathering in MG-VQA -Bench. Left : percentage of the 150 Manipulation -mode episodes per model and task in which at least one grasp or push was used. Center : correctly answered Find questions when the target is absent versus present, 75 episodes each. Right : Count questions answered with an undercount versus an overcount; invalid count answers are excluded. Models are ordered as in Table 1 ; horizontal rule separates open-source from proprietary models.
Model
Ablation
Beneath
Compare
Count
Find
Avg.
Acted /600
Mean manip. /ep
Kimi K3
Baseline
74.7
58.7
40.0
53.3
56.7
371
1.7
+ Search
70.0
64.7
48.7
64.0
61.8
519
2.4
+ Search-persistent
73.3
63.3
62.0
65.3
66.0
527
2.4
+ Multiview
76.0
58.7
54.0
60.7
62.3
385
1.6
Opus 4.8
Baseline
51.3
42.0
25.3
47.3
41.5
225
1.0
+ Search
48.7
48.0
30.0
49.3
44.0
293
1.3
Table 2: Ablations conducted on MG-VQA -Bench (600 questions); values reported in success rate (%). Acted episodes contain at least one grasp / push . Ablations are separate additions.
Model
Count
Find
Beneath
Compare
Gemma 4 31B IT
2 / 0
2 / 0
7 / 2
4 / 0
Kimi K3
3 / 0.5
2 / 0
7 / 2
7 / 2
GLM 5.3
5 / 1
2 / 0
7 / 2
4 / 0.5
Qwen 3.8 Flash Next
2 / 0
2 / 0
5 / 1
7 / 2
Gemini 3.1 Pro
11 / 3
8 / 2
6 / 2
10 / 3
Gemini 3.8 Flash
8 / 2
7 / 2
6 / 2
9 / 2
Table 3: Median model turns / manipulation calls per question, computed over 150 questions per task.
Next manipulation (% of N )
Re-grounding (% of pushes)
Model
N
Push
Retry grasp
None
Scene + seg.
Seg. only
Gemma 4 31B IT
213
76.1
11.3
12.7
32.1
64.2
Kimi K3
246
85.8
7.7
6.5
43.1
55.5
GLM 5.3
246
50.8
2.8
46.3
57.6
35.2
Qwen 3.8 Flash Next
129
86.0
1.6
12.4
61.3
32.4
Gemini 3.1 Pro
454
79.3
13.2
7.5
48.9
48.1
Table 4: Agent actions following failed grasps. N denotes the number of episodes with at least one grasp failure. Next manipulation groups these episodes by the next manipulation action: push , grasp , or none if the model answers directly. For episodes with a subsequent push , Re-grounding shows the percentage in which the model updated the scene graph before segmentation ( Scene + seg. ) or performed segmentation using an outdated scene graph ( Seg. only ). Models are ordered as in Table 1 .
Figure 4: Gemini 3.1 Pro (top) attempts to grasp a flat plate, then displaces a different object with a push and concludes from the still-occluded scene that the target is absent. GPT 6 Astra (bottom) investigates the central pile of objects sequentially, answering only once the target is exposed.
Model
Count
Find
Beneath
Compare
Avg.
Acted /40
Mean manip. /ep.
Open-source models
Kimi K3
0.0
10.0
10.0
50.0
17.5
18
1.125
Qwen 3.8 Flash Next
0.0
0.0
50.0
60.0
27.5
17
0.800
Proprietary models
Gemini 3.1 Pro
10.0
20.0
30.0
50.0
27.5
38
3.750
Gemini 3.8 Flash
0.0
10.0
40.0
50.0
25.0
24
1.575
Table 5: Success rates and manipulation statistics on the MG-VQA -Hard benchmark. Results cover 40 questions, with 10 per task type. Success rates are reported as percentages. Column definitions follow Table 2 . Bold indicates the best result in each column, and underline indicates the second best.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: MG-VQA-Bench: Beneath and Compare. Ten examples per task. Correct answers are bold. Red boxes show privileged ground-truth target extents, including hidden portions; these overlays are not agent inputs.
Figure 6: MG-VQA-Bench: Count and Find. Ten examples per task. Correct answers are in bold. Red boxes show ground-truth target objects; absent targets are not boxed.
Figure 7: Human verification interface. The custom review application displays the initial observation (left), reference object views (center), and a focused verification image (right) for the red-apple Beneath example. The privileged verification view is not exposed to evaluated agents.
Figure 8: System prompt for VLMs evaluated in MG-VQA-Bench, identical across all models. Episodes terminated by five consecutive perception-only calls or by exhausting the manipulation budget are scored as incorrect.
Figure 9: User prompt for VLMs evaluated in MG-VQA-Bench, supplying the current observation, the previous tool result, the question with its answer options, and optional ablation context. Values substituted at runtime are shown as ⟨ placeholders ⟩ .
Figure 10: Sample scenes in MG-VQA-Hard. Three per task. Correct answers are bold. Red boxes show privileged ground-truth target extents, including hidden portions; these overlays are not agent inputs.
Vision-language models (VLMs) increasingly reason over visual evidence that is cropped, segmented, retrieved, or revealed over time. Yet most VQA benchmarks present the complete image and question at once. We ask what models lose when the same information is fragmented. We introduce Layered-VQA, with 93 scenes and 300 questions. Each image is decomposed into ordered RGBA layers that exactly recompose the original scene, and each question is annotated with supporting, minimal-sufficient, and distractor layers. We evaluate eleven open-weight VLMs from 3B to 32B parameters and two proprietary models with a scale of 187,200 conversations, graded by 1.74M open-model cross-judgments. We find three consistent failures. Loss in Composition: fragmenting the question has a small effect, but fragmenting the scene substantially reduces accuracy; recomposing the same layers largely restores performance. Oracle Inversion: even oracle-selected sufficient evidence can perform worse than the complete scene. Loss in Grounding: as more evidence is required, grounding degrades much faster than answer accuracy. Together, these results show that having the right visual evidence is not enough. How that evidence is composed and presented determines whether models can use and ground it. The right evidence is not enough: VLMs need the scene it came from.
L. D. M. S. Sai Teja, Ufaq Khan, N. Siva Gopala Krishna +5
Vision-Language Models (VLMs) have been increasingly applied in real-world scenarios due to their outstanding understanding and reasoning capabilities. Although VLMs have already demonstrated impressive capabilities in common visual question answering and logical reasoning, they still lack the ability to make reasonable decisions in complex real-world environments. We define this ability as spatial logical reasoning, which not only requires understanding the spatial relationships among objects in complex scenes, but also the logical dependencies between steps in multi-step tasks. To bridge this gap, we introduce Spatial Logical Question Answering (SpatiaLQA), a benchmark designed to evaluate the spatial logical reasoning capabilities of VLMs. SpatiaLQA consists of 9,605 question answer pairs derived from 241 real-world indoor scenes. We conduct extensive experiments on 41 mainstream VLMs, and the results show that even the most advanced models still struggle with spatial logical reasoning. To address this issue, we propose a method called recursive scene graph assisted reasoning, which leverages visual foundation models to progressively decompose complex scenes into task-relevant scene graphs, thereby enhancing the spatial logical reasoning ability of VLMs, outperforming all previous methods. Code and dataset are available at https://github.com/xieyc99/SpatiaLQA.
Yuechen Xie, Xiaoyan Zhang, Yicheng Shan +6
Zhejiang University · The University of Sydney · ManyCore +2
Vision-Language Models (VLMs) perform well on commonsense reasoning tasks but struggle with visual spatial reasoning. Most existing solutions introduce extra 3D prior inputs or external spatial encoders, which increase complexity and degrade the underlying VLMs' general-purpose capabilities after spatial fine-tuning. To this end, we propose a parameter-efficient Spatio-vision Language Models (SpatioLM), that enhances spatial intelligence without extra 3D prior inputs or third-party spatial encoders. Concretely, we design a plug-and-play and non-invasive spatio-vision module that elicits the spatial knowledge inherent in VLMs. Furthermore, we innovatively leverage pseudo depth and camera information as supervision to guide the model in learning physically coherent representations. Extensive experiments show that SpatioLM achieves significant improvements in diverse tasks, including spatial perception and understanding while effectively limiting the degradation of general capabilities. Notably, the model achieves an impressive score of 71.6 on the VSI-Bench (the first model to surpass 70). In addition, it attains competitive performance when transferred to embodied manipulation tasks. Code is available at \faGithub~spatio-lm.
Jing Wu, Jianhua Wu, Jiayi Guan +5
Xiaomi EV, Beijing, China · College of Automotive and Energy Engineering, Tongji University, Shanghai, China · Independent Researcher