Vision-language models (VLMs) have shown promising spatial reasoning capabilities from static visual inputs, where the evidence needed to answer a question is available in the provided views. However, in cluttered environments, answer-relevant evidence may be occluded rather than absent: an object may lie beneath a pile, be covered by another object, or have identifying information facing away from the camera. Answering such questions requires physical interaction to reveal the hidden evidence. We formalize this setting as Manipulation-Grounded Visual Question Answering (MG-VQA), where an agent answers a question about an initially cluttered scene by using manipulation as an intermediate evidence-gathering operation. We introduce MG-VQA-Bench, comprising 600 human-verified questions across four spatial reasoning tasks, and evaluate it in a cluttered tabletop interactive environment. Eight VLMs are evaluated under three levels of environment access: Direct (image only), Perception (pointing, segmentation, and scene graphs), and Manipulation (perception, grasping, and pushing). Across all models, Direct and Perception achieve average success rates of 36.2% and 37.6%, respectively, only slightly above an image-blind chance baseline of 32.7%. In contrast, Manipulation raises average success to 56.8% (40.8-83.0% across models). Stronger tool-calling VLMs, GPT6Astra (83.0%) and Gemini3.8Flash (69.3%), search persistently and re-ground after unsuccessful interactions, while weaker models often answer without gathering sufficient evidence or stop prematurely after failed actions. Our results highlight the need for VLM agents that use manipulation for persistent, physically grounded evidence gathering and recovery. Code and benchmark is available here: https://github.com/vineet2104/MG-VQA
Figures & tables
Figure 1: MG-VQA : a simulation environment and benchmark for manipulation-grounded spatial reasoning. Questions in MG-VQA -Bench run on a high-fidelity tabletop simulator (left) with a UR5 + Robotiq platform to physically interact with the scene. MG-VQA -Bench comprises 4 spatial reasoning tasks (right).
Figure 2: Examples where answer options are constructed to confound static question-answering.
Mode
Count
Find
Beneath
Compare
Avg.
Chance
22.0
50.0
26.0
32.7
32.7
Gemma 4 31B IT
Direct
24.0
38.0
33.3
35.3
32.7
Perception
22.7
46.7
28.7
43.3
35.3
Manipulation
32.0
50.7
52.7
42.0
44.3
Kimi K3
Table 1: Success rates (%) on 600 human-verified questions (150 questions per task type). Avg. is the macro-average over the four tasks; bold marks the best result per column, underline the second best.
Figure 3: Manipulation and evidence gathering in MG-VQA -Bench. Left : percentage of the 150 Manipulation -mode episodes per model and task in which at least one grasp or push was used. Center : correctly answered Find questions when the target is absent versus present, 75 episodes each. Right : Count questions answered with an undercount versus an overcount; invalid count answers are excluded. Models are ordered as in Table 1 ; horizontal rule separates open-source from proprietary models.
Model
Ablation
Beneath
Compare
Count
Find
Avg.
Acted /600
Mean manip. /ep
Kimi K3
Baseline
74.7
58.7
40.0
53.3
56.7
371
1.7
+ Search
70.0
64.7
48.7
64.0
61.8
519
2.4
+ Search-persistent
73.3
63.3
62.0
65.3
66.0
527
2.4
+ Multiview
76.0
58.7
54.0
60.7
62.3
385
1.6
Opus 4.8
Baseline
51.3
42.0
25.3
47.3
41.5
225
1.0
+ Search
48.7
48.0
30.0
49.3
44.0
293
1.3
Table 2: Ablations conducted on MG-VQA -Bench (600 questions); values reported in success rate (%). Acted episodes contain at least one grasp / push . Ablations are separate additions.
Model
Count
Find
Beneath
Compare
Gemma 4 31B IT
2 / 0
2 / 0
7 / 2
4 / 0
Kimi K3
3 / 0.5
2 / 0
7 / 2
7 / 2
GLM 5.3
5 / 1
2 / 0
7 / 2
4 / 0.5
Qwen 3.8 Flash Next
2 / 0
2 / 0
5 / 1
7 / 2
Gemini 3.1 Pro
11 / 3
8 / 2
6 / 2
10 / 3
Gemini 3.8 Flash
8 / 2
7 / 2
6 / 2
9 / 2
Table 3: Median model turns / manipulation calls per question, computed over 150 questions per task.
Next manipulation (% of N )
Re-grounding (% of pushes)
Model
N
Push
Retry grasp
None
Scene + seg.
Seg. only
Gemma 4 31B IT
213
76.1
11.3
12.7
32.1
64.2
Kimi K3
246
85.8
7.7
6.5
43.1
55.5
GLM 5.3
246
50.8
2.8
46.3
57.6
35.2
Qwen 3.8 Flash Next
129
86.0
1.6
12.4
61.3
32.4
Gemini 3.1 Pro
454
79.3
13.2
7.5
48.9
48.1
Table 4: Agent actions following failed grasps. N denotes the number of episodes with at least one grasp failure. Next manipulation groups these episodes by the next manipulation action: push , grasp , or none if the model answers directly. For episodes with a subsequent push , Re-grounding shows the percentage in which the model updated the scene graph before segmentation ( Scene + seg. ) or performed segmentation using an outdated scene graph ( Seg. only ). Models are ordered as in Table 1 .
Figure 4: Gemini 3.1 Pro (top) attempts to grasp a flat plate, then displaces a different object with a push and concludes from the still-occluded scene that the target is absent. GPT 6 Astra (bottom) investigates the central pile of objects sequentially, answering only once the target is exposed.
Model
Count
Find
Beneath
Compare
Avg.
Acted /40
Mean manip. /ep.
Open-source models
Kimi K3
0.0
10.0
10.0
50.0
17.5
18
1.125
Qwen 3.8 Flash Next
0.0
0.0
50.0
60.0
27.5
17
0.800
Proprietary models
Gemini 3.1 Pro
10.0
20.0
30.0
50.0
27.5
38
3.750
Gemini 3.8 Flash
0.0
10.0
40.0
50.0
25.0
24
1.575
Table 5: Success rates and manipulation statistics on the MG-VQA -Hard benchmark. Results cover 40 questions, with 10 per task type. Success rates are reported as percentages. Column definitions follow Table 2 . Bold indicates the best result in each column, and underline indicates the second best.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: MG-VQA-Bench: Beneath and Compare. Ten examples per task. Correct answers are bold. Red boxes show privileged ground-truth target extents, including hidden portions; these overlays are not agent inputs.
Figure 6: MG-VQA-Bench: Count and Find. Ten examples per task. Correct answers are in bold. Red boxes show ground-truth target objects; absent targets are not boxed.
Figure 7: Human verification interface. The custom review application displays the initial observation (left), reference object views (center), and a focused verification image (right) for the red-apple Beneath example. The privileged verification view is not exposed to evaluated agents.
Figure 8: System prompt for VLMs evaluated in MG-VQA-Bench, identical across all models. Episodes terminated by five consecutive perception-only calls or by exhausting the manipulation budget are scored as incorrect.
Figure 9: User prompt for VLMs evaluated in MG-VQA-Bench, supplying the current observation, the previous tool result, the question with its answer options, and optional ablation context. Values substituted at runtime are shown as ⟨ placeholders ⟩ .
Figure 10: Sample scenes in MG-VQA-Hard. Three per task. Correct answers are bold. Red boxes show privileged ground-truth target extents, including hidden portions; these overlays are not agent inputs.