Vision-Language Models (VLMs) have demonstrated impressive performance when integrated into robot manipulation pipelines, but real-world evaluation of VLM capabilities on downstream robotics tasks is both difficult and expensive to scale. We propose that spatial and temporal ordering tasks using images from stereo and wrist cameras attached to real-world robots are particularly well-suited for evaluating visual reasoning capabilities, as they assess skills directly applicable to robots. We evaluate VLM reasoning abilities on three questions: ordering shuffled frames conditioned on a language description of the task, ordering objects depthwise based on distance from the camera, and ordering objects from left-to-right in pixel coordinates. The complexity of these tasks can also be modulated by changing the size of the set to sort. Our experiments show that several state-of-the-art vision-language models struggle on ordering tasks. To fix this, we propose a scalable autonomous training data generation pipeline, and show that both supervised finetuning and reinforcement-learning on the generated data can significantly improve VLM reasoning performance both on spatiotemporal ordering tasks as well as downstream robotics and visual reasoning tasks.
Figures & tables
Figure 1: Examples of spatiotemporal ordering tasks in our dataset.
Task Format
Model Name
Base Image
Segmentation Map Image
Bounding Box Image
Low
Medium
High
Low
Medium
High
Low
Medium
High
5 Image Set
GPT-5
15.67
19.67
18.67
11.00
12.00
11.33
8.33
8.67
10.00
GPT-5-Nano
0.67
2.33
1.00
2.00
2.33
0.67
1.00
3.33
2.33
Gemini-ER-1.5
1.33
6.67
7.33
3.67
4.00
5.33
2.33
4.33
5.33
Qwen3-VL-235B
6.67
9.67
7.67
3.00
4.00
4.33
2.67
2.67
3.33
3 Image Set
GPT-5
49.33
47.67
47.33
44.67
45.00
41.33
39.67
38.67
40.67
Table 1: Model Accuracies (%) for Temporal Ordering Tasks.
Task Format
Model Name
Base Image
Segmentation Map Image
Bounding Box Image
Low
Medium
High
Low
Medium
High
Low
Medium
High
5 Object Set
GPT-5
4.33
4.33
5.00
9.33
8.00
6.67
7.67
10.33
7.33
GPT-5-Nano
2.33
4.33
3.00
5.67
3.33
2.67
3.00
3.33
3.67
Gemini-ER-1.5
3.00
6.00
5.67
4.33
8.00
7.33
6.33
9.33
9.33
Qwen3-VL-235B
2.67
3.00
3.33
5.00
3.00
4.00
5.33
5.00
4.67
3 Object Set
GPT-5
34.00
34.00
32.67
42.33
42.33
41.33
43.67
41.67
39.00
Table 2: Model Accuracies (%) for Depth Ordering Tasks
Task Format
Model Name
Base Image
Segmentation Map Image
Bounding Box Image
Low
Medium
High
Low
Medium
High
Low
Medium
High
5 Object Set
GPT-5
21.00
21.00
21.00
17.67
19.33
19.67
20.33
18.00
22.67
GPT-5-Nano
7.00
8.67
11.33
5.33
7.67
10.33
4.41
5.67
7.33
Gemini-ER-1.5
18.79
18.06
19.38
17.58
15.71
18.37
28.77
30.64
27.21
Qwen3-VL-235B
22.33
22.00
19.00
14.33
17.67
23.53
20.67
18.33
21.00
3 Object Set
GPT-5
58.00
57.00
60.00
56.33
55.00
57.00
49.67
52.00
52.67
Table 3: Model Accuracies (%) for Left-to-Right Ordering Tasks
Task Format
Model Name
Temporal Ordering Task
Depth Ordering Task
Left-to-Right Ordering Task
Low
Medium
High
Low
Medium
High
Low
Medium
High
5 Image/Object Set
GPT-5
15.67
19.67
18.67
4.33
4.33
5.00
21.00
21.00
21.00
GPT-5-Nano
0.67
2.33
1.00
2.33
4.33
3.00
7.00
8.67
11.33
Gemini-ER-1.5
1.33
6.67
7.33
3.00
6.00
5.67
18.79
18.06
19.38
Qwen3-VL-235B
6.67
9.67
7.67
2.67
3.00
3.33
22.33
22.00
19.00
4 Image/Object Set
GPT-5
27.00
29.33
30.00
9.67
13.00
11.00
32.67
32.00
37.00
Table 4: Model Accuracies (%) Across Different Set Cardinalities. All results are in the base setting (i.e. no images had any bounding boxes or segmentation map overlays on them).
Model Name
Across All 3 and 5 Set Cardinality Ordering Tasks
Base
Segmentation Map
Bounding Box
GPT-5
1.83 ± 1.49
2.06 ± 1.02
3.17 ± 1.28
GPT-5-Nano
3.33 ± 2.39
3.94 ± 1.39
3.49 ± 2.37
Gemini-ER-1.5
3.96 ± 3.43
3.99 ± 2.37
3.53 ± 1.06
Qwen3-VL-235B
1.61 ± 1.29
2.92 ± 3.11
2.52 ± 2.14
Table 5: Mean and Standard Deviation of Reasoning Difference
Task Format
Model Name
Temporal
Depth
Left-to-Right
Ordering
Ordering
Ordering
5 Image/Object Set
GPT-5
8.00
5.67
23.00
GPT-5-Nano
1.00
5.00
12.67
Gemini-ER-1.5
1.67
6.33
22.33
Qwen3-VL-235B
0.33
3.33
22.00
3 Image/Object Set
GPT-5
42.33
33.00
60.00
Table 6: Model Accuracies (%) for High-Reasoning Merge Sort Based Ordering in Base Image Setting
Figure 2: Comparison of how model performance scales with set cardinality in temporal and depth ordering tasks in the base setting. In the temporal high reasoning setting, model performance scales predictably with increase in set cardinality, maintaining a consistent relative order across all set cardinalities. However, the relative order is inconsistent in the remaining settings in the figure.
Model
3 Image/Object Spatiotemporal Ordering
Downstream Reasoning
Temporal
Depth
Left-to-Right
SAT
RoboFAC (MCQ)
RoboFAC (Open-Ended)
Value Est. (RT-1)
Value Est. (Dobb-E)
Base
18.00
22.33
53.67
70.67
62.55
12.28
0.0066
0.0123
SFT (Temporal)
50.67
26.00
55.33
70.67
60.21
12.92
0.0001
0.0114
SFT (Depth)
22.67
31.67
56.67
68.67
60.26
13.33
0.0047
0.0015
SFT (Left-to-Right)
23.67
24.67
61.33
69.33
58.91
13.16
0.0080
0.0062
GRPO (Temporal)
22.67
26.00
51.67
68.67
63.13
11.94
0.0060
0.0036
Table 7: Finetuned Qwen3VL-8B Results on Spatiotemporal Ordering and Downstream Reasoning Tasks. Value estimation results are average value-order correlations (Appendix B ), while all other results are question-answering accuracies (in %).
Model
Task Format
Training
Temporal
Depth
Left-to-
Name
Method
Right
Qwen2-VL-2B
5 Images/Objects
Base
1.00
0.66
0.67
GRPO
2.00
16.33
17.33
SFT
0.33
10.33
15.33
3 Images/Objects
Base
17.33
15.66
16.00
GRPO
21.67
66.00
67.66
Table 8: Model Accuracies (%) for GRPO and SFT
Model Name
Training Method
Temporal
Depth
Left-to-Right
Qwen2-VL-2B
Base
17.33
15.66
16.00
GRPO-300
19.00
37.00
54.00
GRPO-500
20.00
56.00
63.66
GRPO-700
21.67
66.00
67.66
SmolVLM2-2.2B
Base
13.00
8.00
12.33
GRPO-300
15.33
24.33
45.00
Table 9: 3 Image/Object ordering model accuracies (%) after GRPO under different training data sizes
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Left
Right
Wrist
Multi-
Name
View
View
Camera
View
GPT-5
50.67
46.67
47.67
47.33
GPT-5-Nano
26.33
21.00
22.33
23.67
Gemini-ER-1.5
42.33
42.00
45.00
38.00
Qwen3-VL-235B
36.00
28.67
35.33
32.67
Appendix
Table 10: High Reasoning Model Accuracies (%) for Single-View Image in 3 Image Base Setting for Temporal Ordering
Figure 3: Temporal task, 3 img
Figure 4: Temporal task, 5 image
Figure 5: Depth task,3 Objects
Figure 6: Depth task, 5 objects
Figure 7: Left-to-Right task, 3 Objects
Figure 8: Left-to-Right task, 5 Objects
Figure 9: Task Description: “Remove the lid from the pot | Remove the lid from the black pot. | Remove the glass lid from the pot." Correct Order: 2,3,4,5,1
Figure 10: Task Description: “Pick up the object and put it in the light blue cup | Place the metal object inside the light blue cup | Put the silver object in the light blue cup" Correct Order: 2,1,4,5,3
Figure 11: Task Description: “Pick up the third spice bottle from the right on the bottom shelf of the bottom spice rack and put it on the table | Remove the third container from the right on the bottom rack and put it on the table | Put the spice bottle on the counter" Correct Order: 2,5,3,4,1
Figure 12: Task Description: “Hang the towel on the black stand | Put the towel on the black rod | Pick up the cloth from the table and place it on the black object" Correct Order: 3, 2, 1
Figure 13: Task Description: “Hang the towel on the black stand | Put the towel on the black rod | Pick up the cloth from the table and place it on the black object" Correct Order: 3, 2, 1
Figure 14: “order the following objects from closest to farthest from camera in terms of their centroid in the image: bin, robot arm, bowl, bottle, bowl cup" Correct Order: "robot arm", "bottle", "bowl cup", "bowl", "bin"
Figure 15: “order the following objects from closest to farthest from camera in terms of their centroid in the image: container, cup, bottle, tray, robot arm" Correct Order: "container", "cup", "tray", "robot arm", "bottle"
Figure 16: “order the following objects from closest to farthest from camera in terms of their centroid in the image: sink, faucet, trash bin, cardboard box packet, coffee maker" Correct Order: "cardboard box packet", "trash bin", "coffee maker", "sink", "faucet"
Figure 17: “order the following objects from closest to farthest from camera in terms of their centroid in the image: napkin dispenser, paper towel roll, coffee machine" Correct Order: "paper towel roll", "napkin dispenser", "coffee machine"
Figure 18: “order the following objects from closest to farthest from camera in terms of their centroid in the image: napkin dispenser, paper towel roll, coffee machine" Correct Order: "paper towel roll", "napkin dispenser", "coffee machine"
Figure 19: “Arrange the following objects in order from leftmost to rightmost in the image: rug, chair, plant, table, door" Correct Order: "chair", "table", "plant", "door", "rug"
Figure 20: “Arrange the following objects in order from leftmost to rightmost in the image: robot, paper towel, box, microwave, pumpkin" Correct Order: "robot", "pumpkin", "paper towel", "microwave", "box"
Figure 21: “Arrange the following objects in order from leftmost to rightmost in the image: blanket, lamp, pillow blanket, table, book" Correct Order: "blanket", "pillow blanket", "lamp", "book", "table"
Figure 22: “Arrange the following objects in order from leftmost to rightmost in the image: cable, robot arm, spoon" Correct Order: "cable", "spoon", "robot arm"
Figure 23: “Arrange the following objects in order from leftmost to rightmost in the image: cable, robot arm, spoon" Correct Order: "cable", "spoon", "robot arm"
# Images/Objects
Temporal Ordering Task
Depth Ordering Task
Left-to-Right Ordering Task
Low
Medium
High
Low
Medium
High
Low
Medium
High
2
419
1204
2912
721
1487
5839
960
1664
5376
3
535
1619
3859
788
2067
5778
896
2432
4416
4
668
2062
5012
985
2262
6745
1024
2496
4992
5
801
2496
121,436
1052
2781
7194
1216
3136
6784
Mergesort 3
N/A
N/A
5701
N/A
N/A
2653
N/A
N/A
6400
Appendix
Table 11: Maximum output tokens permitted across reasoning levels and number of ordering elements.
Model
3 Image/Object Spatiotemporal Ordering
Downstream Reasoning
Temporal
Depth
Left-to-Right
SAT
RoboFAC (MCQ)
RoboFAC (Open-Ended)
Value Est. (RT-1)
Value Est. (Dobb-E)
Base
18.33
29.00
45.00
48.00
38.54
00.34
0.0000
0.0005
SFT (Temporal)
44.00
25.33
44.00
52.00
37.55
00.25
0.0000
0.0024
SFT (Depth)
23.67
35.00
49.00
42.67
33.91
00.29
0.0002
0.0022
SFT (Left-to-Right)
20.33
24.33
66.33
52.00
38.91
00.30
0.0183
0.0058
GRPO (Temporal)
16.33
27.67
45.33
45.33
21.51
00.03
0.0000
0.0010
Appendix
Table 12: Finetuned Qwen3VL-8B-Thinking Accuracies on Spatiotemporal Ordering and Downstream Reasoning Tasks
Diverse applications for robotics, such as industry and agriculture, require robots to operate across various embodiments, changing visual conditions, and complex planning. Vision-Language Models (VLMs) offer a promising foundation for general-purpose and interpretable robotic reasoning. Aligning VLMs with diverse robot applications requires a modular understanding of the individual decision components that underlie robotic behavior. Capturing such structure is challenging for conventional robot benchmarks that are primarily based on teleoperated, end-to-end datasets. We propose Robot Question Answering (RQA), a modular evaluation framework and RoboVista, a benchmark curated from real robotic systems, research papers, and expert annotations. RoboVista contains 474 Visual Question Answering (VQA) instances with human annotated reasoning and covers 39 unique task types in agricultural, industrial, domestic, surgical robotics, autonomous driving, and open robot datasets. Experiments on RoboVista show that state-of-the-art VLMs exhibit substantial gaps. Physical robot experiments suggest strong correlation between RoboVista performance and real-world task execution.
Shuangyu Xie, Kaiyuan Chen, Ziyang Chen +8
University of California, Berkeley · Princeton University · Google DeepMind
Understanding user instructions and object spatial relations in surrounding environments is crucial for intelligent robot systems to assist humans in various tasks. The natural language and spatial reasoning capabilities of Vision-Language Models (VLMs) have the potential to enhance the generalization of robot planners on new tasks, objects, and motion specifications. While foundation models have been applied to task planning, it is still unclear the degree to which they have the capability of spatial reasoning required to enforce user preferences or constraints on motion, such as desired distances from objects, topological properties, or motion style preferences. In this paper, we evaluate the capability of four state-of-the-art VLMs at spatial reasoning over robot motion, using four different querying methods. Our results show that, with the highest-performing querying method, Qwen2.5-VL achieves 71.4% accuracy zero-shot and 75% on a smaller model after fine-tuning, and GPT-4o leads to lower performance. We evaluate two types of motion preferences (object-proximity and path-style), and we also analyze the trade-off between accuracy and computation cost in number of tokens. This work shows some promise in the potential of VLM integration with robot motion planning pipelines.
Robots operating in shared human environments must not only navigate, interact, and detect their surroundings, they must also interpret and respond to dynamic, and often unpredictable, human behaviours. Although recent advances have shown promise in enhancing robotic perception and instruction-following using Vision-Language Models (VLMs), they remain limited in addressing the complexities of multimodal human-robot interactions (HRI). Motivated by this challenge, we introduce a lightweight language-to-vision feedback module that closes the loop between an LLM and the vision encoder in VLMs. The module projects image-token hidden states through a gated Multi-Layer Perceptron (MLP) back into the encoder input, prompting a second pass that reinterprets the scene under text context. We evaluate this approach on three robotics-centred tasks: navigation in a simulated environment (Habitat), sequential scene description (Mementos-Robotics), and human-intention recognition (our HRI dataset). Results show that our method improves Qwen 2.5 (7B) by 3.3% (less distance), +0.057 description score, and +2.93% accuracy, with less than 3% extra parameters; Gemma 3 (4B) and LLaVA OV 1.5 (4B) show mixed navigation results but gains +0.111,+0.055 and +10.81%,+4.79% on the latter two tasks. Code is available at https://github.com/alessioGalatolo/VLM-Reasoning-for-Robotics
Alessio Galatolo, Ronald Cumbal, Alexandros Rouchitsas +3
Department of Information Technology, Uppsala University, Sweden.