RoboReason: Boosting Robotics-Relevant VLM Reasoning Through Scalable Data Generation

Sagnik Anupam; Utkarsh Lal; Aneesh Edara; Botong Zhang; Yao Tang; Jiatao Gu; Dinesh Jayaraman; Osbert Bastani

cs.ROOct 24, 2026

RoboReason: Boosting Robotics-Relevant VLM Reasoning Through Scalable Data Generation

Authors: Sagnik Anupam, Utkarsh Lal, Aneesh Edara, Botong Zhang, Yao Tang, Jiatao Gu, Dinesh Jayaraman, Osbert Bastani

Organizations: University of Pennsylvania

Abstract

Vision-Language Models (VLMs) have demonstrated impressive performance when integrated into robot manipulation pipelines, but real-world evaluation of VLM capabilities on downstream robotics tasks is both difficult and expensive to scale. We propose that spatial and temporal ordering tasks using images from stereo and wrist cameras attached to real-world robots are particularly well-suited for evaluating visual reasoning capabilities, as they assess skills directly applicable to robots. We evaluate VLM reasoning abilities on three questions: ordering shuffled frames conditioned on a language description of the task, ordering objects depthwise based on distance from the camera, and ordering objects from left-to-right in pixel coordinates. The complexity of these tasks can also be modulated by changing the size of the set to sort. Our experiments show that several state-of-the-art vision-language models struggle on ordering tasks. To fix this, we propose a scalable autonomous training data generation pipeline, and show that both supervised finetuning and reinforcement-learning on the generated data can significantly improve VLM reasoning performance both on spatiotemporal ordering tasks as well as downstream robotics and visual reasoning tasks.

Figures & tables

Figure 1: Examples of spatiotemporal ordering tasks in our dataset.

Task Format	Model Name	Base Image			Segmentation Map Image			Bounding Box Image
Task Format	Model Name	Low	Medium	High	Low	Medium	High	Low	Medium	High
5 Image Set	GPT-5	15.67	19.67	18.67	11.00	12.00	11.33	8.33	8.67	10.00
	GPT-5-Nano	0.67	2.33	1.00	2.00	2.33	0.67	1.00	3.33	2.33
	Gemini-ER-1.5	1.33	6.67	7.33	3.67	4.00	5.33	2.33	4.33	5.33
	Qwen3-VL-235B	6.67	9.67	7.67	3.00	4.00	4.33	2.67	2.67	3.33
3 Image Set	GPT-5	49.33	47.67	47.33	44.67	45.00	41.33	39.67	38.67	40.67

Table 1: Model Accuracies (%) for Temporal Ordering Tasks.

Task Format	Model Name	Base Image			Segmentation Map Image			Bounding Box Image
Task Format	Model Name	Low	Medium	High	Low	Medium	High	Low	Medium	High
5 Object Set	GPT-5	4.33	4.33	5.00	9.33	8.00	6.67	7.67	10.33	7.33
	GPT-5-Nano	2.33	4.33	3.00	5.67	3.33	2.67	3.00	3.33	3.67
	Gemini-ER-1.5	3.00	6.00	5.67	4.33	8.00	7.33	6.33	9.33	9.33
	Qwen3-VL-235B	2.67	3.00	3.33	5.00	3.00	4.00	5.33	5.00	4.67
3 Object Set	GPT-5	34.00	34.00	32.67	42.33	42.33	41.33	43.67	41.67	39.00

Table 2: Model Accuracies (%) for Depth Ordering Tasks

Task Format	Model Name	Base Image			Segmentation Map Image			Bounding Box Image
Task Format	Model Name	Low	Medium	High	Low	Medium	High	Low	Medium	High
5 Object Set	GPT-5	21.00	21.00	21.00	17.67	19.33	19.67	20.33	18.00	22.67
	GPT-5-Nano	7.00	8.67	11.33	5.33	7.67	10.33	4.41	5.67	7.33
	Gemini-ER-1.5	18.79	18.06	19.38	17.58	15.71	18.37	28.77	30.64	27.21
	Qwen3-VL-235B	22.33	22.00	19.00	14.33	17.67	23.53	20.67	18.33	21.00
3 Object Set	GPT-5	58.00	57.00	60.00	56.33	55.00	57.00	49.67	52.00	52.67

Table 3: Model Accuracies (%) for Left-to-Right Ordering Tasks

Task Format	Model Name	Temporal Ordering Task			Depth Ordering Task			Left-to-Right Ordering Task
Task Format	Model Name	Low	Medium	High	Low	Medium	High	Low	Medium	High
5 Image/Object Set	GPT-5	15.67	19.67	18.67	4.33	4.33	5.00	21.00	21.00	21.00
	GPT-5-Nano	0.67	2.33	1.00	2.33	4.33	3.00	7.00	8.67	11.33
	Gemini-ER-1.5	1.33	6.67	7.33	3.00	6.00	5.67	18.79	18.06	19.38
	Qwen3-VL-235B	6.67	9.67	7.67	2.67	3.00	3.33	22.33	22.00	19.00
4 Image/Object Set	GPT-5	27.00	29.33	30.00	9.67	13.00	11.00	32.67	32.00	37.00

Table 4: Model Accuracies (%) Across Different Set Cardinalities. All results are in the base setting (i.e. no images had any bounding boxes or segmentation map overlays on them).

Model Name	Across All 3 and 5 Set Cardinality Ordering Tasks
	Base	Segmentation Map	Bounding Box
GPT-5	1.83 $\pm$ 1.49	2.06 $\pm$ 1.02	3.17 $\pm$ 1.28
GPT-5-Nano	3.33 $\pm$ 2.39	3.94 $\pm$ 1.39	3.49 $\pm$ 2.37
Gemini-ER-1.5	3.96 $\pm$ 3.43	3.99 $\pm$ 2.37	3.53 $\pm$ 1.06
Qwen3-VL-235B	1.61 $\pm$ 1.29	2.92 $\pm$ 3.11	2.52 $\pm$ 2.14

Table 5: Mean and Standard Deviation of Reasoning Difference

Task Format	Model Name	Temporal	Depth	Left-to-Right
		Ordering	Ordering	Ordering
5 Image/Object Set	GPT-5	8.00	5.67	23.00
	GPT-5-Nano	1.00	5.00	12.67
	Gemini-ER-1.5	1.67	6.33	22.33
	Qwen3-VL-235B	0.33	3.33	22.00
3 Image/Object Set	GPT-5	42.33	33.00	60.00

Table 6: Model Accuracies (%) for High-Reasoning Merge Sort Based Ordering in Base Image Setting

Panel of Figure 2 — Figure 2: Comparison of how model performance scales with set cardinality in temporal and depth ordering tasks in the base setting. In the temporal high reasoning setting, model performance scales predictably with increase in set cardinality, maintaining a consistent relative order across all set cardinalities. However, the relative order is inconsistent in the remaining settings in the figure.

Model	3 Image/Object Spatiotemporal Ordering			Downstream Reasoning
	Temporal	Depth	Left-to-Right	SAT	RoboFAC (MCQ)	RoboFAC (Open-Ended)	Value Est. (RT-1)	Value Est. (Dobb-E)
Base	18.00	22.33	53.67	70.67	62.55	12.28	0.0066	0.0123
SFT (Temporal)	50.67	26.00	55.33	70.67	60.21	12.92	0.0001	0.0114
SFT (Depth)	22.67	31.67	56.67	68.67	60.26	13.33	0.0047	0.0015
SFT (Left-to-Right)	23.67	24.67	61.33	69.33	58.91	13.16	0.0080	0.0062
GRPO (Temporal)	22.67	26.00	51.67	68.67	63.13	11.94	0.0060	0.0036

Table 7: Finetuned Qwen3VL-8B Results on Spatiotemporal Ordering and Downstream Reasoning Tasks. Value estimation results are average value-order correlations (Appendix B ), while all other results are question-answering accuracies (in %).

Model	Task Format	Training	Temporal	Depth	Left-to-
Name		Method			Right
Qwen2-VL-2B	5 Images/Objects	Base	1.00	0.66	0.67
		GRPO	2.00	16.33	17.33
		SFT	0.33	10.33	15.33
	3 Images/Objects	Base	17.33	15.66	16.00
		GRPO	21.67	66.00	67.66

Table 8: Model Accuracies (%) for GRPO and SFT

Model Name	Training Method	Temporal	Depth	Left-to-Right
Qwen2-VL-2B	Base	17.33	15.66	16.00
	GRPO-300	19.00	37.00	54.00
	GRPO-500	20.00	56.00	63.66
	GRPO-700	21.67	66.00	67.66
SmolVLM2-2.2B	Base	13.00	8.00	12.33
	GRPO-300	15.33	24.33	45.00

Table 9: 3 Image/Object ordering model accuracies (%) after GRPO under different training data sizes

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Left	Right	Wrist	Multi-
Name	View	View	Camera	View
GPT-5	50.67	46.67	47.67	47.33
GPT-5-Nano	26.33	21.00	22.33	23.67
Gemini-ER-1.5	42.33	42.00	45.00	38.00
Qwen3-VL-235B	36.00	28.67	35.33	32.67

Appendix

Table 10: High Reasoning Model Accuracies (%) for Single-View Image in 3 Image Base Setting for Temporal Ordering

Panel of Figure 9 — Figure 9: Task Description: “Remove the lid from the pot | Remove the lid from the black pot. | Remove the glass lid from the pot." Correct Order: 2,3,4,5,1

Panel of Figure 10 — Figure 10: Task Description: “Pick up the object and put it in the light blue cup | Place the metal object inside the light blue cup | Put the silver object in the light blue cup" Correct Order: 2,1,4,5,3

Panel of Figure 11 — Figure 11: Task Description: “Pick up the third spice bottle from the right on the bottom shelf of the bottom spice rack and put it on the table | Remove the third container from the right on the bottom rack and put it on the table | Put the spice bottle on the counter" Correct Order: 2,5,3,4,1

Panel of Figure 12 — Figure 12: Task Description: “Hang the towel on the black stand | Put the towel on the black rod | Pick up the cloth from the table and place it on the black object" Correct Order: 3, 2, 1

Panel of Figure 13 — Figure 13: Task Description: “Hang the towel on the black stand | Put the towel on the black rod | Pick up the cloth from the table and place it on the black object" Correct Order: 3, 2, 1

Figure 14: “order the following objects from closest to farthest from camera in terms of their centroid in the image: bin, robot arm, bowl, bottle, bowl cup" Correct Order: "robot arm", "bottle", "bowl cup", "bowl", "bin"

Figure 15: “order the following objects from closest to farthest from camera in terms of their centroid in the image: container, cup, bottle, tray, robot arm" Correct Order: "container", "cup", "tray", "robot arm", "bottle"

Figure 16: “order the following objects from closest to farthest from camera in terms of their centroid in the image: sink, faucet, trash bin, cardboard box packet, coffee maker" Correct Order: "cardboard box packet", "trash bin", "coffee maker", "sink", "faucet"

Figure 17: “order the following objects from closest to farthest from camera in terms of their centroid in the image: napkin dispenser, paper towel roll, coffee machine" Correct Order: "paper towel roll", "napkin dispenser", "coffee machine"

Figure 18: “order the following objects from closest to farthest from camera in terms of their centroid in the image: napkin dispenser, paper towel roll, coffee machine" Correct Order: "paper towel roll", "napkin dispenser", "coffee machine"

Figure 19: “Arrange the following objects in order from leftmost to rightmost in the image: rug, chair, plant, table, door" Correct Order: "chair", "table", "plant", "door", "rug"

Figure 20: “Arrange the following objects in order from leftmost to rightmost in the image: robot, paper towel, box, microwave, pumpkin" Correct Order: "robot", "pumpkin", "paper towel", "microwave", "box"

Figure 21: “Arrange the following objects in order from leftmost to rightmost in the image: blanket, lamp, pillow blanket, table, book" Correct Order: "blanket", "pillow blanket", "lamp", "book", "table"

Figure 22: “Arrange the following objects in order from leftmost to rightmost in the image: cable, robot arm, spoon" Correct Order: "cable", "spoon", "robot arm"

Figure 23: “Arrange the following objects in order from leftmost to rightmost in the image: cable, robot arm, spoon" Correct Order: "cable", "spoon", "robot arm"

# Images/Objects	Temporal Ordering Task			Depth Ordering Task			Left-to-Right Ordering Task
# Images/Objects	Low	Medium	High	Low	Medium	High	Low	Medium	High
2	419	1204	2912	721	1487	5839	960	1664	5376
3	535	1619	3859	788	2067	5778	896	2432	4416
4	668	2062	5012	985	2262	6745	1024	2496	4992
5	801	2496	121,436	1052	2781	7194	1216	3136	6784
Mergesort 3	N/A	N/A	5701	N/A	N/A	2653	N/A	N/A	6400

Appendix

Table 11: Maximum output tokens permitted across reasoning levels and number of ordering elements.

Model	3 Image/Object Spatiotemporal Ordering			Downstream Reasoning
	Temporal	Depth	Left-to-Right	SAT	RoboFAC (MCQ)	RoboFAC (Open-Ended)	Value Est. (RT-1)	Value Est. (Dobb-E)
Base	18.33	29.00	45.00	48.00	38.54	00.34	0.0000	0.0005
SFT (Temporal)	44.00	25.33	44.00	52.00	37.55	00.25	0.0000	0.0024
SFT (Depth)	23.67	35.00	49.00	42.67	33.91	00.29	0.0002	0.0022
SFT (Left-to-Right)	20.33	24.33	66.33	52.00	38.91	00.30	0.0183	0.0058
GRPO (Temporal)	16.33	27.67	45.33	45.33	21.51	00.03	0.0000	0.0010

Appendix

Table 12: Finetuned Qwen3VL-8B-Thinking Accuracies on Spatiotemporal Ordering and Downstream Reasoning Tasks

VLMs for Robotics

Vision-language models for robotics connect visual and linguistic understanding to grounded navigation, manipulation, and robot control.

See all 657 papers

RoboVista: Evaluating Vision Language Models for Diverse Robot Applications
Jul 6, 2026Shuangyu Xie, Kaiyuan Chen, Ziyang Chen +8VLM Evaluation Embodied QA
University of California, Berkeley · Princeton University · Google DeepMind
Evaluating VLMs' Spatial Reasoning Over Robot Motion: A Step Towards Robot Planning with Motion Preferences
Mar 13, 2026Wenxi Wu, Jingjing Zhang, Martim BrandãoVLM Reasoning VLMs for Robotics
King’s College London · University College London
Lightweight Visual Reasoning for Socially-Aware Robots
Mar 4, 2026Alessio Galatolo, Ronald Cumbal, Alexandros Rouchitsas +3VLM Reasoning VLMs for Robotics
Department of Information Technology, Uppsala University, Sweden.

Jul 6, 2026J/K move · Enter open · S save

RoboVista: Evaluating Vision Language Models for Diverse Robot Applications

Shuangyu Xie, Kaiyuan Chen, Ziyang Chen, Simeon Adebola+7

University of California, Berkeley · Princeton University · Google DeepMind

VLM Evaluation Embodied QA VLM Reasoning VLMs for Robotics

Open paper AskPDF arXiv

Diverse applications for robotics, such as industry and agriculture, require robots to operate across various embodiments, changing visual conditions, and complex planning. Vision-Language Models (VLMs) offer a promising foundation for general-purpose and interpretable robotic reasoning. Aligning VLMs with diverse robot applications requires a modular understanding of the individual decision components that underlie robotic behavior. Capturing such structure is challenging for conventional robot benchmarks that are primarily based on teleoperated, end-to-end datasets. We propose Robot Question Answering (RQA), a modular evaluation framework and RoboVista, a benchmark curated from real robotic systems, research papers, and expert annotations. RoboVista contains 474 Visual Question Answering (VQA) instances with human annotated reasoning and covers 39 unique task types in agricultural, industrial, domestic, surgical robotics, autonomous driving, and open robot datasets. Experiments on RoboVista show that state-of-the-art VLMs exhibit substantial gaps. Physical robot experiments suggest strong correlation between RoboVista performance and real-world task execution.

Abstract

VLMs for Robotics

Explore similar work

RoboVista: Evaluating Vision Language Models for Diverse Robot Applications

Evaluating VLMs' Spatial Reasoning Over Robot Motion: A Step Towards Robot Planning with Motion Preferences

Lightweight Visual Reasoning for Socially-Aware Robots