Plan-Bench: Can Existing VLMs Go Well Beyond Perception to Reason and Plan in Embodied Worlds?

Juntong Feng; Qingan Liu; Jingru Dong; Yue Zhu; Weiming Zheng; Yujie Cheng; Xinzhe Han; Shuai Hao

cs.CVOct 13, 2026

Plan-Bench: Can Existing VLMs Go Well Beyond Perception to Reason and Plan in Embodied Worlds?

Authors: Juntong Feng, Qingan Liu, Jingru Dong, Yue Zhu, Weiming Zheng, Yujie Cheng, Xinzhe Han, Shuai Hao, +3 more

Organizations: Dalian University of Technology · Nanyang Technological University

Abstract

Vision-Language Models (VLMs) have achieved strong performance in visual perception and spatial reasoning, leading to growing interest in using them as high-level planners for embodied AI. However, physical tasks require models to integrate spatial information, reason over temporal states, and generate multi-step plans under various constraints, raising the question of whether current VLMs can reliably support such reasoning and planning independently of execution. We introduce Plan-Bench, a benchmark for evaluating the high-level reasoning and multi-step planning capabilities of VLMs relevant to embodied worlds, while isolating the influence of action execution. It covers three complementary dimensions, spanning 16 subclasses and 699 curated samples. Extensive evaluation of 28 VLMs reveals room for improvement, with even the strongest model achieving only 75.0% accuracy. We further find that model scaling generally improves these capabilities. Ablation studies also indicate larger models show increasing potential as high-level planners. Overall, Plan-Bench provides a lightweight probe for evaluating VLMs' reasoning and planning capabilities relevant to embodied environments and assessing their potential as decision makers for physical tasks. Our benchmark and code are available at https://github.com/MinneFeng/Plan-Bench.

Figures & tables

Figure 1: Overview of Plan-Bench. The left illustrates the limitations of existing benchmarks, which either evaluate static VLM capabilities or couple high-level reasoning with low-level execution. The right presents the composition of Plan-Bench, which evaluates execution-decoupled high-level reasoning and planning across Spatial Cognition, Temporal Reasoning, and Multi-Step Planning.

Figure 2: Overview of the Plan-Bench construction pipeline, including capability-driven task definition, dual-source data acquisition, manual annotation, and quality assurance.

Panel of Figure 3 — Figure 3: Illustration of representative samples from the three dimensions of Plan-Bench, including Spatial Cognition, Temporal Reasoning, and Multi-Step Planning.

Model	Spatial	Temporal	Multi-Step	Overall
	Cognition	Reasoning	Planning	(699)
Random Baseline	10.4	4.2	4.3	6.5
$\blacktriangledown$ Commercial Models
Qwen3.8-Max ( Qwen Team, 2026a )	66.4	78.1	81.7	75.0
Kimi-K3 ( Team et al., 2026b )	69.5	78.6	63.5	70.2
GPT-5.6-sol ( OpenAI, 2026 )	56.4	69.5	73.0	65.8

Table 1: Comparison of different VLMs on Plan-Bench. We report performance across the three task categories: Spatial Cognition, Temporal Reasoning, and Multi-Step Planning, together with the overall accuracy. The best results in each column are bolded. The random baseline is computed by assigning 0% to open-ended questions and 25% to single-choice questions. All values are rounded to the nearest tenth of a percentage point and reported as percentages (%).

Figure 4: Fine-grained performance of 8 representative VLMs across the 16 subtasks in Plan-Bench. Each cell reports accuracy (%), with color intensity indicating performance.

Figure 7: Comparison between Thinking and Non-Thinking modes on Plan-Bench across Spatial Cognition, Temporal Reasoning, Multi-Step Planning, and overall accuracy. Blue and orange markers denote Thinking and Non-Thinking results, while the annotated values indicate their gaps.

Figure 8: Representative Examples of Hallucination Errors by Qwen3.8-Max

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Dataset composition of Plan-Bench across three dimensions and 16 subtasks.

Task	Subtask	Single	Open	Data Source
Spatial Cognition	Action Recovery	30	20	Robomme ( Dai et al., 2026 )
	Mirror Distinction	0	35	MSD ( Yang et al., 2019 ) PMD ( Lin et al., 2020 )
	Occlusion Order	18	32	three.js ( Danchilla, 2012 )
	Spatial Relations	0	44	three.js ( Danchilla, 2012 )
	State Recognition	20	20	three.js ( Danchilla, 2012 )
	Target Discovery	40	0	PMD ( Lin et al., 2020 )

Appendix

Table 2: Data sources and question-type distribution across the 16 subtasks in Plan-Bench.

Table 11

Steps	Samples	Percentage
1–3 steps	70	30.43%
4–6 steps	131	56.96%
7–10 steps	29	12.61%
Total	230	100%

Appendix

Table 5: Distribution of planning lengths in the Multi-Step Planning subset of Plan-Bench.

Category	Before Filtering	Removed	Retained
Spatial Cognition	952	693	259
Temporal Reasoning	768	558	210
Multi-Step Planning	860	630	230
Total	2580	1881	699

Appendix

Table 6: Category distribution before and after sample filtering.

Filtering Reason	Removed Samples	Percentage
Insufficient visual evidence or ambiguous answers	637	33.9%
Incorrect annotation or reference answer	364	19.4%
Capability mismatch with target task	297	15.8%
Duplicate or highly correlated scenes	512	27.2%
Other quality issues	71	3.8%
Total removed	1881	72.91%

Appendix

Table 7: Statistics of sample filtering during Plan-Bench construction.

Figure 10: Examples of answer hints for multiple valid answers.

Model	OO	MD	AR	TD	SR	SpR
$\blacktriangledown$ Commercial Models
GPT-5.6-sol	58.0	82.9	54.0	30.0	87.5	31.3
Claude-Opus-4.8	64.0	74.3	28.0	27.5	42.5	22.7
Gemini-3.1-Pro-Preview	22.0	45.7	28.0	30.0	42.5	75.0
$\blacktriangledown$ Large Open-Weight Models
Qwen3.5-27B	18.0	54.3	36.0	27.5	47.5	56.8

Appendix

Table 8: Performance of different VLMs on the six Spatial Cognition subtasks of Plan-Bench.

Panel of Table 10 — Table 10: Performance of different VLMs on the five Multi-Step Planning subtasks of Plan-Bench.

Explore similar work

ViPlan: A Benchmark for Visual Planning with Symbolic Predicates and Vision-Language Models
May 19, 2025Matteo Merler, Nicola Dainese, Minttu Alakuijala +5Large Language Model-Based Robot Planning Symbolic Planning
Fondazione Bruno Kessler · Department of Computer Science, Aalto University · Department of Mathematics, Università degli Studi di Padova
PlanBench-V: A Spatial Planning Map Benchmark for Vision-Language Models
Jun 4, 2026Minxin Chen, He Zhu, Junyou Su +3VLM Evaluation Spatial Reasoning Benchmarks
1Behavioral and Spatial AI Lab, Tongji University · 2Behavioral and Spatial AI Lab, Peking University · College of Architecture and Urban Planning, Tongji University
Planning Takes More Than Token Prediction: Causal Plan for Benchmarking and Building Physically Grounded Embodied Reasoners
Jun 1, 2026Zirui Song, Zheng Lu, Mingqi Gao +8Physical Reasoning AI Agent Benchmarks
MBZUAI · Tsinghua University · Microsoft Research Asia +2

May 19, 2025J/K move · Enter open · S save

ViPlan: A Benchmark for Visual Planning with Symbolic Predicates and Vision-Language Models

Matteo Merler, Nicola Dainese, Minttu Alakuijala, Giovanni Bonetta+4

Fondazione Bruno Kessler · Department of Computer Science, Aalto University · Department of Mathematics, Università degli Studi di Padova

Large Language Model-Based Robot Planning Symbolic Planning Visually Grounded Reasoning

Open paper AskPDF arXiv

Integrating Large Language Models with symbolic planners is a promising direction for obtaining verifiable and grounded plans, with recent works extending this idea to visual domains using Vision-Language Models (VLMs). However, an open-source benchmark for comparing these approaches under matched conditions is missing, due to a lack of visual benchmarks that support symbolic planning. We present ViPlan, the first open-source benchmark for comparing VLM-grounded symbolic approaches (VLM-as-grounder) with direct VLM planning methods (VLM-as-planner). ViPlan introduces a series of increasingly challenging tasks in two visual domains: a visual variant of the classic Blocksworld planning problem and a simulated household robotics environment. Averaged across methods, we find VLM-as-grounders to outperform direct VLM planning in Blocksworld (solving 46% of the tasks against 9%), where image grounding is both crucial and accurate. However, in the household robotics tasks, where linguistic knowledge helps, VLM-as-planner methods are greatly superior to VLM-as-grounder approaches (solving 34% of the tasks against 5%), which are hindered by partial observability. Thus, ViPlan domains capture fundamental shortcomings of both planning approaches, which we further diagnose with a qualitative failure analysis. Finally, across methods, we observe no consistent benefit from Chain-of-Thought prompting, suggesting persistent limitations in current VLMs' visual reasoning abilities.

Abstract

Explore similar work

ViPlan: A Benchmark for Visual Planning with Symbolic Predicates and Vision-Language Models

PlanBench-V: A Spatial Planning Map Benchmark for Vision-Language Models

Planning Takes More Than Token Prediction: Causal Plan for Benchmarking and Building Physically Grounded Embodied Reasoners