Spatially Grounded Long-Horizon Task Planning in the Wild

Sehun Jung; HyunJee Song; Dong-Hee Kim; Reuben Tan; Jianfeng Gao; Yong Jae Lee; Donghyun Kim

cs.ROMar 13, 2026

Spatially Grounded Long-Horizon Task Planning in the Wild

Authors: Sehun Jung, HyunJee Song, Dong-Hee Kim, Reuben Tan, Jianfeng Gao, Yong Jae Lee, Donghyun Kim

Organizations: Korea University · Microsoft Research · University of Wisconsin-Madison

Abstract

Recent advances in robot manipulation increasingly leverage Vision-Language Models (VLMs) for high-level reasoning, such as decomposing task instructions into sequential action plans expressed in natural language that guide downstream low-level motor execution. However, current benchmarks do not assess whether these plans are spatially executable, particularly in specifying the exact spatial locations where the robot should interact to execute the plan, limiting evaluation of real-world manipulation capability. To bridge this gap, we define a novel task of grounded planning and introduce GroundedPlanBench, a newly curated benchmark for spatially grounded long-horizon action planning in the wild. GroundedPlanBench jointly evaluates hierarchical sub-action planning and spatial action grounding (where to act), enabling systematic assessment of whether generated sub-actions are spatially executable for robot manipulation. We further introduce Video-to-Spatially Grounded Planning (V2GP), an automated data generation framework that leverages real-world robot video demonstrations to improve spatially grounded long-horizon planning. Our evaluations reveal that spatially grounded long-horizon planning remains a major bottleneck for current VLMs. Our results demonstrate that V2GP provides a promising approach for improving both action planning and spatial grounding performance, validated on our benchmark as well as through real-world robot manipulation experiments, advancing progress toward spatially actionable planning.

Figures & tables

Panel of Figure 1 — Fig. 1 : Motivation of grounded planning . (a) VLM-as-Planner decomposes high-level instructions into natural language sub-actions, which are then grounded by separate perception modules. However, the lack of explicit spatial specification often leads to ambiguous action grounding. (b) Our GroundedPlanBench jointly annotates and evaluates hierarchical sub-action planning and spatial grounding under both explicit and implicit instructions.

Panel of Figure 2 — Fig. 2 : Task distribution and instruction types in GroundedPlanBench .

Panel of Figure 3 — Fig. 3 : An example of a simplified instruction prompt for spatially grounded planning.

Fig. 4 : V2GP is a training data generation framework designed to enhance spatially grounded sub-action plans from real-world robot demonstration videos through the stages: (1) Temporal sub-action decomposition, where gripper state signals segment demonstrations into sub-action units; (2) Interactive object identification, where a VLM analyzes each segment to identify the actively manipulated objects; (3) Spatial grounding of actions, where SAM3 localizes target objects and placement endpoints using bounding boxes and points; and (4) Spatially grounded task planning, which integrates the grounded sub-action primitives with explicit and implicit task instructions and spatial grounding. The collected data are used to fine-tune VLM-as-Planners, enhancing both hierarchical task planning (what to do) and spatial grounding (where to act) for long-horizon tasks.

Proprietary VLMs
Model	Task Success Rate (TSR) $\uparrow$						Action Recall Rate (ARR) $\uparrow$
	Explicit			Implicit			Explicit			Implicit
	short	medium	long	short	medium	long	short	medium	long	short	medium	long
Decoupled Task Planning + Spatial Grounding
GPT 5.2 + ER1	41.8	13.6	8.4	31.0	10.4	3.6	54.8	48.9	46.7	45.8	42.4	38.4
GPT 5.2 + SAM3 + ER1	36.2	10.1	4.9	28.0	8.7	1.4	47.6	42.4	36.9	41.9	36.2	32.1

TABLE I: Evaluation results on GroundedPlanBench across varying task horizons (short (1–4), medium (5–8), long (9–26)) and instruction types (explicit and implicit instructions).

Fig. 5 : Visualization of decoupled task planning and spatial grounding, where underlined actions are incorrectly grounded due to semantically similar objects ( e.g. , identical napkins). In contrast, V2GP finds the correct correspondence between each action and spatial grounding, enabling sequentially consistent and accurate execution.

Fig. 6 : Qualitative comparison of spatially grounded action plans between ground truth (GT), baseline models, and our method on GroundedPlanBench . Green bounding boxes represent grasp predictions, while red crosses denote placement targets.

Model	TSR (%)	ARR (%)
Qwen3-32B	10.0	48.3
Qwen3-32B + V2GP	70.0	93.3

TABLE II : Performance comparison on real-world experiments.

Fig. 7 : Real-world experimental results from Qwen3-VL-32B+ V2GP . Further results are presented in the video supplement.

Explore similar work

Spa-Bench: A Real-Robot Benchmark for Spatial Grounding in Vision-Language-Action Models
Oct 8, 2026Justin Tien-Smith, Matthew Pidden, Rolandos Alexandros PotamiasVision-Language Grounding Vision-Language-Action Models
Imperial College London
Plan-Bench: Can Existing VLMs Go Well Beyond Perception to Reason and Plan in Embodied Worlds?
Oct 13, 2026Juntong Feng, Qingan Liu, Jingru Dong +8VLM Reasoning VLMs for Robotics
Dalian University of Technology · Nanyang Technological University
RMMBench: A Comprehensive Benchmark for Robotic Mobile Manipulation
Oct 4, 2026Huapeng Li, Fuxiang Feng, Jinqiu Fan +5VLM Evaluation Long-Horizon Robotic Manipulation
School of Control Science and Engineering, Shandong University · Meituan Group

Oct 8, 2026J/K move · Enter open · S save

Spa-Bench: A Real-Robot Benchmark for Spatial Grounding in Vision-Language-Action Models

Justin Tien-Smith, Matthew Pidden, Rolandos Alexandros Potamias

Imperial College London

Vision-Language Grounding Vision-Language-Action Models Robot Manipulation Benchmarks

Open paper AskPDF arXiv

Vision-language-action models (VLAs) offer a promising route toward general-purpose robotics by transferring semantic knowledge from large-scale vision-language pretraining into language-conditioned control. However, most VLA evaluations emphasize aggregate manipulation success under changes in objects, instructions, or environments, leaving unclear how reliably these policies ground spatial descriptions in the physical scenes where they act. We therefore ask a sharper question: how reliably can current VLAs translate spatial descriptions into appropriate real-robot behavior? To investigate this, we introduce Spa-Bench, a real-robot benchmark that evaluates spatial grounding across six tabletop tasks covering Object State, Relative Size, Referential Description, Relational Placement, Ordinal Position, and Counting. Our primary evaluation compares in-distribution (ID) and out-of-distribution (OOD) spatial configurations while retaining training support for the relevant objects, concepts, and actions, reducing object and motor unfamiliarity as alternative explanations for failure. Four diagnostic conditions further probe physical execution, sensitivity to wording, object-task associations, and recognition that no action is required. We release 4,092 episodes collected on the SO-101 robot arm: 1,200 human-teleoperated demonstrations and 2,892 real-robot evaluation rollouts from five state-of-the-art VLA policies. Across the three fully evaluated policies, success falls from 67.5-73.3% on ID spatial configurations to 40.7-52.0% on OOD spatial configurations, while matched direct instructions improve success by 28.3-40.0 percentage points. These results show that the evaluated VLAs can often execute the required manipulations but lack robust spatial grounding. Project materials, including videos, code, and benchmark data, are available at https://justintiensmith.github.io/Spa-Bench/.

Abstract

Explore similar work

Spa-Bench: A Real-Robot Benchmark for Spatial Grounding in Vision-Language-Action Models

Plan-Bench: Can Existing VLMs Go Well Beyond Perception to Reason and Plan in Embodied Worlds?

RMMBench: A Comprehensive Benchmark for Robotic Mobile Manipulation