Spa-Bench: A Real-Robot Benchmark for Spatial Grounding in Vision-Language-Action Models

Justin Tien-Smith; Matthew Pidden; Rolandos Alexandros Potamias

cs.ROOct 8, 2026

Spa-Bench: A Real-Robot Benchmark for Spatial Grounding in Vision-Language-Action Models

Authors: Justin Tien-Smith, Matthew Pidden, Rolandos Alexandros Potamias

Organizations: Imperial College London

Abstract

Vision-language-action models (VLAs) offer a promising route toward general-purpose robotics by transferring semantic knowledge from large-scale vision-language pretraining into language-conditioned control. However, most VLA evaluations emphasize aggregate manipulation success under changes in objects, instructions, or environments, leaving unclear how reliably these policies ground spatial descriptions in the physical scenes where they act. We therefore ask a sharper question: how reliably can current VLAs translate spatial descriptions into appropriate real-robot behavior? To investigate this, we introduce Spa-Bench, a real-robot benchmark that evaluates spatial grounding across six tabletop tasks covering Object State, Relative Size, Referential Description, Relational Placement, Ordinal Position, and Counting. Our primary evaluation compares in-distribution (ID) and out-of-distribution (OOD) spatial configurations while retaining training support for the relevant objects, concepts, and actions, reducing object and motor unfamiliarity as alternative explanations for failure. Four diagnostic conditions further probe physical execution, sensitivity to wording, object-task associations, and recognition that no action is required. We release 4,092 episodes collected on the SO-101 robot arm: 1,200 human-teleoperated demonstrations and 2,892 real-robot evaluation rollouts from five state-of-the-art VLA policies. Across the three fully evaluated policies, success falls from 67.5-73.3% on ID spatial configurations to 40.7-52.0% on OOD spatial configurations, while matched direct instructions improve success by 28.3-40.0 percentage points. These results show that the evaluated VLAs can often execute the required manipulations but lack robust spatial grounding. Project materials, including videos, code, and benchmark data, are available at https://justintiensmith.github.io/Spa-Bench/.

Figures & tables

Task	Grounding Axis	Example Instruction	Success Criterion
Object State	Attribute	“Place the closed white cup in the bowl.”	The same-category object whose object state matches the instruction is placed in the bowl.
Relative Size	Attribute	“Place the smallest object in the bowl.”	The object satisfying the requested relative-size description is placed in the bowl.
Relational Placement	Relation	“Place the green pen to the left of the red block.”	The target finishes within 3 inches of the reference and within the requested $\pm 45^{\circ}$ directional sector.
Referential Description	Relation	“Place the pencil sharpener furthest from the yellow pen in the bowl.”	The object satisfying the specified relation to the reference object is placed in the bowl.
Ordinal Position	Scene structure	“Place the third object from the left in the bowl.”	The object occupying the requested ordinal position is placed in the bowl.
Counting	Scene structure	“Leave exactly two objects in the bowl.”	The requested number of objects remains in the bowl at the end of the episode.

TABLE I: Spa-Bench tasks. All tasks use the same robot, workspace, receptacle, and broad pick-and-place behavior while varying the spatial information needed to determine the correct response.

Fig. 2: Object inventory used in Spa-Bench. The benchmark uses 27 movable object instances in training and evaluation spanning 11 morphological families. These objects are recombined across the six tasks while the robot embodiment, workspace, and receptacle remain fixed.

Fig. 3: Withheld-cell construction for Object State . Rows denote state concepts and columns denote object identities. Observed cells receive state-conditioned instructions during fine-tuning, whereas withheld cells are reserved for OOD evaluation. The concept upside-down appears with the striped cup, and the white cup is manipulated under a direct instruction, but the pairing upside-down white cup is withheld from spatially grounded training.

Fig. 4: Matched-scene diagnostics. The same scene setup and intended object motion are evaluated under four instruction conditions: OOD spatial, direct manipulation, and a paraphrase of each. In this Referential Description example, the direct instruction names the target explicitly, whereas the spatial instruction requires resolving it from the scene.

Fig. 5: Task-level ID spatial-grounding success. Bars show per-task success rates, and whiskers indicate two-sided Wilson 95% confidence intervals. Complete ID evaluations contain 20 trials per task (120 total); VLA-0 contains 72 trials, with at least 10 per task. GR00T-N1.7 F-FT and VLA-0 are reported only as diagnostic ID baselines because their low ID success would make subsequent OOD failures difficult to interpret.

Fig. 6: Task-level OOD spatial-grounding and direct-manipulation success. Colors denote policies; dark and light bars denote OOD spatial and direct-manipulation instructions, respectively. Whiskers show two-sided Wilson 95% confidence intervals. OOD bars use 50 trials per task and direct bars use 20; the matched comparison in Sec. V-B uses only the corresponding 20 OOD scenes.

Fig. 7: Prompt-paraphrase diagnostic on 60 matched scenes. Dark bars show original instructions; light bars show paraphrased instructions for the same scenes. Whiskers show two-sided Wilson 95% confidence intervals.

Fig. 8: Cross-task-object diagnostic. Bars compare ID spatial-grounding success with standard task objects and with familiar objects transferred from other task families. Each condition contains 100 rollouts across five comparable tasks; whiskers show two-sided Wilson 95% confidence intervals.

Fig. 9: No-intervention success in four non-relational diagnostics. The top row shows representative rollouts; the bars summarize all trials. Success requires withholding task-directed movement when the requested count already holds, the requested ordinal position is invalid, or no object satisfies the requested state or referential description. Each task contains 20 rollouts per policy; the four-task aggregate contains 80. Whiskers show two-sided Wilson 95% confidence intervals. Relational Placement is evaluated separately using matched corrective-movement controls.

Fig. 10: Relational Placement action-gating control. The same relation instruction is evaluated when the requested relation already holds and when corrective movement is required. Success is high in the no-movement condition but falls sharply when the policy must correct the relation, indicating that remaining stationary does not necessarily reflect reliable recognition that the goal is satisfied. Each condition contains 20 rollouts per policy; whiskers show two-sided Wilson 95% confidence intervals.

Fig. 11: Primary failure-mode composition for OOD spatial configurations. Bars are normalized within each task–model pair, and $n$ denotes the number of failures. Wrong-instance errors are separated from other wrong-target errors only for Object State and Referential Description .

Fig. 12: Withheld-cell construction with preserved training support. Left: a toy color–object example. Color language training supports the observed pairing yellow ball , while a direct instruction demonstrates manipulation of the evaluation object without naming its color. The pairing yellow cube is withheld from spatially grounded training and reserved for OOD evaluation. Right: the corresponding construction for Object State . The concept upside-down appears with the striped cup, and the white cup is manipulated under a direct instruction, but the pairing upside-down white cup is withheld from spatially grounded training. OOD evaluation requires selecting the upside-down white cup rather than an upright white cup in the same scene.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 13: Complete Spa-Bench concept–argument matrices. Observed cells receive concept-conditioned instructions during fine-tuning; withheld cells are reserved for OOD spatial evaluation. Grey cells denote combinations that are absent or not applicable. The three-object Ordinal Position matrix marks the middle position as absent because those training sequences target only the leftmost and rightmost positions.

Condition	Trials	Diagnostic question
ID spatial configuration	20	Can the policy perform a concept–argument combination explicitly represented during fine-tuning?
OOD spatial configuration	50	Can the policy recombine individually supported concepts, arguments, and physical behaviors in a withheld configuration?
Matched direct manipulation	20	Can the policy execute the corresponding object motion in the same nominal scene when task-specific spatial inference is removed?
Prompt paraphrasing	20	Does performance change when 10 OOD spatial and 10 direct-manipulation instructions are rephrased?
Cross-task object	20	Does an ID spatial behavior transfer to an object associated with a different Spa-Bench task?
No intervention	20	Can the policy refrain from acting when the requested goal already holds or no valid referent exists? Not used for Relative Size .

Appendix

TABLE II: Spa-Bench evaluation conditions. Counts are per task unless otherwise noted.

Policy	Initialization	Data / updates	$H$	Updated components	Action/state processing
$\pi_{0.5}$	lerobot/pi05_base	Full / 76,596	50	Vision-language backbone and action expert	Quantile normalization
MolmoAct2	allenai/MolmoAct2	Full / 76,596	30	VLM and action expert; token embeddings frozen	Min–max normalization
VLA-0	Qwen2.5-VL-3B-Instruct	Trimmed / 77,246	8	Full Qwen backbone; no LoRA or QLoRA	Min–max scaling; 1,000 text bins
GR00T-N1.7 F-LLM	GR00T-N1.7-3B	Trimmed / 71,304	16	Visual encoder, multimodal projector, VLLN, and diffusion-action model; LLM frozen	GR00T-native processing
GR00T-N1.7 F-FT	GR00T-N1.7-3B	Full / 76,596	16	Language, visual, multimodal, and diffusion-action components	GR00T-native processing

Appendix

TABLE III: Resolved model-adaptation settings. Full and trimmed denote the 612,733-frame and 570,386-frame dataset variants, respectively;

H

is the predicted action horizon.

Policy	ID	All OOD	Matched OOD	Matched direct
GR00T-N1.7 F-LLM	81/120 (67.5%, 58.7–75.2)	133/300 (44.3%, 38.8–50.0)	50/120 (41.7%, 33.2–50.6)	98/120 (81.7%, 73.8–87.6)
$\pi_{0.5}$	82/120 (68.3%, 59.6–76.0)	122/300 (40.7%, 35.3–46.3)	54/120 (45.0%, 36.4–53.9)	100/120 (83.3%, 75.7–88.9)
MolmoAct2	88/120 (73.3%, 64.8–80.4)	156/300 (52.0%, 46.4–57.6)	73/120 (60.8%, 51.9–69.1)	107/120 (89.2%, 82.3–93.6)

Appendix

TABLE IV: Aggregate results for the three fully evaluated policies. Matched OOD and matched direct manipulation use the same 120 nominal scene specifications.

Task	GR00T-N1.7 F-LLM	$\pi_{0.5}$	MolmoAct2
Counting	12/20; 60.0% (38.7–78.1)	11/20; 55.0% (34.2–74.2)	12/20; 60.0% (38.7–78.1)
Ordinal Position	9/20; 45.0% (25.8–65.8)	7/20; 35.0% (18.1–56.7)	10/20; 50.0% (29.9–70.1)
Relational Placement	14/20; 70.0% (48.1–85.5)	13/20; 65.0% (43.3–81.9)	16/20; 80.0% (58.4–91.9)
Object State	17/20; 85.0% (64.0–94.8)	16/20; 80.0% (58.4–91.9)	18/20; 90.0% (69.9–97.2)
Relative Size	17/20; 85.0% (64.0–94.8)	19/20; 95.0% (76.4–99.1)	17/20; 85.0% (64.0–94.8)
Referential Description	12/20; 60.0% (38.7–78.1)	16/20; 80.0% (58.4–91.9)	15/20; 75.0% (53.1–88.8)

Appendix

TABLE V: Task-level ID spatial-grounding results. Each cell reports successes/20, success rate, and Wilson 95% confidence interval.

Task	GR00T-N1.7 F-LLM	$\pi_{0.5}$	MolmoAct2
Counting	0/50; 0.0% (0.0–7.1)	0/50; 0.0% (0.0–7.1)	0/50; 0.0% (0.0–7.1)
Ordinal Position	21/50; 42.0% (29.4–55.8)	23/50; 46.0% (33.0–59.6)	27/50; 54.0% (40.4–67.0)
Relational Placement	39/50; 78.0% (64.8–87.2)	36/50; 72.0% (58.3–82.5)	25/50; 50.0% (36.6–63.4)
Object State	17/50; 34.0% (22.4–47.8)	18/50; 36.0% (24.1–49.9)	38/50; 76.0% (62.6–85.7)
Relative Size	29/50; 58.0% (44.2–70.6)	27/50; 54.0% (40.4–67.0)	32/50; 64.0% (50.1–75.9)
Referential Description	27/50; 54.0% (40.4–67.0)	18/50; 36.0% (24.1–49.9)	34/50; 68.0% (54.2–79.2)

Appendix

TABLE VI: Task-level OOD spatial-grounding results. Each cell reports successes/50, success rate, and Wilson 95% confidence interval.

Task	GR00T-N1.7 F-LLM	$\pi_{0.5}$	MolmoAct2
Counting	12/20; 60.0% (38.7–78.1)	11/20; 55.0% (34.2–74.2)	16/20; 80.0% (58.4–91.9)
Ordinal Position	19/20; 95.0% (76.4–99.1)	20/20; 100% (83.9–100)	17/20; 85.0% (64.0–94.8)
Relational Placement	17/20; 85.0% (64.0–94.8)	18/20; 90.0% (69.9–97.2)	17/20; 85.0% (64.0–94.8)
Object State	19/20; 95.0% (76.4–99.1)	18/20; 90.0% (69.9–97.2)	20/20; 100% (83.9–100)
Relative Size	19/20; 95.0% (76.4–99.1)	20/20; 100% (83.9–100)	20/20; 100% (83.9–100)
Referential Description	12/20; 60.0% (38.7–78.1)	13/20; 65.0% (43.3–81.9)	17/20; 85.0% (64.0–94.8)

Appendix

TABLE VII: Task-level matched direct-manipulation results. Each cell reports successes/20, success rate, and Wilson 95% confidence interval.

Condition	GR00T-N1.7 F-LLM	$\pi_{0.5}$	MolmoAct2
Original direct	49/60; 81.7% (70.1–89.4)	52/60; 86.7% (75.8–93.1)	54/60; 90.0% (79.9–95.3)
Paraphrased direct	42/60; 70.0% (57.5–80.1)	52/60; 86.7% (75.8–93.1)	54/60; 90.0% (79.9–95.3)
Original OOD spatial	29/60; 48.3% (36.2–60.7)	29/60; 48.3% (36.2–60.7)	39/60; 65.0% (52.4–75.8)
Paraphrased OOD spatial	23/60; 38.3% (27.1–51.0)	25/60; 41.7% (30.1–54.3)	30/60; 50.0% (37.7–62.3)

Appendix

TABLE VIII: Prompt-paraphrase results on the corresponding 60-scene subsets.

Condition	GR00T-N1.7 F-LLM	$\pi_{0.5}$	MolmoAct2
Standard objects, five-task ID	64/100; 64.0% (54.2–72.7)	66/100; 66.0% (56.3–74.5)	70/100; 70.0% (60.4–78.1)
Cross-task objects, five-task ID	55/100; 55.0% (45.2–64.4)	54/100; 54.0% (44.3–63.4)	53/100; 53.0% (43.3–62.5)
Counting: goal already satisfied	17/20; 85.0% (64.0–94.8)	16/20; 80.0% (58.4–91.9)	16/20; 80.0% (58.4–91.9)
Ordinal Position: invalid ordinal	0/20; 0.0% (0.0–16.1)	0/20; 0.0% (0.0–16.1)	0/20; 0.0% (0.0–16.1)
Object State: no valid target	0/20; 0.0% (0.0–16.1)	0/20; 0.0% (0.0–16.1)	0/20; 0.0% (0.0–16.1)
Referential Description: no candidate	0/20; 0.0% (0.0–16.1)	0/20; 0.0% (0.0–16.1)	3/20; 15.0% (5.2–36.0)

Appendix

TABLE IX: Cross-task-object and action-gating diagnostics. Cross-task rows contain 100 trials; non-relational task rows and relational-control rows contain 20 trials; the four-task aggregates contain 80 trials.

Failure mode	Operational description
No action	The policy does not initiate task-directed arm or gripper motion when movement is required.
Wrong object	The policy approaches or manipulates an object outside the intended target category or otherwise selects the wrong target.
Correct category, wrong instance	The policy selects an object from the requested semantic category but chooses the instance with the wrong state or referential relation. Used only for Object State and Referential Description .
Grasp failure	The policy selects the intended target but fails to establish or maintain a usable grasp before transport.
Incorrect spatial outcome	The intended object is approached or manipulated, but the final count, direction, relation, or placement is incorrect.
Premature release or drop	The policy releases the object before reaching the intended goal or unintentionally drops it during transport.

Appendix

TABLE X: Primary qualitative failure-mode taxonomy.

Fig. 14: Primary failure-mode composition for ID spatial configurations. Bars are normalized within each task–policy pair, and $n$ denotes the number of failed rollouts. Results include all five policies; VLA-0 has fewer ID evaluation trials than the other policies (Fig. 5 ). Wrong-instance errors are separated from other wrong-target errors only for Object State and Referential Description . Some bars represent very few failures and are therefore descriptive.

Explore similar work

Sep 4, 2026

RoboSPA: Can VLA Models Go Beyond Simple Scenes and Short-Horizon Tasks?

Vision-Language-Action (VLA) models have shown promising progress in language-conditioned robotic manipulation. However, existing datasets and benchmarks mainly evaluate task completion under predefined settings, offering limited insight into model reasoning under increasing spatial and procedural complexity. We introduce RoboSPA (Robot Spatial-Procedural Assessment), a large-scale robotic manipulation dataset and benchmark for diagnosing embodied reasoning in VLA models. RoboSPA focuses on two core dimensions, Fine-Grained Spatial Reasoning and Long-Horizon Procedural Planning, covering 10 task categories and 56 base tasks. Each task is instantiated across five difficulty levels, yielding 280 variants with increasing spatial ambiguity and procedural complexity. We collect 527K trajectories across multiple embodiments and diverse scenes. Beyond binary success rate, RoboSPA introduces diagnostic metrics for more detailed evaluation. Experiments on representative VLA models show that current systems still struggle with complex spatial relations, precise low-level execution, and memory-intensive planning. These results establish RoboSPA as a challenging diagnostic benchmark for developing more capable, reliable, and generalizable embodied agents. Our data and code are available at https://github.com/fanzhenxuan/RoboSPA.

Zhenxuan Fan, Bo Zhang, Yutong Lin +9

Zhejiang University · University of Electronic Science and Technology of China · South China Normal University

Robot Manipulation Benchmarks Reasoning for VLA Models Robotic Manipulation Long-Horizon Robotic Manipulation

Ask PDF

Jun 9, 2026

VeriSpace: Spatially Grounded Action Verification for Vision-Language-Action Models

Vision-language-action (VLA) models have shown strong promise for robotic manipulation, but their reliability at test time remains limited by one-shot action prediction, where even small action errors can cause grasp failure, collision, or incorrect task progression. A natural alternative is to equip VLA systems with test-time verification, allowing multiple candidate actions to be proposed and evaluated before execution. However, reliable action verification is challenging because it requires not only distinguishing subtle geometric differences between candidate actions, but also assessing whether an action makes meaningful progress toward the task goal. We present VeriSpace, a 3D-aware action verifier for test-time action selection in VLA systems. VeriSpace evaluates candidate actions through two key components: Dual-Path 3D-Injected Scene Encoding, which constructs a scene representation that jointly preserves visual semantics and explicit 3D geometry, and Spatially-Grounded Action Reasoning, which evaluates each action by reasoning over task-relevant spatial relations, geometric validity, and expected goal progress. Together, these components enable more reliable discrimination between subtle yet outcome-critical action candidates while remaining fully compatible with existing VLA policies. Experiments on public benchmarks and real-world robotic manipulation tasks show that VeriSpace consistently improves decision reliability over both underlying VLA policies and prior verification-based methods, yielding substantial gains in both in-distribution and out-of-distribution settings.

Guiyu Zhao, Longteng Guo, Junyou Zhu +6

Institute of Automation, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

Visual Spatial Reasoning Robotic Manipulation Vision-Language-Action Models

Ask PDF

Mar 13, 2026

Spatially Grounded Long-Horizon Task Planning in the Wild

Recent advances in robot manipulation increasingly leverage Vision-Language Models (VLMs) for high-level reasoning, such as decomposing task instructions into sequential action plans expressed in natural language that guide downstream low-level motor execution. However, current benchmarks do not assess whether these plans are spatially executable, particularly in specifying the exact spatial locations where the robot should interact to execute the plan, limiting evaluation of real-world manipulation capability. To bridge this gap, we define a novel task of grounded planning and introduce GroundedPlanBench, a newly curated benchmark for spatially grounded long-horizon action planning in the wild. GroundedPlanBench jointly evaluates hierarchical sub-action planning and spatial action grounding (where to act), enabling systematic assessment of whether generated sub-actions are spatially executable for robot manipulation. We further introduce Video-to-Spatially Grounded Planning (V2GP), an automated data generation framework that leverages real-world robot video demonstrations to improve spatially grounded long-horizon planning. Our evaluations reveal that spatially grounded long-horizon planning remains a major bottleneck for current VLMs. Our results demonstrate that V2GP provides a promising approach for improving both action planning and spatial grounding performance, validated on our benchmark as well as through real-world robot manipulation experiments, advancing progress toward spatially actionable planning.

Sehun Jung, HyunJee Song, Dong-Hee Kim +4

Korea University · Microsoft Research · University of Wisconsin-Madison

Robot Task Planning Long-Horizon Robotic Manipulation Robot Manipulation Benchmarks VLMs for Robotics

Ask PDF