BEHAVIOR-Spatial Benchmark: Evaluating Spatial Understanding for Robotics

Sirui Chen; Huang Huang; Qineng Wang; Wenlong Huang; Manling Li; Ruohan Zhang; Li Fei-Fei; Jiajun Wu

cs.ROOct 25, 2026

BEHAVIOR-Spatial Benchmark: Evaluating Spatial Understanding for Robotics

Authors: Sirui Chen, Huang Huang, Qineng Wang, Wenlong Huang, Manling Li, Ruohan Zhang, Li Fei-Fei, Jiajun Wu

Organizations: Stanford University · Northwestern University

Abstract

Robot tasks require grounding spatial language in physical scenes, but task success alone cannot establish whether spatial instructions are followed. We present BEHAVIOR-Spatial, a benchmark built on BEHAVIOR-1K that measures spatial question answering, point-based target grounding, and robot action to examine how VLM spatial understanding relates to VLA behavior. It comprises 3,256 spatial questions across 49 activities, 1,240 grounding queries across 14 manipulation tasks, and action evaluations covering those 14 tasks and 10 navigation tasks. Our manipulation protocol evaluates held-out configurations and instruction following when the requested target changes within a fixed training layout. The baseline achieves 41.6% on held-out configurations but only 16.1% under changed instructions. VLM adaptation and robot-data pretraining provide limited improvements, whereas oracle visual target cues raise these rates to 60.3% and 38.0%, respectively. Perception evaluations reveal policy-dependent correlations between VLM question-answering and grounding accuracy and VLA manipulation performance. Together, these findings motivate evaluating spatial instruction following alongside task completion and examining when spatial perception performance translates into robot behavior. Project website: https://arielchen07.github.io/behavior-spatial/.

Figures & tables

Figure 1: BEHAVIOR-Spatial links VQA (Understand), target grounding (Ground), and execution (Act), testing whether VLM spatial capabilities predict VLA task performance.

Figure 2: BEHAVIOR-Spatial Dataset Showcase. Left: Paired task VQA, grounding, and action examples cover direction, distance, and size; navigation shows three context views and the current view. Right: Spatial taxonomy, VQA counts, grounding/IR representations, and task coverage.

Figure 3: Data Construction Pipeline. Demonstration observations and simulator geometry support general VQA and curated spatial tasks. Task instances produce task-grounded VQA, grounding labels, and VLA datasets. Selection criteria guide curation; each suite validates its samples.

Figure 4: BEHAVIOR-Spatial-VLA Architecture. Task specific views share a Qwen3.5-4B backbone and flow matching action expert. Only manipulation uses optional oracle grounding.

Proprietary
Model	Single-frame				Multi-frame
	All	Dir.	Dist.	Size	All	Dir.
GPT-6 Astra	82.1	85.8	77.3	77.7	74.4	74.6
Gemini 3	77.8	80.5	75.2	72.7	71.1	69.6
Fable5.1	70.5	69.3	70.2	76.3	68.4	66.8
GPT-5	64.7	61.7	68.1	69.1	63.2	59.6

Table 1: VLM accuracy on BEHAVIOR-Spatial-VQA. Single-frame Direction includes its locate sub-type; All is pooled over all questions.

Table 6

IR-conditioned
Policy	Held-out		Prompt-probe
	+Soft $\uparrow$	S $\uparrow$	+Soft $\uparrow$	S $\uparrow$	W $\downarrow$
$\pi_{0.5}$ -Qwen (frozen VLM)	41.6	27.9	16.1	10.0	70.0
+ LoRA-VLM	43.5	30.9	17.9	9.6	68.1
+ DROID pretrain init	40.1	24.7	16.6	8.9	66.0
+ dot IR	60.3	44.8	38.0	24.9	43.9

Table 4: Success rates of all policy variants on the two evaluation sets. +Soft = soft success; S = strict success; W = fraction of episodes committing to a non-instructed object.

Writer	+Soft $\uparrow$	S $\uparrow$	W $\downarrow$
Proprietary
Fable 5.1	66.1	51.4	23.5
GPT-6	61.2	47.3	23.0
Gemini 3	50.3	33.3	28.8
Open-weight
Qwen3.5-9B	39.0	5.3	8.7

Table 5: Code-as-policy agent performance. +Soft = soft success; S = strict success; W = fraction of episodes committing to a non-instructed object.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Category	Template	Answer space	$n$
single-frame	direction	“From the viewpoint of {frame_idx} , where is {objA} relative to {objB / robot arm} ?”	left / right / front / back / above / below	300
single-frame	locate	“From the viewpoint of {frame_idx} , where is {objA} located on {objB} ?”	container region (layer/level/grid)	300
single-frame	distance	“Measuring from the center of each object, which object is the closest to {objA / camera / robot arm} ?”	candidate objects	339
single-frame	size	“Measuring from the length of the longest dimension (length, width, or height), which object is the longest?”	candidate objects	139
multi-frame	direction	“From the viewpoint of the last frame, measuring from room centers, where is {objA} relative to {objB} ?”	front-left / front-right / back-left / back-right	401
multi-frame	distance	“Based on the given frames, measuring from room centers, which room is the closest to the robot’s position at the last frame?”	candidate rooms	77

Appendix

Table S1: BEHAVIOR-Spatial-VQA general subset templates. Question templates, answer spaces, and evaluation counts.

Figure S1: Manipulation tasks of BEHAVIOR-Spatial. Representative head-camera views of the 14 tasks used for both training and evaluation, plus one training-only task. Color badges indicate the queried spatial concept. Table S2 lists instructions and prompt variants.

Figure S2: The 10 navigation tasks of BEHAVIOR-Spatial. Each row shows ten evenly sampled frames from a source demonstration, from the starting view to the instructed target. Routes include in-room approaches, open-area traversals, and passage through doorways. Instructions follow the template “Move to the {target}” (Table S2 ).

Task	Category	Example instruction	Instruction variants	$n$
Manipulation
reach_food_item	size	“Reach the largest food item on the chopping board.”	{largest, 2nd largest, smallest, 2nd smallest}	32
reach_item	size	“Reach the smallest food item on the countertop.”	{largest, 2nd largest, smallest, 2nd smallest}	32
reach_plate	size	“Reach the largest plate from the countertop.”	{largest, 2nd largest, smallest, 2nd smallest}	31
reach_toys	size	“Reach the second largest toy on the desk.”	{largest, 2nd largest, 3rd largest, smallest, 2nd smallest}	40
reach_dice	distance	“Reach the dice closest to the board game on the bed.”	{closest, farthest} $\times$ {board game, wall}	32

Appendix

Table S2: Task inventory of BEHAVIOR-Spatial. The benchmark comprises 14 manipulation tasks and 10 navigation tasks; one additional manipulation task is used only for training. Manipulation variants change the spatial relation or reference object;

\times

denotes all combinations. Navigation evaluation varies validated starting poses rather than instructions.

n

denotes held-out evaluation instances.

Figure S3: Complete navigation context tours (1/2). Context frames for Two-floor house, main floor; Single-floor house, toy room; and Single-floor house, bedroom. Within each scene, frames are arranged in input order, from left to right and then top to bottom. Numbers indicate one-based positions in the context sequence. The remaining scenes appear in Figure S4 .

Figure S4: Complete navigation context tours (2/2). Context frames for Single-floor house, open living area; and Restaurant diner. Using the ordering and numbering convention of Figure S3 .

Policy	Batch	Peak LR	LR schedule	Checkpoint
Manipulation
$\pi_{0.5}$ -Qwen (frozen VLM)	256	$4\times 10^{-4}$	cosine	100k
+ LoRA-VLM	128	$2.8\times 10^{-4}$	cosine	110k
+ DROID pretrain init	256	$4\times 10^{-4}$	cosine	90k
+ IR variants (all)	256	$4\times 10^{-4}$	cosine	100k
Navigation

Appendix

Table S3: Policy training hyperparameters. Checkpoint is the training step whose weights produce all reported results.

Figure S5: Spatial intermediate representations. Examples from distance (top) and size (bottom) tasks. Each row shows the RGB head-camera view, a dot IR marking the instructed target, and a Gaussian attention map centered at the same location.

Panel of Figure 17 — Figure S6: Navigation initial poses and scene layouts. Blue arrows show evaluation starting positions and headings; orange points show cached training-trajectory poses for the corresponding house task. Training trajectories are unavailable for the diner tasks. Gray polygons show projected furniture bounding boxes, yellow polygons highlight target objects, and stars mark destination reference points. Bounding-box projections are schematic, not exact collision boundaries.

Figure S7: IR scoring on dataset frames. (a) Strict and tolerant reach scoring for the same model prediction. (b) Directional and container placement scoring, with illustrative points A–C.

Family	Strict success (correct)	Additional soft credit (for none)	Tasks

Reach (contact)	The end-effector touches the instructed candidate. Touching a non-instructed candidate is wrong .	The minimum end-effector distance to the instructed target over the episode is smaller than the minimum distance to every distractor.	food, item, plate, toys, dice, paperback, pizza, storage, tool, hardback
Reach (base proximity)	The end-effector touches the instructed can, or the base ends within $0.70$ m of it and satisfies the nearest-candidate rule.	Same minimum-distance rule as contact reach.	soda
Directional placement	The object is released onto the countertop, at least $0.17$ m from the reference, on the instructed side along the robot-right axis. The opposite side is wrong ; releases within $0.17$ m or off the countertop are on-ref failures.	At episode end, the still-held object lies on the instructed side along the robot-right axis.	carving, saucer
Container placement	The object rests inside the instructed compartment’s height band. A different compartment is wrong ; outside the container is none .	The object’s peak height within the container footprint falls within the permitted band around the instructed compartment. Passing through the band before overshooting does not qualify.	tupperware, bowl
Navigation	The robot base satisfies the instructed goal criterion (binary subtask_success ); no strict/soft split applies.	—	all move_to

Appendix

Table S4: Task-family success criteria. Soft success includes strict success plus eligible non-committed episodes; floor drops receive no soft credit.

Figure S8: Qualitative examples of VQA and IR predictions. Qwen3.5-4B predictions on held-out task frames, with VQA answers above and IR predictions below. Columns show both correct, VQA-only correct, IR-only correct, and both incorrect. Blue points indicate IR predictions; orange masks mark reference targets. IR is correct only when the predicted point falls inside the target mask. Questions and options are reproduced verbatim; images are cropped for readability.

Task	$n$	Strict $\uparrow$	Soft $\uparrow$	Wrong $\downarrow$	Cand.	Size (px)	Frames (k)	Len.
reach_food_item	32	53.1	56.2	34.4	4.0	52	319	318
reach_storage_box	32	46.9	68.8	25.0	2.0	85	145	1365
reach_soda_can	42	38.1	52.4	42.9	3.0	35	97	696
reach_plate	31	35.5	35.5	61.3	3.7	124	55	212
reach_hardback	48	31.2	37.5	37.5	4.3	66	237	421
reach_item	32	31.2	34.4	53.1	7.0	98	307	311

Appendix

Table S5: Per-task held-out results of the frozen-VLM baseline with covariates. Strict/Soft = strict/soft success (%); Wrong = wrong-target commitments (%); Cand. = mean same-category candidates per scene; Size = mean apparent size of the target object (square root of its 2D bounding-box area in the

720^{2}

head-camera evaluation frames, px); Frames = training frames (k); Len. = mean demonstration length (frames). A dash indicates that candidate count is not reported for that task. All rates use the task’s

n

evaluated episodes.

Task	$N$	Proprietary writers			Open-weight writers
		Fable 5.1	GPT-6	Gemini 3	Qwen3.5-9B	Qwen3.5-4B
place_bowl_in_cabinet	40	72 / 72	48 / 32	28 / 28	20 / 20	18 / 3
place_carving_knife	24	100 / 58	75 / 58	38 / 21	50 / 8	71 / 0
place_tupperware_in_fridge	22	18 / 18	59 / 9	5 / 5	18 / 9	5 / 5
reach_dice	32	97 / 94	91 / 88	78 / 56	66 / 0	56 / 0
reach_food_item	32	69 / 59	59 / 56	59 / 53	47 / 6	44 / 6

Appendix

Table S6: Per-task code-as-policy performance. Each cell reports soft / strict success (%).

N

is the number of test episodes. Bold indicates the highest soft success within each task, including ties. Per-task values are rounded to whole percentages; the pooled row is computed from aggregate episode counts.

Figure S9: Task-level correlation between perception and policy success. Each point represents a manipulation task for a policy variant, pairing a Qwen3.5-4B perception score with the policy’s soft held-out success rate. Lines show descriptive linear fits.

IR source	Soft $\uparrow$	Strict $\uparrow$
Baseline	41.6	27.9
Predicted dot
Qwen3.5-4B	48.8	34.1
Qwen3.5-9B	47.3	34.5
Molmo2-8B	47.1	33.5
Oracle dot	60.3	44.8

Appendix

Table S7: Rollout with VLM-predicted dot IR. Success (%) on the same held-out set. All rows use the Qwen3.5-4B policy backbone; predicted rows replace the ground-truth dot IR at inference with a dot predicted by the listed grounding VLM.

Figure S10: Original and augmented paperback demonstrations. Matched observations show (a) the original starting pose, (b) the augmented starting pose, and (c) an intermediate frame during the added approach motion. The augmented demonstration proceeds from (b) through (c) to (a), then follows the original interaction. The instruction and target remain unchanged. All panels are recorded head-camera frames from the same source demonstration and its augmented variant.

Training data	Prompt-probe
	Soft $\uparrow$	Strict $\uparrow$	Wrong $\downarrow$
Original	11.6	10.1	79.7
+ prefix augmentation	36.2	10.1	43.5

Appendix

Table S8: Paperback demonstration-augmentation case study results. Policies trained on original demonstrations or with approach-motion prefix augmentation are evaluated on the same 69 prompt-probe instances. All values are percentages; Wrong denotes wrong-target commitments.

Task	Base	Dot IR
		100%	50%	30%
reach_food_item	53.1/56.2	65.6 / 71.9	65.6 /68.8	59.4/68.8
reach_plate	35.5/35.5	71.0 / 74.2	48.4/64.5	51.6/67.7
reach_soda_can	38.1/52.4	71.4/78.6	69.0/ 81.0	73.8 / 81.0
reach_storage_box	46.9/68.8	68.8 / 78.1	65.6/ 78.1	65.6/ 78.1
reach_item	31.2/34.4	50.0 / 50.0	50.0 / 50.0	43.8/43.8

Appendix

Table S9: Per-task held-out strict/soft success (%). Base = frozen-VLM baseline; Dot IR = dot-conditioned policy, at the indicated training-data fraction. Bold marks the highest value in each row for each metric separately, including ties. Pooled rates aggregate evaluated episodes.

Task	Base	Dot IR
		100%	50%	30%
reach_food_item	16.0/16.0	40.0 / 40.0	28.0/30.0	22.0/24.0
reach_soda_can	12.0/16.0	62.0/62.0	76.0 / 76.0	66.0/68.0
reach_storage_box	4.0/4.0	52.0 / 54.0	26.0/28.0	30.0/34.0
reach_plate	6.0/6.0	20.0/24.0	8.0/12.0	26.0 / 28.0
reach_tool	8.0/8.0	28.0 /32.0	22.0/ 36.0	26.0/32.0

Appendix

Table S10: Per-task prompt-probe strict/soft success (%) for the same configurations as Table S9 . Bold marks the highest value in each row for each metric separately, including ties.

Task	Scene / region	VQA $\uparrow$	Success $\uparrow$
move_to_bookcase	single-floor house, toy room	30.0	92.5
move_to_bathroom	single-floor house, bedroom	40.0	90.0
move_to_corner_booth	restaurant diner	23.3	87.5
move_to_table_in_kitchen	two-floor house, main floor	38.2	75.0
move_to_table_near_sofa	single-floor house, open living area	23.9	73.2
move_to_bar	restaurant diner	25.0	50.0

Appendix

Table S11: Navigation performance across 10 tasks. VQA = Qwen3.5-4B accuracy on scene-based direction questions; Success = BEHAVIOR-Spatial-VLA navigation success on spawn-validated held-out instances. All values are percentages.

Figure S11: Qualitative examples of VLA generalization and instruction following. Start and end frames from four independent baseline evaluations. Rows show held-out layouts and swapped instructions in training layouts; columns show successes and wrong-target failures. Orange circles mark acceptable targets. In the failed instruction-swap probe, the policy reaches for the original training target (chip bag) instead of the instructed largest item (oatmeal box).

Figure S12: Qualitative examples of navigation outcomes. Each row shows selected scene-context and rollout frames from a held-out episode using the 10-task policy. The top three episodes succeed; the bottom three fail to reach their instructed goals.

Figure S13: Qualitative examples of navigation VQA predictions. Correct and incorrect Qwen3.5-4B answers on held-out navigation questions. Each example shows four selected context frames and the current view, which is the last input frame referenced by the question.

Spatial Reasoning Benchmarks

Spatial reasoning benchmarks test whether AI systems can perceive, represent, and act on geometric, relational, and viewpoint-dependent information.

See all 162 papers

Spa-Bench: A Real-Robot Benchmark for Spatial Grounding in Vision-Language-Action Models
Oct 8, 2026Justin Tien-Smith, Matthew Pidden, Rolandos Alexandros PotamiasSpatial Grounding for VLA Models Visual Grounding for Robotics
Imperial College London
SpatialAct: Probing Spatial Reasoning-to-Action Capabilities of VLM Agents in 3D Scenes
May 29, 2026Tianhui Liu, Jie Feng, Zhiheng Zheng +6Vision-Language Agents Visual Spatial Reasoning
1The Hong Kong University of Science and Technology (Guangzhou) · 2Zhongguancun Academy · 3Tsinghua University +1
Spatially Grounded Long-Horizon Task Planning in the Wild
Mar 13, 2026Sehun Jung, HyunJee Song, Dong-Hee Kim +4Robot Task Planning Long-Horizon Robotic Manipulation
Korea University · Microsoft Research · University of Wisconsin-Madison

Oct 8, 2026J/K move · Enter open · S save

Spa-Bench: A Real-Robot Benchmark for Spatial Grounding in Vision-Language-Action Models

Justin Tien-Smith, Matthew Pidden, Rolandos Alexandros Potamias

Imperial College London

Spatial Grounding for VLA Models Visual Grounding for Robotics Vision-Language Grounding Vision-Language-Action Models Robot Manipulation Benchmarks

Open paper AskPDF arXiv

Vision-language-action models (VLAs) offer a promising route toward general-purpose robotics by transferring semantic knowledge from large-scale vision-language pretraining into language-conditioned control. However, most VLA evaluations emphasize aggregate manipulation success under changes in objects, instructions, or environments, leaving unclear how reliably these policies ground spatial descriptions in the physical scenes where they act. We therefore ask a sharper question: how reliably can current VLAs translate spatial descriptions into appropriate real-robot behavior? To investigate this, we introduce Spa-Bench, a real-robot benchmark that evaluates spatial grounding across six tabletop tasks covering Object State, Relative Size, Referential Description, Relational Placement, Ordinal Position, and Counting. Our primary evaluation compares in-distribution (ID) and out-of-distribution (OOD) spatial configurations while retaining training support for the relevant objects, concepts, and actions, reducing object and motor unfamiliarity as alternative explanations for failure. Four diagnostic conditions further probe physical execution, sensitivity to wording, object-task associations, and recognition that no action is required. We release 4,092 episodes collected on the SO-101 robot arm: 1,200 human-teleoperated demonstrations and 2,892 real-robot evaluation rollouts from five state-of-the-art VLA policies. Across the three fully evaluated policies, success falls from 67.5-73.3% on ID spatial configurations to 40.7-52.0% on OOD spatial configurations, while matched direct instructions improve success by 28.3-40.0 percentage points. These results show that the evaluated VLAs can often execute the required manipulations but lack robust spatial grounding. Project materials, including videos, code, and benchmark data, are available at https://justintiensmith.github.io/Spa-Bench/.

Abstract

Spatial Reasoning Benchmarks

Explore similar work

Spa-Bench: A Real-Robot Benchmark for Spatial Grounding in Vision-Language-Action Models

SpatialAct: Probing Spatial Reasoning-to-Action Capabilities of VLM Agents in 3D Scenes

Spatially Grounded Long-Horizon Task Planning in the Wild