cs.ROOct 8, 2026
SaveSpa-Bench: A Real-Robot Benchmark for Spatial Grounding in Vision-Language-Action Models
Organizations: Imperial College London
Abstract
Vision-language-action models (VLAs) offer a promising route toward general-purpose robotics by transferring semantic knowledge from large-scale vision-language pretraining into language-conditioned control. However, most VLA evaluations emphasize aggregate manipulation success under changes in objects, instructions, or environments, leaving unclear how reliably these policies ground spatial descriptions in the physical scenes where they act. We therefore ask a sharper question: how reliably can current VLAs translate spatial descriptions into appropriate real-robot behavior? To investigate this, we introduce Spa-Bench, a real-robot benchmark that evaluates spatial grounding across six tabletop tasks covering Object State, Relative Size, Referential Description, Relational Placement, Ordinal Position, and Counting. Our primary evaluation compares in-distribution (ID) and out-of-distribution (OOD) spatial configurations while retaining training support for the relevant objects, concepts, and actions, reducing object and motor unfamiliarity as alternative explanations for failure. Four diagnostic conditions further probe physical execution, sensitivity to wording, object-task associations, and recognition that no action is required. We release 4,092 episodes collected on the SO-101 robot arm: 1,200 human-teleoperated demonstrations and 2,892 real-robot evaluation rollouts from five state-of-the-art VLA policies. Across the three fully evaluated policies, success falls from 67.5-73.3% on ID spatial configurations to 40.7-52.0% on OOD spatial configurations, while matched direct instructions improve success by 28.3-40.0 percentage points. These results show that the evaluated VLAs can often execute the required manipulations but lack robust spatial grounding. Project materials, including videos, code, and benchmark data, are available at https://justintiensmith.github.io/Spa-Bench/.
Figures & tables
| Task | Grounding Axis | Example Instruction | Success Criterion |
|---|---|---|---|
| Object State | Attribute | “Place the closed white cup in the bowl.” | The same-category object whose object state matches the instruction is placed in the bowl. |
| Relative Size | Attribute | “Place the smallest object in the bowl.” | The object satisfying the requested relative-size description is placed in the bowl. |
| Relational Placement | Relation | “Place the green pen to the left of the red block.” | The target finishes within 3 inches of the reference and within the requested directional sector. |
| Referential Description | Relation | “Place the pencil sharpener furthest from the yellow pen in the bowl.” | The object satisfying the specified relation to the reference object is placed in the bowl. |
| Ordinal Position | Scene structure | “Place the third object from the left in the bowl.” | The object occupying the requested ordinal position is placed in the bowl. |
| Counting | Scene structure | “Leave exactly two objects in the bowl.” | The requested number of objects remains in the bowl at the end of the episode. |
TABLE I: Spa-Bench tasks. All tasks use the same robot, workspace, receptacle, and broad pick-and-place behavior while varying the spatial information needed to determine the correct response.
Fig. 2: Object inventory used in Spa-Bench. The benchmark uses 27 movable object instances in training and evaluation spanning 11 morphological families. These objects are recombined across the six tasks while the robot embodiment, workspace, and receptacle remain fixed.
Fig. 3: Withheld-cell construction for Object State . Rows denote state concepts and columns denote object identities. Observed cells receive state-conditioned instructions during fine-tuning, whereas withheld cells are reserved for OOD evaluation. The concept upside-down appears with the striped cup, and the white cup is manipulated under a direct instruction, but the pairing upside-down white cup is withheld from spatially grounded training.
Fig. 4: Matched-scene diagnostics. The same scene setup and intended object motion are evaluated under four instruction conditions: OOD spatial, direct manipulation, and a paraphrase of each. In this Referential Description example, the direct instruction names the target explicitly, whereas the spatial instruction requires resolving it from the scene.
Fig. 5: Task-level ID spatial-grounding success. Bars show per-task success rates, and whiskers indicate two-sided Wilson 95% confidence intervals. Complete ID evaluations contain 20 trials per task (120 total); VLA-0 contains 72 trials, with at least 10 per task. GR00T-N1.7 F-FT and VLA-0 are reported only as diagnostic ID baselines because their low ID success would make subsequent OOD failures difficult to interpret.
Fig. 6: Task-level OOD spatial-grounding and direct-manipulation success. Colors denote policies; dark and light bars denote OOD spatial and direct-manipulation instructions, respectively. Whiskers show two-sided Wilson 95% confidence intervals. OOD bars use 50 trials per task and direct bars use 20; the matched comparison in Sec. V-B uses only the corresponding 20 OOD scenes.
Fig. 7: Prompt-paraphrase diagnostic on 60 matched scenes. Dark bars show original instructions; light bars show paraphrased instructions for the same scenes. Whiskers show two-sided Wilson 95% confidence intervals.
Fig. 8: Cross-task-object diagnostic. Bars compare ID spatial-grounding success with standard task objects and with familiar objects transferred from other task families. Each condition contains 100 rollouts across five comparable tasks; whiskers show two-sided Wilson 95% confidence intervals.
Fig. 9: No-intervention success in four non-relational diagnostics. The top row shows representative rollouts; the bars summarize all trials. Success requires withholding task-directed movement when the requested count already holds, the requested ordinal position is invalid, or no object satisfies the requested state or referential description. Each task contains 20 rollouts per policy; the four-task aggregate contains 80. Whiskers show two-sided Wilson 95% confidence intervals. Relational Placement is evaluated separately using matched corrective-movement controls.
Fig. 10: Relational Placement action-gating control. The same relation instruction is evaluated when the requested relation already holds and when corrective movement is required. Success is high in the no-movement condition but falls sharply when the policy must correct the relation, indicating that remaining stationary does not necessarily reflect reliable recognition that the goal is satisfied. Each condition contains 20 rollouts per policy; whiskers show two-sided Wilson 95% confidence intervals.
Fig. 11: Primary failure-mode composition for OOD spatial configurations. Bars are normalized within each task–model pair, and denotes the number of failures. Wrong-instance errors are separated from other wrong-target errors only for Object State and Referential Description .
Fig. 12: Withheld-cell construction with preserved training support. Left: a toy color–object example. Color language training supports the observed pairing yellow ball , while a direct instruction demonstrates manipulation of the evaluation object without naming its color. The pairing yellow cube is withheld from spatially grounded training and reserved for OOD evaluation. Right: the corresponding construction for Object State . The concept upside-down appears with the striped cup, and the white cup is manipulated under a direct instruction, but the pairing upside-down white cup is withheld from spatially grounded training. OOD evaluation requires selecting the upside-down white cup rather than an upright white cup in the same scene.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. 13: Complete Spa-Bench concept–argument matrices. Observed cells receive concept-conditioned instructions during fine-tuning; withheld cells are reserved for OOD spatial evaluation. Grey cells denote combinations that are absent or not applicable. The three-object Ordinal Position matrix marks the middle position as absent because those training sequences target only the leftmost and rightmost positions.
| Condition | Trials | Diagnostic question |
|---|---|---|
| ID spatial configuration | 20 | Can the policy perform a concept–argument combination explicitly represented during fine-tuning? |
| OOD spatial configuration | 50 | Can the policy recombine individually supported concepts, arguments, and physical behaviors in a withheld configuration? |
| Matched direct manipulation | 20 | Can the policy execute the corresponding object motion in the same nominal scene when task-specific spatial inference is removed? |
| Prompt paraphrasing | 20 | Does performance change when 10 OOD spatial and 10 direct-manipulation instructions are rephrased? |
| Cross-task object | 20 | Does an ID spatial behavior transfer to an object associated with a different Spa-Bench task? |
| No intervention | 20 | Can the policy refrain from acting when the requested goal already holds or no valid referent exists? Not used for Relative Size . |
TABLE II: Spa-Bench evaluation conditions. Counts are per task unless otherwise noted.
| Policy | Initialization | Data / updates | Updated components | Action/state processing | |
|---|---|---|---|---|---|
| lerobot/pi05_base | Full / 76,596 | 50 | Vision-language backbone and action expert | Quantile normalization | |
| MolmoAct2 | allenai/MolmoAct2 | Full / 76,596 | 30 | VLM and action expert; token embeddings frozen | Min–max normalization |
| VLA-0 | Qwen2.5-VL-3B-Instruct | Trimmed / 77,246 | 8 | Full Qwen backbone; no LoRA or QLoRA | Min–max scaling; 1,000 text bins |
| GR00T-N1.7 F-LLM | GR00T-N1.7-3B | Trimmed / 71,304 | 16 | Visual encoder, multimodal projector, VLLN, and diffusion-action model; LLM frozen | GR00T-native processing |
| GR00T-N1.7 F-FT | GR00T-N1.7-3B | Full / 76,596 | 16 | Language, visual, multimodal, and diffusion-action components | GR00T-native processing |
TABLE III: Resolved model-adaptation settings. Full and trimmed denote the 612,733-frame and 570,386-frame dataset variants, respectively; is the predicted action horizon.
| Policy | ID | All OOD | Matched OOD | Matched direct |
|---|---|---|---|---|
| GR00T-N1.7 F-LLM | 81/120 (67.5%, 58.7–75.2) | 133/300 (44.3%, 38.8–50.0) | 50/120 (41.7%, 33.2–50.6) | 98/120 (81.7%, 73.8–87.6) |
| 82/120 (68.3%, 59.6–76.0) | 122/300 (40.7%, 35.3–46.3) | 54/120 (45.0%, 36.4–53.9) | 100/120 (83.3%, 75.7–88.9) | |
| MolmoAct2 | 88/120 (73.3%, 64.8–80.4) | 156/300 (52.0%, 46.4–57.6) | 73/120 (60.8%, 51.9–69.1) | 107/120 (89.2%, 82.3–93.6) |
TABLE IV: Aggregate results for the three fully evaluated policies. Matched OOD and matched direct manipulation use the same 120 nominal scene specifications.
| Task | GR00T-N1.7 F-LLM | MolmoAct2 | |
|---|---|---|---|
| Counting | 12/20; 60.0% (38.7–78.1) | 11/20; 55.0% (34.2–74.2) | 12/20; 60.0% (38.7–78.1) |
| Ordinal Position | 9/20; 45.0% (25.8–65.8) | 7/20; 35.0% (18.1–56.7) | 10/20; 50.0% (29.9–70.1) |
| Relational Placement | 14/20; 70.0% (48.1–85.5) | 13/20; 65.0% (43.3–81.9) | 16/20; 80.0% (58.4–91.9) |
| Object State | 17/20; 85.0% (64.0–94.8) | 16/20; 80.0% (58.4–91.9) | 18/20; 90.0% (69.9–97.2) |
| Relative Size | 17/20; 85.0% (64.0–94.8) | 19/20; 95.0% (76.4–99.1) | 17/20; 85.0% (64.0–94.8) |
| Referential Description | 12/20; 60.0% (38.7–78.1) | 16/20; 80.0% (58.4–91.9) | 15/20; 75.0% (53.1–88.8) |
TABLE V: Task-level ID spatial-grounding results. Each cell reports successes/20, success rate, and Wilson 95% confidence interval.
| Task | GR00T-N1.7 F-LLM | MolmoAct2 | |
|---|---|---|---|
| Counting | 0/50; 0.0% (0.0–7.1) | 0/50; 0.0% (0.0–7.1) | 0/50; 0.0% (0.0–7.1) |
| Ordinal Position | 21/50; 42.0% (29.4–55.8) | 23/50; 46.0% (33.0–59.6) | 27/50; 54.0% (40.4–67.0) |
| Relational Placement | 39/50; 78.0% (64.8–87.2) | 36/50; 72.0% (58.3–82.5) | 25/50; 50.0% (36.6–63.4) |
| Object State | 17/50; 34.0% (22.4–47.8) | 18/50; 36.0% (24.1–49.9) | 38/50; 76.0% (62.6–85.7) |
| Relative Size | 29/50; 58.0% (44.2–70.6) | 27/50; 54.0% (40.4–67.0) | 32/50; 64.0% (50.1–75.9) |
| Referential Description | 27/50; 54.0% (40.4–67.0) | 18/50; 36.0% (24.1–49.9) | 34/50; 68.0% (54.2–79.2) |
TABLE VI: Task-level OOD spatial-grounding results. Each cell reports successes/50, success rate, and Wilson 95% confidence interval.
| Task | GR00T-N1.7 F-LLM | MolmoAct2 | |
|---|---|---|---|
| Counting | 12/20; 60.0% (38.7–78.1) | 11/20; 55.0% (34.2–74.2) | 16/20; 80.0% (58.4–91.9) |
| Ordinal Position | 19/20; 95.0% (76.4–99.1) | 20/20; 100% (83.9–100) | 17/20; 85.0% (64.0–94.8) |
| Relational Placement | 17/20; 85.0% (64.0–94.8) | 18/20; 90.0% (69.9–97.2) | 17/20; 85.0% (64.0–94.8) |
| Object State | 19/20; 95.0% (76.4–99.1) | 18/20; 90.0% (69.9–97.2) | 20/20; 100% (83.9–100) |
| Relative Size | 19/20; 95.0% (76.4–99.1) | 20/20; 100% (83.9–100) | 20/20; 100% (83.9–100) |
| Referential Description | 12/20; 60.0% (38.7–78.1) | 13/20; 65.0% (43.3–81.9) | 17/20; 85.0% (64.0–94.8) |
TABLE VII: Task-level matched direct-manipulation results. Each cell reports successes/20, success rate, and Wilson 95% confidence interval.
| Condition | GR00T-N1.7 F-LLM | MolmoAct2 | |
|---|---|---|---|
| Original direct | 49/60; 81.7% (70.1–89.4) | 52/60; 86.7% (75.8–93.1) | 54/60; 90.0% (79.9–95.3) |
| Paraphrased direct | 42/60; 70.0% (57.5–80.1) | 52/60; 86.7% (75.8–93.1) | 54/60; 90.0% (79.9–95.3) |
| Original OOD spatial | 29/60; 48.3% (36.2–60.7) | 29/60; 48.3% (36.2–60.7) | 39/60; 65.0% (52.4–75.8) |
| Paraphrased OOD spatial | 23/60; 38.3% (27.1–51.0) | 25/60; 41.7% (30.1–54.3) | 30/60; 50.0% (37.7–62.3) |
TABLE VIII: Prompt-paraphrase results on the corresponding 60-scene subsets.
| Condition | GR00T-N1.7 F-LLM | MolmoAct2 | |
|---|---|---|---|
| Standard objects, five-task ID | 64/100; 64.0% (54.2–72.7) | 66/100; 66.0% (56.3–74.5) | 70/100; 70.0% (60.4–78.1) |
| Cross-task objects, five-task ID | 55/100; 55.0% (45.2–64.4) | 54/100; 54.0% (44.3–63.4) | 53/100; 53.0% (43.3–62.5) |
| Counting: goal already satisfied | 17/20; 85.0% (64.0–94.8) | 16/20; 80.0% (58.4–91.9) | 16/20; 80.0% (58.4–91.9) |
| Ordinal Position: invalid ordinal | 0/20; 0.0% (0.0–16.1) | 0/20; 0.0% (0.0–16.1) | 0/20; 0.0% (0.0–16.1) |
| Object State: no valid target | 0/20; 0.0% (0.0–16.1) | 0/20; 0.0% (0.0–16.1) | 0/20; 0.0% (0.0–16.1) |
| Referential Description: no candidate | 0/20; 0.0% (0.0–16.1) | 0/20; 0.0% (0.0–16.1) | 3/20; 15.0% (5.2–36.0) |
TABLE IX: Cross-task-object and action-gating diagnostics. Cross-task rows contain 100 trials; non-relational task rows and relational-control rows contain 20 trials; the four-task aggregates contain 80 trials.
| Failure mode | Operational description |
|---|---|
| No action | The policy does not initiate task-directed arm or gripper motion when movement is required. |
| Wrong object | The policy approaches or manipulates an object outside the intended target category or otherwise selects the wrong target. |
| Correct category, wrong instance | The policy selects an object from the requested semantic category but chooses the instance with the wrong state or referential relation. Used only for Object State and Referential Description . |
| Grasp failure | The policy selects the intended target but fails to establish or maintain a usable grasp before transport. |
| Incorrect spatial outcome | The intended object is approached or manipulated, but the final count, direction, relation, or placement is incorrect. |
| Premature release or drop | The policy releases the object before reaching the intended goal or unintentionally drops it during transport. |
TABLE X: Primary qualitative failure-mode taxonomy.
Fig. 14: Primary failure-mode composition for ID spatial configurations. Bars are normalized within each task–policy pair, and denotes the number of failed rollouts. Results include all five policies; VLA-0 has fewer ID evaluation trials than the other policies (Fig. 5 ). Wrong-instance errors are separated from other wrong-target errors only for Object State and Referential Description . Some bars represent very few failures and are therefore descriptive.