Vision-language action (VLA) policies often report strong manipulation benchmark performance with relatively few demonstrations, but it remains unclear whether this reflects robust language-to-object grounding or reliance on object--location correlations that do not transfer beyond the training distribution. We present a controlled multi-object picking study that progressively increases object placement variability up to full workspace randomization and evaluates held-out object--location pairings that break familiar associations without increasing spatial difficulty. Across these stress tests and data scaling, we find that for representative VLA policies, including SmolVLA and π0.5, execution of the manipulation primitive remains substantially more reliable than instruction-conditioned task success in harder regimes, suggesting that manipulation skill acquisition is decoupled from instruction following. We recommend augmenting manipulation benchmarks with task ladders and decomposed metrics that separately measure primitive execution and instruction-conditioned success to better diagnose instruction-grounded generalization.
Figures & tables
Benchmark
Policy (# Params)
Success Rate (%) – Simulation
LIBERO
Spatial
Object
Goal
Long
Avg.
π0 ( 3.3 B)
90
86
95
73
86.0
π0.5 ( 3.6 B)
97
99
98
96
97.5
SmolVLA ( 0.45 B)
90
96
92
71
87.3
MetaWorld
Easy
Medium
Hard
Very Hard
Avg.
π0 ( 3.5 B)
71.8
48.2
41.7
30.0
47.9
Table 1: Success rates on LIBERO, Meta-World, and our high-randomization ManiSkill-based benchmark. Figures for LIBERO and MetaWorld are taken from LeRobot ( Shukor et al., 2025 ; HuggingFace, 2026 ) , while results on our benchmark are obtained using the same models and training code. Performance decreases sharply as region-based layout regularities are progressively removed in our environment.
Figure 1: Object placement distributions under increasing randomization. Object placements progress from fixed regions with limited jitter to full workspace randomization. Each panel shows sampled initial object positions, with colors indicating different objects. In the jittered regimes, object placements remain largely separated across regions, enabling shortcut learning based on location. In contrast, full workspace randomization produces substantial overlap between object placement distributions, eliminating reliable positional cues.
Figure 2: Multi-object YCB environment in ManiSkill. The VLA observes two RGB camera streams: a wrist-mounted camera (top left) and a fixed frontal camera (top right). The bottom image shows the full environment for visualization only and is not provided as input to the policy.
Model
Regime
Success
Grasp-any
Reach
SmolVLA
Small jitter
90
98
100
Medium jitter
37
48
93
Large jitter
41
50
100
Full random
2
12
4
π0.5
Large jitter
10
11
60
Full random
6
15
16
Table 2: Performance along the spatial task ladder. The ladder progressively increases spatial variability from benchmark-like small jitter to full workspace randomization.
Split
Success
Grasp-any
Reach
Pairings (ID)
44
57
100
Pairings (OOD)
0
31
2
Table 3: Compositional generalization to held-out object–region pairings. SmolVLA is trained under the small-jitter regime with each object appearing in three regions. Performance is evaluated on seen object–region pairings (ID) and on the two held-out regions per object (OOD).
Model
Regime
Success
Grasp-any
Reach
SmolVLA
Med. jitter
37→42
48→54
93→100
Lrg. jitter
41→48
50→55
100→100
Full random
2→1
12→10
4→4
π0.5
Lrg. jitter
10→18
11→22
60→67
Full random
4→6
13→15
11→16
Table 4: Response to data scaling under increasing spatial variability. Values report metric changes as dataset size increases. Med. jitter: 0.5k→1k , Lrg. jitter: 1k→5k , Full random: 10k→100k .
# Objects
Success
Grasp-any
Reach
1
15
24
38
2
2
4
4
3
1
3
6
4
2
7
4
5
2
12
4
Table 5: Object-count ablation under full random placement. Results are obtained with a SmolVLA model trained on 10,000 trajectories.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Value
Total timesteps
30,000,000
Parallel environments
1024
Max episode steps
50
Optimizer
AdamW
Learning rate
1×10−4
Weight decay
1×10−5
Appendix
Table 6: RL expert training hyperparameters. State-based PPO agent trained with privileged information.
Figure 3: Example of a trajectory produced by a π0.5 model trained on 100k expert demonstrations in the Full Random environment.
Figure 4: Example of a trajectory produced by a π0.5 model trained on 100k expert demonstrations in the Full Random environment.
Vision-language-action (VLA) models have made general-purpose robot manipulation increasingly plausible by conditioning robot actions on natural-language instructions. A key test of such generality is whether policies actually follow language instructions. Yet many manipulation benchmarks leave this ability underdetermined: the intended object or destination is often visually salient or uniquely feasible, allowing policies to succeed without grounding the instruction. We argue that instruction-following evaluation should be text-indispensable: multiple actions should be visually and physically plausible, while only one should be consistent with the language instruction. We introduce InstructMove, a text-indispensable benchmark for instruction-following manipulation. InstructMove instantiates this principle in pick-and-place scenes with semantic distractors, decomposing instruction following into category identification, attribute discrimination, spatial reasoning, and compositional pick-and-place. InstructMove supports a train-eval protocol with InstructMove training data and held-out evaluation tasks, with additional diagnostics for language dependence. Experiments with representative VLA policies show that InstructMove provides a controlled testbed for diagnosing visual shortcuts and that InstructMove simulation data can improve real-world instruction-following manipulation performance. Code: https://github.com/HorizonRobotics/RoboOrchardSim
Mengao Zhao, Ziang Li, Chaodong Huang +15
Horizon Robotics · Southeast University · Huazhong University of Science and Technology +1
Vision-Language-Action (VLA) systems, built on pretrained vision-language models (VLMs), have shown rapidly improving performance on robot manipulation benchmarks. These gains are commonly interpreted as evidence that semantic representations learned from internet-scale data transfer to physical execution generalization. This position paper argues that the assumption underlying this interpretation -- that semantic generalization is sufficient to support physical action decisions -- has not been independently verified and cannot be tested under current evaluation protocols. We support this claim by decomposing VLA policies into semantic mapping and physical action decision, and showing that task success rate -- the dominant evaluation metric -- cannot distinguish between these two sources of capability. As a result, improvements in benchmark performance are consistent with multiple competing explanations, including semantic matching, distributional overlap, and genuine physical generalization. We further argue that this identifiability gap has been reinforced through narrative drift, whereby successive systems inherit and strengthen prior interpretations of performance gains without isolating the underlying causal mechanism. To address this limitation, we propose a research direction based on evaluation designs that introduce controlled variation to separately measure semantic and physical generalization. Such designs make it possible to causally attribute performance without requiring access to model internals, and to empirically assess the role of VLM backbones as semantic interfaces rather than implicit sources of physical competence. Our goal is not to refute the role of VLMs in robotics, but to clarify the conditions under which claims of physical generalization can be meaningfully evaluated.
Taozhao Chen, Ian Manchester, Huaming Chen
School of Electrical and Computer Engineering The University of Sydney Sydney, NSW, Australia · School of Aerospace, Mechanical and Mechatronic Engineering The University of Sydney Sydney, NSW, Australia
Instruction following is central to language-conditioned robot policies: language should determine what to do when the same scene permits multiple valid actions. Yet successful execution alone cannot establish whether a policy follows the instruction or infers the task from the scene. We study this ambiguity through scene-preserving instruction interventions, using valid target substitutions, arbitrary nouns, and unrelated sentences while holding the scene fixed. We evaluate vision-language-action (VLA) policies and world-action models (WAMs) in simulation and in real-world experiments. Our analysis addresses three questions: (a) Does task success imply instruction following? When instructions request a different visible object, all evaluated policies predominantly approach and pick up the incorrect original target associated with the scene. (b) Is this failure caused by language insensitivity? Instruction perturbations affect task performance. A layerwise action lens shows intermediate action predictions respond to these perturbations. Linear probes accurately recover instructed targets, indicating modified instructions are encoded despite rarely determining target selection. (c) Why does encoded language fail to control action? Attention analysis indicates weak instruction-token contributions to action generation. Target-token attention can remain focused on the original object, revealing a mismatch between target encoding and visual grounding. UMAP and shared non-negative matrix factorization show target information remains accessible within representations increasingly organized by scene identity. Our findings expose a grounding gap concealed by nominal success and provide a diagnostic framework. They further establish a concrete criterion for progress: policies should reliably follow valid changes in user intent, even when they conflict with scene-favored behavior.