Robust Skills, Brittle Grounding: Diagnosing Restricted Generalization in Vision-Language Action Policies via Multi-Object Picking

David Emukpere; Romain Deffayet; Jean-Michel Renders

cs.ROFeb 27, 2026

Robust Skills, Brittle Grounding: Diagnosing Restricted Generalization in Vision-Language Action Policies via Multi-Object Picking

Authors: David Emukpere, Romain Deffayet, Jean-Michel Renders

Organizations: Naver Labs Europe, 6 Chem. de Maupertuis, Meylan, France.

Abstract

Vision-language action (VLA) policies often report strong manipulation benchmark performance with relatively few demonstrations, but it remains unclear whether this reflects robust language-to-object grounding or reliance on object--location correlations that do not transfer beyond the training distribution. We present a controlled multi-object picking study that progressively increases object placement variability up to full workspace randomization and evaluates held-out object--location pairings that break familiar associations without increasing spatial difficulty. Across these stress tests and data scaling, we find that for representative VLA policies, including SmolVLA and $π_{0.5}$ , execution of the manipulation primitive remains substantially more reliable than instruction-conditioned task success in harder regimes, suggesting that manipulation skill acquisition is decoupled from instruction following. We recommend augmenting manipulation benchmarks with task ladders and decomposed metrics that separately measure primitive execution and instruction-conditioned success to better diagnose instruction-grounded generalization.

Figures & tables

Benchmark	Policy (# Params)	Success Rate (%) – Simulation
LIBERO		Spatial	Object	Goal	Long	Avg.
	$\pi_{0}$ ( $3.3$ B)	$90$	$86$	$95$	$73$	$86.0$
	$\pi_{0.5}$ ( $3.6$ B)	$97$	$99$	$98$	$96$	$97.5$
	SmolVLA ( $0.45$ B)	$90$	$96$	$92$	$71$	$87.3$
MetaWorld		Easy	Medium	Hard	Very Hard	Avg.
	$\pi_{0}$ ( $3.5$ B)	$71.8$	$48.2$	$41.7$	$30.0$	$47.9$

Table 1: Success rates on LIBERO, Meta-World, and our high-randomization ManiSkill-based benchmark. Figures for LIBERO and MetaWorld are taken from LeRobot ( Shukor et al., 2025 ; HuggingFace, 2026 ) , while results on our benchmark are obtained using the same models and training code. Performance decreases sharply as region-based layout regularities are progressively removed in our environment.

Panel of Figure 1 — Figure 1: Object placement distributions under increasing randomization. Object placements progress from fixed regions with limited jitter to full workspace randomization. Each panel shows sampled initial object positions, with colors indicating different objects. In the jittered regimes, object placements remain largely separated across regions, enabling shortcut learning based on location. In contrast, full workspace randomization produces substantial overlap between object placement distributions, eliminating reliable positional cues.

Figure 2: Multi-object YCB environment in ManiSkill. The VLA observes two RGB camera streams: a wrist-mounted camera (top left) and a fixed frontal camera (top right). The bottom image shows the full environment for visualization only and is not provided as input to the policy.

Model	Regime	Success	Grasp-any	Reach
SmolVLA	Small jitter	$90$	$98$	$100$
	Medium jitter	$37$	$48$	$93$
	Large jitter	$41$	$50$	$100$
	Full random	$2$	$12$	$4$
$\pi_{0.5}$	Large jitter	$10$	$11$	$60$
$\pi_{0.5}$	Full random	$6$	$15$	$16$

Table 2: Performance along the spatial task ladder. The ladder progressively increases spatial variability from benchmark-like small jitter to full workspace randomization.

Split	Success	Grasp-any	Reach
Pairings (ID)	$44$	$57$	$100$
Pairings (OOD)	$0$	$31$	$2$

Table 3: Compositional generalization to held-out object–region pairings. SmolVLA is trained under the small-jitter regime with each object appearing in three regions. Performance is evaluated on seen object–region pairings (ID) and on the two held-out regions per object (OOD).

Model	Regime	Success	Grasp-any	Reach
SmolVLA	Med. jitter	$37\!\rightarrow\!42$	$48\!\rightarrow\!54$	$93\!\rightarrow\!100$
	Lrg. jitter	$41\!\rightarrow\!48$	$50\!\rightarrow\!55$	$100\!\rightarrow\!100$
	Full random	$2\!\rightarrow\!1$	$12\!\rightarrow\!10$	$4\!\rightarrow\!4$
$\pi_{0.5}$	Lrg. jitter	$10\!\rightarrow\!18$	$11\!\rightarrow\!22$	$60\!\rightarrow\!67$
$\pi_{0.5}$	Full random	$4\!\rightarrow\!6$	$13\!\rightarrow\!15$	$11\!\rightarrow\!16$

Table 4: Response to data scaling under increasing spatial variability. Values report metric changes as dataset size increases. Med. jitter:

0.5\mathrm{k}\!\rightarrow\!1\mathrm{k}

, Lrg. jitter:

1\mathrm{k}\!\rightarrow\!5\mathrm{k}

, Full random:

10\mathrm{k}\!\rightarrow\!100\mathrm{k}

# Objects	Success	Grasp-any	Reach
$1$	$15$	$24$	$38$
$2$	$2$	$4$	$4$
$3$	$1$	$3$	$6$
$4$	$2$	$7$	$4$
$5$	$2$	$12$	$4$

Table 5: Object-count ablation under full random placement. Results are obtained with a SmolVLA model trained on

10{,}000

trajectories.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Value
Total timesteps	$30{,}000{,}000$
Parallel environments	$1024$
Max episode steps	$50$
Optimizer	AdamW
Learning rate	$1\times 10^{-4}$
Weight decay	$1\times 10^{-5}$

Appendix

Table 6: RL expert training hyperparameters. State-based PPO agent trained with privileged information.

Panel of Figure 3 — Figure 3: Example of a trajectory produced by a $\pi_{0.5}$ model trained on $100k$ expert demonstrations in the Full Random environment.

Panel of Figure 4 — Figure 4: Example of a trajectory produced by a $\pi_{0.5}$ model trained on $100k$ expert demonstrations in the Full Random environment.

Explore similar work

InstructMove: A Text-Indispensable Benchmark for Instruction-Following Manipulation
Aug 24, 2026Mengao Zhao, Ziang Li, Chaodong Huang +15Robot Manipulation Benchmarks Language-Conditioned Robot Manipulation
Horizon Robotics · Southeast University · Huazhong University of Science and Technology +1
Position: Vision-Language-Action Models Cannot Be Verified to Perform Physical Reasoning
Jun 28, 2026Taozhao Chen, Ian Manchester, Huaming ChenVLM Evaluation Physical Reasoning
School of Electrical and Computer Engineering The University of Sydney Sydney, NSW, Australia · School of Aerospace, Mechanical and Mechatronic Engineering The University of Sydney Sydney, NSW, Australia
Encoded but Not in Control: Revealing the Grounding Gap in Vision-Language Robot Policies
Oct 5, 2026Shaohan Jiang, Jiahang Cao, Qiduo He +9Vision-Language Grounding Vision-Language-Action Models
HKU · CUHK(SZ) · Shanghai AI Lab +3

Aug 24, 2026J/K move · Enter open · S save

InstructMove: A Text-Indispensable Benchmark for Instruction-Following Manipulation

Mengao Zhao, Ziang Li, Chaodong Huang, Mengchen Ma+14

Horizon Robotics · Southeast University · Huazhong University of Science and Technology+1

Robot Manipulation Benchmarks Language-Conditioned Robot Manipulation Robotic Manipulation Vision-Language-Action Models

Open paper AskPDF arXiv

Vision-language-action (VLA) models have made general-purpose robot manipulation increasingly plausible by conditioning robot actions on natural-language instructions. A key test of such generality is whether policies actually follow language instructions. Yet many manipulation benchmarks leave this ability underdetermined: the intended object or destination is often visually salient or uniquely feasible, allowing policies to succeed without grounding the instruction. We argue that instruction-following evaluation should be text-indispensable: multiple actions should be visually and physically plausible, while only one should be consistent with the language instruction. We introduce InstructMove, a text-indispensable benchmark for instruction-following manipulation. InstructMove instantiates this principle in pick-and-place scenes with semantic distractors, decomposing instruction following into category identification, attribute discrimination, spatial reasoning, and compositional pick-and-place. InstructMove supports a train-eval protocol with InstructMove training data and held-out evaluation tasks, with additional diagnostics for language dependence. Experiments with representative VLA policies show that InstructMove provides a controlled testbed for diagnosing visual shortcuts and that InstructMove simulation data can improve real-world instruction-following manipulation performance. Code: https://github.com/HorizonRobotics/RoboOrchardSim

Abstract

Explore similar work

InstructMove: A Text-Indispensable Benchmark for Instruction-Following Manipulation

Position: Vision-Language-Action Models Cannot Be Verified to Perform Physical Reasoning

Encoded but Not in Control: Revealing the Grounding Gap in Vision-Language Robot Policies