InstructMove: A Text-Indispensable Benchmark for Instruction-Following Manipulation

Mengao Zhao; Ziang Li; Chaodong Huang; Mengchen Ma; Haoyi Jiang; Yiwei Jin; Xinjie Wang; Yun Du

cs.ROAug 24, 2026

InstructMove: A Text-Indispensable Benchmark for Instruction-Following Manipulation

Authors: Mengao Zhao, Ziang Li, Chaodong Huang, Mengchen Ma, Haoyi Jiang, Yiwei Jin, Xinjie Wang, Yun Du, +10 more

Organizations: Horizon Robotics · Southeast University · Huazhong University of Science and Technology · WuwenAI

Abstract

Vision-language-action (VLA) models have made general-purpose robot manipulation increasingly plausible by conditioning robot actions on natural-language instructions. A key test of such generality is whether policies actually follow language instructions. Yet many manipulation benchmarks leave this ability underdetermined: the intended object or destination is often visually salient or uniquely feasible, allowing policies to succeed without grounding the instruction. We argue that instruction-following evaluation should be text-indispensable: multiple actions should be visually and physically plausible, while only one should be consistent with the language instruction. We introduce InstructMove, a text-indispensable benchmark for instruction-following manipulation. InstructMove instantiates this principle in pick-and-place scenes with semantic distractors, decomposing instruction following into category identification, attribute discrimination, spatial reasoning, and compositional pick-and-place. InstructMove supports a train-eval protocol with InstructMove training data and held-out evaluation tasks, with additional diagnostics for language dependence. Experiments with representative VLA policies show that InstructMove provides a controlled testbed for diagnosing visual shortcuts and that InstructMove simulation data can improve real-world instruction-following manipulation performance. Code: https://github.com/HorizonRobotics/RoboOrchardSim

Figures & tables

Figure 1: InstructMove is a text-indispensable benchmark for instruction-following manipulation. When a scene makes the target obvious, the correct pick-and-place action can be inferred from vision alone, reducing the instruction to a task identifier. InstructMove instead introduces semantic distractors—objects and destinations that are visually plausible and physically feasible, yet are disambiguated by the instruction—making language necessary for action selection.

Dimension	RLBench	CALVIN	VIMA	LIBERO	VLABench	SimplerEnv	SimplerEnv- Instruct	RoboTwin	LIBERO-CF	Spatial VL	InstructMove
Visual & Policy Setting
Visual realism	✗	✗	✗	✗	$\blacktriangle$	$\blacktriangle$	$\blacktriangle$	$\blacktriangle$	✗	$\blacktriangle$	$\blacktriangle$
Language-conditioned manipulation	$\blacktriangle$	✓	✓	✓	✓	✓	✓	$\blacktriangle$	✓	✗	✓
Protocol	Train eval.	Train eval.	Train eval.	Train eval.	Train eval.	ZS eval.	ZS eval.	Train eval.	Diag.	QA	Train eval.
Grounding & Distractor Design
Semantic distractors	✗	✗	$\blacktriangle$	$\blacktriangle$	$\blacktriangle$	$\blacktriangle$	$\blacktriangle$	$\blacktriangle$	$\blacktriangle$	✗	✓

Table 1: Comparison of representative manipulation and spatial vision-language benchmarks. ✓ denotes an explicit design goal,

\blacktriangle

denotes partial or indirect support, and ✗ denotes that the dimension is not a primary focus. “ZS eval.” denotes zero-shot policy evaluation and “Diag.” denotes diagnostic evaluation. Spatial VL summarizes non-closed-loop spatial vision-language resources such as EmbSpatial-Bench and RoboSpatial. Unlike prior benchmarks, InstructMove evaluates whether language is indispensable for selecting the correct executable action among multiple visually plausible alternatives.

Figure 2: Overview of the InstructMove benchmark instantiation pipeline. The benchmark is built through three aligned components: a validated asset library, task templates with constraint-based scene/layout generation across four task families, and expert trajectory synthesis for generating training demonstrations.

Model	pick_category Seen	pick_attribute	pick_spatial	place_a2b
	PiperX		Franka
$\pi_{0}$	0.34/0.04( 0.19 )	0.44/0.05( 0.24 )	0.36/0.02( 0.19 )	0.64/0.07/0.05/0.03( 0.19 )
$\pi_{0.5}$	0.82/0.57( 0.69 )	0.81/0.62( 0.71 )	0.46/0.15( 0.30 )	0.88/0.48/0.37/0.36( 0.52 )
GR00T N1.7	0.76/0.17( 0.47 )	0.76/0.07( 0.41 )	0.50/0.01( 0.25 )	0.60/0.06/0.05/0.05( 0.19 )
Motus	0.41/0.10( 0.26 )	0.69/0.17( 0.43 )	0.51/0.02( 0.26 )	0.57/0.02/0.02/0.01( 0.16 )

Table 2: Main train-eval results after robot-specific multi-task training. The pick_category and pick_attribute tasks are evaluated on PiperX, and pick_spatial and place_a2b are evaluated on Franka. Pick-only entries are reported as Reach/Lift (

S_{\mathrm{pick}}

), and pick-and-place entries as ReachPick/LiftPick/ReachPlace/Place (

S_{\mathrm{pnp}}

); red values in parentheses denote the stage score.

Model	Unseen Inst.	Unseen Cat.
$\pi_{0}$	0.32/0.05( 0.18 )	0.34/0.05( 0.19 )
$\pi_{0.5}$	0.74/0.48( 0.61 )	0.81/0.54( 0.67 )
GR00T N1.7	0.70/0.09( 0.40 )	0.67/0.18( 0.43 )
Motus	0.45/0.06( 0.26 )	0.63/0.09( 0.36 )

Table 3: Pick-category generalization results on PiperX for held-out instances and categories. Entries are reported as Reach/Lift (

S_{\mathrm{pick}}

); red values in parentheses denote the stage score.

Figure 3: Qualitative examples in simulated and real evaluation cases.

Instruction condition	Target Reach/Lift ( $S_{\mathrm{pick}}$ )	Any-object Reach	Any-object Lift
Normal	0.74/0.48( 0.61 )	–	–
Alternative target	0.72/0.53( 0.62 )	–	–
Generic object	–	1.00	0.66
Empty	–	1.00	0.64
Absent object	–	0.99	0.66

Table 4: Counterfactual instruction-ablation results for

\pi_{0.5}

over 100 fixed pick_category unseen-instance scenes, with each scene evaluated under every instruction condition. For target-specific instructions, entries are reported as Reach/Lift (

S_{\mathrm{pick}}

); red values in parentheses denote the stage score. For instructions without a unique valid target, we report whether the policy reaches or lifts any object.

Training set	Reach	Lift
200 real	19/50	11/50
200 sim	26/50	11/50
2,000 sim	28/50	15/50
2,000 sim + 200 real	34/50	18/50

Table 5: Real-world Pick Category transfer with

\pi_{0.5}

on the dual-arm PiperX. The evaluation contains 50 trials across five target objects, with at least 5 visually or semantically similar distractors per trial.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Overview of the benchmark data generation pipeline, including Task Specification, Scene Generation, Motion Generation, Validator-Based Post-Processing, and Recorded Data Output.

Figure 5: Automated asset curation and management pipeline: (a) asset generation, (b) VLM-based annotation, (c) quality curation with human-in-the-loop review, and (d) reproducible sampling.

Figure 6: Distribution of the asset corpus used in the benchmark and experiments. Left : domain proportions. Right : asset count per super-category (18 in total).


Domain	Super-Category	Fine-Grained Categories	# Categories
desktop_supplies	office_stationery	stapler , ballpoint_pen , pen_holder , glue_stick , eraser , correction_tape , marker , sharpener , notebook	9
desktop_supplies	office_tools	scissors , scotch_tape , magnifier , cutter_knife , stamp , staples_box , sticky_notes	7
desktop_supplies	electronic_accessories	mouse , tv_remote , ac_remote , presenter_clicker , gamepad , earbud_case , charging_head , charging_cable , usb_drive	9
desktop_supplies	decorations	mini_sculpture , phone_stand , vase	3
kitchen_supplies	cutting_tools	fruit_knife , peeler , garlic_press , wire_cutters	4

Appendix

Table 6: Detailed asset category inventory. The count column reports the number of fine-grained categories within each super-category; object instances are summarized separately in Figure 6 .

Figure 7: Domain randomization on a fixed manipulation scene. (a) Baseline with default lighting and textures. (b) Randomized illumination. (c) Randomized object surface textures. (d) Randomized table surface material. Scene layout and embodiment are held fixed so that each panel differs in a single factor.

Panel of Figure 8 — Figure 8: Real-world experimental setup. Left : dual-arm AgileX PiperX robot with one static camera and two wrist-mounted cameras. Right : ALOHA teleoperation setup for data collection.

Figure 9: Two representative Reach-failed $\pi_{0.5}$ episodes on the pick_category seen split. Blue boxes mark the instruction target; orange boxes mark the similar distractor selected by the policy. The right panels show the robot approaching the distractor.

Model	Init checkpoint	Trainable scope	Batch / Steps
$\pi_{0}$	OpenPI pi0_base	LoRA only	32/GPU, 5k
$\pi_{0.5}$	OpenPI pi05_base	LoRA only	32/GPU, 5k
GR00T N1.7	GR00T-N1.7-3B	Proj./diff./VL-norm	24/GPU, 5k
Motus	motus-robotics/Motus	Qwen3-VL frozen	8/GPU, 5k

Appendix

Table 7: Summary of model-specific training recipes. Unlogged details are qualified in the text below.

Explore similar work

Robust Skills, Brittle Grounding: Diagnosing Restricted Generalization in Vision-Language Action Policies via Multi-Object Picking
Feb 27, 2026David Emukpere, Romain Deffayet, Jean-Michel RendersInstruction Generalization in VLA Models Generalization in Robotic Manipulation
Naver Labs Europe, 6 Chem. de Maupertuis, Meylan, France.
MobileManiBench: Simplifying Model Verification for Mobile Manipulation
Feb 5, 2026Wenbo Wang, Fangyun Wei, QiXiu Li +5Mobile Manipulation Robotics Simulation
University of Sydney · 1Microsoft Research Asia · 3Tsinghua University
FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies
May 26, 2026Xintong Hu, Xuhong Huang, Jinyu Zhang +11Language-Conditioned Robot Manipulation Vision-Language-Action Models
XLANG Lab, The University of Hong Kong · Qwen Team, Alibaba Inc.

Feb 27, 2026J/K move · Enter open · S save

Robust Skills, Brittle Grounding: Diagnosing Restricted Generalization in Vision-Language Action Policies via Multi-Object Picking

David Emukpere, Romain Deffayet, Jean-Michel Renders

Naver Labs Europe, 6 Chem. de Maupertuis, Meylan, France.

Instruction Generalization in VLA Models Generalization in Robotic Manipulation Vision-Language Grounding Robot Manipulation Benchmarks

Open paper AskPDF arXiv

Vision-language action (VLA) policies often report strong manipulation benchmark performance with relatively few demonstrations, but it remains unclear whether this reflects robust language-to-object grounding or reliance on object--location correlations that do not transfer beyond the training distribution. We present a controlled multi-object picking study that progressively increases object placement variability up to full workspace randomization and evaluates held-out object--location pairings that break familiar associations without increasing spatial difficulty. Across these stress tests and data scaling, we find that for representative VLA policies, including SmolVLA and

π_{0.5}

, execution of the manipulation primitive remains substantially more reliable than instruction-conditioned task success in harder regimes, suggesting that manipulation skill acquisition is decoupled from instruction following. We recommend augmenting manipulation benchmarks with task ladders and decomposed metrics that separately measure primitive execution and instruction-conditioned success to better diagnose instruction-grounded generalization.

Abstract

Explore similar work

Robust Skills, Brittle Grounding: Diagnosing Restricted Generalization in Vision-Language Action Policies via Multi-Object Picking

MobileManiBench: Simplifying Model Verification for Mobile Manipulation

FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies