InstructMove: A Text-Indispensable Benchmark for Instruction-Following Manipulation

Mengao Zhao; Ziang Li; Chaodong Huang; Mengchen Ma; Haoyi Jiang; Yiwei Jin; Xinjie Wang; Yun Du

cs.ROAug 24, 2026

InstructMove: A Text-Indispensable Benchmark for Instruction-Following Manipulation

Authors: Mengao Zhao, Ziang Li, Chaodong Huang, Mengchen Ma, Haoyi Jiang, Yiwei Jin, Xinjie Wang, Yun Du, +10 more

Organizations: Horizon Robotics · Southeast University · Huazhong University of Science and Technology · WuwenAI

Abstract

Vision-language-action (VLA) models have made general-purpose robot manipulation increasingly plausible by conditioning robot actions on natural-language instructions. A key test of such generality is whether policies actually follow language instructions. Yet many manipulation benchmarks leave this ability underdetermined: the intended object or destination is often visually salient or uniquely feasible, allowing policies to succeed without grounding the instruction. We argue that instruction-following evaluation should be text-indispensable: multiple actions should be visually and physically plausible, while only one should be consistent with the language instruction. We introduce InstructMove, a text-indispensable benchmark for instruction-following manipulation. InstructMove instantiates this principle in pick-and-place scenes with semantic distractors, decomposing instruction following into category identification, attribute discrimination, spatial reasoning, and compositional pick-and-place. InstructMove supports a train-eval protocol with InstructMove training data and held-out evaluation tasks, with additional diagnostics for language dependence. Experiments with representative VLA policies show that InstructMove provides a controlled testbed for diagnosing visual shortcuts and that InstructMove simulation data can improve real-world instruction-following manipulation performance. Code: https://github.com/HorizonRobotics/RoboOrchardSim

Figures & tables

Figure 1: InstructMove is a text-indispensable benchmark for instruction-following manipulation. When a scene makes the target obvious, the correct pick-and-place action can be inferred from vision alone, reducing the instruction to a task identifier. InstructMove instead introduces semantic distractors—objects and destinations that are visually plausible and physically feasible, yet are disambiguated by the instruction—making language necessary for action selection.

Dimension	RLBench	CALVIN	VIMA	LIBERO	VLABench	SimplerEnv	SimplerEnv- Instruct	RoboTwin	LIBERO-CF	Spatial VL	InstructMove
Visual & Policy Setting
Visual realism	✗	✗	✗	✗	$\blacktriangle$	$\blacktriangle$	$\blacktriangle$	$\blacktriangle$	✗	$\blacktriangle$	$\blacktriangle$
Language-conditioned manipulation	$\blacktriangle$	✓	✓	✓	✓	✓	✓	$\blacktriangle$	✓	✗	✓
Protocol	Train eval.	Train eval.	Train eval.	Train eval.	Train eval.	ZS eval.	ZS eval.	Train eval.	Diag.	QA	Train eval.
Grounding & Distractor Design
Semantic distractors	✗	✗	$\blacktriangle$	$\blacktriangle$	$\blacktriangle$	$\blacktriangle$	$\blacktriangle$	$\blacktriangle$	$\blacktriangle$	✗	✓

Table 1: Comparison of representative manipulation and spatial vision-language benchmarks. ✓ denotes an explicit design goal,

\blacktriangle

denotes partial or indirect support, and ✗ denotes that the dimension is not a primary focus. “ZS eval.” denotes zero-shot policy evaluation and “Diag.” denotes diagnostic evaluation. Spatial VL summarizes non-closed-loop spatial vision-language resources such as EmbSpatial-Bench and RoboSpatial. Unlike prior benchmarks, InstructMove evaluates whether language is indispensable for selecting the correct executable action among multiple visually plausible alternatives.

Figure 2: Overview of the InstructMove benchmark instantiation pipeline. The benchmark is built through three aligned components: a validated asset library, task templates with constraint-based scene/layout generation across four task families, and expert trajectory synthesis for generating training demonstrations.

Model	pick_category Seen	pick_attribute	pick_spatial	place_a2b
	PiperX		Franka
$\pi_{0}$	0.34/0.04( 0.19 )	0.44/0.05( 0.24 )	0.36/0.02( 0.19 )	0.64/0.07/0.05/0.03( 0.19 )
$\pi_{0.5}$	0.82/0.57( 0.69 )	0.81/0.62( 0.71 )	0.46/0.15( 0.30 )	0.88/0.48/0.37/0.36( 0.52 )
GR00T N1.7	0.76/0.17( 0.47 )	0.76/0.07( 0.41 )	0.50/0.01( 0.25 )	0.60/0.06/0.05/0.05( 0.19 )
Motus	0.41/0.10( 0.26 )	0.69/0.17( 0.43 )	0.51/0.02( 0.26 )	0.57/0.02/0.02/0.01( 0.16 )

Table 2: Main train-eval results after robot-specific multi-task training. The pick_category and pick_attribute tasks are evaluated on PiperX, and pick_spatial and place_a2b are evaluated on Franka. Pick-only entries are reported as Reach/Lift (

S_{\mathrm{pick}}

), and pick-and-place entries as ReachPick/LiftPick/ReachPlace/Place (

S_{\mathrm{pnp}}

); red values in parentheses denote the stage score.

Model	Unseen Inst.	Unseen Cat.
$\pi_{0}$	0.32/0.05( 0.18 )	0.34/0.05( 0.19 )
$\pi_{0.5}$	0.74/0.48( 0.61 )	0.81/0.54( 0.67 )
GR00T N1.7	0.70/0.09( 0.40 )	0.67/0.18( 0.43 )
Motus	0.45/0.06( 0.26 )	0.63/0.09( 0.36 )

Table 3: Pick-category generalization results on PiperX for held-out instances and categories. Entries are reported as Reach/Lift (

S_{\mathrm{pick}}

); red values in parentheses denote the stage score.

Figure 3: Qualitative examples in simulated and real evaluation cases.

Instruction condition	Target Reach/Lift ( $S_{\mathrm{pick}}$ )	Any-object Reach	Any-object Lift
Normal	0.74/0.48( 0.61 )	–	–
Alternative target	0.72/0.53( 0.62 )	–	–
Generic object	–	1.00	0.66
Empty	–	1.00	0.64
Absent object	–	0.99	0.66

Table 4: Counterfactual instruction-ablation results for

\pi_{0.5}

over 100 fixed pick_category unseen-instance scenes, with each scene evaluated under every instruction condition. For target-specific instructions, entries are reported as Reach/Lift (

S_{\mathrm{pick}}

); red values in parentheses denote the stage score. For instructions without a unique valid target, we report whether the policy reaches or lifts any object.

Training set	Reach	Lift
200 real	19/50	11/50
200 sim	26/50	11/50
2,000 sim	28/50	15/50
2,000 sim + 200 real	34/50	18/50

Table 5: Real-world Pick Category transfer with

\pi_{0.5}

on the dual-arm PiperX. The evaluation contains 50 trials across five target objects, with at least 5 visually or semantically similar distractors per trial.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Overview of the benchmark data generation pipeline, including Task Specification, Scene Generation, Motion Generation, Validator-Based Post-Processing, and Recorded Data Output.

Figure 5: Automated asset curation and management pipeline: (a) asset generation, (b) VLM-based annotation, (c) quality curation with human-in-the-loop review, and (d) reproducible sampling.

Figure 6: Distribution of the asset corpus used in the benchmark and experiments. Left : domain proportions. Right : asset count per super-category (18 in total).


Domain	Super-Category	Fine-Grained Categories	# Categories
desktop_supplies	office_stationery	stapler , ballpoint_pen , pen_holder , glue_stick , eraser , correction_tape , marker , sharpener , notebook	9
desktop_supplies	office_tools	scissors , scotch_tape , magnifier , cutter_knife , stamp , staples_box , sticky_notes	7
desktop_supplies	electronic_accessories	mouse , tv_remote , ac_remote , presenter_clicker , gamepad , earbud_case , charging_head , charging_cable , usb_drive	9
desktop_supplies	decorations	mini_sculpture , phone_stand , vase	3
kitchen_supplies	cutting_tools	fruit_knife , peeler , garlic_press , wire_cutters	4

Appendix

Table 6: Detailed asset category inventory. The count column reports the number of fine-grained categories within each super-category; object instances are summarized separately in Figure 6 .

Figure 7: Domain randomization on a fixed manipulation scene. (a) Baseline with default lighting and textures. (b) Randomized illumination. (c) Randomized object surface textures. (d) Randomized table surface material. Scene layout and embodiment are held fixed so that each panel differs in a single factor.

Panel of Figure 8 — Figure 8: Real-world experimental setup. Left : dual-arm AgileX PiperX robot with one static camera and two wrist-mounted cameras. Right : ALOHA teleoperation setup for data collection.

Figure 9: Two representative Reach-failed $\pi_{0.5}$ episodes on the pick_category seen split. Blue boxes mark the instruction target; orange boxes mark the similar distractor selected by the policy. The right panels show the robot approaching the distractor.

Model	Init checkpoint	Trainable scope	Batch / Steps
$\pi_{0}$	OpenPI pi0_base	LoRA only	32/GPU, 5k
$\pi_{0.5}$	OpenPI pi05_base	LoRA only	32/GPU, 5k
GR00T N1.7	GR00T-N1.7-3B	Proj./diff./VL-norm	24/GPU, 5k
Motus	motus-robotics/Motus	Qwen3-VL frozen	8/GPU, 5k

Appendix

Table 7: Summary of model-specific training recipes. Unlogged details are qualified in the text below.

Explore similar work

Feb 27, 2026

Robust Skills, Brittle Grounding: Diagnosing Restricted Generalization in Vision-Language Action Policies via Multi-Object Picking

Vision-language action (VLA) policies often report strong manipulation benchmark performance with relatively few demonstrations, but it remains unclear whether this reflects robust language-to-object grounding or reliance on object--location correlations that do not transfer beyond the training distribution. We present a controlled multi-object picking study that progressively increases object placement variability up to full workspace randomization and evaluates held-out object--location pairings that break familiar associations without increasing spatial difficulty. Across these stress tests and data scaling, we find that for representative VLA policies, including SmolVLA and

π_{0.5}

, execution of the manipulation primitive remains substantially more reliable than instruction-conditioned task success in harder regimes, suggesting that manipulation skill acquisition is decoupled from instruction following. We recommend augmenting manipulation benchmarks with task ladders and decomposed metrics that separately measure primitive execution and instruction-conditioned success to better diagnose instruction-grounded generalization.

David Emukpere, Romain Deffayet, Jean-Michel Renders

Naver Labs Europe, 6 Chem. de Maupertuis, Meylan, France.

Instruction Generalization in VLA Models Generalization in Robotic Manipulation Vision-Language Grounding Robot Manipulation Benchmarks

Ask PDF

Feb 5, 2026

MobileManiBench: Simplifying Model Verification for Mobile Manipulation

Vision-language-action models have advanced robotic manipulation but remain constrained by reliance on the large, teleoperation-collected datasets dominated by the static, tabletop scenes. We propose a simulation-first framework to verify VLA architectures before real-world deployment and introduce MobileManiBench, a large-scale benchmark for mobile-based robotic manipulation. Built on NVIDIA Isaac Sim and powered by reinforcement learning, our pipeline autonomously generates diverse manipulation trajectories with rich annotations (language instructions, multi-view RGB-depth-segmentation images, synchronized object/robot states and actions). MobileManiBench features 2 mobile platforms (parallel-gripper and dexterous-hand robots), 2 synchronized cameras (head and right wrist), 630 objects in 20 categories, 5 skills (open, close, pull, push, pick) with over 100 tasks performed in 100 realistic scenes, yielding 300K trajectories. This design enables controlled, scalable studies of robot embodiments, sensing modalities, and policy architectures, accelerating research on data efficiency and generalization. We benchmark representative VLA models and report insights into perception, reasoning, and control in complex simulated environments, with all code, datasets, and models publicly released.

Wenbo Wang, Fangyun Wei, QiXiu Li +5

University of Sydney · 1Microsoft Research Asia · 3Tsinghua University

Mobile Manipulation Robotics Simulation Synthetic Data Generation for Robotics Vision-Language-Action Models Robot Manipulation Benchmarks

Ask PDF

May 26, 2026

FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies

Vision-Language-Action (VLA) models are increasingly expected to not only complete robot tasks, but also follow human instructions about how those tasks should be executed. However, existing robot datasets usually pair trajectories with coarse goal-level language, leaving execution-critical details such as active arm, approach direction, and contact region unspecified. This limits steerable policy learning and robotic video understanding. We introduce FineVLA, an open framework for action-aligned fine-grained VLA supervision. The framework includes: (1) a data construction tool that unifies 972,247 trajectories across 85K tasks from 10 open-source robot datasets and builds FineVLA-Data, a human-verified dataset of 47,159 fine-grained trajectories; (2) a held-out benchmark with 500 videos, 11,631 atomic facts, and 1,030 VQA questions; (3) a robotics-specialized VLM annotator for scalable fine-grained annotation; and (4) a steerable VLA policy trained with controlled mixtures of fine-grained and raw goal-level instructions. Our experiments yield three findings. First, fine-grained supervision does not sacrifice goal-level success: FG-only improves over Raw-only by +1.4 to +8.1 success-rate points across settings. Second, fine-grained and raw instructions are complementary, following a consistent inverted-U trend peaking at FG:Raw = 1:2 to 1:1. The best mixed setting reaches 86.8%/82.5% in RoboTwin simulation and 62.7/100 in real-world dual-arm manipulation (vs. 49.9 Raw-only). Third, fine-grained supervision improves steerable control: the largest real-world gains appear on pose (+23), color (+18), and approach direction (+18)--factors where goal-level instructions provide no guidance. Overall, fine-grained language should augment goal-level instructions: specifying how to execute alongside what to achieve. Project page: https://finevla.xlang.ai/

Xintong Hu, Xuhong Huang, Jinyu Zhang +11

XLANG Lab, The University of Hong Kong · Qwen Team, Alibaba Inc.

Language-Conditioned Robot Manipulation Vision-Language-Action Models Robot Manipulation Benchmarks Language-Conditioned Robot Control

Ask PDF