VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Models

Zixuan Wang; Yuxin Chen; Yuqi Liu; Jinhui Ye; Pengguang Chen; Changsheng Lu; Shu Liu; Bei Yu

cs.ROMar 23, 2026

VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Models

Authors: Zixuan Wang, Yuxin Chen, Yuqi Liu, Jinhui Ye, Pengguang Chen, Changsheng Lu, Shu Liu, Bei Yu, +1 more

Organizations: HKUST · CUHK · SmartMore

Abstract

Vision-Language-Action (VLA) models typically map visual observations and linguistic instructions directly to control signals. This "black-box" mapping forces a single forward pass to simultaneously handle instruction interpretation, spatial grounding, and low-level control, often leading to poor spatial precision and limited robustness in out-of-distribution scenarios. To address these limitations, we propose VP-VLA, a dual-system framework that decouples high-level reasoning and low-level execution via a structured visual prompting interface. Specifically, a "System 2 Planner" decomposes complex instructions into sub-tasks and identifies relevant target objects and goal locations. These spatial anchors are rendered directly within the native RGB observation space as modality-consistent visual prompts, such as crosshairs and bounding boxes. This avoids the modality mismatch introduced by dense masks, affordance maps, or additional control-specific representations. Guided by these prompts and enhanced by a novel auxiliary visual grounding objective during training, a "System 1 Controller" reliably generates precise low-level execution motions. Extensive experiments in simulation and real world demonstrate that VP-VLA surpasses state-of-the-art end-to-end baselines including QwenOFT and GR00T-N1.6. Project page: https://visualprompt-vla.github.io/

Figures & tables

Figure 1: VP-VLA leverages a dual-system architecture to bridge high-level reasoning and low-level control, maintaining competitive performance across a wide variety of tasks on in-distribution and out-of-distribution settings.

Figure 2: When facing (a) novel objects and (b) unseen spatial configurations , existing VLA models often fail to achieve precise localization ( Upper part ), whereas our VP-VLA leverages visual prompts to enhance generalization, ensuring accurate placement for target objects ( Lower part ).

Figure 3: Overview of VP-VLA pipeline. Our VP-VLA leverages a dual-system architecture to bridge high-level reasoning and low-level control. The System 2 planner first decomposes a language instruction into subtasks and generates visual prompts as interaction anchors and spatial constraints. The System 1 controller then utilizes these grounded visual cues to generate precise sensorimotor trajectories for complex, multi-stage manipulation tasks, while a grounding loss aligns the policy with the highlighted regions during training.

Figure 4: Overview of our real-world task and robot setting. (a) Robot setting for the experiments. We use an external camera A for categorization task and pick colored egg task, while using the external camera B for egg carton placement task. (b) We collect real-world robot demonstration for three task suites. (c) We present examples of each task with the OOD setting illustration.

Figure 5: Results of real-world robot manipulation. (a) Categorization result; (b) Pick colored egg result and egg carton placement result.

Task	w/o grounding	w/ all frame grounding	w/ point	w/ direct overlay	Ours (Full)
PnP * to * Close (Avg)	49.7	46.3	38.7	47.7	54.3
PnP Novel From Cuttingboard To * (Avg)	54.4	53.6	48.8	54.0	60.8
PnP Novel From Placemat To * (Avg)	46.0	45.0	46.0	46.5	54.5
PnP Novel From Tray To * (Avg)	43.2	46.0	49.6	50.0	46.0
PnP Novel From Plate To * (Avg)	54.0	58.0	56.5	56.5	53.5
Average	49.4	49.5	47.3	50.8	53.8

Table 3: Ablation results of evaluating the VLA models with the GR1 robot in the RoboCasa Tabletop simulation environment

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Models	Success Rate (%)
QwenOFT	30
Ours	50

Appendix

Table 4: Comparison of Scoop Beans Performance

Figure 6: Inference visualization on real-world tasks.

Section	Prompt Content
Task Description	{task_description}
Instruction	Decompose this robotic manipulation task into sequential atomic subtasks.
Subtask Types	• Pick action (e.g., “pick up the cup”) • Place action (e.g., “place the cup on the table”) • Manipulation action (e.g., “close the drawer”, “open the microwave”)
Rules	• Keep subtask descriptions short and clear. • Preserve the object and location names from the original task description.
Output Format	{ "subtasks": ["subtask 1", "subtask 2", ...] }

Appendix

Table 7: Prompt structure used for task decomposition. The VLM planner decomposes a high-level task description into atomic robotic manipulation subtasks.

Section	Prompt Content
Task	{task_description}
Current Subtask	{current_subtask}
Next Subtask	{next_subtask} or “None (this is the last subtask)”
Visual Context	• Image A: frame when the current subtask started • Image B: current frame Use visual differences between Image A and Image B to judge whether the current subtask has been completed.
Gripper State	LEFT_GRIPPER = {state} RIGHT_GRIPPER = {state}
	Optional notification if gripper state changes between frames: GRIPPER STATE CHANGE DETECTED: LEFT_GRIPPER: {prev_left} $\to$ {curr_left} RIGHT_GRIPPER: {prev_right} $\to$ {curr_right} This change may indicate that the current subtask has been completed. Consider both the visual evidence and this gripper state change when deciding whether to proceed to the next subtask.

Appendix

Table 8: Prompt structure used for subtask completion detection. The VLM planner determines whether the current subtask has been completed using visual evidence and gripper state information.

Figure 7: Inference visualization on the SimplerEnv simulation environment.

Figure 8: Inference visualization on the RoboCasa Tabletop simulation environment.

Figure 9: Inference visualization on real-world tasks.

Object	Ours	QwenOFT	Object	Ours	QwenOFT
Green Box			Red Box
Bottle (ID)	7	7	Banana (ID)	8	6
Beige cup (ID)	9	8	Orange (ID)	9	9
Black shoe (ID)	10	9	Corn (ID)	10	9
Toy chicken (ID)	8	7	Potato (ID)	10	10
Blue cup (OOD)	8	8	Pear (OOD)	9	8

Appendix

Table 9: Categorization results (10k steps). Successes out of 10 trials. Bold indicates best performance.

	In-Domain			OOD (Color)			OOD (Pos.)
Color	Ours	QwenOFT		Ours	QwenOFT		Ours	QwenOFT
Blue	10	7
Pink	8	6	Purple	9	4	Blue	9	6
Red	10	8	Green	9	3	Red	9	7
Yellow	9	7
Avg (%)	77.1	58.3		75.0	29.2		75.0	54.2

Appendix

Table 10: Pick colored egg results. Performance (raw score/12) of our method vs. QwenOFT. We evaluate across In-Domain (ID) and two OOD scenarios (Color and Position change).

In-Domain (ID)			Out-of-Distribution (OOD)
Pos.	Ours	QwenOFT	Pos.	Ours	QwenOFT
L1 C2	5.00	4.25	L1 C3	2.50	1.75
L1 C4	5.00	4.50	L2 C1	2.25	2.50
L2 C4	5.00	2.25	L3 C2	5.00	3.25
L2 C2	4.00	4.25	L4 C3	4.00	3.50
L3 C1	4.50	4.50	Summary Statistics

Appendix

Table 11: Egg carton placement results. Successes are reported out of 5 trials. L

X

Y

denotes the position at Line

X

, Column

Y

. We report both In-Domain (ID) and Out-of-Distribution (OOD) performance.

Explore similar work

Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models
Mar 19, 2026Bryce Grant, Xijia Zhao, Peng WangVision-Language-Action Models Robotics
Case Western Reserve University
Mind-VLA: Instruction-Aware Spatial Representation Alignment for Vision-Language-Action Models
Aug 5, 2026Xingyu Ding, Yuzhong Zhao, Yang Wu +4Robotic Manipulation Vision-Language-Action Models
Nanjing University, Nanjing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Look Before Acting: Enhancing Vision Foundation Representations for Vision-Language-Action Models
Mar 16, 2026Yulin Luo, Hao Chen, Zhuangzhe Wu +10Vision-Language-Action Models Vision-Language Models
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · The Chinese University of Hong Kong · Simplexity Robotics

Mar 19, 2026J/K move · Enter open · S save

Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models

Bryce Grant, Xijia Zhao, Peng Wang

Case Western Reserve University

Vision-Language-Action Models Robotics

Open paper AskPDF arXiv

A fine-tuned Vision-Language-Action (VLA) policy can easily place the alphabet soup in the basket on command, just as it saw during fine-tuning. However, if the basket is shifted five centimeters to the left, it drops the soup off the table. We developed an interpretability pipeline to investigate what such a policy is computing. Injecting task A's activations into the action expert while the policy looks at task B's scene makes pi0.5 execute task A's motor trajectory in 99.6% of episodes (

n{=}1{,}968

). The injected program is bound to absolute workspace coordinates and ignores the visible scene, which provides a mechanistic account of the perturbation brittleness that concurrent benchmarks report. The binding holds on a physical UR10e: directed replay redirects the commanded task in 26/26 rollouts in one direction and 18/26 in the other, with no-injection and random-direction controls at the floor, and once the source object is moved 10--15,cm the arm grasps and releases at the recorded coordinates in 50/50 trials, picking up the wrong block when one sits there and closing on empty air when none does. Applying the same interventions to six VLAs on LIBERO, MetaWorld, SimplerEnv, and ALOHA over 420{,}000+ rollouts yields three further findings: every architecture encodes the instruction, and four of the five weight it below vision whenever the scene alone identifies the goal; under the standard replacement write-back the SAE pooling that preserves behavior is architecture-specific (pi0.5 per-token, X-VLA mean-pool, SmolVLA indifferent), and goal selection and motor generation separate into VLM and expert pathways wherever the two are distinguishable. We release Action Atlas (https://action-atlas.com), an interactive platform covering all 520+ trained SAEs, 79 identified concepts, and the scored UR10e rollouts. Code: https://github.com/CWRU-AISM/action-atlas

Abstract

Explore similar work

Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models

Mind-VLA: Instruction-Aware Spatial Representation Alignment for Vision-Language-Action Models

Look Before Acting: Enhancing Vision Foundation Representations for Vision-Language-Action Models