DeicticVLA: Unifying Instruction Modes Based on Language and Deictic Gestures in a Single VLA

Kango Yanagida; Tatsuya Aoki; Yuichiro Yoshikawa; Takato Horii

cs.ROAug 28, 2026

DeicticVLA: Unifying Instruction Modes Based on Language and Deictic Gestures in a Single VLA

Authors: Kango Yanagida, Tatsuya Aoki, Yuichiro Yoshikawa, Takato Horii

Organizations: Dept. of Systems Innovation, Graduate School of Engineering Science, The University of Osaka, Japan. · International Research Center for Neurointelligence, The University of Tokyo, Japan.

Abstract

Vision-Language-Action models (VLAs) allow users to specify manipulation tasks in natural language, but distinguishing a target or placement goal among objects of the same category or similar appearance requires detailed expressions that VLAs may not use reliably. We propose DeicticVLA, which canonicalizes Language Instruction (LI), Vision-Language Instruction (VLI), and Visual Instruction (VI) into a text prompt and deictic masks through text-prompt completion and deictic gesture grounding, enabling a single pretrained VLA to handle all three instruction modes. With a shared backbone, demonstrations, and matched training steps, we compare two RGB visual prompting methods, two separate-channel mask prompting methods, and three training strategies in simulation. Under two-stage training, the four prompting methods achieve high in-distribution success but differ in their ability to use deictic masks in unseen layouts. Across methods, training-strategy ablations show that two-stage training improves such use, while retaining second-stage LI data mitigates forgetting without reducing VLI and VI performance. In three real-world tasks, one policy supports all modes. VLI and VI outperform LI under unseen expressions, appearance changes, and novel objects. For unseen categories, both achieve 100% success, compared with 16.7% for jointly trained LI. These results demonstrate the unified three-mode interface and guide DeicticVLA design.

Figures & tables

Fig. 1: Overview of the DeicticVLA system and the prompting methods evaluated in this work. (a) The user specifies a task through LI, VLI, or VI, which provide language only, language with a deictic gesture, or a deictic gesture only, respectively. Instruction canonicalization converts these user inputs into a canonical representation consisting of a text prompt and deictic masks through text-prompt completion and deictic gesture grounding. DeicticVLA uses this representation together with RGB observations and the robot state to generate an action chunk. (b) VP-BBox and VP-Fade construct RGB visual prompts by drawing a bounding box or fading regions outside the deictic mask on the RGB observation, respectively. MP-Early and MP-Late encode the mask separately and fuse the resulting mask prompt embeddings before and after the visual encoder, respectively.

Fig. 2: Roles of deictic masks in task specification. Mask-T specifies the object to be manipulated, Mask-G specifies the placement goal, and Mask-R specifies a reference object used in a spatial referring expression. For example, in “put the green tea on the tray,” the green-tea bottle is Mask-T and the tray is Mask-G. In “put the tea next to this on the tray,” the deictically indicated green-tea bottle serves as Mask-R.

Fig. 3: Example inputs for the three instruction modes in (a) LIBERO-Object, (b) LIBERO-Spatial, and (c) LIBERO-Goal. LI uses the original LIBERO text prompt. VLI uses Mask-T, Mask-R, or Mask-G depending on the suite. VI uses the fixed text prompt “follow visual instruction” with Mask-T+G, which consists of separate Mask-T and Mask-G deictic masks shown together in the same panel.

Optimizer	AdamW ( $\beta_{1}=0.9$ , $\beta_{2}=0.95$ , $\epsilon=10^{-8}$ )
Weight decay	$10^{-10}$
Gradient clipping	$\\|\mathbf{g}\\|_{2}\leq 1.0$
LR schedule	Cosine decay (warmup: 1k steps)
Learning rate	$2.5\times 10^{-5}\to 2.5\times 10^{-6}$
Total training steps	60,000
Batch size	32

TABLE I: Training Hyperparameters

		In-distribution SR				In-distribution $\Delta\mathrm{SR}$		Object-ZS $\Delta\mathrm{SR}$		Spatial-ZS $\Delta\mathrm{SR}$
Training	Prompting	LI	VLI	VI	Mean	VLI	VI	VLI	VI	VLI	VI
2S	MP-Early	93.0	96.3	96.9	95.4	+23.1	+31.7	+9.1	+10.0	+3.1	+4.4
	MP-Late	91.5	95.5	95.3	94.1	+22.3	+33.4	+29.2	+25.2	+0.9	+1.7
	VP-Fade	93.1	96.2	95.6	95.0	+25.6	+35.2	+26.2	+24.8	+1.3	+5.8
	VP-BBox	94.0	96.8	96.1	95.6	+25.6	+34.0	+29.2	+27.8	+16.0	+11.1
1S	MP-Early	90.5	92.3	91.4	91.4	+19.1	+24.1	+3.5	+3.4	+0.2	+0.9

TABLE II: Simulation results. Mean SR is averaged over LI, VLI, and VI. Higher

\Delta

SR indicates greater contribution from deictic masks.

Fig. 4: Real-world experiment setup and task environments. (a) Hardware setup with a UR5e, Robotiq 2F-85 with Fin-ray fingers, overhead camera, and wrist camera. (b) PickBlock : pick up a specified block from six colored blocks on the table. (c) PutBlock : put a block into a specified bowl among visually identical white bowls arranged in a row. (d) OrganizeToy : pick up a specified stuffed toy and place it into a specified basket; left shows the standard setting, right shows the VC-Surface condition with a tablecloth not seen during training. (e) Stuffed toys used in OrganizeToy across two super-categories: animal and vegetable. TL1 denotes seen objects used for both training and evaluation, while NO-Instance and NO-Category denote novel objects used only for evaluation.

TL	PickBlock	PutBlock	OrganizeToy
TL1	pick up …	put the block into …	put the …
	the {yellow, black, blue, red} block,	the {1st, 2nd} bowl from the right ( $N{=}2\text{--}5$ ),	{animal, vegetable} into the
	the {1st, 2nd, 3rd, 4th} block from the left,	the 3rd bowl from the right ( $N{=}3\text{--}5$ ).	{front, back left, back right} basket,
	the {yellow, blue} block on the right,		<category> into the
	the {black, red} block on the left.		{front, back left, back right} basket.
TL2	pick up …	put the block into …

TABLE III: Task templates for each task and condition.

Condition	Animal	Vegetable
TL1	dachshund (dog),	carrot-bunch (carrot),
	chihuahua (dog),	lettuce, green onion,
	hedgehog, bird, elephant, pig.	tomato.
NO-Instance	bulldog (dog), wanwan (dog),	carrot-single (carrot),
	pink bear (animal),	cucumber (vegetable),
	monkey (animal),	mushroom (vegetable),

TABLE IV: Stuffed toys used in OrganizeToy. For entries containing parentheses, the toy name appears first, followed by the category label used in LI in parentheses.

	PickBlock			PutBlock			OrganizeToy
Instruction mode	TL1	TL2	TL3	TL1	TL2	TL3	TL1	VC-Surface	NO-Instance	NO-Category
	( $n{=}24$ )	( $n{=}16$ )	( $n{=}16$ )	( $n{=}15$ )	( $n{=}15$ )	( $n{=}20$ )	( $n{=}40$ )	( $n{=}40$ )	( $n{=}24$ )	( $n{=}12$ )
LI (baseline)	70.8 (17)	12.5 ( 0 2)	18.8 ( 0 3)	80.0 (12)	0 6.7 ( 0 1)	30.0 ( 0 6)	67.5 (27)	65.0 (26)	66.7 (16)	33.3 ( 0 4)
LI	29.2 ( 0 7)	12.5 ( 0 2)	18.8 ( 0 3)	40.0 ( 0 6)	26.7 ( 0 4)	20.0 ( 0 4)	62.5 (25)	60.0 (24)	62.5 (15)	16.7 ( 0 2)
VLI	50.0 (12)	62.5 (10)	68.8 (11)	73.3 (11)	86.7 (13)	80.0 (16)	77.5 (31)	87.5 (35)	95.8 (23)	0. 100 (12)
VI	45.8 (11)	62.5 (10)	93.8 (15)	53.3 ( 0 8)	86.7 (13)	75.0 (15)	75.0 (30)	90.0 (36)	95.8 (23)	0. 100 (12)

TABLE V: Success rates in real-world experiments. Values outside and inside parentheses denote success rates (%) and numbers of successful trials, respectively;

n

denotes the total number of trials in each column.

Fig. 5: Evaluation GUI for aligning the initial scene configuration. Before each trial, the evaluator adjusts the objects so that their positions in the live overhead-camera image (Live) match those in the overlaid reference image (Reference).

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

	SR				$\Delta\mathrm{SR}$
Training	LI	VLI	VI	Mean	VLI	VI
2S	62.0	41.4	22.0	41.8	+1.6	$-$ 2.7
2S-NoLI	49.9	34.6	25.9	36.8	$-$ 2.7	$-$ 7.1
1S	69.5	43.2	25.9	46.2	$-$ 1.5	$-$ 5.2

Appendix

TABLE VI: In-distribution results for MP-Inter-3.

Vision-Language-Action Models

Vision-language-action models turn visual observations and natural-language instructions into actions for robots and other embodied agents.

See all 1,154 papers

VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Models
Mar 23, 2026Zixuan Wang, Yuxin Chen, Yuqi Liu +6Vision-Language-Action Models Reasoning for VLA Models
HKUST · CUHK · SmartMore
Mind-VLA: Instruction-Aware Spatial Representation Alignment for Vision-Language-Action Models
Aug 5, 2026Xingyu Ding, Yuzhong Zhao, Yang Wu +4Robotic Manipulation Vision-Language-Action Models
Nanjing University, Nanjing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use
Aug 6, 2026Jiarui Yang, Wen Huang, Jiale Zhang +2ICL for Robotics Continual Learning for VLA Models
Nankai University · Tsinghua University · Swiss Federal Institute of Technology in Lausanne (EPFL)

Mar 23, 2026J/K move · Enter open · S save

VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Models

Zixuan Wang, Yuxin Chen, Yuqi Liu, Jinhui Ye+5

HKUST · CUHK · SmartMore

Vision-Language-Action Models Reasoning for VLA Models Robustness of VLA Models Visual Grounding

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models typically map visual observations and linguistic instructions directly to control signals. This "black-box" mapping forces a single forward pass to simultaneously handle instruction interpretation, spatial grounding, and low-level control, often leading to poor spatial precision and limited robustness in out-of-distribution scenarios. To address these limitations, we propose VP-VLA, a dual-system framework that decouples high-level reasoning and low-level execution via a structured visual prompting interface. Specifically, a "System 2 Planner" decomposes complex instructions into sub-tasks and identifies relevant target objects and goal locations. These spatial anchors are rendered directly within the native RGB observation space as modality-consistent visual prompts, such as crosshairs and bounding boxes. This avoids the modality mismatch introduced by dense masks, affordance maps, or additional control-specific representations. Guided by these prompts and enhanced by a novel auxiliary visual grounding objective during training, a "System 1 Controller" reliably generates precise low-level execution motions. Extensive experiments in simulation and real world demonstrate that VP-VLA surpasses state-of-the-art end-to-end baselines including QwenOFT and GR00T-N1.6. Project page: https://visualprompt-vla.github.io/

Abstract

Vision-Language-Action Models

Explore similar work

VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Models

Mind-VLA: Instruction-Aware Spatial Representation Alignment for Vision-Language-Action Models

In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use