Do VLAs Really Follow Instructions? Diagnosing How-Level Instruction Following in Vision-Language-Action Models

Sangwu Park; Wonjoong Kim; Yeonjun In; Chanyoung Park

cs.CVOct 15, 2026

Do VLAs Really Follow Instructions? Diagnosing How-Level Instruction Following in Vision-Language-Action Models

Authors: Sangwu Park, Wonjoong Kim, Yeonjun In, Chanyoung Park

Organizations: KAIST

Abstract

Vision-language-action (VLA) models are expected to follow instructions like their language backbones, yet prior work has focused on What-level instructions that specify the task rather than How-level instructions that specify how to carry it out. Through our diagnoses, we find that How-level instructions do not help current VLAs and lower task success. We hypothesize that pairing each training episode with a single instruction limits How-level instruction following, and show that training with segment-level enriched instructions makes actions respond to such instructions without consistently raising task success. To measure How-level instruction following beyond standard benchmarks, we introduce SIMPLER-IN, where success requires conditions given only in the instruction. It shows that the gained responsiveness does not extend to demanding or out-of-distribution How-level instructions. How-level instruction following thus remains an open challenge.

Figures & tables

Figure 1: Overview of this work. (a), (b): Our diagnoses show that VLAs do not clearly follow How-level instructions. (c): Examples of enriched instructions.

General VLAs
	Offline correction						Online intervention
Model	$N$	Retry	Correction	$p$ (C vs. R)	Inverted	$p$ (C vs. I)	No intervention	Corrector	$\Delta$	Unified	$\Delta$
$\pi_{0}$	133	36.1	17.3	.001	11.3	.185	70.8 (2.8)	72.9 (4.2)	+2.1	58.0 (3.0)	$-$ 12.8
InternVLA-M1	115	26.1	13.0	.028	18.3	.307	72.2 (0.9)	70.5 (1.3)	$-$ 1.7	58.0 (1.3)	$-$ 14.2
QwenGR00T	142	22.5	8.5	<.001	7.0	.815	65.3 (0.9)	61.1 (3.6)	$-$ 4.2	53.5 (2.1)	$-$ 11.8
Pooled	390	28.2	12.8	–	11.8	–	69.4 (1.0)	68.2 (0.3)	$-$ 1.2	56.5 (1.8)	$-$ 12.9

Table 1: Diagnosis results for public VLA checkpoints (%). Offline correction:

N

is the number of failed episodes pooled over three seeds, and

p

compares Correction (C) with Retry (R) and Inverted (I), with bold for

p<.05

. Online intervention: Entries show the mean over the three runs with standard errors in parentheses, and

\Delta

is the change from No intervention, with bold for positive values. Pooled rows aggregate the models in each group.

Model	$\pi_{0}$	InternVLA-M1	QwenGR00T	InstructVLA	LangForce
Ratio	0.91	0.70	0.72	0.41	1.03

Table 2: Ratio of the attention.

Control
	Offline correction						Online intervention
Model	$N$	Retry	Correction	$p$	Inverted	$p$	No intervention	Corrector	$\Delta$	Unified	$\Delta$
$\pi_{0}$	195	25.1	12.3	.001	6.7	.080	63.5 (1.2)	60.1 (2.5)	$-$ 3.4	45.5 (1.3)	$-$ 18.0
InternVLA-M1	218	5.5	2.8	.146	2.8	1.000	52.4 (2.4)	49.7 (2.1)	$-$ 2.7	42.0 (1.4)	$-$ 10.4
QwenGR00T	194	10.8	7.2	.296	4.6	.267	57.3 (2.4)	55.6 (2.4)	$-$ 1.7	36.8 (1.7)	$-$ 20.5
InstructVLA	236	11.0	2.5	<.001	2.5	1.000	27.8 (3.6)	26.7 (0.9)	$-$ 1.1	24.3 (0.7)	$-$ 3.5

Table 3: Results of the enriched models and their controls, following the notation of Table 1 . † marks

p<.05

with Correction higher, as expected from Eq. ( 1 ).

Figure 2: Action-level analysis. Error bars show 95% confidence intervals, the line marks chance level, and All combines the five model families.

Figure 3: Overview of SIMPLER-IN . (a) Task examples with their conditions and instructions. (b) The Path scene with the barrier shown (left) and as seen (right).

Control
Model	Path	Grasp	Place
$\pi_{0}$	25.0 / 0.0	83.3 / 50.0	50.0 / 50.0
InternVLA-M1	0.0 / –	50.0 / 50.0	54.2 / 38.5
QwenGR00T	12.5 / 0.0	58.3 / 50.0	0.0 / –
InstructVLA	0.0 / –	12.5 / 66.7	8.3 / 50.0
LangForce	4.2 / 0.0	45.8 / 27.3	4.2 / 100.0

Table 4: Results on SIMPLER-IN (%). Each entry is task success rate / instruction following rate among successful episodes. Table 14 reports each public model.

Figure 4: Frequency of How-level content in the ECoT annotations and the enriched instructions (log scale).

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Group	$N$	Cause	Executable
Public	17	15	17
Control	17	13	17
Enriched	16	16	16
All	50	44	50

Appendix

Table 5: Number of corrections that identify the cause of the failure and that can be executed in the manual check.

Group	Start	$N$	R	C	I
Public	First	274	19.0	7.3	6.9
	Later	410	22.2	12.2	9.0
Control	First	458	14.0	4.4	2.4
	Later	588	12.4	7.1	4.9
Enriched	First	452	12.2	8.8	6.4
	Later	511	16.4	15.1	12.3

Appendix

Table 6: Rerun success rate (%) by whether the critical segments start at the first step, in the notation of Table 1 .

Figure 5: Interface for the manual check of corrections.

		Retry	Seed $\xi$		Seed $\xi^{\prime}$
Group	$N$	( $\xi^{\prime}$ )	C	I	C	I
Public	684	20.9	10.2	8.2	10.8	10.5
Control	1046	13.1	5.9	3.8	6.6	5.3
Enriched	963	14.4	12.1	9.6	11.4	10.5

Appendix

Table 7: Rerun success rate (%) under each seed, in the notation of Table 1 .

Model	Initialization	Trained parameters	Control	Enriched	Budget
$\pi_{0}$	PaliGemma-3B	LoRA (r=32) on LM, full ViT and action expert	48k	51.2k	100k
InternVLA-M1	InternVLA-M1 (stage 1)	LoRA (r=32) on LLM, full Q-Former and action head	60k	100k	120k
QwenGR00T	Qwen3-VL-4B	LoRA (r=32) on VLM, action head	40k	40k	50k
InstructVLA	Eagle2-2B (expert stage)	LoRA (r=128) on LLM, action expert	40k	95k	100k
LangForce	Qwen3-VL-4B	LoRA (r=32) on VLM, action head, action queries	40k	40k	50k

Appendix

Table 8: Training of the enriched models and their controls, which share the recipe. The Control and Enriched columns give the steps of the checkpoints with the highest task success in SIMPLER. Budget gives the total training steps.

Model	Attention	Query
$\pi_{0}$	VLM, 18 layers	All prefix
InternVLA-M1	Q-Former cross-attn.	64 queries
QwenGR00T	VLM, 36 layers	All
InstructVLA	VLM, 6 layers	64 meta
LangForce	VLM, 36 layers	64 action

Appendix

Table 9: Where the attention in Table 2 is measured, averaged over heads, query tokens, and layers.

Group	$N$	R	C	I	Neutral
Public	684	20.9	10.2	8.2	19.7
$\pi_{0}$	133	36.1	17.3	11.3	31.6
InternVLA-M1	115	26.1	13.0	18.3	35.7
QwenGR00T	142	22.5	8.5	7.0	13.4
InstructVLA	210	11.9	7.1	4.3	10.0
LangForce	84	9.5	6.0	1.2	14.3

Appendix

Table 10: Rerun success rate (%) with the neutral sentence, in the notation of Table 1 .

Model	Public	Control	Enriched
$\pi_{0}$	52.3 (109)	52.9 (174)	89.3 (169)
InternVLA-M1	55.4 (101)	48.0 (179)	87.9 (149)
QwenGR00T	52.6 (114)	50.9 (159)	88.1 (159)
InstructVLA	56.0 (182)	62.0 (213)	80.2 (197)
LangForce	48.0 (75)	47.4 (171)	77.7 (166)
All	53.5 (581)	52.7 (896)	84.4 (840)

Appendix

Table 11: Fraction (%) of failed episodes that follow the directions of the correction, with the number of episodes in parentheses. The bootstrap 95% confidence intervals of All are [49.8, 57.4], [49.2, 56.1], and [81.8, 86.9].

Model	Public	Control	Enriched
$\pi_{0}$	0.25 / 31.2	0.21 / 28.7	4.33 / 79.3
InternVLA-M1	0.18 / 29.7	0.03 / 24.6	1.05 / 52.3
QwenGR00T	0.18 / 30.7	0.11 / 32.7	2.09 / 62.9
InstructVLA	0.32 / 35.2	0.34 / 19.2	1.39 / 57.4
LangForce	0.09 / 34.7	$-$ 0.01 / 25.1	1.60 / 57.2
All	0.21 / 32.5	0.15 / 25.7	1.74 / 61.9

Appendix

Table 12: Median displacement (cm) in the specified direction / fraction (%) of episodes that move more than 1 cm in every specified direction.

Task	Instruction
Path	“put carrot on plate. there is a transparent barrier. approach the carrot from the far side, behind the carrot, away from the robot , and exit the same way.”
	“put carrot on plate. there is a transparent barrier. approach the carrot from the near side, in front of the carrot, close to the robot , and exit the same way.”
Grasp	“put carrot on plate. grasp the carrot by the green end .”
	“put carrot on plate. grasp the carrot by the orange end .”
Place	“put the spoon on the towel. place it on the far part of the towel, away from the robot.”
	“put the spoon on the towel. place it on the near part of the towel, close to the robot.”

Appendix

Table 13: Instructions in SIMPLER-IN , each consisting of the task instruction and the How-level instruction, with the condition in bold.

Model	Path	Grasp	Place
$\pi_{0}$	33.3 / 0.0	70.8 / 47.1	83.3 / 60.0
InternVLA-M1	20.8 / 0.0	75.0 / 55.6	37.5 / 44.4
QwenGR00T	12.5 / 0.0	41.7 / 50.0	58.3 / 57.1
InstructVLA	4.2 / 0.0	25.0 / 50.0	16.7 / 50.0
LangForce	0.0 / –	62.5 / 66.7	58.3 / 42.9
Pooled	14.2 / 0.0	55.0 / 54.5	50.8 / 52.5

Appendix

Table 14: Results of each public checkpoint on SIMPLER-IN , in the notation of Table 4 .

Content	Pattern	ECoT	Enriched
Motion direction	left, right, up, down, upward, downward, forward, backward, lower, lift, raise	40.3	83.5
Approach side	from the left / right / front / back / far / near / top / other side / end	0.016	0.19
Part by attribute	color word + end / tip / part / side / half	0.0007	0.004
Far / near part	far / near + part / side / half / end / edge / corner	0.019	0.24
Relative to robot	away from / toward(s) / close(r) to + (the) robot / you / camera / base	0.014	0.08

Appendix

Table 15: Patterns and frequencies (%) of the How-level content in Fig. 4 . Color words are green, orange, red, blue, yellow, white, black, brown, purple, and pink.

Figure 6: Prompt for generating corrections with Gemini 3.1 Pro. Placeholders in braces are filled for each failed episode.

Figure 7: Prompt for generating inverted corrections. The inputs before the correction are the same as in Fig. 6 .

Figure 8: Prompts of System 2 for (a) the corrector and (b) the unified mode.

Figure 9: Prompt for generating enriched instructions with GPT-5.4-mini.

Vision-Language-Action Models

Vision-language-action models turn visual observations and natural-language instructions into actions for robots and other embodied agents.

See all 1,168 papers

Flatness Preserves Instruction Following in Vision-Language-Action Models
Jun 22, 2026Haochen Zhang, Yonatan BiskSharpness-Aware Minimization Vision-Language-Action Models
Carnegie Mellon University
Do Vision-Language-Action Models Understand Instructions? A Mechanistic Interpretability Study on Language Grounding
Oct 7, 2026Theodor Wulff, Angelo CangelosiVision-Language-Action Models Robustness of VLA Models
Department of Computer Science The University of Manchester Manchester, United Kingdom
Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models
Mar 19, 2026Bryce Grant, Xijia Zhao, Peng WangVision-Language-Action Models Robotics
Case Western Reserve University

Jun 22, 2026J/K move · Enter open · S save

Flatness Preserves Instruction Following in Vision-Language-Action Models

Haochen Zhang, Yonatan Bisk

Carnegie Mellon University

Sharpness-Aware Minimization Vision-Language-Action Models Flat Minima VLMs for Robotics

Open paper AskPDF arXiv

Vision-language-action (VLA) models have the potential for open-world generalization by leveraging pretrained vision-language representations, yet downstream finetuning on limited robot data often degrades these representations, leading to brittle policies that ignore language instructions in favor of visual shortcuts, a failure mode we term instruction blindness. We hypothesize that standard finetuning with limited data applies gradients to a sparse set of points, which manifests as a sharp loss landscape with high-curvature minima. We propose to address this directly through flatness-preserving optimization while finetuning on the exact same data, where learning a flatter landscape results in a model more robust to perturbations in the weight space. Specifically, we demonstrate that simply applying sharpness-aware minimization during VLA finetuning significantly improves instruction following by over 60% across multiple simulation and real-world benchmarks without additional data, architectural modification, or retraining. We further analyze the effect of selective sharpness, quantify its effects, and show that our approach is complementary to existing guidance techniques. Project page can be found at https://haochenz11.github.io/papers/flatness-vla/.

Abstract

Vision-Language-Action Models

Explore similar work

Flatness Preserves Instruction Following in Vision-Language-Action Models

Do Vision-Language-Action Models Understand Instructions? A Mechanistic Interpretability Study on Language Grounding

Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models