Rephrase Before You Act: Characterizing and Mitigating Language Sensitivity in Vision-Language-Action Models

Mikey Watts; Yuchen Cui

cs.ROOct 7, 2026

Rephrase Before You Act: Characterizing and Mitigating Language Sensitivity in Vision-Language-Action Models

Authors: Mikey Watts, Yuchen Cui

Organizations: Independent Researcher · University of California, Los Angeles

Abstract

Vision-language-action models (VLAs) are strikingly sensitive to instruction phrasing and do not inherit the language robustness of the vision-language models they are built on. A one-word edit can move success by tens of points: $π_{0.5}$ turns on a LIBERO stove 100% of the time for "switch on the stove" and 2% for "switch on the hot plate", and a $π_0$ checkpoint finetuned with rephrase augmentation still shows swings of up to 61 points. We characterize this sensitivity with statistically tested single-edit swings and an oracle phrase search, which shows that phrasing alone nearly closes the 21-point gap between in-distribution and out-of-distribution tasks. We then reduce it without modifying the policy. Because the sensitivity is systematic, it can be expressed as explicit rules: we score many phrasings of a few training tasks, have a large language model distill the evidence into ten to twenty rephrasing rules, and at deployment rewrite each incoming instruction once under these rules. The rules improve the frozen $π_0$ by 16 to 27% relative on twelve held-out tasks across adversarial, VLM-generated, and human-generated phrasings, with gains concentrated on out-of-distribution tasks. The pipeline replicates on $π_{0.5}$ and LIBERO, lifting in-finetune success from 93.6% to 97.8%. The method requires no retraining and no per-step verification, and applies zero-shot to unseen tasks and instructions. Project website: https://sttawm.github.io/rephrase-before-you-act

Figures & tables

Fig. 1 : Method overview. Evidence gathering : a VLM rephrases each training task, and the frozen VLA’s rollout success on each phrasing forms the evidence $\mathcal{D}$ . Rule distillation : an LLM distills $\mathcal{D}$ , once and offline, into 10–20 explicit phrasing rules $R$ . Rule application : at deployment a VLM conditioned on $R$ rewrites each incoming instruction once ( $f_{R}$ ) before passing it to the unchanged VLA. The phrasings, scores, and rules shown are illustrative only.

Fig. 2 : Representative SIMPLER Bridge first-frame scenes. Tabletop tasks share one setup and differ only in the source and destination objects; the rightmost panel is the benchmark’s sink scene.

Fig. 3 : Representative LIBERO first-frame scenes. libero_goal and libero_spatial share one scene per suite, libero_object places groceries beside a basket, and libero_90 scenes vary by task.

Fig. 4 : Single-edit swings for

\pi_{0}

on SIMPLER Bridge: 26 significant sets over 16 tasks, the largest per task and edit category. Each row is one set whose phrases differ by one edit; dots are success rates over n = 72 rollouts (basket task n = 180), the filled dot is the best phrasing, and the right column is the swing in points. Every set has

p<0.05

under a two-proportion z test.

Fig. 5 : Single-edit swings for

\pi_{0.5}

on LIBERO, formatted as Fig. 4 : 15 sets over 13 tasks, n = 50 per phrase, with eight further significant sets removed as scene-ambiguous. Swings reach 100 pp, as “fire up the stove” succeeds 100% of the time and “turn on the stove” 6%.

Fig. 6 : Success of the rephrase-augmented $\pi_{0}$ on the 12 sealed Bridge tasks by phrasing condition, over 24 fixed layouts. Solid bars pool all tasks; thin flanks are the in-distribution (left, 5 tasks) and out-of-distribution (right, 7) strata. Repetitions per layout: oracle and canonical 12, adversarial 2, naturals 1. Oracle phrasing shrinks the 21-point strata gap to 3 points.

Fig. 7 : Success of $\pi_{0.5}$ on LIBERO by phrasing condition over 50 layouts per phrase. Solid bars pool all 28 tasks; thin flanks are the in-finetune (left, 10 tasks) and out-of-finetune (right, 18) halves. In-finetune, adversarial rephrasing costs 50 pp (97 to 47); out-of-finetune, the canonical, natural, and adversarial tiers are indistinguishable (52 / 50 / 48). Oracle search finds headroom in both regimes (99 / 66).

Fig. 8 : Main results: success of the rephrase-augmented $\pi_{0}$ on the 12 sealed SIMPLER tasks, one panel per phrase set and one bar group per applier. Dashed red = no-rephraser baseline; grey = no-rules rephraser; greens = the out-of-finetune, both, and in-finetune rulebooks, each the mean over three draws; thin flanks = in-distribution (left, 5 tasks) and out-of-distribution (right, 7) strata. Out-of-finetune rulebooks are strongest on every non-canonical set, with gains concentrated out-of-distribution; the canonical panel, with one phrase per task, is statistically underpowered and should be read with caution.

	Adversarial	Human-Generated	VLM-Generated
	(n=72)	Naturals (n=363)	Naturals (n=186)
Claude	+26.0% (p=0.0083)	+21.7% (p<0.0001)	+20.5% (p<0.0001)
Gemini	+26.5% (p=0.0074)	+20.3% (p<0.0001)	+17.8% (p<0.0001)
Qwen	+24.4% (p=0.0112)	+25.2% (p<0.0001)	+16.1% (p=0.0001)

TABLE I : Relative improvement of the out-of-finetune rulebook over the no-rephraser baseline: mean paired difference as a percent of the baseline rate, with permutation-test p-values (§ VI-C ); n is the number of base instructions.

	Adversarial	Human-Generated	VLM-Generated
	(n=72)	Naturals (n=363)	Naturals (n=186)
Claude	+26.6% (p=0.0001)	+11.2% (p<0.0001)	+4.7% (p=0.0924)
Gemini	+29.1% (p=0.0001)	+9.3% (p=0.0001)	+6.7% (p=0.0167)
Qwen	+23.1% (p=0.0001)	+6.0% (p=0.0201)	+7.3% (p=0.0271)

TABLE II : As Table I , but against the no-rules rephraser, isolating the effect of the distilled rules from rephrasing alone.

	no rephraser	no-rules rephraser	CoVer
Adversarial	24.5	24.3	21.6
VLM-Generated Naturals	26.0	28.8	22.8
Canonical	36.1	30.5	26.4

TABLE III : CoVer against our two baselines on the sealed tasks: pooled success (%) at step 60, all arms using identical Gemini reasoning traces. CoVer trails both baselines on every set.

Explore similar work

Do Vision-Language-Action Models Understand Instructions? A Mechanistic Interpretability Study on Language Grounding
Oct 7, 2026Theodor Wulff, Angelo Cangelosi
Department of Computer Science The University of Manchester Manchester, United Kingdom
Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steering
Jun 10, 2026Hyun Joe Jeong, Gokul Swamy, Andrea BajcsySteering
Robotics Institute Carnegie Mellon University
When Does Language Matter? Multilingual Instructions Reveal Step-wise Language Sensitivity in Vision-Language-Action Models
Jun 10, 2026Xuan Dong, Zhe Han, Tianhao Niu +2Language Pairs
Harbin Institute of Technology

Oct 7, 2026, cs.ROJ/K move · Enter open · S save

Do Vision-Language-Action Models Understand Instructions? A Mechanistic Interpretability Study on Language Grounding

Theodor Wulff, Angelo Cangelosi

Department of Computer Science The University of Manchester Manchester, United Kingdom

Open paper AskPDF arXiv

Vision-Language-Action models are designed to generalise across environments and task descriptions, raising the question of whether their action generation actually depends on the language instruction, or whether they largely rely on visual cues and superficial correlations. Robustness to variance in the visual and linguistic observation space is critical for real-world deployment, yet VLAs lack explicit grounding modules and instead rely on the intrinsic language grounding capabilities of their Vision-Language model backbones. For this reason, we conduct a controlled mechanistic interpretability study on the language grounding capabilities of two state-of-the-art Vision-Language-Action models,

π_{0.5}

and GR00T N1.7, by applying activation and attribution patching to the residual stream of the action generation modules. We systematically corrupt the task instruction of input samples of the LIBERO benchmark following five strategies: synonym replacement, semantic scaling, directional corruption, random object substitution, and empty string. Our experiments find that both models are comparatively insensitive to abstract rephrasing and to referencing non-existent objects, but react strongly to empty task descriptions and, especially, to directional language. During action generation, this sensitivity is concentrated in different loci for each model: mainly in the early, periodic cross-attention layers for GR00T N1.7, versus distributed across the earliest and selected later layers for

π_{0.5}

. For GR00T N1.7, directional perturbations drive some of the largest causal effects while leaving the internal representational geometry comparatively unchanged, a dissociation we do not observe clearly for

π_{0.5}

. Finally, the reliability of attribution patching is model-dependent: it closely tracks activation patching for GR00T N1.7 but not for

π_{0.5}

Abstract

Explore similar work

Do Vision-Language-Action Models Understand Instructions? A Mechanistic Interpretability Study on Language Grounding

Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steering

When Does Language Matter? Multilingual Instructions Reveal Step-wise Language Sensitivity in Vision-Language-Action Models