Encoded but Not in Control: Revealing the Grounding Gap in Vision-Language Robot Policies

Shaohan Jiang; Jiahang Cao; Qiduo He; Fengting Deng; Kun Wu; Jingkai Sun; Jiaxu Wang; Qiang Zhang

cs.ROOct 5, 2026

Encoded but Not in Control: Revealing the Grounding Gap in Vision-Language Robot Policies

Authors: Shaohan Jiang, Jiahang Cao, Qiduo He, Fengting Deng, Kun Wu, Jingkai Sun, Jiaxu Wang, Qiang Zhang, +4 more

Organizations: HKU · CUHK(SZ) · Shanghai AI Lab · Horizon Robotics · CUHK · USTC

Abstract

Instruction following is central to language-conditioned robot policies: language should determine what to do when the same scene permits multiple valid actions. Yet successful execution alone cannot establish whether a policy follows the instruction or infers the task from the scene. We study this ambiguity through scene-preserving instruction interventions, using valid target substitutions, arbitrary nouns, and unrelated sentences while holding the scene fixed. We evaluate vision-language-action (VLA) policies and world-action models (WAMs) in simulation and in real-world experiments. Our analysis addresses three questions: (a) Does task success imply instruction following? When instructions request a different visible object, all evaluated policies predominantly approach and pick up the incorrect original target associated with the scene. (b) Is this failure caused by language insensitivity? Instruction perturbations affect task performance. A layerwise action lens shows intermediate action predictions respond to these perturbations. Linear probes accurately recover instructed targets, indicating modified instructions are encoded despite rarely determining target selection. (c) Why does encoded language fail to control action? Attention analysis indicates weak instruction-token contributions to action generation. Target-token attention can remain focused on the original object, revealing a mismatch between target encoding and visual grounding. UMAP and shared non-negative matrix factorization show target information remains accessible within representations increasingly organized by scene identity. Our findings expose a grounding gap concealed by nominal success and provide a diagnostic framework. They further establish a concrete criterion for progress: policies should reliably follow valid changes in user intent, even when they conflict with scene-favored behavior.

Figures & tables

Figure 1: Target replacement reveals a gap in instruction following. Holding the scene fixed, we change the instruction to request another visible and reachable object. In both LIBERO-Object and real-world manipulation, policies rarely grasp the currently instructed target ( dashed green trajectories ) and predominantly grasp the target named by the original instruction ( solid red trajectories ).

Environment	Model	Current instructed target		Original target
Environment	Model	Tend	Success	Tend	Success
LIBERO-Object	$\pi_{0.5}$	$13.0$	$11.5$	$86.5$	$69.5$
	GR00T N1.7	$1.3$	$0.5$	$96.5$	$90.9$
	FastWAM	$0.5$	$0.0$	$98.3$	$95.0$
	LingBot-VA	$1.8$	$1.3$	$96.3$	$84.4$
Real World	$\pi_{0.5}$	$10.8$	$5.8$	$80.8$	$74.2$

Table 1: Task completion under controlled target replacement. The current instructed target is another object in the same scene requested by the modified instruction; the original target is named by the original instruction. “Tend” denotes a committed approach or manipulation attempt, and “Success” denotes successful task completion. All values are percentages (%). Policies predominantly retain the original target despite the modified instruction.

Benchmark	Task / Setting	Instruction	VLA		WAM
Benchmark	Task / Setting	Instruction	$\pi_{0.5}$	GR00T N1.7	FastWAM	LingBot-VA
LIBERO	Spatial	Original	$98.8$	$97.7$	$98.2$	$98.5$
		Random noun	$99.0$ ↑0.2	$96.6$ ↓1.1	$73.6$ ↓25.1	$90.4$ ↓8.2
		Random sentence	$66.4$ ↓32.8	$53.2$ ↓45.5	$37.4$ ↓61.9	$75.2$ ↓23.7
	Object	Original	$98.2$	$98.5$	$100.0$	$99.6$
		Random noun	$96.6$ ↓1.6	$94.4$ ↓4.1	$92.6$ ↓7.4	$83.4$ ↓16.3

Table 2: Task performance under instruction corruption. Main values denote success rates (%); colored subscripts indicate relative decreases (

\downarrow

) or increases (

\uparrow

) from the original-instruction baseline (%). The results demonstrate that instruction corruption affects performance, with sensitivity varying across tasks and evaluation conditions.

Figure 2: Layerwise action predictions under instruction interventions. Curves show mean squared deviations from original-instruction predictions at matched layers and denoising steps. Random sentences (red) generally produce the largest deviations, followed by in-scene target replacements (blue) and out-of-scene nouns (orange), indicating language sensitivity within generation.

Figure 3: Linear probing of target and scene identity. Probes on mean-pooled instruction-token representations achieve near-ceiling accuracy in

\pi_{0.5}

and GR00T before and after fine-tuning on LIBERO-Object and real-world data, showing that both identities remain accessible despite instruction-following failures.

Figure 4: Modality contributions to action generation. Instruction tokens contribute relatively weakly to action generation in both

\pi_{0.5}

and GR00T N1.7, with visual or robot-state inputs accounting for larger shares across simulation and real-world settings.

Figure 5: Target-token visual attention under target replacement. When the instructed target changes from tomato sauce to BBQ sauce, attention remains focused on the original tomato sauce, which the policy grasps under both instructions. Red boxes mark the currently instructed target.

Figure 6: UMAP projections of

\pi_{0.5}

instruction representations. Columns show layers 1, 6, 13, and 18; rows compare Base and SFT. Shape denotes target identity; color denotes scene. Base representations cluster by target. After fine-tuning, the first layer retains target clustering, intermediate layers group by scene, and the final layer is more dispersed with residual target grouping.

Figure 7: Shared semi-NMF components before and after fine-tuning. Arrows connect components before (open circles) and after fine-tuning (filled circles). Across both settings, components shift from target-dominated to scene-dominated organization, with greater scene-unique and lower target-unique variance.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Pool	Members
LIBERO object vocabulary used for eligibility (45)	bowl, plate, ramekin, cup, mug, box, cabinet, drawer, basket, book, butter, cheese, chocolate, cream, sauce, can, bottle, pan, pot, lid, tray, rack, shelf, table, desk, chair, cloth, towel, bag, container, toy, block, cube, ball, hammer, brush, fork, spoon, knife, glass, jar, kettle, pitcher, bin, bucket.
Replacement and random-sentence nouns (99)	cat, dog, bird, fish, horse, lion, tiger, bear, wolf, rabbit, deer, eagle, shark, whale, frog, snake, turtle, rock, stone, tree, leaf, cloud, river, mountain, beach, forest, lake, desert, island, volcano, cave, valley, lamp, clock, mirror, pillow, blanket, carpet, curtain, vase, candle, picture, statue, phone, remote, cable, apple, banana, orange, grape, pizza, bread, rice, soup, cookie, cake, sandwich, salad, pasta, egg, milk, wrench, nail, screw, drill, rope, chain, key, lock, pipe, wire, tape, glue, paint, wheel, gear, shirt, hat, shoe, sock, coat, glove, belt, scarf, coin, stamp, ticket, card, map, flag, sign, label, button, switch, ring, medal, trophy, balloon, umbrella.
Random-sentence verbs (10)	observe, locate, inspect, examine, survey, approach, avoid, ignore, study, scan.
Random-sentence adjectives (10)	curious, ancient, shiny, purple, enormous, tiny, mysterious, hollow, rusty, invisible.

Appendix

Table 3: Lexical pools used for the Random Noun and Random Sentence interventions.

No.	Template
1	The {adj} {noun1} is resting next to the {noun2}.
2	A {noun1} and a {noun2} are placed on the {noun3}.
3	Please {verb} the {adj} {noun1} carefully.
4	There is a {noun1} near the {noun2} and a {noun3} on the floor.
5	The robot should {verb} the {noun1} without touching the {noun2}.
6	A {adj} {noun1} has been spotted beside the {noun2}.

Appendix

Table 4: Templates used to construct the Random Sentence intervention.

	Scene	Target set		Scene	Target set
1		$\mathcal{T}_{1}=\{$ alphabet soup , salad dressing , cream cheese , milk , tomato sauce , butter $\}$	2		$\mathcal{T}_{2}=\{$ cream cheese , alphabet soup , milk , tomato sauce , butter , orange juice $\}$
3		$\mathcal{T}_{3}=\{$ milk , cream cheese , tomato sauce , butter , orange juice , chocolate pudding $\}$	4		$\mathcal{T}_{4}=\{$ tomato sauce , milk , butter , orange juice , chocolate pudding , bbq sauce $\}$
5		$\mathcal{T}_{5}=\{$ butter , tomato sauce , orange juice , chocolate pudding , bbq sauce , ketchup $\}$	6		$\mathcal{T}_{6}=\{$ orange juice , butter , chocolate pudding , bbq sauce , ketchup , salad dressing $\}$
7		$\mathcal{T}_{7}=\{$ chocolate pudding , orange juice , bbq sauce , ketchup , salad dressing , alphabet soup $\}$	8		$\mathcal{T}_{8}=\{$ bbq sauce , chocolate pudding , ketchup , salad dressing , alphabet soup , cream cheese $\}$
9		$\mathcal{T}_{9}=\{$ ketchup , bbq sauce , salad dressing , alphabet soup , cream cheese , milk $\}$	10		$\mathcal{T}_{10}=\{$ salad dressing , ketchup , alphabet soup , cream cheese , milk , tomato sauce $\}$

Appendix

Table 5: LIBERO-Object scene–target sets. Each scene

s

is associated with a six-element instructed-target set

\mathcal{T}_{s}

Panel of Table 6 — Table 6: Real-world scene layouts. All six scenes use the same instructed-target set $\mathcal{T}_{\mathrm{real}}$ and the same instruction template.

Model	Denoising steps	Layers / step	Evaluated action region	Instantiation of $\mathcal{R}_{m,s}$
$\pi_{0.5}$	10	18	$10\times 7$	Timestep-conditioned adaptive normalization followed by the native action_out_proj
GR00T N1.7	4	32	$16\times 7$	Native output normalization and timestep-conditioned modulation, followed by the output projection and LIBERO-specific embodiment mapping
FastWAM	10	30	$32\times 7$	Shared trained ActionDiT action head used in the native inference pathway
LingBot-VA	10	30	$16\times 7$ *	Timestep-conditioned norm_out followed by action_proj_out

Appendix

Table 7: Model-specific action-lens readouts.

\mathcal{R}_{m,s}

is instantiated using the native output operations of each model. The evaluated action region contains only the valid LIBERO action coordinates used in the MSE computation. * For LingBot-VA, four context-fixed positions in the first conditioned chunk are excluded, yielding a

12\times 7

evaluated region for that chunk.

Explore similar work

Oct 7, 2026cs.RO

Do Vision-Language-Action Models Understand Instructions? A Mechanistic Interpretability Study on Language Grounding

Vision-Language-Action models are designed to generalise across environments and task descriptions, raising the question of whether their action generation actually depends on the language instruction, or whether they largely rely on visual cues and superficial correlations. Robustness to variance in the visual and linguistic observation space is critical for real-world deployment, yet VLAs lack explicit grounding modules and instead rely on the intrinsic language grounding capabilities of their Vision-Language model backbones. For this reason, we conduct a controlled mechanistic interpretability study on the language grounding capabilities of two state-of-the-art Vision-Language-Action models,

π_{0.5}

and GR00T N1.7, by applying activation and attribution patching to the residual stream of the action generation modules. We systematically corrupt the task instruction of input samples of the LIBERO benchmark following five strategies: synonym replacement, semantic scaling, directional corruption, random object substitution, and empty string. Our experiments find that both models are comparatively insensitive to abstract rephrasing and to referencing non-existent objects, but react strongly to empty task descriptions and, especially, to directional language. During action generation, this sensitivity is concentrated in different loci for each model: mainly in the early, periodic cross-attention layers for GR00T N1.7, versus distributed across the earliest and selected later layers for

π_{0.5}

. For GR00T N1.7, directional perturbations drive some of the largest causal effects while leaving the internal representational geometry comparatively unchanged, a dissociation we do not observe clearly for

π_{0.5}

. Finally, the reliability of attribution patching is model-dependent: it closely tracks activation patching for GR00T N1.7 but not for

π_{0.5}

Theodor Wulff, Angelo Cangelosi

Department of Computer Science The University of Manchester Manchester, United Kingdom

Topic pending

Ask PDF

Mar 6, 2026cs.RO

Restoring Linguistic Grounding in VLA Models via Train-Free Attention Recalibration

Vision-Language-Action (VLA) models enable robots to perform manipulation tasks directly from natural language instructions and are increasingly viewed as a foundation for generalist robotic policies. However, their reliability under Out-of-Distribution (OOD) instructions remains underexplored. In this paper, we reveal a critical failure mode in which VLA policies continue executing visually plausible actions even when the language instruction contradicts the scene. We refer to this phenomenon as linguistic blindness, where VLA policies prioritize visual priors over instruction semantics during action generation. To systematically analyze this issue, we introduce ICBench, a diagnostic benchmark constructed from the LIBERO dataset that probes language-action coupling by injecting controlled OOD instruction contradictions while keeping the visual environment unchanged. Evaluations on three representative VLA architectures, including Pi0, Pi0.5 and OpenVLA OFT, show that these models frequently succeed at tasks despite logically impossible instructions, revealing a strong visual bias in action generation. To mitigate this issue, we propose Instruction-Guided Attention Recalibration (IGAR), a train-free inference-time mechanism that rebalances attention distributions to restore the influence of language instructions. IGAR operates without retraining or architectural modification and can be directly applied to existing VLA models. Experiments across 30 LIBERO tasks demonstrate that IGAR substantially reduces erroneous execution under OOD contradictory instructions while preserving baseline task performance. We additionally validate the approach on a real Franka robotic arm, where IGAR effectively prevents manipulation triggered by inconsistent instructions.

Ninghao Zhang, Bin Zhu, Shijie Zhou +1

Tsinghua University · Singapore Management University · Institute of Trustworthy Embodied AI, Fudan University +1

Robot Policies Linguistics

Ask PDF

Sep 22, 2026cs.RO

RoboFollow: Unveiling the Instruction Following Mirage in Embodied Agents

Modern embodied agents achieve impressive success rates, yet their actual instruction-following ability is far weaker than these numbers suggest. We trace this illusion to a structural property we term low scene entropy: when a visual scene admits only one valid task, language becomes redundant and a policy can score highly while barely using it. We introduce RoboFollow, a diagnostic benchmark with three principles: (1) High Scene Entropy: each training scene supports multiple kinematically distinct task branches, making vision alone insufficient and forcing reliance on language. (2) Hierarchical Diagnostic Protocol: a four-level protocol (L0--L3) progressively perturbs visual layout and semantics, probing whether equivalent instructions yield consistent behavior and distinct ones yield discriminable behavior across spatial relations, attributes, trajectory constraints, and logic. (3) Confound-Controlled Diagnosis: we simplify interaction objects, restrict actions to the trained repertoire and report stage-wise Intent and Execution scores, isolating comprehension from motor execution. Evaluation of nine VLA and WAM policies shows that strong L0 performance, where attained, does not reliably transfer to L1--L3 under our fine-tuning setup. Representative mitigations, including stronger VLM backbones, QA co-training, LangForce, and Classifier-Free Guidance, all fail to close this gap. RoboFollow exposes genuine instruction following as a critical, overlooked bottleneck. Code and dataset are available at https://github.com/AutoLab-SAI-SJTU/RoboFollow and https://huggingface.co/datasets/AutoLab-SJTU/robofollow-data.

Chang Guo, Yukun Xie, Bohan Tan +6

AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University · Research Lab, Anyverse Dynamics

Embodied Agents Robocasa Instruction Recent Vision-Language Models

Ask PDF