The Human--VLA Communication Gap: What Humans Leave Implicit and What Policies Cannot Recover

Yiqing Xu; Jiajun Wu

cs.ROOct 13, 2026

The Human--VLA Communication Gap: What Humans Leave Implicit and What Policies Cannot Recover

Authors: Yiqing Xu, Jiajun Wu

Organizations: Department of Computer Science, Stanford University, Stanford, CA, USA.

Abstract

A robot may know how to perform a task yet fail when a human asks for it naturally. We study this human--VLA communication gap by separating an information gap, where task semantics, grounding, or event structure remain unresolved, from a residual execution gap, where even a fully specified task depends on how it is organized and invoked. To do this, we characterize how people naturally express demonstrated objectives and evaluate controlled variants of the same physical tasks, with detailed analysis on $π_{0.5}$ and consistent trends on $π_0$ -FAST and Cosmos3-Edge. Our study reveals that humans increasingly compress instructions as tasks grow, relying more on contextual grounding and implicit execution structure. Current policies are misaligned with this behavior: they tolerate grounding dependence relatively well, but struggle when task semantics or event structure must be reconstructed. Even after the information gap is resolved, execution remains sensitive to command granularity and invocation state. These findings identify concrete targets for human-centered policy learning and interfaces that better bridge natural human objectives to robot behavior. Project website: https://commgap.github.io/.

Figures & tables

Fig. 1: Human–VLA communication gap. For the same physical task, human specifications vary in task content, reference, and composition, producing substantially different VLA performance (top). We separate this mismatch into an information gap , where task semantics, grounding, or event structure remain unresolved, and a residual execution gap , where the fully resolved task remains sensitive to command organization and invocation state (bottom), distinguishing recovering the intended task from reliably executing it through the policy’s learned interface.

Factor	Form	Controlled variation	Example
Task content	Action	Action performed	“ Put all the yellow objects in the left bin and all the blue objects in the right bin.”
	Outcome	Desired physical goal state	“All yellow objects should be in the left bin, and all blue objects should be in the right bin.”
	Intent / preference	Desired effect or preference	“ I want the objects sorted by color .”
Reference form	Direct	Directly named referent	“Put the yellow mug in the left bin and the blue cup in the right bin .”
	Grounding-dependent	Scene-resolved referent	“Put the yellow mug in the bin on the left and the blue cup in the bin on the right .”
	Missing	Referent omitted	“Put the yellow mug and the blue cup away.” (destination omitted)

TABLE I: Factorized taxonomy of human objective specification. All example variants describe the same physical task; highlighted spans mark the varied element.

Content $\downarrow$	Reference $\downarrow$	$\boldsymbol{\eta_{\tau}}$	$\boldsymbol{\gamma_{\tau}}$	Composition $\rightarrow$
				Enumerated ( $\bar{\delta}{=}1.00$ , $N{=}120$ )		Grouped ( $\bar{\delta}{=}0.47{\pm}0.08$ , $N{=}54$ )		Collective ( $\bar{\delta}{=}0.41{\pm}0.04$ , $N{=}50$ )
				Success (%)	$\Delta_{\mathrm{Success}}^{\mathrm{info}}$	Success (%)	$\Delta_{\mathrm{Success}}^{\mathrm{info}}$	Success (%)	$\Delta_{\mathrm{Success}}^{\mathrm{info}}$
Action	Direct	1.00	1.00	38.0 $\pm$ 5.4	0.0 (ref.)	15.9 $\pm$ 6.0	7.6 $\pm$ 3.1	13.1 $\pm$ 5.8	10.4 $\pm$ 3.5
	Grounding-dep.	1.00	0.33	35.2 $\pm$ 5.3	2.8 $\pm$ 1.8	13.5 $\pm$ 5.6	10.0 $\pm$ 3.3	12.4 $\pm$ 5.7	11.1 $\pm$ 3.6
	Missing	0.33	0.33	26.9 $\pm$ 4.9	11.1 $\pm$ 2.3	7.1 $\pm$ 4.2	16.4 $\pm$ 3.9	6.5 $\pm$ 4.2	17.0 $\pm$ 4.1
Outcome	Direct	0.67	0.67	33.5 $\pm$ 5.2	4.5 $\pm$ 1.9	11.8 $\pm$ 5.3	11.7 $\pm$ 3.5	9.7 $\pm$ 5.1	13.8 $\pm$ 3.8

TABLE II:

\pi_{0.5}

performance across the full factorial space of taxonomy-controlled specifications. Less explicit specifications (lighter blue, less decomposed columns) yield lower success and larger information-gap penalties (darker red); bold marks one-factor ablations from the fully specified Action–Direct–Enumerated reference. Rows vary content and reference; columns vary composition (enumerated, grouped, and collective forms on

N=120

54

, and

50

tasks). Mean

\pm

95% CI over task-level means after averaging rollouts and surface realizations. Results for

\pi_{0}

-FAST and Cosmos3-Edge, which show similar trends, are in Appendix -F .

Fig. 2: Humans increasingly compress execution structure as tasks grow. Instruction length increases with the number of events (left), but the executable structure made explicit decreases (right): coverage declines moderately, while groundedness and decomposition fall sharply.

Fig. 3: Behavioral cost of unresolved task structure. Left: For all three policies, grounding dependence is cheap, while missing action/object/destination information and undecomposed events cost far more. Right: With task complexity, coverage and decomposition become more strongly associated with the success gap than groundedness.

Fig. 4: Residual execution gap. On the same fully specified Action–Direct–Enumerated tasks, issuing atomic events sequentially improves success and partial progress over a single upfront prompt for all three policies, and resetting the arm to its home configuration between events adds a further gain.

Fig. 5: Information and execution gaps in human–VLA communication. Left: The policy succeeds when object identity or action is implicit, but fails when task semantics must be inferred. Right: Multi-step execution can fail through incorrect object selection; returning to a home pose between subtasks enables completion in the illustrated rollout. Solid blue: targets; dashed blue: destinations; red: incorrect objects; dashed gray: home pose.

Task structure	Operational criterion	Information retained by compact form	RoboLab examples
Atomic	$K_{\tau}=1$	One grounded $(A,O,D)$ event	BananaInBowlTask ; BowlInBinTask
Compound, enumerative	$K_{\tau}>1$ and critical event-specific arguments or ordering must still be enumerated	Individual object identities and/or event ordering	RubiksCubeAndBananaTask ; BananaThenRubiksCubeTask ; RubiksCubeThenBananaTask
Compound, rule-compressible	$K_{\tau}>1$ and a grounded rule $P_{\tau}$ generates the required events without event-wise enumeration	Observable set membership, relation, shared state, or attribute–destination mapping	BlackItemsInBinTask ; FoodPackingByColorTask ; ReorientAllMugsTask ; BlockStackingSpecifiedOrderTask
Compound, intent-compressible	$K_{\tau}>1$ and a semantic effect $F_{\tau}$ compactly characterizes the objective while leaving concrete event structure to be inferred	Desired purpose or task-level effect rather than complete grounded $(O,D)$ assignments	CleanUpToysTask ; CookingClearPlateTask ; RecycleCartonTask ; ThrowAwaySnacksTask

TABLE III: Operational definition of task structure used to analyze human objective specification. Event count measures physical manipulation complexity, whereas compression affordance characterizes the structure available for expressing those events compactly.

Task structure	Tasks	Fraction (%)
Atomic	64	$53.3\%$
Compound, enumerative	3	$2.5\%$
Compound, rule-compressible	42	$35.0\%$
Compound, intent-compressible	11	$9.2\%$
Total	120	$100.0\%$

TABLE IV: Structural distribution of RoboLab-120 under our object-centric event and compression-affordance definitions.

Factor	Agreement (%)	Coverage (%)
Task content	$91.3$	$98.0$
Reference form	$96.0$	$99.3$
Composition	$93.3$	$98.7$
Complete specification	–	$96.7$

TABLE V: Coverage of naturally elicited specifications by the proposed taxonomy and agreement between independent annotators.

Fig. 6: From human specifications to measurable information gaps. The same physical objective can be expressed through substantially different human specifications. Our taxonomy classifies each instruction by its task content, reference form, and composition, and maps these choices to the executable information made available to the policy: which actions, objects, and destinations are directly specified, require grounding, or are omitted. This yields quantitative measures of semantic coverage, groundedness, and decomposition ratio. Rather than studying instructions only as lexical variants, we can therefore relate VLA performance to the information they carry and identify which missing structure the policy can recover and which it cannot.

Task structure	Source ID. (%)	Correctness (%)
Atomic	$50.8\%$	$98.5\%$
Compound, enumerative	$51.9\%$	$97.3\%$
Compound, rule-compressible	$53.7\%$	$95.4\%$
Compound, intent-compressible	$57.2\%$	$90.6\%$
Overall	$52.8\%$	$96.0\%$

TABLE VI: Human-source identification and semantic correctness of taxonomy-matched generated specifications. Chance source identification is

50\%

Task complexity	Mean sentences	Mean words	Coverage $\eta_{\tau}$	Groundedness $\gamma_{\tau}$	Decomp ratio $\delta_{\tau}$
$K_{\tau}=1$	$1.08\pm 0.04$	$9.1\pm 0.6$	$0.95\pm 0.03$	$0.89\pm 0.04$	$1.00$
$K_{\tau}=2$	$1.27\pm 0.06$	$12.7\pm 0.8$	$0.86\pm 0.04$	$0.58\pm 0.06$	$0.66\pm 0.06$
$K_{\tau}\geq 3$	$1.56\pm 0.08$	$15.8\pm 1.0$	$0.74\pm 0.05$	$0.27\pm 0.06$	$0.21\pm 0.05$

TABLE X: Natural specification length and explicit task structure as the number of required atomic events increases. Values are mean

\pm

expected

95\%

confidence-interval half-width over human specifications.

Reasoning	Example	Unresolved structure	Required recovery
Visual grounding	“Put the banana in the bowl on the left.”	Grounded $O_{k}$ or $D_{k}$	Resolve the referenced entity from the observed scene.
Action inference	“The banana should be in the bowl.”	Action $A_{k}$	Infer the required manipulation from the stated object and desired physical state.
Task decomposition	“Put all the fruit in the bowl.”	Event structure $E_{\tau}$	Expand the compact objective into its constituent grounded manipulation events.
Intent inference	“I need room on this table.”	Physical objective	Infer the intended physical goal before recovering the corresponding grounded events.

TABLE XII: Reasoning operations required to recover executable structure left unresolved by a specification. These operations may co-occur in natural specifications; the controlled analysis in Table XIII isolates each requirement as closely as possible.

Human-Robot Interaction

Human-robot interaction studies how people and robots communicate, coordinate, and safely work or live together.

See all 449 papers

Encoded but Not in Control: Revealing the Grounding Gap in Vision-Language Robot Policies
Oct 5, 2026Shaohan Jiang, Jiahang Cao, Qiduo He +9Vision-Language Grounding Vision-Language-Action Models
HKU · CUHK(SZ) · Shanghai AI Lab +3
Distill: Uncovering the True Intent behind Human-Robot Communication
May 14, 2026Ting Li, David PorfirioIntent Classification Human-Robot Interaction
Computer Science George Mason University Fairfax, Virginia, USA
Causeway: Restoring Task Accessibility for Instruction Switching in VLA Policies
Sep 25, 2026Qingzi Wang, Kaixi Feng, Guangyao Shi +3Vision-Language-Action Models Test-Time Adaptation of VLA Models
University of Maryland, College Park, MD, USA. · University of Southern California, Los Angeles, CA, USA.

Oct 5, 2026J/K move · Enter open · S save

Encoded but Not in Control: Revealing the Grounding Gap in Vision-Language Robot Policies

Shaohan Jiang, Jiahang Cao, Qiduo He, Fengting Deng+8

HKU · CUHK(SZ) · Shanghai AI Lab+3

Vision-Language Grounding Vision-Language-Action Models Reasoning for VLA Models

Open paper AskPDF arXiv

Instruction following is central to language-conditioned robot policies: language should determine what to do when the same scene permits multiple valid actions. Yet successful execution alone cannot establish whether a policy follows the instruction or infers the task from the scene. We study this ambiguity through scene-preserving instruction interventions, using valid target substitutions, arbitrary nouns, and unrelated sentences while holding the scene fixed. We evaluate vision-language-action (VLA) policies and world-action models (WAMs) in simulation and in real-world experiments. Our analysis addresses three questions: (a) Does task success imply instruction following? When instructions request a different visible object, all evaluated policies predominantly approach and pick up the incorrect original target associated with the scene. (b) Is this failure caused by language insensitivity? Instruction perturbations affect task performance. A layerwise action lens shows intermediate action predictions respond to these perturbations. Linear probes accurately recover instructed targets, indicating modified instructions are encoded despite rarely determining target selection. (c) Why does encoded language fail to control action? Attention analysis indicates weak instruction-token contributions to action generation. Target-token attention can remain focused on the original object, revealing a mismatch between target encoding and visual grounding. UMAP and shared non-negative matrix factorization show target information remains accessible within representations increasingly organized by scene identity. Our findings expose a grounding gap concealed by nominal success and provide a diagnostic framework. They further establish a concrete criterion for progress: policies should reliably follow valid changes in user intent, even when they conflict with scene-favored behavior.

Abstract

Human-Robot Interaction

Explore similar work

Encoded but Not in Control: Revealing the Grounding Gap in Vision-Language Robot Policies

Distill: Uncovering the True Intent behind Human-Robot Communication

Causeway: Restoring Task Accessibility for Instruction Switching in VLA Policies