A robot may know how to perform a task yet fail when a human asks for it naturally. We study this human--VLA communication gap by separating an information gap, where task semantics, grounding, or event structure remain unresolved, from a residual execution gap, where even a fully specified task depends on how it is organized and invoked. To do this, we characterize how people naturally express demonstrated objectives and evaluate controlled variants of the same physical tasks, with detailed analysis on π0.5 and consistent trends on π0-FAST and Cosmos3-Edge. Our study reveals that humans increasingly compress instructions as tasks grow, relying more on contextual grounding and implicit execution structure. Current policies are misaligned with this behavior: they tolerate grounding dependence relatively well, but struggle when task semantics or event structure must be reconstructed. Even after the information gap is resolved, execution remains sensitive to command granularity and invocation state. These findings identify concrete targets for human-centered policy learning and interfaces that better bridge natural human objectives to robot behavior. Project website: https://commgap.github.io/.
Figures & tables
Fig. 1: Human–VLA communication gap. For the same physical task, human specifications vary in task content, reference, and composition, producing substantially different VLA performance (top). We separate this mismatch into an information gap , where task semantics, grounding, or event structure remain unresolved, and a residual execution gap , where the fully resolved task remains sensitive to command organization and invocation state (bottom), distinguishing recovering the intended task from reliably executing it through the policy’s learned interface.
Factor
Form
Controlled variation
Example
Task content
Action
Action performed
“ Put all the yellow objects in the left bin and all the blue objects in the right bin.”
Outcome
Desired physical goal state
“All yellow objects should be in the left bin, and all blue objects should be in the right bin.”
Intent / preference
Desired effect or preference
“ I want the objects sorted by color .”
Reference form
Direct
Directly named referent
“Put the yellow mug in the left bin and the blue cup in the right bin .”
Grounding-dependent
Scene-resolved referent
“Put the yellow mug in the bin on the left and the blue cup in the bin on the right .”
Missing
Referent omitted
“Put the yellow mug and the blue cup away.” (destination omitted)
TABLE I: Factorized taxonomy of human objective specification. All example variants describe the same physical task; highlighted spans mark the varied element.
Content ↓
Reference ↓
ητ
γτ
Composition →
Enumerated ( δˉ=1.00 , N=120 )
Grouped ( δˉ=0.47±0.08 , N=54 )
Collective ( δˉ=0.41±0.04 , N=50 )
Success (%)
ΔSuccessinfo
Success (%)
ΔSuccessinfo
Success (%)
ΔSuccessinfo
Action
Direct
1.00
1.00
38.0 ± 5.4
0.0 (ref.)
15.9 ± 6.0
7.6 ± 3.1
13.1 ± 5.8
10.4 ± 3.5
Grounding-dep.
1.00
0.33
35.2 ± 5.3
2.8 ± 1.8
13.5 ± 5.6
10.0 ± 3.3
12.4 ± 5.7
11.1 ± 3.6
Missing
0.33
0.33
26.9 ± 4.9
11.1 ± 2.3
7.1 ± 4.2
16.4 ± 3.9
6.5 ± 4.2
17.0 ± 4.1
Outcome
Direct
0.67
0.67
33.5 ± 5.2
4.5 ± 1.9
11.8 ± 5.3
11.7 ± 3.5
9.7 ± 5.1
13.8 ± 3.8
TABLE II: π0.5 performance across the full factorial space of taxonomy-controlled specifications. Less explicit specifications (lighter blue, less decomposed columns) yield lower success and larger information-gap penalties (darker red); bold marks one-factor ablations from the fully specified Action–Direct–Enumerated reference. Rows vary content and reference; columns vary composition (enumerated, grouped, and collective forms on N=120 , 54 , and 50 tasks). Mean ± 95% CI over task-level means after averaging rollouts and surface realizations. Results for π0 -FAST and Cosmos3-Edge, which show similar trends, are in Appendix -F .
Fig. 2: Humans increasingly compress execution structure as tasks grow. Instruction length increases with the number of events (left), but the executable structure made explicit decreases (right): coverage declines moderately, while groundedness and decomposition fall sharply.
Fig. 3: Behavioral cost of unresolved task structure. Left: For all three policies, grounding dependence is cheap, while missing action/object/destination information and undecomposed events cost far more. Right: With task complexity, coverage and decomposition become more strongly associated with the success gap than groundedness.
Fig. 4: Residual execution gap. On the same fully specified Action–Direct–Enumerated tasks, issuing atomic events sequentially improves success and partial progress over a single upfront prompt for all three policies, and resetting the arm to its home configuration between events adds a further gain.
Fig. 5: Information and execution gaps in human–VLA communication. Left: The policy succeeds when object identity or action is implicit, but fails when task semantics must be inferred. Right: Multi-step execution can fail through incorrect object selection; returning to a home pose between subtasks enables completion in the illustrated rollout. Solid blue: targets; dashed blue: destinations; red: incorrect objects; dashed gray: home pose.
Task structure
Operational criterion
Information retained by compact form
RoboLab examples
Atomic
Kτ=1
One grounded (A,O,D) event
BananaInBowlTask ; BowlInBinTask
Compound, enumerative
Kτ>1 and critical event-specific arguments or ordering must still be enumerated
TABLE III: Operational definition of task structure used to analyze human objective specification. Event count measures physical manipulation complexity, whereas compression affordance characterizes the structure available for expressing those events compactly.
Task structure
Tasks
Fraction (%)
Atomic
64
53.3%
Compound, enumerative
3
2.5%
Compound, rule-compressible
42
35.0%
Compound, intent-compressible
11
9.2%
Total
120
100.0%
TABLE IV: Structural distribution of RoboLab-120 under our object-centric event and compression-affordance definitions.
Factor
Agreement (%)
Coverage (%)
Task content
91.3
98.0
Reference form
96.0
99.3
Composition
93.3
98.7
Complete specification
–
96.7
TABLE V: Coverage of naturally elicited specifications by the proposed taxonomy and agreement between independent annotators.
Fig. 6: From human specifications to measurable information gaps. The same physical objective can be expressed through substantially different human specifications. Our taxonomy classifies each instruction by its task content, reference form, and composition, and maps these choices to the executable information made available to the policy: which actions, objects, and destinations are directly specified, require grounding, or are omitted. This yields quantitative measures of semantic coverage, groundedness, and decomposition ratio. Rather than studying instructions only as lexical variants, we can therefore relate VLA performance to the information they carry and identify which missing structure the policy can recover and which it cannot.
Task structure
Source ID. (%)
Correctness (%)
Atomic
50.8%
98.5%
Compound, enumerative
51.9%
97.3%
Compound, rule-compressible
53.7%
95.4%
Compound, intent-compressible
57.2%
90.6%
Overall
52.8%
96.0%
TABLE VI: Human-source identification and semantic correctness of taxonomy-matched generated specifications. Chance source identification is 50% .
Task complexity
Mean sentences
Mean words
Coverage ητ
Groundedness γτ
Decomp ratio δτ
Kτ=1
1.08±0.04
9.1±0.6
0.95±0.03
0.89±0.04
1.00
Kτ=2
1.27±0.06
12.7±0.8
0.86±0.04
0.58±0.06
0.66±0.06
Kτ≥3
1.56±0.08
15.8±1.0
0.74±0.05
0.27±0.06
0.21±0.05
TABLE X: Natural specification length and explicit task structure as the number of required atomic events increases. Values are mean ± expected 95% confidence-interval half-width over human specifications.
Reasoning
Example
Unresolved structure
Required recovery
Visual grounding
“Put the banana in the bowl on the left.”
Grounded Ok or Dk
Resolve the referenced entity from the observed scene.
Action inference
“The banana should be in the bowl.”
Action Ak
Infer the required manipulation from the stated object and desired physical state.
Task decomposition
“Put all the fruit in the bowl.”
Event structure Eτ
Expand the compact objective into its constituent grounded manipulation events.
Intent inference
“I need room on this table.”
Physical objective
Infer the intended physical goal before recovering the corresponding grounded events.
TABLE XII: Reasoning operations required to recover executable structure left unresolved by a specification. These operations may co-occur in natural specifications; the controlled analysis in Table XIII isolates each requirement as closely as possible.
Instruction following is central to language-conditioned robot policies: language should determine what to do when the same scene permits multiple valid actions. Yet successful execution alone cannot establish whether a policy follows the instruction or infers the task from the scene. We study this ambiguity through scene-preserving instruction interventions, using valid target substitutions, arbitrary nouns, and unrelated sentences while holding the scene fixed. We evaluate vision-language-action (VLA) policies and world-action models (WAMs) in simulation and in real-world experiments. Our analysis addresses three questions: (a) Does task success imply instruction following? When instructions request a different visible object, all evaluated policies predominantly approach and pick up the incorrect original target associated with the scene. (b) Is this failure caused by language insensitivity? Instruction perturbations affect task performance. A layerwise action lens shows intermediate action predictions respond to these perturbations. Linear probes accurately recover instructed targets, indicating modified instructions are encoded despite rarely determining target selection. (c) Why does encoded language fail to control action? Attention analysis indicates weak instruction-token contributions to action generation. Target-token attention can remain focused on the original object, revealing a mismatch between target encoding and visual grounding. UMAP and shared non-negative matrix factorization show target information remains accessible within representations increasingly organized by scene identity. Our findings expose a grounding gap concealed by nominal success and provide a diagnostic framework. They further establish a concrete criterion for progress: policies should reliably follow valid changes in user intent, even when they conflict with scene-favored behavior.
As robots become increasingly integrated into everyday environments, intuitive communication paradigms such as natural language and end-user programming have become indispensable for specifying autonomous robot behavior. However, these mechanisms are ineffective at fully capturing user intent: natural language is imprecise and ambiguous, whereas end-user programming can be overly specific. As a result, understanding what users truly mean when they interact with robots remains a central challenge for human-AI communication systems. To address this issue, we propose the Distill approach for human-robot communication interfaces. Given a task specification provided by the user, Distill (1) removes unnecessary steps; (2) generalizes the meaning behind individual steps; and (3) relaxes ordering constraints between steps. We implemented Distill on a web interface and, through a crowdsourcing study, demonstrated its ability to elicit and refine user intent from initial task specifications.
Ting Li, David Porfirio
Computer Science George Mason University Fairfax, Virginia, USA
Vision-language-action (VLA) policies can execute many tasks from standard initial states, yet a new instruction may fail after another task has altered the robot's physical state. We study instruction switching, where a new task is issued during or after the execution of a different one. We observe that a target task that is reliably completed from its standard initial states can become inaccessible from states produced by a preceding task. We call such states task islands. We propose Causeway, a training-free inference-time intervention. Given the current state and a re-entry pose for the target task, Causeway back-propagates through the frozen decoding computation and applies a state-directed write within the action-stream representation. The VLA decodes the return motion itself, without parameter updates, a new action head, or external action generation. Across 71 cross-object pairs, three switch timings, and three VLA architectures on LIBERO-Goal, Causeway raises bare-switch success from 3-26% to 47-65% and increases the rate of reaching the handoff neighborhood by 42-72 percentage points across models. Additional experiments on LIBERO-Object and a real xArm platform show that the recovery extends beyond the main LIBERO-Goal setting, both in simulation and on a robot.
Qingzi Wang, Kaixi Feng, Guangyao Shi +3
University of Maryland, College Park, MD, USA. · University of Southern California, Los Angeles, CA, USA.