Vision-language models (VLMs) have become a powerful foundation for robotic scene understanding, task planning, and language-conditioned control. A natural question follows: can these models also decode what a human operator intends while teleoperating a robot, without any explicit instruction? We formulate this as hierarchical intent prediction, jointly inferring the ongoing atomic action and the broader mission it serves from a short visual teleoperation history. To study this, we introduce VI AMO (Vision Intention Action and Mission Ongoing), a dataset of 6,600 episodes from eleven robot sources with 13,653 temporally aligned action annotations and 9,702 mission annotations. Large proprietary models struggle with this problem: on held-out episodes, GPT-4o reaches Sentence-BERT similarities of only 0.563 and 0.470 for action and mission, and GPT-5.6 Luna achieves only 0.581 and 0.554. Fine-tuning a VLM on VI AMO yields our Vision-Intention Model (VIM), which raises these scores to 0.825 and 0.821 and still reaches 0.726 and 0.717 on the 35.4% of test actions with both labels absent from training. Finally, as a downstream application, VIM's predicted mission enables a vision-language-action (VLA) policy to take over from the operator mid-task in a real-world environment unseen during VIM fine-tuning, reducing human teleoperation time by 34.6% relative to GPT-4o and 39.8% relative to GPT-5.6 Luna, while cutting intent-prediction latency by roughly 70% on average.
Figures & tables
Fig. 2: From robot demonstrations to hierarchical intent. (a) Annotation pipeline. A VLM decomposes each episode instruction into ordered atomic actions and groups consecutive actions into the mission they serve; coarse-to-fine video segmentation localizes every action in time, with gripper signals guiding fine boundaries where available, followed by human review. (b) Overview of our proposed VI AMO dataset. VI AMO pairs each atomic action with its mission across 6,600 episodes from eleven robot sources (13,653 actions, 9,702 missions), providing hierarchical supervision for intent prediction. (c) VIM is fine-tuned on VI AMO to generate the ongoing action and the ongoing mission from a short visual history and the completed-action context.
Source
Train
Val
Test
Actions
Missions
Berkeley Fanuc [ 29 ]
23
2
2
48
32
Bridge V2 [ 11 ]
2,574
259
258
4,950
3,445
CMU Play Fusion [ 30 ]
186
19
19
725
448
DROID [ 12 ]
1,702
169
170
6,372
4,452
FastUMI-100K [ 31 ]
22
2
2
100
52
Fractal (RT-1) [ 32 ]
483
48
48
656
579
TABLE I: VI AMO source composition with annotation summary.
Ongoing action
Ongoing mission
Dataset
Model
F1 ↑
SBERT ↑
SigLIP ↑
Top-10 ↑
F1 ↑
SBERT ↑
SigLIP ↑
Top-10 ↑
VI AMO test
GPT-5.6 Luna
0.484
0.581
0.741
39.14
0.399
0.554
0.716
39.33
VI AMO test
GPT-4o
0.463
0.563
0.733
35.32
0.330
0.470
0.685
29.91
VI AMO test
Qwen Base (27B)
0.493
0.581
0.744
36.20
0.424
0.564
0.722
37.27
VI AMO test
Qwen Base (9B)
0.464
0.529
0.718
31.43
0.368
0.496
0.684
28.32
VI AMO test
RynnBrain 1.1-9B
0.365
0.505
0.721
34.80
0.381
0.499
0.712
35.49
TABLE II: OA and OM prediction on the VI AMO test set.
Fig. 3: OA and OM prediction across action progress. We evaluate ongoing action (OA) and ongoing mission (OM) predictions on the VI AMO test set at nine observation cutoffs spanning 0–95% of the current atomic action. Panels (a,b) report cosine similarity between predicted and ground-truth texts using SBERT embeddings, while panels (c,d) use SigLIP text embeddings. Higher values indicate closer semantic agreement with the ground-truth intent. Curves connect the mean score at each observation cutoff, and all seven models use the same test examples, observation cutoffs, prompts, and execution context.
Fig. 4: Seen versus unseen labels on the VI AMO test set. (a) Both OA and OM labels are seen during training; (b) both are unseen. Each polygon represents one model; the blue outline denotes VIM. Axes run clockwise from the top: I: OA SBERT; II: OM SBERT; III: OA SigLIP; IV: OM SigLIP; V: OA Top-10; VI: OM Top-10.
Ongoing action
Ongoing mission
Method
F1 ↑
SBERT ↑
SigLIP ↑
Top-10 ↑
F1 ↑
SBERT ↑
SigLIP ↑
Top-10 ↑
Front-only
0.753
0.798
0.861
70.13
0.730
0.798
0.855
73.75
No context
0.704
0.767
0.840
66.46
0.719
0.788
0.851
73.63
Single @ 0%
0.750
0.793
0.858
70.39
0.724
0.792
0.850
72.87
VIM (Ours)
0.784
0.825
0.879
76.59
0.756
0.821
0.872
78.77
TABLE III: Ablations on the VI AMO test set.
Fig. 5: Real-world task quantitative results. Bars show mean ± SD for (a) teleoperation time and (b) VLM latency.
Fig. 6: VIM-guided control handoffs across real-world tasks. The top row shows, from left to right, the visualization of our shared autonomy setup, egg placement, and trash sweeping task. The bottom row presents front-view keyframes of three successive missions in a multi-mission rollout: placing the cherry, apple, and banana on the plate. Mode bars indicate intervals of human teleoperation and VLA control, while mission bars identify the active mission. These examples illustrate the handoff process: VIM infers the ongoing mission from partial teleoperation, the operator confirms it, and the VLA continues execution.
Vision-Language-Action (VLA) models can turn multimodal context into robot actions, but their action decoders are still trained largely by behavior cloning. This supervises which motor command was demonstrated while leaving implicit the local objective served by the behavior under the instruction. Future-based supervision enriches action learning with frames, latent observations, trajectories, or motion representations, but these signals capture particular realizations of what may happen rather than the shared semantic objective of the forthcoming behavior. We propose Intention Distillation (INDI), which distills behavior-level intent into the action decoder. During training, a frozen teacher VLM interprets a demonstrated segment from the current observation, instruction, coarse action summary, and corresponding execution video. From its standard inputs, the deployed VLA recovers the resulting multimodal intent representation at an intermediate decoder layer and uses it to organize action prediction together with representations of how the behavior unfolds and what it achieves. On SimplerEnv-Bridge, INDI improves GR00T-N1.7 from 64.3% to 84.7%, and on RoboCasa Kitchen it improves the controlled GR00T-N1.7 baseline from 64.1% to 70.3%, with consistent gains on π0.5 across both benchmarks. In real-world tasks, INDI improves average success from 62.0% to 68.7%, with gains of up to 12.0 pp on longer-horizon tasks. Further analyses show that the recovered latent is used by the decoder, captures behavior objective and execution progress, and organizes downstream predictions in an objective-dependent manner. These results show that action decoders benefit from explicitly modeling the semantic objective of the behavior they generate.
Generalist robot manipulation can benefit from pretrained vision-language models (VLMs) for semantic reasoning and video generation models (VGMs) for world modeling. However, existing unified models that directly fuse them through shared attention at multiple layers can make it harder to draw on their specialized capabilities. We introduce PINT, a unified model that coordinates semantic reasoning, world modeling, and action generation through a shared chunk-level Predictive Intent while preserving expert-specific pathways. PINT inserts learnable Intent Tokens into the VLM sequence to form this representation, conditioning both the VGM and an Action Expert (AE). It replaces only the VGM's prompt conditioning and retains the VGM at inference, where the AE reads its representations through one-way attention. Rather than prescribing what Predictive Intent should encode, PINT learns it jointly from the native prediction objectives of the VLM, VGM, and AE, which provide complementary views of the same upcoming task execution. Training uses two stages: Stage 1 freezes the VGM so that future video prediction shapes Predictive Intent with task-relevant future dynamics, while Stage 2 fixes the learned Intent and adapts the VGM and AE. Without embodied pretraining, PINT achieves strong in-distribution (ID) performance and substantially improves out-of-distribution (OOD) generalization, reaching 15.5% on RoboTwin 2.0-Hard compared with 1.8% for Fast-WAM and 3.2% for StarVLA, and 79.5% on LIBERO-Plus compared with 51.5% and 75.0%, respectively. Across five real-world bimanual tasks, PINT reaches 68% ID and 24% OOD success, compared with 35%/9% for pretrained Motus and 67%/35% for pretrained π0.5.
Xinyu Guo, Chao Jing, Li Yi +2
Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Zhongguancun Academy +2
The prevailing recipe for Vision-Language-Action (VLA) models couples a pretrained VLM with a separately trained flow-matching action expert. This makes the VLM a context encoder rather than a decision-maker. We introduce G0.5, a pretrained autoregressive VLA in which a single transformer decoder emits reasoning and action tokens under a single objective. Three components make this tractable at foundation-model scale: a learnable cross-embodiment action tokenizer that maps heterogeneous robot actions into a shared vocabulary; a native chain-of-thought stream interleaving task decomposition, object grounding, and action hints with action tokens; and a visual memory module that injects multi-second history through the vision encoder. Because reasoning and action share a single set of weights, the pretrained VLM's capabilities carry over to physical behavior: the model follows instructions closely, and prompts directly steer action granularity, task horizon, and out-of-distribution scene handling without further training. Pretrained on a large collection of robot datasets together with VQA samples, G0.5 surpasses state-of-the-art models across 7 independent regimes: real-world fine-tuning on R1lite and R1pro robots (76.7% vs.\ 53.3% for π0.5 and 24.4% for GR00T-N1.7), the 2025 BEHAVIOR Challenge on 50 long-horizon household mobile manipulation tasks using a generalist policy (31.4% vs.\ 26.3% for π0.5 and 26.1% for the challenge winner), DROID post-training followed by zero-shot transfer to an unseen environment and objects (82.5%), a language-following Pick-and-Place benchmark, LIBERO (98.9%), RoboTwin 2.0 (93.3%), and SimplerEnv-Bridge (87.3%).