I See What You Want: Decoding Hierarchical Operator Intent from Visual Teleoperation History
Organizations: ETH Zurich · Chinese University of Hong Kong
Abstract
Vision-language models (VLMs) have become a powerful foundation for robotic scene understanding, task planning, and language-conditioned control. A natural question follows: can these models also decode what a human operator intends while teleoperating a robot, without any explicit instruction? We formulate this as hierarchical intent prediction, jointly inferring the ongoing atomic action and the broader mission it serves from a short visual teleoperation history. To study this, we introduce VI AMO (Vision Intention Action and Mission Ongoing), a dataset of 6,600 episodes from eleven robot sources with 13,653 temporally aligned action annotations and 9,702 mission annotations. Large proprietary models struggle with this problem: on held-out episodes, GPT-4o reaches Sentence-BERT similarities of only 0.563 and 0.470 for action and mission, and GPT-5.6 Luna achieves only 0.581 and 0.554. Fine-tuning a VLM on VI AMO yields our Vision-Intention Model (VIM), which raises these scores to 0.825 and 0.821 and still reaches 0.726 and 0.717 on the 35.4% of test actions with both labels absent from training. Finally, as a downstream application, VIM's predicted mission enables a vision-language-action (VLA) policy to take over from the operator mid-task in a real-world environment unseen during VIM fine-tuning, reducing human teleoperation time by 34.6% relative to GPT-4o and 39.8% relative to GPT-5.6 Luna, while cutting intent-prediction latency by roughly 70% on average.
Figures & tables
| Source | Train | Val | Test | Actions | Missions |
|---|---|---|---|---|---|
| Berkeley Fanuc [ 29 ] | 23 | 2 | 2 | 48 | 32 |
| Bridge V2 [ 11 ] | 2,574 | 259 | 258 | 4,950 | 3,445 |
| CMU Play Fusion [ 30 ] | 186 | 19 | 19 | 725 | 448 |
| DROID [ 12 ] | 1,702 | 169 | 170 | 6,372 | 4,452 |
| FastUMI-100K [ 31 ] | 22 | 2 | 2 | 100 | 52 |
| Fractal (RT-1) [ 32 ] | 483 | 48 | 48 | 656 | 579 |
| Ongoing action | Ongoing mission | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Dataset | Model | F1 | SBERT | SigLIP | Top-10 | F1 | SBERT | SigLIP | Top-10 |
| VI AMO test | GPT-5.6 Luna | 0.484 | 0.581 | 0.741 | 39.14 | 0.399 | 0.554 | 0.716 | 39.33 |
| VI AMO test | GPT-4o | 0.463 | 0.563 | 0.733 | 35.32 | 0.330 | 0.470 | 0.685 | 29.91 |
| VI AMO test | Qwen Base (27B) | 0.493 | 0.581 | 0.744 | 36.20 | 0.424 | 0.564 | 0.722 | 37.27 |
| VI AMO test | Qwen Base (9B) | 0.464 | 0.529 | 0.718 | 31.43 | 0.368 | 0.496 | 0.684 | 28.32 |
| VI AMO test | RynnBrain 1.1-9B | 0.365 | 0.505 | 0.721 | 34.80 | 0.381 | 0.499 | 0.712 | 35.49 |
| Ongoing action | Ongoing mission | |||||||
|---|---|---|---|---|---|---|---|---|
| Method | F1 | SBERT | SigLIP | Top-10 | F1 | SBERT | SigLIP | Top-10 |
| Front-only | 0.753 | 0.798 | 0.861 | 70.13 | 0.730 | 0.798 | 0.855 | 73.75 |
| No context | 0.704 | 0.767 | 0.840 | 66.46 | 0.719 | 0.788 | 0.851 | 73.63 |
| Single @ 0% | 0.750 | 0.793 | 0.858 | 70.39 | 0.724 | 0.792 | 0.850 | 72.87 |
| VIM (Ours) | 0.784 | 0.825 | 0.879 | 76.59 | 0.756 | 0.821 | 0.872 | 78.77 |
VLMs for Robotics
Vision-language models for robotics connect visual and linguistic understanding to grounded navigation, manipulation, and robot control.
See all 717 papers