Organizations: Department of Automation, Tsinghua University, Beijing, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Institute for Interdisciplinary Information Sciences, Tsinghua University, Beijing, China · Computer Science and Engineering Department, Chinese University of Hong Kong, Hong Kong, China
Enabling robots to explore and act in unfamiliar environments under ambiguous human instructions by interactively identifying task-relevant objects (e.g., cups or beverages for "I'm thirsty") remains challenging for existing vision-language model (VLM)-based methods, due to inefficient reasoning and limited environmental interaction. These factors hinder real-time task planning and execution. To address this, we propose Affordance-Aware Interactive Decision-Making and Execution for Ambiguous Instructions (AIDE), a human-inspired dual-stream framework that integrates interactive exploration with vision-language reasoning: mirroring the complementary deliberative and intuitive processes of human cognition, Multi-Stage Inference (MSI) serves as the decision-making stream and Accelerated Decision-Making (ADM) as the execution stream, enabling zero-shot affordance analysis and interpretation of ambiguous instructions. Experiments in simulation and real-world settings show that AIDE achieves over 80% task planning success rate and more than 95% closed-loop continuous execution accuracy at 10 Hz, surpassing existing VLM-based methods in diverse open scenarios.
Figures & tables
Fig. 1: Method objective. Given ambiguous instruction and unfamiliar scene as the input task, the objective of the AIDE is to find the required tool and locate its operational and functional regions within the scene, enabling accurate tool manipulation and task completion.
Fig. 2: Framework of AIDE. Given the input task, the Multi-Stage Inference (MSI) stream uses Multimodal CoT Module (MM-CoT) and Exploration Policy to generate keyframe-based task planning result, which is projected into the Instruction-Tool Relationship Space via affordance scoring to enhance cross-modal understanding of the input instruction and robustness to GPT-5 hallucinations. Building on this projection, the Accelerated Decision-Making (ADM) stream employs Efficient Retrieval Scheme (ERS) and Exploration Policy for real-time, closed-loop task execution over continuous frames.
Fig. 3: Affordance vector distributions for instructions and tools after t-SNE projection into the three-dimensional space. The image shows that taking cutting/cleaning/drinking/heating-related tasks as representative examples, the affordance vector distributions for instructions and tools largely overlap.
Method
Qwen3-Next-80B
Llama3-70B
GPT-5
AIDE
6.43
6.67
7.45
IntroPlan
3.22
4.12
5.99
MOKA
1.47
2.67
3.27
TABLE I: Instruction Naturalness Analysis. Average naturalness scores (1-10) of instructions from different methods, as evaluated by different LLMs.
Method
Functionality
G-Dataset
R-Dataset
CL
AA
RT
TSR
OSR
FSR
WSR
ASR
FPS
TSR
OSR
FSR
WSR
ASR
FPS
Molmo
10.5
6.5
1.0
0.5
0.0
0.05
15.5
9.5
4.0
1.5
0.0
0.05
Qwen3-VL
42.0
23.5
13.5
12.0
8.0
0.04
70.0
31.0
34.5
21.0
5.0
0.04
GPT-5
50.5
34.5
37.5
29.5
12.0
0.03
78.0
37.0
45.5
33.5
10.0
0.04
Magma
✓
1.5
0.0
0.5
0.0
0.0
1.03
1.0
0.5
0.0
0.0
0.0
0.94
Robopoint
✓
64.0
38.0
34.0
21.5
4.0
0.14
77.5
41.5
38.5
36.5
0.0
0.19
TABLE II: Zero-shot performance comparison on G-Dataset and R-Dataset.
Method
G-Dataset
R-Dataset
TSR
OSR
FSR
WSR
ASR
FPS
TSR
OSR
FSR
WSR
ASR
FPS
MM-CoT
73.5
72.0
67.5
63.5
-
0.03
82.0
77.5
76.0
73.0
-
0.03
ERS
79.5
66.5
71.5
61.0
-
9.50
83.0
72.0
76.0
69.0
-
9.80
AIDE (w/o ADM)
90.5
89.0
84.5
80.5
68.0
0.03
96.0
91.5
90.0
87.0
70.0
0.03
AIDE (w/o MSI)
92.5
79.5
84.5
74.0
52.0
9.70
93.0
82.0
86.0
79.0
50.0
9.50
AIDE (Full)
94.5
88.0
89.0
83.0
64.0
9.70
95.0
92.0
91.0
88.0
65.0
9.50
TABLE III: Ablation study on different components of AIDE.
Fig. 4: Examples of real-world experiment results. Top left and bottom middle: Identified handle (blue dot) and body (red dot) regions of the target tool. Bottom left: Visible exploration region (blue box). Top middle: Invisible exploration region (green box). Top right and bottom right: tool region (red box).
No.
Instruction (abbr.)
Valid Frames
ESR (%)
1
Thirsty
937
96.1
2
Clean dust
656
96.8
3
Crack walnuts
1180
97.9
4
Drink cold
850
97.2
5
Close up boxes
924
97.4
6
Support waist
333
99.7
TABLE IV: Zero-shot performance on Instructions in real-world experiments.
Stream
G-Dataset
R-Dataset
EDR
ERR
EDR
ERR
ADM
63.5
-
57.1
-
MSI
-
41.0
-
42.3
TABLE V: Error Detection and Recovery Performance of AIDE.