Fly as Instructed: Refining World-Action Policies for UAV Vision-Language Navigation
Organizations: Harbin Institute of Technology · National University of Singapore · Harbin Institute of Technology, Shenzhen
Abstract
Natural language-guided UAV navigation enables intuitive human-robot interaction and flexible autonomous flight in complex environments. However, reliable navigation requires accurate instruction following, continuous task progress tracking, and safe obstacle avoidance, which remain challenging for existing world-action models that primarily focus on action prediction. To address these limitations, we propose a world-action learning framework that integrates three complementary mechanisms: a language-conditioned action interface with targeted post-training for instruction-responsive motion, recurrent instruction-consumption prediction to track task progress and determine termination, and Gaussian-scene post-training to enhance obstacle avoidance through trajectory imitation and differentiable geometric feedback. Closed-loop evaluations and ablation studies demonstrate improvements in sequential task completion, safe goal reaching, and timely termination. Furthermore, direct one-step denoising reduces inference overhead, and real-world indoor and outdoor flights validate the framework's effectiveness.
Figures & tables
| Method | nDTW | OC | SFR | CFR | ||
|---|---|---|---|---|---|---|
| Ours (Full System) | 436 (83.7) | 401 (77.0) | 0.648 | 416 (79.8) | 399 (76.6) | 452 (86.8) |
| Ours (Action Policy) | 284 (54.5) | 0 (0.0) | 0.503 | 227 (43.6) | 210 (40.3) | 318 (61.0) |
| WorldVLN | 236 (45.3) | 164 (31.5) | 0.448 | 181 (34.7) | 158 (30.3) | 305 (58.5) |
| FastWAM | 221 (42.4) | 0 (0.0) | 0.431 | 168 (32.2) | 149 (28.6) | 287 (55.1) |
| HugeBench | 207 (39.7) | 139 (26.7) | 0.417 | 155 (29.8) | 137 (26.3) | 293 (56.2) |
| Pi0-UAV | 178 (34.2) | 97 (18.6) | 0.386 | 131 (25.1) | 118 (22.6) | 301 (57.8) |
| Configuration | Safety SFR | Progress OC | Progress | General |
|---|---|---|---|---|
| Action Policy | 52 (30.6) | 41 (24.1) | 0 (0.0) | 139 (76.8) |
| + Safety Adapter | 133 (78.2) | 42 (24.7) | 0 (0.0) | 138 (76.2) |
| + Consumption Head | 54 (31.8) | 138 (81.2) | 131 (77.1) | 151 (83.4) |
| + Both (Full System) | 133 (78.2) | 138 (81.2) | 131 (77.1) | 152 (84.0) |
| Method | Instruction SR (%) | Contrastive SR (%) | Paraphrase SR (%) |
|---|---|---|---|
| Pi0-UAV | 33.2 | 20.8 | 9.2 |
| WorldVLN | 42.2 | 18.4 | 5.2 |
| OpenVLA-UAV | 26.4 | 13.6 | 8.2 |
| Ours | 80.6 | 56.4 | 38.2 |
| Configuration | OC | |
|---|---|---|
| Full Instruction | 41 (24.1) | 0 (0.0) |
| Timed Switching | 89 (52.4) | 81 (47.6) |
| Hard Switching | 119 (70.0) | 104 (61.2) |
| w/o GRU Memory | 98 (57.6) | 84 (49.4) |
| Soft Mask (Ours) | 138 (81.2) | 131 (77.1) |
| Configuration | SFR | Collision rate | Task retention |
|---|---|---|---|
| Base | 52 (30.6) | 109 (64.1) | 139 (76.8) |
| w/o Persistent Exploration | 79 (46.5) | 84 (49.4) | 134 (74.0) |
| w/o Outcome Filtering | 68 (40.0) | 88 (51.8) | 124 (68.5) |
| w/o Auxiliary Signals | 101 (59.4) | 60 (35.3) | 141 (77.9) |
| w/o Constrained Updates | 124 (72.9) | 33 (19.4) | 97 (53.6) |
| Full Method | 133 (78.2) | 29 (17.1) | 138 (76.2) |
| Model | Params (M) | Head (ms) | Full (ms) |
|---|---|---|---|
| FastWAM | 1802.59 | 84.13 | 99.67 |
| OpenVLA-UAV | 7541.24 | 183.42 | 210.27 |
| WorldVLN | 9181.06 | 302.51 | 6167.84 |
| Ours | 815.28 | 30.55 | 91.84 |
Aerial VLN
Aerial vision-and-language navigation (VLN) enables UAVs to follow natural-language goals using visual perception and flight control.
See all 67 papers