Organizations: Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China, Shenzhen, China · Department of Computer Science, Hofstra University, Hempstead, USA
Vision-Language Navigation (VLN) for Unmanned Aerial Vehicles (UAVs) demands complex visual interpretation and continuous control in dynamic 3D environments. Existing hierarchical approaches rely on dense oracle guidance or auxiliary object detectors, creating semantic gaps and limiting genuine autonomy. We propose AerialVLA, a minimalist end-to-end Vision-Language-Action framework mapping raw visual observations and fuzzy linguistic instructions directly to continuous physical control signals. First, we introduce a streamlined dual-view perception strategy that reduces visual redundancy while preserving essential cues for forward navigation and precise grounding, which additionally facilitates future simulation-to-reality transfer. To reclaim genuine autonomy, we deploy a fuzzy directional prompting mechanism derived solely from onboard sensors, completely eliminating the dependency on dense oracle guidance. Ultimately, we formulate a unified control space that integrates continuous 3-Degree-of-Freedom (3-DoF) kinematic commands with an intrinsic landing signal, freeing the agent from external object detectors for precision landing. Extensive experiments on the TravelUAV benchmark demonstrate that AerialVLA achieves state-of-the-art performance in seen environments. Furthermore, it exhibits superior generalization in unseen scenarios by achieving nearly three times the success rate of leading baselines, validating that a minimalist, autonomy-centric paradigm captures more robust visual-motor representations than complex modular systems.
Figures & tables
Figure 1: Comparison of UAV navigation paradigms. While existing modular methods (red) rely on oracle guidance and external detectors, AerialVLA (green) achieves agile autonomous navigation and precise landing via a unified end-to-end policy driven by fuzzy onboard hints and intrinsic stopping.
Figure 2: The architecture of AerialVLA. The framework processes multimodal inputs to generate continuous control signals end-to-end. (a) Language Input constructs prompts with fuzzy directional hints derived from the IMU, eliminating oracle reliance. (b) Visual Input fuses front and down views via a vertical mosaic. (c) The AerialVLA Model utilizes a Llama-2 backbone with LoRA to autoregressively predict numerical tokens. (d) Output and Action Stage decodes tokens into spatial offsets for velocity control, or triggers the dual-condition landing.
Angle Range ∣θ∣
Fuzzy Hint ( θ>0 / θ<0 )
0∘≤∣θ∣≤15∘
“straight ahead”
15∘<∣θ∣≤60∘
“forward-right” / “forward-left”
60∘<∣θ∣≤120∘
“to your right” / “to your left”
120∘<∣θ∣≤180∘
“to your right rear” / “to your left rear”
Table 1: Fuzzy directional hint mapping from onboard IMU-derived relative angle.
Figure 3: AerialVLA prompt formulation. The structured prompt comprises four components: (i) an <image> token for visual input, (ii) a fuzzy directional hint (red), (iii) a detailed target description, and (iv) the corresponding numerical control actions (blue).
Method
Full
Easy
Hard
NE ↓
SR ↑
OSR ↑
SPL ↑
NE ↓
SR ↑
OSR ↑
SPL ↑
NE ↓
SR ↑
OSR ↑
SPL ↑
Human
14.15
94.51
94.51
77.84
11.68
95.44
95.44
76.19
17.16
93.37
93.37
79.85
Random Action
222.20
0.14
0.21
0.07
142.07
0.26
0.39
0.13
320.12
0.00
0.00
0.00
Fixed Action
188.61
2.27
8.16
1.40
121.36
3.48
11.48
2.14
270.69
0.79
4.09
0.49
CMA [ 1 ]
135.73
8.37
18.72
7.90
84.89
11.48
24.52
10.68
197.77
4.57
11.65
4.51
TravelUAV-DA [ 34 ]
98.66
17.45
48.87
15.76
66.40
20.26
51.23
18.10
138.04
14.02
45.98
12.90
Table 2: Comparison on the Test Seen Set. SR, OSR, and SPL are reported in percentage (%). Bold and underline denote the best and second-best model results.
Method
Full
Easy
Hard
NE ↓
SR ↑
OSR ↑
SPL ↑
NE ↓
SR ↑
OSR ↑
SPL ↑
NE ↓
SR ↑
OSR ↑
SPL ↑
Random Action
260.14
0.16
0.16
0.16
174.10
0.48
0.48
0.48
302.96
0.00
0.00
0.00
Fixed Action
212.84
3.66
9.54
2.16
151.66
6.70
13.88
3.72
243.29
2.14
7.38
1.38
CMA [ 1 ]
155.79
9.06
16.06
8.68
102.92
14.83
22.49
13.90
182.09
6.19
12.86
6.08
TravelUAV [ 34 ]
118.11
22.42
46.90
20.51
86.12
24.40
49.28
22.03
134.03
21.43
45.71
19.75
NavFoM [ 39 ]
108.04
29.83
47.99
27.20
70.51
32.54
50.72
29.54
133.01
28.03
46.18
25.64
Table 3: Comparison on the Test Unseen Object Set. SR, OSR, and SPL are reported in percentage (%). Bold and underline denote the best and second-best results, respectively.
Method
Full
Easy
Hard
NE ↓
SR ↑
OSR ↑
SPL ↑
NE ↓
SR ↑
OSR ↑
SPL ↑
NE ↓
SR ↑
OSR ↑
SPL ↑
Random Action
202.98
0.00
0.00
0.00
158.46
0.00
0.00
0.00
265.88
0.00
0.00
0.00
Fixed Action
180.47
0.52
2.61
0.39
132.89
0.89
4.28
0.67
247.72
0.00
0.25
0.00
CMA [ 1 ]
141.68
2.30
10.02
2.16
102.29
3.57
14.26
3.33
197.35
0.50
4.03
0.50
TravelUAV [ 34 ]
138.80
4.18
20.77
3.84
102.94
4.63
22.82
4.24
189.46
3.53
17.88
3.28
NavFoM [ 39 ]
125.10
6.30
18.95
5.68
102.41
6.77
20.07
6.04
170.58
5.36
15.71
4.97
Table 4: Comparison on the Test Unseen Map Set. SR, OSR, and SPL are reported in percentage (%). Bold and underline denote the best and second-best results, respectively.
Figure 4: Qualitative visualization of our proposed AerialVLA. We display the vertical mosaic inputs (Front/Down) at key timesteps. The agent demonstrates precision maneuvering in clutter (Top) and active error correction against distractors (Bottom), validating the robustness of the end-to-end policy.
Training Data / Method
Seen
Unseen Object
Unseen Map
SR ↑
SPL ↑
SR ↑
SPL ↑
SR ↑
SPL ↑
Baseline LongFly [ 12 ]
36.39
31.07
43.87
38.39
11.27
9.32
Ours - custom tokens
39.84
31.73
38.79
32.77
26.51
19.98
Ours - 5-view input
41.54
32.69
51.51
42.33
21.71
13.46
Ours w/o filtering
40.90
32.59
51.99
42.67
32.15
22.67
Ours
47.96
38.54
56.60
46.61
37.58
28.22
Table 5: Ablation on data curation, perception views, and action tokenization. We compare raw noisy data, 5-view input, and custom action token variants against our default formulation. Metrics are in percentage (%).
Vision-language navigation (VLN) for UAVs demands grounding free-form instructions into 6-DoF flight under partial observability. While Vision-Language-Action (VLA) models excel at semantic reasoning, they suffer from brittleness due to geometric inconsistency and dynamics mismatch. To address this, we propose ImagineUAV, an imagination-driven framework leveraging cascaded world-action modeling. Instead of direct regression, ImagineUAV employs a latent video diffusion model to generate instruction-conditioned future observations, explicitly imagining environmental evolution, from which 6-DoF motions are inferred via an action extractor. A kinodynamic planner then refines these estimates into collision-free trajectories. Additionally, a step-distilled inference pipeline ensures real-time execution. With only 1.3B parameters, ImagineUAV outperforms prior VLN and VLA baselines on benchmarks and real-world flights, validating the practicality of imagination-driven aerial navigation.
Xuchen Liu, Jiawei Huang, Shihao Xia +3
Pengcheng Laboratory, Shenzhen, Guangdong, China · School of Computer Science and Cyber Engineering, Guangzhou University, Guangzhou, Guangdong, China · Southern University of Science and Technology, Shenzhen, Guangdong, China
Language-guided UAV agents must execute long-horizon semantic instructions while producing smooth, physically feasible continuous flight commands, yet existing Vision-Language Navigation (VLN) benchmarks typically use discrete or coarse actions and existing UAV Vision-Language-Action (VLA) tasks focus on short, atomic maneuvers. To address this gap in UAV task settings, we introduce FLIGHT, a Fine-grained Long-horizon Instruction-Guided benchmark for Hybrid UAV navigation and reasoning Tasks, which combines multi-stage instructions with dense 6-DoF trajectory annotations across two dataset splits: Fine-grained VLN and Long-horizon Flow. To endow the UAV agent with the capability of real-time in-flight reasoning over task execution status and mission planning, while simultaneously accommodating high-frequency, real-time precise control, we further propose FLIGHT VLA, an asynchronous architecture that decouples a low-frequency Streaming Pilot Vision-Language Model (VLM) for task-state reasoning from a high-frequency diffusion action model for continuous control, supervised by explicit Pilot Reasoning texts that summarize the current flight state and anticipate the next subgoal. In closed-loop evaluation, FLIGHT VLA consistently surpasses representative VLN and VLA baselines on our FLIGHT benchmarks, achieving stronger multi-stage completion, subgoal adherence, and terminal control. Its trained Streaming Pilot Reasoning VLM further improves UAV video reasoning, validating the effectiveness of our design.
Xiangyi Zheng, Xiangyu Wang, Qinan Liao +6
Colab, Beihang University · National University of Singapore · Meituan
Vision-Language Navigation (VLN) enables UAV autonomous navigation in unknown environments by mapping language instructions to real-time visual inputs. Compared with GPS-dependent or pre-programmed navigation, VLN supports intuitive human-machine interaction and stronger environmental adaptability, requiring tight integration of high-level semantic reasoning and low-latency flight control.Existing methods suffer from structural misalignment between global multimodal understanding and sequential action generation, causing jittery trajectories and severe decision latency for long-horizon aerial navigation. To solve this issue, we propose FSD-VLN, a fast-slow dual-system architecture disentangling semantic reasoning and low-latency flight command generation.The framework has two asynchronous branches: a slow stream extracting stable semantic priors from pre-trained vision-language models, and a Diffusion Transformer (DiT) fast stream modeling cross-temporal action distributions to produce consistent flight outputs. We further introduce a time-aware adaptive optimizer to stabilize long-sequence training and reduce gradient oscillation.Large-scale low-altitude simulation experiments show FSD-VLN achieves up to 2X higher navigation success rates on unseen scenes than SOTA methods, while cutting single-action inference delay and total task runtime by over 50%. Our work validates the benefit of decoupled semantic-control modeling and provides a practical paradigm for long-horizon aerial VLN.
Xueke Zhu, Qingyan Meng, Liutao Yu +4
Pengcheng Laboratory · Shenzhen Institutes of Advanced Technology · Peking University