AerialVLA: A Vision-Language-Action Model for UAV Navigation via Minimalist End-to-End Control

Peng Xu; Zhengnan Deng; Jiayan Deng; Zonghua Gu; Shaohua Wan

cs.CVMar 15, 2026

AerialVLA: A Vision-Language-Action Model for UAV Navigation via Minimalist End-to-End Control

Authors: Peng Xu, Zhengnan Deng, Jiayan Deng, Zonghua Gu, Shaohua Wan

Organizations: Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China, Shenzhen, China · Department of Computer Science, Hofstra University, Hempstead, USA

Abstract

Vision-Language Navigation (VLN) for Unmanned Aerial Vehicles (UAVs) demands complex visual interpretation and continuous control in dynamic 3D environments. Existing hierarchical approaches rely on dense oracle guidance or auxiliary object detectors, creating semantic gaps and limiting genuine autonomy. We propose AerialVLA, a minimalist end-to-end Vision-Language-Action framework mapping raw visual observations and fuzzy linguistic instructions directly to continuous physical control signals. First, we introduce a streamlined dual-view perception strategy that reduces visual redundancy while preserving essential cues for forward navigation and precise grounding, which additionally facilitates future simulation-to-reality transfer. To reclaim genuine autonomy, we deploy a fuzzy directional prompting mechanism derived solely from onboard sensors, completely eliminating the dependency on dense oracle guidance. Ultimately, we formulate a unified control space that integrates continuous 3-Degree-of-Freedom (3-DoF) kinematic commands with an intrinsic landing signal, freeing the agent from external object detectors for precision landing. Extensive experiments on the TravelUAV benchmark demonstrate that AerialVLA achieves state-of-the-art performance in seen environments. Furthermore, it exhibits superior generalization in unseen scenarios by achieving nearly three times the success rate of leading baselines, validating that a minimalist, autonomy-centric paradigm captures more robust visual-motor representations than complex modular systems.

Figures & tables

Figure 1: Comparison of UAV navigation paradigms. While existing modular methods (red) rely on oracle guidance and external detectors, AerialVLA (green) achieves agile autonomous navigation and precise landing via a unified end-to-end policy driven by fuzzy onboard hints and intrinsic stopping.

Figure 2: The architecture of AerialVLA. The framework processes multimodal inputs to generate continuous control signals end-to-end. (a) Language Input constructs prompts with fuzzy directional hints derived from the IMU, eliminating oracle reliance. (b) Visual Input fuses front and down views via a vertical mosaic. (c) The AerialVLA Model utilizes a Llama-2 backbone with LoRA to autoregressively predict numerical tokens. (d) Output and Action Stage decodes tokens into spatial offsets for velocity control, or triggers the dual-condition landing.

Angle Range $\|\theta\|$	Fuzzy Hint ( $\theta>0$ / $\theta<0$ )
$0^{\circ}\leq\|\theta\|\leq 15^{\circ}$	“straight ahead”
$15^{\circ}<\|\theta\|\leq 60^{\circ}$	“forward-right” / “forward-left”
$60^{\circ}<\|\theta\|\leq 120^{\circ}$	“to your right” / “to your left”
$120^{\circ}<\|\theta\|\leq 180^{\circ}$	“to your right rear” / “to your left rear”

Table 1: Fuzzy directional hint mapping from onboard IMU-derived relative angle.

Figure 3: AerialVLA prompt formulation. The structured prompt comprises four components: (i) an <image> token for visual input, (ii) a fuzzy directional hint (red), (iii) a detailed target description, and (iv) the corresponding numerical control actions (blue).

Method	Full				Easy				Hard
Method	NE $\downarrow$	SR $\uparrow$	OSR $\uparrow$	SPL $\uparrow$	NE $\downarrow$	SR $\uparrow$	OSR $\uparrow$	SPL $\uparrow$	NE $\downarrow$	SR $\uparrow$	OSR $\uparrow$	SPL $\uparrow$
Human	14.15	94.51	94.51	77.84	11.68	95.44	95.44	76.19	17.16	93.37	93.37	79.85
Random Action	222.20	0.14	0.21	0.07	142.07	0.26	0.39	0.13	320.12	0.00	0.00	0.00
Fixed Action	188.61	2.27	8.16	1.40	121.36	3.48	11.48	2.14	270.69	0.79	4.09	0.49
CMA [ 1 ]	135.73	8.37	18.72	7.90	84.89	11.48	24.52	10.68	197.77	4.57	11.65	4.51
TravelUAV-DA [ 34 ]	98.66	17.45	48.87	15.76	66.40	20.26	51.23	18.10	138.04	14.02	45.98	12.90

Table 2: Comparison on the Test Seen Set. SR, OSR, and SPL are reported in percentage (%). Bold and underline denote the best and second-best model results.

Method	Full				Easy				Hard
Method	NE $\downarrow$	SR $\uparrow$	OSR $\uparrow$	SPL $\uparrow$	NE $\downarrow$	SR $\uparrow$	OSR $\uparrow$	SPL $\uparrow$	NE $\downarrow$	SR $\uparrow$	OSR $\uparrow$	SPL $\uparrow$
Random Action	260.14	0.16	0.16	0.16	174.10	0.48	0.48	0.48	302.96	0.00	0.00	0.00
Fixed Action	212.84	3.66	9.54	2.16	151.66	6.70	13.88	3.72	243.29	2.14	7.38	1.38
CMA [ 1 ]	155.79	9.06	16.06	8.68	102.92	14.83	22.49	13.90	182.09	6.19	12.86	6.08
TravelUAV [ 34 ]	118.11	22.42	46.90	20.51	86.12	24.40	49.28	22.03	134.03	21.43	45.71	19.75
NavFoM [ 39 ]	108.04	29.83	47.99	27.20	70.51	32.54	50.72	29.54	133.01	28.03	46.18	25.64

Table 3: Comparison on the Test Unseen Object Set. SR, OSR, and SPL are reported in percentage (%). Bold and underline denote the best and second-best results, respectively.

Method	Full				Easy				Hard
Method	NE $\downarrow$	SR $\uparrow$	OSR $\uparrow$	SPL $\uparrow$	NE $\downarrow$	SR $\uparrow$	OSR $\uparrow$	SPL $\uparrow$	NE $\downarrow$	SR $\uparrow$	OSR $\uparrow$	SPL $\uparrow$
Random Action	202.98	0.00	0.00	0.00	158.46	0.00	0.00	0.00	265.88	0.00	0.00	0.00
Fixed Action	180.47	0.52	2.61	0.39	132.89	0.89	4.28	0.67	247.72	0.00	0.25	0.00
CMA [ 1 ]	141.68	2.30	10.02	2.16	102.29	3.57	14.26	3.33	197.35	0.50	4.03	0.50
TravelUAV [ 34 ]	138.80	4.18	20.77	3.84	102.94	4.63	22.82	4.24	189.46	3.53	17.88	3.28
NavFoM [ 39 ]	125.10	6.30	18.95	5.68	102.41	6.77	20.07	6.04	170.58	5.36	15.71	4.97

Table 4: Comparison on the Test Unseen Map Set. SR, OSR, and SPL are reported in percentage (%). Bold and underline denote the best and second-best results, respectively.

Figure 4: Qualitative visualization of our proposed AerialVLA. We display the vertical mosaic inputs (Front/Down) at key timesteps. The agent demonstrates precision maneuvering in clutter (Top) and active error correction against distractors (Bottom), validating the robustness of the end-to-end policy.

Training Data / Method	Seen		Unseen Object		Unseen Map
Training Data / Method	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$
Baseline LongFly [ 12 ]	36.39	31.07	43.87	38.39	11.27	9.32
Ours - custom tokens	39.84	31.73	38.79	32.77	26.51	19.98
Ours - 5-view input	41.54	32.69	51.51	42.33	21.71	13.46
Ours w/o filtering	40.90	32.59	51.99	42.67	32.15	22.67
Ours	47.96	38.54	56.60	46.61	37.58	28.22

Table 5: Ablation on data curation, perception views, and action tokenization. We compare raw noisy data, 5-view input, and custom action token variants against our default formulation. Metrics are in percentage (%).

Explore similar work

ImagineUAV: Aerial Vision-Language Navigation via World-Action Modeling and Kinodynamic Planning
May 31, 2026Xuchen Liu, Jiawei Huang, Shihao Xia +3UAV Navigation Vision-Language Navigation
Pengcheng Laboratory, Shenzhen, Guangdong, China · School of Computer Science and Cyber Engineering, Guangzhou University, Guangzhou, Guangdong, China · Southern University of Science and Technology, Shenzhen, Guangdong, China
Think Like a Pilot: Fine-Grained Long-Horizon UAV Navigation
Jun 5, 2026Xiangyi Zheng, Xiangyu Wang, Qinan Liao +6UAV Navigation Vision-Language Navigation
Colab, Beihang University · National University of Singapore · Meituan
FSD-VLN: Fast-Slow Dual-System Modeling for Aerial Long-Horizon Vision-Language Navigation
Jul 9, 2026Xueke Zhu, Qingyan Meng, Liutao Yu +4Vision-Language Models UAV Navigation
Pengcheng Laboratory · Shenzhen Institutes of Advanced Technology · Peking University

May 31, 2026J/K move · Enter open · S save

ImagineUAV: Aerial Vision-Language Navigation via World-Action Modeling and Kinodynamic Planning

Xuchen Liu, Jiawei Huang, Shihao Xia, Bingxi Liu+2

Pengcheng Laboratory, Shenzhen, Guangdong, China · School of Computer Science and Cyber Engineering, Guangzhou University, Guangzhou, Guangdong, China · Southern University of Science and Technology, Shenzhen, Guangdong, China

UAV Navigation Vision-Language Navigation World Action Models Motion Planning Aerial VLN

Open paper AskPDF arXiv

Vision-language navigation (VLN) for UAVs demands grounding free-form instructions into 6-DoF flight under partial observability. While Vision-Language-Action (VLA) models excel at semantic reasoning, they suffer from brittleness due to geometric inconsistency and dynamics mismatch. To address this, we propose ImagineUAV, an imagination-driven framework leveraging cascaded world-action modeling. Instead of direct regression, ImagineUAV employs a latent video diffusion model to generate instruction-conditioned future observations, explicitly imagining environmental evolution, from which 6-DoF motions are inferred via an action extractor. A kinodynamic planner then refines these estimates into collision-free trajectories. Additionally, a step-distilled inference pipeline ensures real-time execution. With only 1.3B parameters, ImagineUAV outperforms prior VLN and VLA baselines on benchmarks and real-world flights, validating the practicality of imagination-driven aerial navigation.

Abstract

Explore similar work

ImagineUAV: Aerial Vision-Language Navigation via World-Action Modeling and Kinodynamic Planning

Think Like a Pilot: Fine-Grained Long-Horizon UAV Navigation

FSD-VLN: Fast-Slow Dual-System Modeling for Aerial Long-Horizon Vision-Language Navigation