Organizations: School of Artificial Intelligence, Sun Yat-Sen University, Zhuhai, China. · Southern University of Science and Technology, Shenzhen, China · The Hong Kong University of Science and Technology, Guangzhou, China.
Aerial vision-language navigation (AVLN) enables UAVs to follow natural-language instructions in complex 3D environments. However, existing zero-shot AVLN methods often suffer from unstable single-stream Vision-Language Model decision-making, unreliable long-horizon progress monitoring, and a trade-off between safety and efficiency. We propose OnFly, a fully onboard, real-time framework for zero-shot AVLN. OnFly adopts a shared-perception dual-agent architecture that decouples high-frequency target generation from low-frequency progress monitoring, thereby stabilizing decision-making. It further employs a hybrid keyframe-recent-frame memory to preserve global trajectory context while maintaining KV-cache prefix stability, enabling reliable long-horizon monitoring with termination and recovery signals. In addition, a semantic-geometric verifier refines VLM-predicted targets for instruction consistency and geometric safety using VLM features and depth cues, while a receding-horizon planner generates optimized collision-free trajectories under geometric safety constraints, improving both safety and efficiency. In simulation, OnFly improves task success from 26.4% to 67.8%, compared with the strongest state-of-the-art baseline, while fully onboard real-world flights validate its feasibility for real-time deployment. The code will be released at https://github.com/Robotics-STAR-Lab/OnFly
Figures & tables
Fig. 1: Real-world example of onboard language-guided UAV navigation. The figure illustrates the UAV platform (left), the first-person onboard view (inset), and the executed flight trajectory (right).
Fig. 2: System pipeline of OnFly. OnFly is a fully onboard system for AVLN with dual-agent decision-making and hybrid memory-based progress monitoring. The task manager splits a long instruction into subtasks and outputs the current one. The dual-agent module predicts candidate goals and monitors progress from onboard observations, where the Hybrid Memo ry consists of the initial frame, keyframes, and the latest frame, to preserve global trajectory context and KV-cache prefix stability. Semantic–geometric verification and a safety-aware planner then generate an executable real-time trajectory.
Fig. 3: Hybrid Memory construction. The monitoring memory is built from the initial frame, selected keyframes, and the latest frame. It maintains a de-duplicated keyframe pool, selects representative keyframes for global coverage, and serializes them in a prefix-stable order for KV-cache reuse.
Fig. 4: Simulation environments for benchmarking. (a) Examples of the 10 high-fidelity Unreal Engine scenes, covering diverse indoor and outdoor settings. (b) Task composition of the benchmark, including object navigation, precise navigation, and long-range navigation.
Fig. 5: Simulation visualizations. Each row shows a navigation episode, with the language instruction at the top and representative first-person UAV keyframes along the trajectory.
Fig. 6: Real-world flight experiments with language instructions. Each row corresponds to one task. The middle panel shows the executed 3D trajectory, and the right panel presents representative first-person RGB snapshots, with the corresponding language instruction shown on the left. The tasks include pedestrian tracking along a roughly circular route, long-range corridor navigation with sparse landmarks, precise outdoor navigation across a stepped lawn edge, and multi-level navigation with obstacle avoidance to reach a second-floor platform and stop near the specified person. Additional results are provided in the supplementary video.
Method
SR (%) ↑
OSR (%) ↑
CR (%) ↓
FT (s) ↓
TypeFly [ 6 ]
5.7
17.5
66.5
35.4
PIVOT [ 2 ]
10.3
44.2
64.1
32.9
SPF [ 1 ]
26.4
61.5
42.7
39.2
Ours
67.8
78.1
2.7
27.1
Table I: Zero-shot AVLN Benchmark Results.
Method
SR (%) ↑
OSR (%) ↑
CR (%) ↓
FT (s) ↓
SPF w/ planner
43.2
80.3
4.7
50.2
Ours
67.8
78.1
2.7
27.1
Table II: Obstacle-Avoidance Planner Control Study.
Setting
SR (%) ↑
OSR (%) ↑
CR (%) ↓
FT (s) ↓
Architecture Ablation
Ours
67.8
78.1
2.7
27.1
w/o Dual Agent
48.6
77.9
2.9
44.1
w/o Verification
63.7
72.6
3.2
28.3
w/o Planner
35.8
69.4
37.5
32.2
Memory Ablation
Table III: Ablation Study Results.
Strategy
Decision
Monitoring
Cost (s) ↓
Spd. ↑
Cost (s) ↓
Spd. ↑
Baseline
3.83
1.00 ×
7.47
1.00 ×
Edge-Deploy
0.81
4.73 ×
1.98
3.77 ×
w/ Hybrid Memory
0.81
4.73 ×
1.15
6.50 ×
Table IV: Onboard Latency Breakdown on Jetson Orin NX.