UAV-Flow++: A Unified Suite for Benchmarking and Learning Language-Guided UAV Flight
Authors: Donglin Yang, Xiangyu Wang, Yantong Zhong, Wenhao Zheng, Zihan Ding, Yue Liao, Hongsheng Li, Ke Xu, Si Liu
Organizations: Beihang University, Beijing, China 100191 · National University of Singapore, Singapore 119077 · MMLab, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong, China
Language-guided UAV navigation has largely focused on long-horizon route planning, leaving short-range, object-relative flight maneuvers underexplored. To address this gap, we formulate Flying-on-a-Word (Flow), a task requiring a UAV to convert concise natural-language instruction, egocentric visual observation, and current state dynamics into continuous flight actions. Flow encompasses both primitive motion commands and complex object-interactive behaviors, assessing motion intent comprehension and spatial context grounding. To enable policy learning and closed-loop evaluation, we introduce UAV-Flow, a comprehensive dataset comprising 30,692 expert-piloted real-world trajectories across eight distinct motion categories, alongside 10,109 simulation trajectories in the UAV-Flow-Sim environment equipped with task-specific completion criteria. Furthermore, we develop FlowVLA, which undergoes imitation pretraining on mixed real-sim demonstrations and online post-training via Tail-Aware GRPO (TGRPO). By leveraging group reward distributions, TGRPO effectively reinforces high-reward action sequences. Evaluated on 273 held-out UAV-Flow-Sim episodes, pretrained FlowVLA achieves a mean success rate of 75.58%, outperforming the strongest baseline by 6.72 percentage points, which TGRPO post-training further elevates to 93.16% with superior sample efficiency. Physical UAV experiments demonstrate robust execution of object-relative commands in both indoor and outdoor real-world scenes. Together, Flow establishes a novel benchmark and learning framework for language-conditioned flight maneuvers.
Figures & tables
Figure 1: Overview of UAV-Flow++. UAV-Flow++ comprises real-world flight demonstrations, an interactive simulation suite, and a vision-language-action policy for language-conditioned UAV maneuvers. The real-world dataset pairs language instructions with onboard observations, UAV states, and expert actions across primitive motion and object-interactive tasks. UAV-Flow-Sim supports closed-loop policy training and evaluation. FlowVLA predicts position-yaw action chunks from language instructions, initial and current images, and UAV states, using imitation pretraining on mixed real and simulated demonstrations followed by reinforcement learning with trajectory-level rewards.
Benchmark
Action Type
#DoF
#Traj
Collection Setting
Task Granularity
AVDN [ 31 ]
Continuous
3
3K
Sim
Long-horizon dialog navigation
AerialVLN [ 17 ]
Discrete
4
8K
Sim
Long-horizon navigation
CityNav [ 18 ]
Discrete
4
32K
Sim
Goal-directed navigation
OpenUAV [ 20 ]
Continuous
6
12K
Sim
Goal-directed navigation
OpenFly [ 19 ]
Discrete
4
100K
Sim
Long-horizon navigation
IndoorUAV [ 32 ]
Discrete
4
50K
Sim
Long-horizon VLN / short-horizon VLA
Table 1: Comparison of language-guided UAV benchmarks.
Figure 2: Representative Flow tasks. Examples show scene observations, natural-language instructions, and corresponding 3D flight trajectories for hovering, approaching, retreating, passing, orbiting, and landing. Multiple expert trajectories for the same instruction illustrate the diversity of demonstrated flight behaviors.
Figure 3: Overview of real-world UAV-Flow dataset construction. Expert pilots collect onboard videos and flight logs, followed by temporal synchronization and 6-DoF pose registration. Each flight demonstration is assigned a task label and a language instruction, followed by annotation review. The resulting UAV-Flow dataset pairs egocentric videos and flight trajectories with the corresponding instructions.
Category
Action Type
Real
Sim
Primitive Motion
Shift
4.3k (13.9%)
3.0k (29.4%)
Rotate
1.8k (5.9%)
1.0k (10.3%)
Hover
3.2k (10.6%)
0 (0%)
Retreat
1.7k (5.5%)
1.3k (12.6%)
Object Interaction
Surround
1.0k (3.4%)
0.2k (2.3%)
Approach
8.6k (28.1%)
2.6k (25.6%)
Table 2: Trajectory distribution across action types in UAV-Flow and UAV-Flow-Sim. Percentages indicate the proportion of trajectories within each dataset.
Figure 4: Overview of the two-stage training pipeline. Stage one trains a language-conditioned UAV policy through imitation learning on mixed real-world and simulated demonstrations. In stage two, the policy undergoes interactive reinforcement learning via TGRPO, leveraging simulated closed-loop rollouts and tail-aware advantages derived from task completion scores for policy optimization.
Method
Overall
Pass
Rotate
Land
Approach
SR (%)
NDTW
SR (%) / CS
NDTW
SR (%) / CS
NDTW
SR (%) / CS
NDTW
SR (%) / CS
NDTW
TravelUAV
56.78
0.237
42.50 / 0.610
0.173
46.67 / 0.753
0.396
59.26 / 0.806
0.121
47.62 / 0.681
0.116
OpenVLA-UAV
53.85
0.345
30.00 / 0.513
0.239
13.33 / 0.527
0.269
22.22 / 0.493
0.118
47.62 / 0.581
0.243
π0 -UAV
68.86
0.431
57.50 / 0.722
0.384
46.67 / 0.807
0.538
68.52 / 0.911
0.281
54.76 / 0.790
0.288
MemoryVLA-UAV
62.64
0.342
62.50 / 0.720
0.319
6.67 / 0.588
0.177
72.22 / 0.880
0.110
52.38 / 0.740
0.339
π0.5 -UAV
66.67
0.398
47.50 / 0.637
0.369
66.67 / 0.967
0.662
70.37 / 0.930
0.251
54.76 / 0.793
0.317
Table 3: Closed-loop performance comparison on UAV-Flow-Sim. We report overall performance and results on the four RL training tasks: Pass, Rotate, Land, and Approach. FlowVLA represents the imitation-pretrained policy; FlowVLA-GRPO and FlowVLA-TGRPO denote its RL post-training variants. Metrics are aggregated over three evaluation runs, reported as mean ± std for overall performance and as means for individual tasks. Bold text highlights the best Success Rate (SR).
Figure 6: Performance comparison of RL post-training algorithms on UAV-Flow-Sim. (a) Left: mean within-group training reward against policy update steps. Right: overall test-set metrics over time, showing SR as solid lines on the left axis and nDTW as dashed lines on the right axis. (b) SR gains relative to the pretrained baseline in percentage points. The top row reports performance across the four RL training tasks, whereas the bottom row displays the mean gain over the remaining tasks excluded from post-training, with shaded regions denoting the min–max range across these tasks.
RL Algorithm
Overall
Pass
Rotate
Land
Approach
SR (%)
NDTW
SR (%) / CS
NDTW
SR (%) / CS
NDTW
SR (%) / CS
NDTW
SR (%) / CS
NDTW
DAPO
82.91±1.18
0.352±0.031
92.50 / 0.912
0.354
53.33 / 0.700
0.181
91.36 / 0.954
0.245
84.13 / 0.917
0.306
GSPO
89.13±1.29
0.394±0.007
91.67 / 0.907
0.299
40.00 / 0.698
0.527
95.68 / 0.968
0.273
85.71 / 0.965
0.263
GDPO
89.87±2.44
0.406±0.018
93.33 / 0.926
0.321
100.00 / 1.000
0.664
89.51 / 0.876
0.230
88.89 / 0.944
0.300
GRPO
92.06±0.56
0.434±0.005
85.83 / 0.879
0.391
100.00 / 1.000
0.660
95.06 / 0.994
0.301
84.13 / 0.952
0.335
Table 4: Comparison of RL algorithms on UAV-Flow-Sim. All variants start from the same pretrained FlowVLA policy. We report overall performance and results on the four RL training tasks: Pass, Rotate, Land, and Approach. Overall metrics are reported as mean ± standard deviation across three selected evaluations per variant; task-wise metrics are means. Bold denotes the highest SR.
Variant
Data
SR (%)
NDTW
FlowVLA
Sim
73.14±0.56
0.399±0.016
FlowVLA
Mixed
75.58±1.12
0.402±0.006
FlowVLA w/ FAST head
Mixed
64.10±0.63
0.340±0.007
FlowVLA w/ all layers
Mixed
76.19±0.73
0.419±0.022
Table 5: Ablation of training data and model variants on UAV-Flow-Sim. Results are reported as mean ± standard deviation across three evaluations.
Figure 7: Outdoor flight demonstrations with π0 -UAV. Third-person image sequences show passing a tree from the left side and approaching the right side of a sign. Colored circles indicate the UAV position in each frame.
Table 7: RL optimization and rollout hyperparameters.
Figure 9: Paired closed-loop trajectories for Shift, Rotate, Ascend/Descend, Land, Approach, Retreat, Pass, and Surround. Each task includes an instruction and adjacent plots for FlowVLA and FlowVLA + TGRPO. Solid blue lines denote executed trajectories, and dashed lines denote references. Stars mark target objects, open circles mark initial states, and filled blue dots mark endpoints. Green check marks and red crosses indicate successful and failed episodes, respectively. Spatial coordinates are in meters. The Rotate plots show clockwise yaw change relative to the initial heading, with arrows indicating the final headings and annotations giving the executed rotation angles.
Figure 10: Hardware setup for physical-flight tests , including the P600 UAV platform, DJI Tello, RTK equipment, remote control, and ground station [ 30 ] .
Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China · Department of Mechanical and Energy Engineering, Southern University of Science and Technology, Shenzhen, China · Differential Robotics