FlyingDrones: A Dataset and Benchmark for Optical Flow Estimation from UAV motion
Organizations: Digital Safety and Security; Data Science and Artificial Intelligence; Austrian Institute of Technology · School of Computation, Information and Technology; Technical University of Munich; Munich Center for Machine Learning (MCML)
Abstract
Optical flow is an important cue for drone analysis in aerial surveillance, airspace monitoring, and long-range target tracking, where motion remains informative when appearance is weak. Estimating the motion of small Unmanned Aerial Vehicles, however, is difficult because distant targets occupy few pixels, have low contrast, and move against complex dynamic backgrounds. Standard optical-flow benchmarks emphasize large, object-centric motions and therefore do not characterize distant aerial targets well. We introduce FlyingDrones, a generated benchmark with dense ground-truth optical flow and segmentation masks derived from physically based rendering, and define resolution-normalized metrics (EPE Norm) that normalize flow by image dimensions. We evaluate 16 state-of-the-art model families on a motion- and size-stratified subset of 999 flow pairs at three resolutions. Every family recovers drone motion better than the zero-flow reference, but the gap between families widens with resolution: at 1920x1080, global-matching, recurrent, transformer, and short-window temporal models separate clearly from pyramid, classical, self-supervised, lightweight, and diffusion models. Within-family checkpoint spread is comparable to or larger than between-family spread, so the training curriculum is as important as the architecture. Two complementary studies test the benchmark beyond synthetic accuracy: a ground-truth-free motion-alignment evaluation on 21 cameras of real UAV footage shows that the synthetic ranking carries a moderate, significant signal about real-world performance (Spearman \r{ho} = 0.63 over 41 checkpoints) while the exact top ranks differ, and a fine-tuning study shows that 800 FlyingDrones pairs reduce held-out drone-region error by up to 37% and improve drone localization on the real footage for all four architectures tested.
Figures & tables
| General property | Amount |
|---|---|
| Total frames | 9001 |
| Flow pairs | 9000 |
| Frames with visible drones | 7908 |
| Drone instances | 19,999 |
| Drone flow magnitude | [px/frame] |
| Mean (per frame) | 8.39 |
| Cls | Family | 480 270 | 960 432 | 1920 1080 | |
|---|---|---|---|---|---|
| T | UniMatch | 0.952 | 1.293 | 1.557 ‡ | 1.7 |
| M | VideoFlow † | 0.997 | 1.425 | 1.582 ‡ | 3.0 |
| M | MemFlow † | 0.948 | 1.484 | 1.965 ‡ | 3.3 |
| T | FlowFormer++ | 1.108 | 1.513 | 1.840 ‡ | 5.3 |
| E | ARFlow | 0.873 | 1.581 | 3.175 | 5.7 |
| C | SEA-RAFT | 1.117 | 1.557 | 1.883 | 6.3 |
| Model | Synth. drone EPE ( ) | Real | Real Hit@50 |
|---|---|---|---|
| FlowFormer++ | ( ) | ||
| GMFlow | ( ) | ||
| RAFT | ( ) | ||
| VideoFlow MOF | ( ) |
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Asset | Type | Official source | License / terms |
|---|---|---|---|
| FlyingChairs | Dataset | University of Freiburg [ 7 ] | Non-commercial / research use only; no standard open-data license stated. |
| FlyingThings3D | Dataset | University of Freiburg Scene Flow Datasets [ 20 ] | Non-commercial / research use only; no standard open-data license stated. |
| MPI Sintel | Dataset / source movie assets | MPI Sintel benchmark; Blender Durian project [ 2 ] | Derived from the open movie Sintel ; movie and published project assets are released under CC BY 3.0; dataset published under PS:License 1.0. |
| KITTI | Dataset | KITTI Vision Benchmark Suite [ 22 ] | CC BY-NC-SA 3.0. |
| TartanAir | Dataset | TartanAir documentation [ 34 ] | CC BY 4.0 for the dataset; accompanying toolkit under MIT. |
| Spring | Dataset | Spring benchmark / DaRUS release [ 21 ] | CC BY 4.0. |
| Family | Checkpoint(s) | Dataset order used in training |
|---|---|---|
| FlowNet2 | FlowNet2-C_checkpoint | FlyingChairs |
| FlowNet2 | FlowNet2-CS_checkpoint | FlyingChairs FlyingThings3D |
| FlowNet2 | FlowNet2-CSS-ft-sd_checkpoint , FlowNet2_checkpoint | FlyingChairs FlyingThings3D ChairsSDHom / small-displacement refinement |
| FlowNet2 | FlowNet2-S_checkpoint , FlowNet2-SD_checkpoint | Published single-stream / small-displacement release checkpoints |
| PyFlow | accurate , default , fast | No learned schedule; classical coarse-to-fine optimization |
| PWC-Net | pwc_net_chairs , pwc_net | FlyingChairs published PWC-Net release refinement |
| Cls | Model | Checkpoint | EPE [95% CI] | EPE | Px1 | Px3 | Px5 | Fl |
|---|---|---|---|---|---|---|---|---|
| E | ARFlow | KITTI12/pwclite_ar_mv | 0.873 [0.77,0.99] | 0.215 | 19.95 | 6.779 | 3.543 | 6.779 |
| M | MemFlow † | MemFlowNet_spring | 0.948 [0.85,1.05] | 0.235 | 23.05 | 7.776 | 3.343 | 7.776 |
| T | UniMatch | gmflow-scale2-regrefine6-mixdata-train320x576-4e7b215d | 0.952 [0.86,1.05] | 0.237 | 24.55 | 7.568 | 3.310 | 7.568 |
| P | PyFlow (C2F) | fast | 0.977 [0.85,1.11] | 0.240 | 18.53 | 7.751 | 4.785 | 7.751 |
| M | VideoFlow † | MOF_things_288960noise | 0.997 [0.91,1.09] | 0.251 | 26.17 | 8.343 | 3.065 | 8.343 |
| C | WAFT | tar-c-t-spring-540p | 1.000 [0.89,1.11] | 0.245 | 24.82 | 8.815 | 3.802 | 8.815 |
| Cls | Model | Checkpoint | EPE [95% CI] | EPE | Px1 | Px3 | Px5 | Fl |
|---|---|---|---|---|---|---|---|---|
| T | UniMatch | gmflow-scale2-regrefine6-things-776ed612 | 1.293 [1.18,1.41] | 0.175 | 26.70 | 11.60 | 5.926 | 11.60 |
| M | VideoFlow † | BOF_sintel | 1.425 [1.31,1.55] | 0.192 | 30.82 | 13.16 | 6.967 | 13.16 |
| M | MemFlow † | MemFlowNet_spring | 1.484 [1.32,1.66] | 0.192 | 27.03 | 12.60 | 7.300 | 12.60 |
| T | FlowFormer++ | things_288960 | 1.513 [1.37,1.66] | 0.205 | 33.06 | 13.41 | 7.099 | 13.41 |
| C | SEA-RAFT | Tartan-C-T-TSKH432x960-M (sintel) | 1.557 [1.40,1.72] | 0.204 | 30.26 | 14.48 | 7.701 | 14.48 |
| E | ARFlow | KITTI12/pwclite_ar_mv | 1.581 [1.36,1.82] | 0.199 | 24.74 | 12.14 | 7.919 | 12.14 |
| Cls | Model | Checkpoint | EPE [95% CI] | EPE | Px1 | Px3 | Px5 | Fl |
|---|---|---|---|---|---|---|---|---|
| T | UniMatch ‡ | gmflow-scale2-regrefine6-sintelft-6e39e2b9 | 1.557 [1.42,1.70] | 0.097 | 25.26 | 12.12 | 7.931 | 12.12 |
| M | VideoFlow † ‡ | BOF_sintel | 1.582 [1.45,1.72] | 0.099 | 26.66 | 12.06 | 8.111 | 12.05 |
| T | FlowFormer++ ‡ | things_288960 | 1.840 [1.67,2.02] | 0.115 | 29.26 | 13.92 | 9.244 | 13.92 |
| C | SEA-RAFT | Tartan-C-T-TSKH432x960-M (sintel) | 1.883 [1.70,2.08] | 0.115 | 27.80 | 14.37 | 10.14 | 14.37 |
| M | MemFlow † ‡ | MemFlowNet_T_sintel | 1.965 [1.77,2.18] | 0.120 | 29.76 | 15.17 | 10.64 | 15.17 |
| T | GMFlow ‡ | gmflow_with_refine_sintel-3ed1cf48 | 1.996 [1.82,2.19] | 0.124 | 32.64 | 15.65 | 9.978 | 15.65 |
| Model | Checkpoint | Res. | EPE | EPE | Px1 | Px3 | Px5 | Fl |
|---|---|---|---|---|---|---|---|---|
| ARFlow | CityScapes/pwclite_ar | 480 270 | 0.331 | 0.091 | 7.451 | 0.645 | 0.025 | 0.645 |
| ARFlow | KITTI12/pwclite_ar | 480 270 | 0.168 | 0.048 | 2.342 | 0.177 | 0.064 | 0.177 |
| ARFlow | KITTI12/pwclite_ar_mv | 480 270 | 0.150 | 0.042 | 2.371 | 0.128 | 0.025 | 0.128 |
| ARFlow | KITTI15/pwclite_ar | 480 270 | 0.189 | 0.051 | 2.962 | 0.513 | 0.256 | 0.513 |
| ARFlow | KITTI15/pwclite_ar_mv | 480 270 | 0.157 | 0.043 | 2.373 | 0.273 | 0.112 | 0.273 |
| ARFlow | KITTI15/pwclite_raw | 480 270 | 0.225 | 0.059 | 2.524 | 0.371 | 0.271 | 0.371 |
| Model | Checkpoint | Res. | EPE | EPE | Px1 | Px3 | Px5 | Fl |
|---|---|---|---|---|---|---|---|---|
| ARFlow | CityScapes/pwclite_ar | 480 270 | 0.940 | 0.231 | 21.28 | 7.538 | 3.650 | 7.538 |
| ARFlow | KITTI12/pwclite_ar | 480 270 | 0.896 | 0.222 | 20.74 | 6.834 | 3.433 | 6.834 |
| ARFlow | KITTI12/pwclite_ar_mv | 480 270 | 0.873 | 0.215 | 19.95 | 6.779 | 3.543 | 6.779 |
| ARFlow | KITTI15/pwclite_ar | 480 270 | 0.897 | 0.222 | 20.12 | 7.009 | 3.501 | 7.009 |
| ARFlow | KITTI15/pwclite_ar_mv | 480 270 | 0.886 | 0.217 | 19.92 | 7.011 | 3.632 | 7.011 |
| ARFlow | KITTI15/pwclite_raw | 480 270 | 0.896 | 0.222 | 20.61 | 6.864 | 3.448 | 6.864 |
| Model | Ckpt | Window | Strat. | Model | Ckpt | Window | Strat. | ||
|---|---|---|---|---|---|---|---|---|---|
| PyFlow | fast | 1.395 | 0.977 | -0.42 | VideoFlow | BOF_sintel | 1.797 | 1.425 | -0.37 |
| UniMatch | scale2-regrefine6-mixdata-train320x576 | 1.472 | 0.952 | -0.52 | UniMatch | scale2-regrefine6-things | 1.954 | 1.293 | -0.66 |
| ARFlow | KITTI12/pwclite_ar_mv | 1.516 | 0.873 | -0.64 | GMFlow | gmflow_with_refine_sintel | 2.154 | 1.665 | -0.49 |
| MemFlow | MemFlowNet_spring | 1.634 | 0.948 | -0.69 | FlowFormer++ | things_288960 | 2.438 | 1.513 | -0.93 |
| VideoFlow | MOF_things_288960noise | 1.641 | 0.997 | -0.64 | SEA-RAFT | TSKH432x960-M (sintel) | 2.472 | 1.557 | -0.92 |
| Family | Checkpoint | Hit@50 | Contr. | cos | pt-EPE | ||
|---|---|---|---|---|---|---|---|
| MemFlow | MemFlowNet_spring | 21 | 27.3 10.2 | 0.62 | 303.8 | 0.60 | 1.33 |
| PyFlow | fast | 21 | 26.1 9.9 | 0.71 | 74.6 | 0.79 | 1.01 |
| SEA-RAFT | Tartan-C-T-TSKH432x960-M (sintel) | 21 | 16.5 8.9 | 0.45 | 35.4 | 0.46 | 1.52 |
| MemFlow | MemFlowNet_sintel | 21 | 16.3 7.7 | 0.46 | 42.8 | 0.34 | 1.80 |
| RAFT | raft-sintel | 21 | 15.5 8.4 | 0.40 | 23.7 | 0.41 | 1.75 |
| FlowFormer++ | things_288960 ‡ | 21 | 15.5 8.7 | 0.56 | 12.8 | 0.54 | 1.58 |
| Checkpoint | sub-px | low | mid | high |
|---|---|---|---|---|
| ( px) | ( – ) | ( – ) | ( ) | |
| MemFlow / spring | 0.24 | 0.63 | 0.86 | 0.90 |
| PyFlow / fast | 0.54 | 0.74 | 0.81 | 0.88 |
| FlowFormer++ / things_288960 | 0.22 | 0.48 | 0.84 | 0.91 |
| GMFlow / with_refine_kitti | 0.15 | 0.22 | 0.47 | 0.85 |
| VideoFlow MOF / sintel | 0.07 | 0.11 | 0.25 | 0.39 |
| Val drone-EPE | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | Init | Eval res. | Crop | Batch | LR | Best | step 0 | best | 1500 | |
| FlowFormer++ | things_288960 | 0.85 | 600 | 1.042 | 0.986 | 0.989 | ||||
| GMFlow | gmflow_chairs | 0.90 | 300 | 2.239 | 1.856 | 1.967 | ||||
| RAFT | raft-things | 0.85 | 150 | 1.105 | 1.107 | 1.195 | ||||
| VideoFlow MOF | MOF_sintel | 0.85 | 1500 | 1.116 | 0.840 | 0.840 | ||||
| Synthetic test (FlyingDrones) | Real footage | |||||
|---|---|---|---|---|---|---|
| Model | Res. | Drone EPE | Full EPE | Hit@50 | ||
| FlowFormer++ | ||||||
| GMFlow | ||||||
| RAFT | ||||||
| VideoFlow MOF | ||||||
Optical Flow
Optical flow estimates the apparent motion of pixels between images, supporting video understanding, tracking, robotics, and image synthesis.
See all 64 papers