Autonomous driving has made remarkable progress through imitation learning with massive human demonstration data. However, a trained planner often degrades severely when applied to a new environment zero-shot, because of domain shifts in traffic regulations, road layout and driving behaviors. Therefore, adapting a trajectory planner to a new city typically requires resource-demanding local data collection with a vehicle sensor suite. In this work, we show that driving behavior can be learned from a scalable and efficient alternative. We introduce SkyDrive, a framework that utilizes drone-based traffic monitoring to provide efficient supervision for autonomous driving agents in a new environment. While vehicle-based data collection logs the ego and its surroundings, an aerial platform naturally observes many road users simultaneously over an extended field of view. As a result, every vehicle can be a data source with grounded driving behavior, effectively scaling up the amount of supervision. Based on 137 hours of aerial traffic monitoring footage, we extract 650K driving samples and construct a benchmark for trajectory planners and motion predictors. Zero-shot experiments with multiple models reveal significant cross-city domain gaps, but many of them can be alleviated by limited supervision from the sky, e.g., 30 minutes of monitoring per location. Our findings show that aerial traffic monitoring is an efficient and scalable data source for adapting autonomous driving systems in new cities. Data and code will be made publicly available.
Figures & tables
Figure 1: Overview of SkyDrive . We utilize accurate vehicle trajectory data from drone-based traffic monitoring research, study trajectory planning and motion prediction tasks, and evaluate accuracy, regulatory compliance and safety.
Figure 2: Map visualizations for two example intersections. The lane bounding boxes are labeled in Songdo Traffic, and the stop lines, lane centerlines, and drivable area are derived using trajectory data.
Split
Sessions
Time (h)
Valid veh.
Filtered
Ego segs.
Train
680
116.23
313075
200440
549929
Test
120
20.97
54731
33910
99629
Overall
800
137.20
367806
234350
649558
Table 1: Statistics of the standard data split
Dataset
# unique tracks
Avg track length
#sce.
Scenario duration
Total time
NuScenes
4.3k
–
50K
8s
5.5h
Argoverse2
13.9m
5.16s
250K
11s
763h
Waymo
7.64m
7.04s
576K
9.1s
574h
Interaction
40k
19.8s
–
–
16.5h
SinD
13.2k
–
–
–
7.02h
HeteroD
64.5k
–
–
–
17.5h
Table 2: Comparison with other datasets.
Figure 3: Example of semantic views during a lane-changing event in busy traffic. The leftmost panel shows a bird’s-eye view of the scene, where the ego is rendered as a red star. The perspective views from left to right are front, back, left and right.
Figure 4: The trajectory types in the trajectory planning task
Task
Input data
Input horizon
Output data
Output horizon
Frequency
Metrics
Trajectory planning
Rendered multi-camera semantic views, ego motion history, high-level direction
2s
Ego trajectory
4s
2Hz
ADE, FDE, TTC infraction rate, NCT
Motion prediction
Map structures and ego/surrounding- vehicle trajectories
2s
6 ego trajectories and probabilities
6s
10Hz
minADE, minFDE, Miss Rate, BrierFDE
Table 3: Summary of the trajectory planning and motion prediction tasks.
Figure 5: Evaluation results by Kalman difficulty. Easy ∈ [0, 10), medium ∈ [10, 20), hard ≥ 20.
Figure 6: Trajectory planning results by trajectory type
Figure 7: Scaling experiment results. The experiments with 1%, 5%, 10%, 20% and 50% training data have 6, 34, 68, 136 and 340 monitoring sessions respectively, corresponding to 1.0, 5.8, 11.7, 23.3 and 58.3 hours of approximate traffic monitoring time.
Method
ADE (m) ↓
FDE (m) ↓
TTC (%) ↓
NCT (%) ↓
DrivoR (S)
1.485
3.514
8.44
4.72
DrivoR (C)
1.535
3.659
7.94
7.87
RAP (S)
1.145
2.447
4.07
0.09
RAP (C)
1.170
2.506
5.09
0.06
Table 5: Matched standard (S) and cross-intersection (C) results on overlapping samples
Method
BrierFDE
minADE
minFDE
Miss Rate
MTR*
3.953
1.497
3.369
0.494
AutoBot
1.847
0.588
1.197
0.154
Wayformer
1.763
0.564
1.157
0.148
MTR
1.902
0.680
1.516
0.273
Table 6: Motion prediction results. Best values in bold.
Method
Easy [0, 30)
Medium [30, 50)
Hard [50, 100]
MTR*
3.248
8.088
16.975
AutoBot
1.722
2.600
3.564
Wayformer
1.645
2.482
3.242
MTR
1.765
2.716
4.107
Table 7: BrierFDE by Kalman difficulty. Best values in bold.
Type / Method
MTR*
AutoBot
Wayformer
MTR
Stationary
2.017
0.623
0.562
0.457
Straight
2.704
1.791
1.690
1.817
Straight-right
5.337
2.700
2.631
2.985
Straight-left
4.549
2.423
2.262
2.668
Right turn
7.825
2.459
2.399
2.667
Left turn
7.131
2.396
2.350
2.554
Table 8: BrierFDE by trajectory type. Best values in bold.
Autonomous driving develops vehicles that perceive road scenes, predict interactions, and plan and control safe movement with limited or no human input.