Organizations: School of Vehicle and Mobility, Tsinghua University, Beijing, China · State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University, Beijing, China · Xibu Zhiche (Chongqing) Technology Co., Ltd., Chongqing, China
Existing autonomous driving datasets and benchmarks provide extensive resources for routine driving, yet real-world data covering safety-critical scenarios and extreme environmental conditions remain limited. To address this gap, we present ExtremeDrive, a real-world dataset and benchmark covering emergency maneuvers, complex traffic interactions, and adverse conditions such as slippery or snow-covered roads, alongside routine driving. The dataset comprises 589 driving scenes curated from a larger pool of real-world driving recordings, with calibrated and aligned multisensor data and reviewed annotations of road structure and traffic participants. We further establish a benchmark to evaluate planned trajectories through simulated vehicle execution, measuring safety-related metrics and driving performance. Experiments at equal training set sizes show that incorporating critical driving data reduces trajectory prediction errors and improves mean critical-success rates in critical scenarios. ExtremeDrive provides a unified resource for learning and evaluating autonomous driving in safety-critical scenarios and extreme environmental conditions.
Figures & tables
Fig. 2: Multisensor views of a complex traffic scene in ExtremeDrive. Camera detections accompany a LiDAR reconstruction and a BEV view of road and traffic annotations.
Dataset
Scale
Camera
LiDAR
Vehicle signals
Emergency
Urban
Suburban
Rural
Highway
nuPlan [ 2 ]
1,282 h / 4 cities
✓
✓
—
—
✓
—
—
—
INTERACTION [ 6 ]
11 locations
×
×
—
✓
✓
—
—
✓
DoTA [ 13 ]
4,677 videos
✓
×
—
△
—
—
—
—
KITTI (Odometry) [ 18 ]
39 km / 22 seqs
✓
✓
—
—
✓
✓
×
×
UrbanLoco [ 19 ]
40 km / 13 seqs
✓
✓
—
—
✓
×
×
×
MulRan [ 20 ]
124 km / 12 seqs
×
✓
—
—
✓
—
—
—
TABLE I: Comparison of selected driving datasets.
Fig. 3: ExtremeDrive data collection platform and sensor layout.
Parameter
Symbol
Value
Total mass
m
2178kg
Yaw moment of inertia
Iz
3216kg⋅m2
CG-to-front-axle distance
lf
1.526m
CG-to-rear-axle distance
lr
1.374m
Wheel rotational inertia
Iw
2.5kg⋅m2
TABLE II: Key vehicle parameters of the Changan Deepal SL03 platform.
Fig. 4: Dataset composition and maneuver intensity. (a) Condition shares among 589 driving scenes. (b) Action shares among critical scenes. Percentages are rounded. (c)–(d) Normalized peak-deceleration and absolute yaw rate densities for routine and critical clips over 4 s horizons; dashed lines mark empirical P90 values. Each density panel uses a shared bandwidth and reflection at zero.
Method
Input
ADE/FDE (m) ↓
S+↑
Collision ↓
Road ↑
TTC ↑
Progress ↑
C↑
ES ↑
HyperDiffusion [ 21 ]
V
1.17/2.79
80.57
2.53
85.85
88.88
92.82
99.82
61.70
SimLingo [ 22 ]
V
1.47/3.44
77.58
3.97
82.19
87.95
90.20
99.78
51.77
Latent TransFuser [ 23 ]
V
1.18/2.91
75.46
3.38
80.33
88.45
92.05
99.90
58.16
DiffusionDrive [ 24 ]
V+L
1.28/3.33
74.85
3.30
79.86
88.65
92.51
96.88
57.45
TransFuser [ 23 ]
V+L
1.09/2.71
74.26
3.46
77.95
88.57
92.60
99.93
56.03
TABLE III: Planning and execution results.
Fig. 5: Annotation example from a multi-vehicle scene. Scene labels and CoC summarize the original annotations; the BEV shows accepted road annotations and in-road vehicle footprints aligned with the camera frame. Shading distinguishes the exterior of the solid boundaries.
Condition
Model
Position
Long. vel.
Lat. vel.
(m)
(m/s)
(m/s)
Dry
Kinematic bicycle
0.697
0.378
0.081
Dry
Calibrated 7DOF
0.688
0.373
0.055
Wet
Kinematic bicycle
0.717
0.428
0.058
Wet
Calibrated 7DOF
0.698
0.415
0.045
TABLE IV: Vehicle-model replay errors on held-out driving logs. All metrics are RMSE ( ↓ ).
Model / training
Critical
ADE
FDE
ES (%)
MLP-Routine only
4.11 ± 0.03
8.06 ± 0.15
42.35 ± 0.26
MLP-Mixed
3.16 ± 0.11
6.06 ± 0.32
59.10 ± 0.54
TransFuser-Routine only
3.28 ± 0.09
8.28 ± 0.20
47.43 ± 0.20
TransFuser-Mixed
2.31 ± 0.16
5.53 ± 0.51
57.45 ± 2.33
DiffusionDrive-Routine only
3.53 ± 0.11
8.72 ± 0.22
46.35 ± 1.55
TABLE V: Contribution of critical data at equal training size.
End-to-end autonomous driving has witnessed rapid progress, yet existing benchmarks are increasingly saturated, with state-of-the-art models achieving near-perfect scores on widely used open-loop and closed-loop benchmarks. This saturation does not mean that the problem has been solved; instead, it reveals that current benchmarks remain limited in scenario diversity, object variety, and the breadth of driving capabilities they evaluate. In particular, they lack sufficient long-tail scenarios involving rare but safety-critical objects and fail to assess advanced decision-making such as legal compliance, ethical reasoning, and emergency response. To address these gaps, we propose HiDrive, a new closed-loop benchmark for end-to-end autonomous driving that emphasizes long-tail scenarios and a richer evaluation of driving capabilities. HiDrive introduces a diverse set of rare objects and uncommon traffic situations, and expands evaluation from basic driving skills to more advanced capabilities, including rule compliance, moral reasoning, and context-dependent emergency maneuvers. Correspondingly, we extend previous collision-avoidance-centered metrics into a comprehensive evaluation system that encompasses collision and braking, traffic-rule compliance, and moral-reasoning indicators. Built on a more advanced physics engine, HiDrive provides physically realistic lighting and high-fidelity visual rendering, offering a more challenging and realistic testbed for assessing whether autonomous driving systems can handle the complexity of real-world deployment. The HiDrive software, source code, digital assets, and documentation are available at https://github.com/VDIGPKU/HiDrive.
Zhongyu Xia, Guanyu Zhu, Guo Tang +2
Wangxuan Institute of Computer Technology, Peking University
Collecting a high-quality dataset is a critical task that demands meticulous attention to detail, as overlooking certain aspects can render the entire dataset unusable. Autonomous driving challenges remain a prominent area of research, requiring further exploration to enhance the perception and planning performance of vehicles. However, existing datasets are often incomplete. For instance, datasets that include perception information generally lack planning data, while planning datasets typically consist of extensive driving sequences where the ego vehicle predominantly drives forward, offering limited behavioral diversity. In addition, many real datasets struggle to evaluate their models, especially for planning tasks, since they lack a proper closed-loop evaluation setup. The CARLA Leaderboard 2.0 challenge, which provides a diverse set of scenarios to address the long-tail problem in autonomous driving, has emerged as a valuable alternative platform for developing perception and planning models in both open-loop and closed-loop evaluation setups. Nevertheless, existing datasets collected on this platform present certain limitations. Some datasets appear to be tailored primarily for limited sensor configuration, with particular sensor configurations. To support end-to-end autonomous driving research, we have collected a new dataset comprising over 2.85 million frames using the CARLA simulation environment for the diverse Leaderboard 2.0 challenge scenarios. Our dataset is designed not only for planning tasks but also supports dynamic object detection, lane divider detection, centerline detection, traffic light recognition, prediction tasks and visual language action models . Furthermore, we demonstrate its versatility by training various models using our dataset. Moreover, we also provide numerical rarity scores to understand how rarely the current state occurs in the dataset.
Tugrul Gorgulu, Atakan Dag, M. Esat Kalfaoglu +4
The authors are currently with Trutek AI. · This work was conducted at Trutek AI. The author is currently with Ultralytics. · This work was conducted at Trutek AI. The author is currently with Amazon.
The Boreas Road Trip (Boreas-RT) dataset extends the multi-season Boreas dataset to new and diverse locations that pose challenges for modern autonomous driving algorithms. Boreas-RT comprises 60 sequences collected over 9 real-world routes, totalling 643 km of driving. Each route is traversed multiple times, enabling evaluation in identical environments under varying traffic and, in some cases, weather conditions. The data collection platform includes a 5MP FLIR Blackfly S camera, a 360 degree Navtech RAS6 Doppler-enabled spinning radar, a 128-channel 360 degree Velodyne Alpha Prime lidar, an Aeva Aeries II FMCW Doppler-enabled lidar, a Silicon Sensing DMU41 inertial measurement unit, and a Dynapar wheel encoder. Centimetre-level ground truth is provided via post-processed Applanix POS LV GNSS-INS data. The dataset includes precise extrinsic and intrinsic calibrations, a publicly available development kit, and a live leaderboard for odometry and metric localization. Benchmark results show that many state-of-the-art odometry and localization algorithms overfit to simple driving environments and degrade significantly on the more challenging Boreas-RT routes. Boreas-RT provides a unified dataset for evaluating multi-modal algorithms across diverse road conditions. The dataset, leaderboard, and development kit are available at www.boreas.utias.utoronto.ca.
Daniil Lisus, Katya M. Papais, Cedric Le Gentil +4
University of Toronto Institute of Aerospace Studies, Toronto, Canada · Trimble, Canada