TaCarla: A comprehensive benchmarking dataset for end-to-end autonomous driving

Tugrul Gorgulu; Atakan Dag; M. Esat Kalfaoglu; Halil Ibrahim Kuru; Baris Can Cam; Halil Ibrahim Ozturk; Ozsel Kilinc

cs.ROFeb 26, 2026

TaCarla: A comprehensive benchmarking dataset for end-to-end autonomous driving

Authors: Tugrul Gorgulu, Atakan Dag, M. Esat Kalfaoglu, Halil Ibrahim Kuru, Baris Can Cam, Halil Ibrahim Ozturk, Ozsel Kilinc

Organizations: The authors are currently with Trutek AI. · This work was conducted at Trutek AI. The author is currently with Ultralytics. · This work was conducted at Trutek AI. The author is currently with Amazon.

Abstract

Collecting a high-quality dataset is a critical task that demands meticulous attention to detail, as overlooking certain aspects can render the entire dataset unusable. Autonomous driving challenges remain a prominent area of research, requiring further exploration to enhance the perception and planning performance of vehicles. However, existing datasets are often incomplete. For instance, datasets that include perception information generally lack planning data, while planning datasets typically consist of extensive driving sequences where the ego vehicle predominantly drives forward, offering limited behavioral diversity. In addition, many real datasets struggle to evaluate their models, especially for planning tasks, since they lack a proper closed-loop evaluation setup. The CARLA Leaderboard 2.0 challenge, which provides a diverse set of scenarios to address the long-tail problem in autonomous driving, has emerged as a valuable alternative platform for developing perception and planning models in both open-loop and closed-loop evaluation setups. Nevertheless, existing datasets collected on this platform present certain limitations. Some datasets appear to be tailored primarily for limited sensor configuration, with particular sensor configurations. To support end-to-end autonomous driving research, we have collected a new dataset comprising over 2.85 million frames using the CARLA simulation environment for the diverse Leaderboard 2.0 challenge scenarios. Our dataset is designed not only for planning tasks but also supports dynamic object detection, lane divider detection, centerline detection, traffic light recognition, prediction tasks and visual language action models . Furthermore, we demonstrate its versatility by training various models using our dataset. Moreover, we also provide numerical rarity scores to understand how rarely the current state occurs in the dataset.

Figures & tables

Panel of Figure 1 — Figure 1: These images include views from 6 cameras, point cloud from LiDAR sensors, and satellite-like BEV RGB image. We visualize the ground-truth 3D bounding boxes for vehicles, and 2D bounding boxes for traffic lights on camera-view. Additionally, ground-truths are visualized on a BEV grid, including centerlines, lane guidance, and orientation for vehicles.

Condition	Clean (%)	Low (%)	Medium (%)	Heavy (%)
Cloudiness	35.49	7.65	34.22	22.62
Fog Density	68.45	3.31	23.82	4.40
Precipitation	51.77	3.76	33.99	10.46
Precipitation Deposits	33.65	3.37	49.16	13.80
Wetness	79.87	0.41	10.79	8.91

Table 1: Weather Condition Distributions in the TaCarla Dataset

Panel of Figure 2 — Figure 2: Distribution of the ego vehicle’s ground-truth location.

Scenario	TaCarla	Bench2Drive	PDM-Lite
Accident	353	28	88
AccidentTwoWays	410	201	97
BlockedIntersection	43	582	96
ConstructionObstacle	306	27	92
ConstructionObstacleTwoWays	310	138	94
ControlLoss	255	18	266

Table 2: Number of scenarios in TaCarla compared to Bench2Drive and PDM-Lite

Panel of Figure 3 — Figure 3: Outputs of the TaCarla Label Viewer

	AP	ATE	ASE	AOE	AVE
Car	0.459	0.444	0.147	0.012	0.559
Crossbike	0.324	0.242	0.094	0.057	0.165
Walker	0.426	0.456	0.885	1.333	0.292
Police	0.381	0.249	0.056	0.011	0.048
Construction	0.419	0.665	0.812	1.125	0.065
Ambulance	0.098	0.440	0.132	0.065	0.525

Table 3: Camera-only 3D object detection performance.

	AP	ATE	ASE	AOE	AVE
Car	0.716	0.173	0.125	0.022	0.399
Crossbike	0.556	0.113	0.086	0.076	0.119
Walker	0.527	0.152	0.885	1.304	0.277
Police	0.486	0.082	0.052	0.018	0.038
Construction	0.657	0.253	0.821	1.125	0.098
Ambulance	0.428	0.254	0.100	0.074	0.283

Table 4: Camera-LiDAR 3D object detection performance.

Detection Task	$\text{AP}_{f}$	$\text{AP}_{c}$	F1 1.5
Centerline Detection	39.6	41.7	67.3
Lane Divider Detection	N/A	32.1	64.3

Table 5: Centerline and Lane Divider Detection Results of TopoBDA architecture for TaCarla Dataset.

Figure 4: Bird’s Eye View (BEV) results demonstrating the performance of TopoBDA on the TaCarla dataset. GT denotes the ground truth, and Pred denotes the predictions. GT + Pred shows the overlaid results of both, facilitating visual assessment of localization performance.

Model	$\mathrm{AP}$	$\mathrm{AP_{50}}$
FCOS [ 43 ]	$59.5$	$88.2$

Table 6: Traffic Light Detection task results in TaCarla.

Figure 5: Waypoints from ground truth(Top Left), PlanT(Top Right), Transfuser(Bottom Left), and DiffusionDrive (Bottom Right) models.

Model	$\mathcal{H}$	ADE	FDE	AHE	FHE
DiffusionDrive [ 28 ]	4	2.69	5.58	0.27	0.21
	2	1.14	2.14	0.21	0.21
	1	0.51	0.78	0.12	0.12
PlanT [ 38 ]	4	-	-	-	-
	2	1.03	1.71	0.36	0.34
	1	-	-	-	-

Table 7: Open-loop results on the validation set of unseen Town13 with 1s, 2s, and 4s planning horizons (H).

Model	Driving	Route	Penalty
DiffusionDrive [ 28 ]	22.35	62.06	0.339
Transfuser [ 8 ]	17.18	65.67	0.283
PlanT [ 38 ]	52.95	81.67	0.658
PlanT* [ 38 ]	59.25	81.59	0.705

Table 8: Closed-loop metrics in unseen Town13 (Driving Score ↑, Route Score ↑, Penalty ↑) on validation routes. PlanT* is trained with the samples whose corresponding rarity scores are greater than 0.

Panel of Figure 6 — Figure 6: Outputs of the FCOS traffic light model

Scenario Name	Model	ADE ↓	FDE ↓	AHE ↓	FHE ↓
Accident	DiffusionDrive	2.1493	4.0263	0.1040	0.1156
	PlanT	1.4002	2.2346	0.1368	0.1934
	Transfuser	1.8716	3.4446	0.1109	0.0963
AccidentTwoWays	DiffusionDrive	0.8104	1.5032	0.6924	0.6760
	PlanT	0.5810	2.2346	0.1368	0.1934
	Transfuser	0.5422	1.0367	0.8329	1.0033

Table 9: Open-loop metrics (ADE ↓, FDE ↓, AHE ↓, FHE ↓) for each scenario.

Explore similar work

HiDrive: A Closed-Loop Benchmark for High-Level Autonomous Driving
May 11, 2026Zhongyu Xia, Guanyu Zhu, Guo Tang +2Autonomous Driving Benchmarks End-to-End Autonomous Driving
Wangxuan Institute of Computer Technology, Peking University
A2RL V\textsubscript{max}: The A2RL autonomous racing dataset for long-range, high-speed perception and multi-vehicle interaction
Jul 20, 2026Marvin Klemp, Dominic Ebner, Cornelius Schröder +15Autonomous Driving Perception Autonomous Racing
Institute of Measurement and Control Systems, Karlsruhe Institute of Technology · Code 19 Racing · Institute of Automotive Technology, Technical University of Munich +8
StandardE2E: A Unified Framework for End-to-End Autonomous Driving Datasets
Jun 2, 2026Stepan KonevAutonomous Driving Datasets End-to-End Autonomous Driving

May 11, 2026J/K move · Enter open · S save

HiDrive: A Closed-Loop Benchmark for High-Level Autonomous Driving

Zhongyu Xia, Guanyu Zhu, Guo Tang, Wenhao Chen+1

Wangxuan Institute of Computer Technology, Peking University

Autonomous Driving Benchmarks End-to-End Autonomous Driving Autonomous Driving Safety Evaluation Autonomous Driving Autonomous Driving Planning Autonomous Driving Simulation

Open paper AskPDF arXiv

End-to-end autonomous driving has witnessed rapid progress, yet existing benchmarks are increasingly saturated, with state-of-the-art models achieving near-perfect scores on widely used open-loop and closed-loop benchmarks. This saturation does not mean that the problem has been solved; instead, it reveals that current benchmarks remain limited in scenario diversity, object variety, and the breadth of driving capabilities they evaluate. In particular, they lack sufficient long-tail scenarios involving rare but safety-critical objects and fail to assess advanced decision-making such as legal compliance, ethical reasoning, and emergency response. To address these gaps, we propose HiDrive, a new closed-loop benchmark for end-to-end autonomous driving that emphasizes long-tail scenarios and a richer evaluation of driving capabilities. HiDrive introduces a diverse set of rare objects and uncommon traffic situations, and expands evaluation from basic driving skills to more advanced capabilities, including rule compliance, moral reasoning, and context-dependent emergency maneuvers. Correspondingly, we extend previous collision-avoidance-centered metrics into a comprehensive evaluation system that encompasses collision and braking, traffic-rule compliance, and moral-reasoning indicators. Built on a more advanced physics engine, HiDrive provides physically realistic lighting and high-fidelity visual rendering, offering a more challenging and realistic testbed for assessing whether autonomous driving systems can handle the complexity of real-world deployment. The HiDrive software, source code, digital assets, and documentation are available at https://github.com/VDIGPKU/HiDrive.

Abstract

Explore similar work

HiDrive: A Closed-Loop Benchmark for High-Level Autonomous Driving

A2RL V\textsubscript{max}: The A2RL autonomous racing dataset for long-range, high-speed perception and multi-vehicle interaction

StandardE2E: A Unified Framework for End-to-End Autonomous Driving Datasets