TaCarla: A comprehensive benchmarking dataset for end-to-end autonomous driving

Tugrul Gorgulu; Atakan Dag; M. Esat Kalfaoglu; Halil Ibrahim Kuru; Baris Can Cam; Halil Ibrahim Ozturk; Ozsel Kilinc

cs.ROFeb 26, 2026

TaCarla: A comprehensive benchmarking dataset for end-to-end autonomous driving

Authors: Tugrul Gorgulu, Atakan Dag, M. Esat Kalfaoglu, Halil Ibrahim Kuru, Baris Can Cam, Halil Ibrahim Ozturk, Ozsel Kilinc

Organizations: The authors are currently with Trutek AI. · This work was conducted at Trutek AI. The author is currently with Ultralytics. · This work was conducted at Trutek AI. The author is currently with Amazon.

Abstract

Collecting a high-quality dataset is a critical task that demands meticulous attention to detail, as overlooking certain aspects can render the entire dataset unusable. Autonomous driving challenges remain a prominent area of research, requiring further exploration to enhance the perception and planning performance of vehicles. However, existing datasets are often incomplete. For instance, datasets that include perception information generally lack planning data, while planning datasets typically consist of extensive driving sequences where the ego vehicle predominantly drives forward, offering limited behavioral diversity. In addition, many real datasets struggle to evaluate their models, especially for planning tasks, since they lack a proper closed-loop evaluation setup. The CARLA Leaderboard 2.0 challenge, which provides a diverse set of scenarios to address the long-tail problem in autonomous driving, has emerged as a valuable alternative platform for developing perception and planning models in both open-loop and closed-loop evaluation setups. Nevertheless, existing datasets collected on this platform present certain limitations. Some datasets appear to be tailored primarily for limited sensor configuration, with particular sensor configurations. To support end-to-end autonomous driving research, we have collected a new dataset comprising over 2.85 million frames using the CARLA simulation environment for the diverse Leaderboard 2.0 challenge scenarios. Our dataset is designed not only for planning tasks but also supports dynamic object detection, lane divider detection, centerline detection, traffic light recognition, prediction tasks and visual language action models . Furthermore, we demonstrate its versatility by training various models using our dataset. Moreover, we also provide numerical rarity scores to understand how rarely the current state occurs in the dataset.

Figures & tables

Panel of Figure 1 — Figure 1: These images include views from 6 cameras, point cloud from LiDAR sensors, and satellite-like BEV RGB image. We visualize the ground-truth 3D bounding boxes for vehicles, and 2D bounding boxes for traffic lights on camera-view. Additionally, ground-truths are visualized on a BEV grid, including centerlines, lane guidance, and orientation for vehicles.

Condition	Clean (%)	Low (%)	Medium (%)	Heavy (%)
Cloudiness	35.49	7.65	34.22	22.62
Fog Density	68.45	3.31	23.82	4.40
Precipitation	51.77	3.76	33.99	10.46
Precipitation Deposits	33.65	3.37	49.16	13.80
Wetness	79.87	0.41	10.79	8.91

Table 1: Weather Condition Distributions in the TaCarla Dataset

Panel of Figure 2 — Figure 2: Distribution of the ego vehicle’s ground-truth location.

Scenario	TaCarla	Bench2Drive	PDM-Lite
Accident	353	28	88
AccidentTwoWays	410	201	97
BlockedIntersection	43	582	96
ConstructionObstacle	306	27	92
ConstructionObstacleTwoWays	310	138	94
ControlLoss	255	18	266

Table 2: Number of scenarios in TaCarla compared to Bench2Drive and PDM-Lite

Panel of Figure 3 — Figure 3: Outputs of the TaCarla Label Viewer

	AP	ATE	ASE	AOE	AVE
Car	0.459	0.444	0.147	0.012	0.559
Crossbike	0.324	0.242	0.094	0.057	0.165
Walker	0.426	0.456	0.885	1.333	0.292
Police	0.381	0.249	0.056	0.011	0.048
Construction	0.419	0.665	0.812	1.125	0.065
Ambulance	0.098	0.440	0.132	0.065	0.525

Table 3: Camera-only 3D object detection performance.

	AP	ATE	ASE	AOE	AVE
Car	0.716	0.173	0.125	0.022	0.399
Crossbike	0.556	0.113	0.086	0.076	0.119
Walker	0.527	0.152	0.885	1.304	0.277
Police	0.486	0.082	0.052	0.018	0.038
Construction	0.657	0.253	0.821	1.125	0.098
Ambulance	0.428	0.254	0.100	0.074	0.283

Table 4: Camera-LiDAR 3D object detection performance.

Detection Task	$\text{AP}_{f}$	$\text{AP}_{c}$	F1 1.5
Centerline Detection	39.6	41.7	67.3
Lane Divider Detection	N/A	32.1	64.3

Table 5: Centerline and Lane Divider Detection Results of TopoBDA architecture for TaCarla Dataset.

Figure 4: Bird’s Eye View (BEV) results demonstrating the performance of TopoBDA on the TaCarla dataset. GT denotes the ground truth, and Pred denotes the predictions. GT + Pred shows the overlaid results of both, facilitating visual assessment of localization performance.

Model	$\mathrm{AP}$	$\mathrm{AP_{50}}$
FCOS [ 43 ]	$59.5$	$88.2$

Table 6: Traffic Light Detection task results in TaCarla.

Figure 5: Waypoints from ground truth(Top Left), PlanT(Top Right), Transfuser(Bottom Left), and DiffusionDrive (Bottom Right) models.

Model	$\mathcal{H}$	ADE	FDE	AHE	FHE
DiffusionDrive [ 28 ]	4	2.69	5.58	0.27	0.21
	2	1.14	2.14	0.21	0.21
	1	0.51	0.78	0.12	0.12
PlanT [ 38 ]	4	-	-	-	-
	2	1.03	1.71	0.36	0.34
	1	-	-	-	-

Table 7: Open-loop results on the validation set of unseen Town13 with 1s, 2s, and 4s planning horizons (H).

Model	Driving	Route	Penalty
DiffusionDrive [ 28 ]	22.35	62.06	0.339
Transfuser [ 8 ]	17.18	65.67	0.283
PlanT [ 38 ]	52.95	81.67	0.658
PlanT* [ 38 ]	59.25	81.59	0.705

Table 8: Closed-loop metrics in unseen Town13 (Driving Score ↑, Route Score ↑, Penalty ↑) on validation routes. PlanT* is trained with the samples whose corresponding rarity scores are greater than 0.

Panel of Figure 6 — Figure 6: Outputs of the FCOS traffic light model

Scenario Name	Model	ADE ↓	FDE ↓	AHE ↓	FHE ↓
Accident	DiffusionDrive	2.1493	4.0263	0.1040	0.1156
	PlanT	1.4002	2.2346	0.1368	0.1934
	Transfuser	1.8716	3.4446	0.1109	0.0963
AccidentTwoWays	DiffusionDrive	0.8104	1.5032	0.6924	0.6760
	PlanT	0.5810	2.2346	0.1368	0.1934
	Transfuser	0.5422	1.0367	0.8329	1.0033

Table 9: Open-loop metrics (ADE ↓, FDE ↓, AHE ↓, FHE ↓) for each scenario.

Explore similar work

May 11, 2026

HiDrive: A Closed-Loop Benchmark for High-Level Autonomous Driving

End-to-end autonomous driving has witnessed rapid progress, yet existing benchmarks are increasingly saturated, with state-of-the-art models achieving near-perfect scores on widely used open-loop and closed-loop benchmarks. This saturation does not mean that the problem has been solved; instead, it reveals that current benchmarks remain limited in scenario diversity, object variety, and the breadth of driving capabilities they evaluate. In particular, they lack sufficient long-tail scenarios involving rare but safety-critical objects and fail to assess advanced decision-making such as legal compliance, ethical reasoning, and emergency response. To address these gaps, we propose HiDrive, a new closed-loop benchmark for end-to-end autonomous driving that emphasizes long-tail scenarios and a richer evaluation of driving capabilities. HiDrive introduces a diverse set of rare objects and uncommon traffic situations, and expands evaluation from basic driving skills to more advanced capabilities, including rule compliance, moral reasoning, and context-dependent emergency maneuvers. Correspondingly, we extend previous collision-avoidance-centered metrics into a comprehensive evaluation system that encompasses collision and braking, traffic-rule compliance, and moral-reasoning indicators. Built on a more advanced physics engine, HiDrive provides physically realistic lighting and high-fidelity visual rendering, offering a more challenging and realistic testbed for assessing whether autonomous driving systems can handle the complexity of real-world deployment. The HiDrive software, source code, digital assets, and documentation are available at https://github.com/VDIGPKU/HiDrive.

Zhongyu Xia, Guanyu Zhu, Guo Tang +2

Wangxuan Institute of Computer Technology, Peking University

Autonomous Driving Benchmarks End-to-End Autonomous Driving Autonomous Driving Safety Evaluation Autonomous Driving Autonomous Driving Planning Autonomous Driving Simulation

Ask PDF

Jul 20, 2026

A2RL V\textsubscript{max}: The A2RL autonomous racing dataset for long-range, high-speed perception and multi-vehicle interaction

In autonomous driving development, a perception dataset is crucial, as it provides fundamental data for training, testing, and validating algorithms for an autonomous vehicle's multimodal perception systems. So far, most research has concentrated on providing datasets for well-structured urban environments. This work introduces the A2RL V\textsubscript{max} open-source dataset, specifically designed for perception tasks in high-speed autonomous driving and multi-vehicle interaction. The dataset was captured during the 2024 Abu Dhabi Autonomous Racing League (A2RL), held at the Yas Marina F1 Circuit, with participation from all competing teams. It contains diverse scenarios, including single-vehicle data at varying speeds, multi-vehicle sessions, and the full final four-vehicle race. The dataset contains almost 30,000 professionally annotated LiDAR point clouds, along with RADAR point clouds. In particular, it is the first large-scale dataset in autonomous racing to feature professionally annotated LiDAR point clouds, enabling deep learning-based perception research. The data is provided in a developer-friendly format, enabling easy implementation and evaluation in future research. We provide implementation and evaluation for off-the-shelf 3D detection and tracking methods. Although baseline methods show promising results for both 3D detection and tracking, specialized methods are required to address the unique challenges of high-speed autonomous driving. For a detailed description of the dataset, please visit the A2RL Vmax Dataset Website

Marvin Klemp, Dominic Ebner, Cornelius Schröder +15

Institute of Measurement and Control Systems, Karlsruhe Institute of Technology · Code 19 Racing · Institute of Automotive Technology, Technical University of Munich +8

Autonomous Driving Perception Autonomous Racing 3D Object Detection Multi-Object Tracking

Ask PDF

Jun 2, 2026

StandardE2E: A Unified Framework for End-to-End Autonomous Driving Datasets

Autonomous driving has shifted from modular perception-prediction-planning stacks toward end-to-end (E2E) models that map sensor inputs directly to vehicle control, often regularized by auxiliary tasks such as 3D detection, motion forecasting, and HD-map perception. Progress is driven by a fast-growing ecosystem of sensor-rich driving datasets, yet each ships its own file formats, APIs, coordinate conventions, and modality coverage, leaving cross-dataset experimentation and even basic per-dataset preprocessing to be re-implemented per project. We present StandardE2E, a framework that provides a single unified interface over E2E driving datasets. StandardE2E (i) standardizes per-dataset preprocessing under one shared data schema; (ii) combines multiple datasets in a single PyTorch DataLoader for cross-dataset pretraining, auxiliary-task supervision, and scenario-level filtering; and (iii) reduces adding a new dataset to a single per-dataset mapping from raw frames to the canonical schema, leaving the entire downstream pipeline unchanged. The framework supports six datasets out of the box: Waymo End-to-End, Waymo Perception, Argoverse 2 Sensor, Argoverse 2 LiDAR, NAVSIM (OpenScene-v1.1), and WayveScenes101, and is released as the open-source standard-e2e Python package, available at https://github.com/stepankonev/StandardE2E.

Stepan Konev

Autonomous Driving Datasets End-to-End Autonomous Driving Autonomous Driving

Ask PDF