Toward Deep Representation Learning for Event-Enhanced Visual Autonomous Perception: the eAP Dataset

Jinghang Li; Shichao Li; Qing Lian; Peiliang Li; Xiaozhi Chen; Yi Zhou

Datasets	Year	Duration (in hours)	#3D Bbox	TTC Anno	#Class	#Resolution (RGB)	#Resolution (Event)	Aim	Data source	Scenarios
MVSEC [ 14 ]	2018	0.5	-	-	-	$752\times 480$	$346\times 260$	Eval	Collected	Urban
DDD20 [ 15 ]	2020	51	-	-	-	$346\times 260$	$346\times 260$	Train&Eval	Collected	Expressways, Highways, Urban
DSEC-Det [ 4 ]	2021	1	-	-	6	$1440\times 1080$	$640\times 480$	Train&Eval	Collected	Urban, Suburban, Rural areas
VECtor [ 16 ]	2022	0.36	-	-	-	$1224\times 1024$	$640\times 480$	Eval	Collected	Indoor, School
M3ED [ 17 ]	2023	3.42 (car 2) 1	-	-	11	$1280\times 800$	$1280\times 720$	Train&Eval	Collected	Urban, Forest
DSEC-3DOD [ 18 ] 2	2025	0.15	✓	-	2	$1440\times 1080$	$640\times 480$	Eval	Collected	Urban, Suburban, Highways

TABLE I: A comprehensive overview of recent collision prediction and TTC estimation datasets. eAP features a large dataset scale, diverse scenes, high resolution, a narrow-baseline setup for cross-modal tasks, along with object-level 3D bounding box and TTC annotations.

Fig. 0: Sensor configuration for the eAP dataset. The event camera and RGB camera are rigidly mounted with a narrow baseline of 3 cm, as highlighted in the zoomed-in image. The positions and coordinate systems of the sensors are depicted in the bird’s eye view.

Fig. 1: Projected point cloud on the RGB image after using Calib-Anything [ 56 ] for calibrating the extrinsic parameters between the LiDAR and the RGB camera. Different colors represent the distinct clusters identified during the clustering process in [ 56 ] . The ground points are omitted for clearer visualization.

Fig. 3: (Top) Bird’s eye view (BEV) visualization of the LiDAR point cloud with annotated 3D bounding boxes demonstrating the 3D annotation precision; (Bottom) Plot of velocity curves of exemplar object trajectories.

Fig. 4: Exemplar frames, event data, and object TTC annotations from the eAP dataset under varying weather, location, time, and illumination conditions. The 2D bounding boxes show projections of the annotated 3D boxes, along with categories and ground truth TTC values. Event data demonstrates superior quality in over-exposed and low-light scenes, highlighting the benefits of the event modality. Bounding boxes of objects with high visibility are visualized, and a complete visualization of all boxes is included in the supplementary video.

Fig. 5: Visualization of additional ground-truth annotation types planned for future releases of the eAP dataset, including (A) LiDAR point clouds, (B) depth maps, (C) optical flow, (D) 2D annotations, (E) dense TTC maps, and (F) segmentation maps.

Fig. 6: Statistics of the eAP dataset. Left: distribution of the annotated objects with respect to category and weather. Right: The TTC ground truth distribution shown across the eight object classes and under the four different lighting conditions.

Region	Distance [km]	Illumination	Sequences (Train/Test)	Time [min] (Train/Test)
Highways	178.44	Sunny Cloudy Twilight	13/3 10/1 7/2	65/15 50/5 35/10
Urban	52.61 km	Sunny Cloudy Twilight Night	5/1 4/1 1/1 5/2	25/5 20/5 5/5 25/10
Low-light	5.01 km	Night	1/1	5/5
Total	236.06 km	–	58	290

TABLE III: Statistics of the eAP dataset, categorized by region and illumination. The table details the total traveled distance [km], along with the train/test split for both sequence counts and duration [min].

Fig. 7: Diagram of the BEV 3D vehicle detection model. The RGB and event encoders extract FV features from the input images and events. The view transform module builds a BEV scene representation based on the FV features. For the event-enhanced model, the FV features extracted from RGB images and events are fused via concatenation. For the temporal model, the BEV features from past frames are warped to the current frame and concatenated with the BEV feature extracted from the current frame. An anchor-based detection head predicts the detected vehicles from the BEV feature.

Fig. 8: Architecture of (a) the baseline model for direct TTC regression, (b) the height-ratio network leveraging TTC geometry, (c) a late-fusion strategy, and (d) an extended version incorporating object foreground estimation during training, supervised by a teacher model.

Fig. 9: Diagram illustrating the change in visual object height from time

t_{1}

to

t_{2}

as the object moves closer to the camera.

Fig. 10: Qualitative visualization of 3D vehicle detection performance on the eAP testing set. The figure presents two representative sequences (Frames A-C and D-F). The first three columns compare the detection results generated from different inputs: (a) Visual-only, (b) Event-only, and (c) Our proposed event-enhanced multi-model. The fourth column (d) provides a Birds-Eye-View (BEV) visualization, comparing the predicted boxes with the ground truth. Best viewed in color and zoomed in for more details.

Methods	Modality	Frame Number	Driving				HDR
Methods	Modality	Frame Number	AP $~\uparrow$	ATE $~\downarrow$	ASE $~\downarrow$	AOE $~\downarrow$	AP $~\uparrow$	ATE $~\downarrow$	ASE $~\downarrow$	AOE $~\downarrow$
Visual-only	V	1	0.510	0.497	0.075	0.021	0.403	0.584	0.018	0.087
Event-only	E	1	0.200	0.794	0.096	0.023	0.138	0.872	0.118	0.031
Fusion	V+E	1	0.515	0.482	0.076	0.021	0.460	0.503	0.082	0.020
Fusion-temporal	V+E	3	0.531	0.400	0.072	0.018	0.558	0.363	0.085	0.015
CenterNet3D Visual-only	V	1	0.423	0.495	0.099	0.027	0.389	0.511	0.099	0.023

TABLE IV: Quantitative evaluation of 3D vehicle detection performance on the eAP testing set for the Car category using our anchor-based model and another anchor-free model, CenterNet3D [ 68 ] . V: Visual (RGB), E: Event, V+E: Visual (RGB) + Event. Best and second best results are highlighted and underlined , respectively. Models trained using different input modalities and numbers of input frames are compared in different driving scenes. Driving indicates normal scenes while HDR indicates driving scenes with challenging illumination.

Methods	Type	Modality	MiD ${}_{\text{c}}\downarrow$ (FR $\downarrow$ )	MiD ${}_{\text{s}}\downarrow$ (FR $\downarrow$ )	MiD ${}_{\text{l}}\downarrow$ (FR $\downarrow$ )	MiD ${}_{\text{n}}\downarrow$ (FR $\downarrow$ )	RTE ${}_{\text{c}}\downarrow$	RTE ${}_{\text{s}}\downarrow$	RTE ${}_{\text{l}}\downarrow$	RTE ${}_{\text{n}}\downarrow$
FAITH [ 27 ]	M	E	606.8 (1.2%)	490.8 (0.9%)	319.0 (0.9%)	376.4 (0.5%)	282.7	311.8	515.7	428.9
ETTCM ${}_{\text{scaling-3}}$ [ 30 ]	M	E	402.2 (0.0%)	279.5 (0.0%)	263.9 (0.1%)	207.5 (0.0%)	414.5	181.6	288.2	132.9
ETTCM ${}_{\text{six-dof-3}}$ [ 30 ]	M	E	226.1 (0.0%)	326.2 (0.0%)	321.6 (0.0%)	223.8 (0.0%)	222.6	17703.4	784.8	660.1
CMax [ 26 ]	M	E	632.8 (6.7%)	1583.6 (5.5%)	1528.0 (9.6%)	1187.0 (7.9%)	48.2	107.3	132.4	235.7
STRTTC [ 23 ]	M	E	237.2 (0.0%)	532.9 (2.7%)	954.3 (21.6%)	348.9 (5.3%)	37.3	54.1	209.4	48.3
Garl-TTC (Ours)	L	E+V	53.1 (0.0%)	37.6 (0.0%)	40.6 (0.0%)	31.3 (0.0%)	16.6	20.0	34.1	28.2

TABLE V: Comparison of Motion-in-depth Error (MiD) for different ranges of TTC ground truth on the eAP testing set. Failure Ratio (FR) indicates the proportion of invalid TTC predictions of previous approaches that fall out of the evaluation range. E: Event, V+E: Visual (RGB) + Event, M: Model-based, L: Learning-based. Best and second best results are highlighted and underlined , respectively.

Fig. 11: TTC estimation results of our approach for different object trajectories on the eAP dataset. Input frames and event data are shown for several timestamps of the trajectory. Our predicted and ground-truth TTC curves are shown in blue and red, respectively. Best viewed in color.

Method	Type	Modality	CCRs2-medium		CCRs2-high		CCRm-medium		Average
Method	Type	Modality	RTE (%) $\downarrow$	Runtime (ms) $\downarrow$	RTE (%) $\downarrow$	Runtime (ms) $\downarrow$	RTE (%) $\downarrow$	Runtime (ms) $\downarrow$	RTE (%) $\downarrow$	Runtime (ms) $\downarrow$
FAITH [ 27 ]	M	E	47.45	191	49.62	165	56.26	110	51.11	155
ETTCM [ 30 ]	M	E	52.84	98	49.46	127	58.00	121	53.43	115
CMax [ 26 ]	M	E	11.20	3449	11.10	3826	14.73	3657	12.34	3644
STRTTC [ 23 ]	M	E	11.83	24	13.68	28	10.36	23	11.96	25
Garl-TTC (Ours)	L	E+V	8.31	13	10.56	13	12.93	12	10.60	13

TABLE VI: Performance of multiple TTC estimation algorithms on the EvTTC benchmark [ 22 ] . E: Event, V+E: Visual (RGB) + Event, M: Model-based, L: Learning-based. Best and second best results are highlighted and underlined , respectively.

Fig. 12: Comparison of predicted TTC with ground truth on the FCWD benchmark [ 23 ] without model fine-tuning.

		FCWD1 ${}_{\text{RTE}}$ $\downarrow$	FCWD2 ${}_{\text{RTE}}$ $\downarrow$	FCWD3 ${}_{\text{RTE}}$ $\downarrow$	eAP ${}_{\text{MiD}}$ $\downarrow$
Event-based	ETTCM ${}_{\text{6-dof}}$ [ 30 ]	15.5	18.4	19.0	265.4
Event-based	STRTTC [ 23 ]	9.8	11.5	14.0	408.7
Frame-based	DeepScale [ 21 ]	25.4	19.6	21.7	81.9
Ours	Visual-only baseline	48.5	47.6	46.8	160.6
	Visual-only + LHR	13.3	16.7	16.6	68.3
	Ours (Full)	5.2	6.1	5.4	45.0

TABLE VII: Quantitative evaluation of object TTC estimation performance on the FCWD benchmark [ 23 ] and the eAP dataset.

Experimental Setting	Input Modality		Approach		Fusion Strategy		MiD ${}_{\text{c}}\downarrow$	MiD ${}_{\text{s}}\downarrow$	MiD ${}_{\text{l}}\downarrow$	MiD ${}_{\text{n}}\downarrow$	MiD ${}_{\text{overall}}\downarrow$
Experimental Setting	RGB	Event	LHR	FS	Early	Late	MiD ${}_{\text{c}}\downarrow$	MiD ${}_{\text{s}}\downarrow$	MiD ${}_{\text{l}}\downarrow$	MiD ${}_{\text{n}}\downarrow$	MiD ${}_{\text{overall}}\downarrow$
Visual-only baseline	✓	✗	✗	✗	-	-	222.9	105.7	93.9	80.4	160.6
Event-only baseline	✗	✓	✗	✗	-	-	93.9	65.2	89.6	42.4	79.7
Multi-modal baseline	✓	✓	✗	✗	✓	✗	164.9	97.6	117.5	66.1	130.1
Visual-only + LHR	✓	✗	✓	✗	-	-	95.6	47.4	30.5	32.4	68.3
Event-only + LHR	✗	✓	✓	✗	-	-	90.5	44.8	43.0	32.2	66.2

TABLE VIII: Motion-in-depth error (MiD) on the testing split for different ranges of TTC ground truth on the eAP testing set. In each experiment, ✓ indicates a proposed technique is used and ✗ indicates otherwise. LHR: height-ratio-aware representation learning. FS: foreground supervision.

Fig. 13: The predicted mask of our student model (blue) and that of SAM [ 67 ] (red) used as ground truth for supervision.

Model	Device	Module	Avg. (ms)	Total (ms)
Garl-TTC	NVIDIA A100 (PyTorch)	RGB Encoder	7.11	21.05
		Event Encoder	7.08
		Height Head	0.15
	NVIDIA Orin NX16G (ONNX, MaxN)	All Modules	4.55	4.55
3D Det.	NVIDIA A100 (PyTorch)	RGB Encoder	27.48	125.07
		Event Encoder	23.35

TABLE IX: Runtime analysis of different model components (in ms).

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Toward Deep Representation Learning for Event-Enhanced Visual Autonomous Perception: the eAP Dataset

Abstract

Figures & tables

Autonomous Driving Perception

EventDrive: Event Cameras for Vision-Language Driving Intelligence

Think as Needed: Geometry-Driven Adaptive Perception for Autonomous Driving

Efficient Multi-Timescale Event Representations for Feed-Forward Object Detection

Abstract

Autonomous Driving Perception

Explore similar work

EventDrive: Event Cameras for Vision-Language Driving Intelligence

Think as Needed: Geometry-Driven Adaptive Perception for Autonomous Driving

Efficient Multi-Timescale Event Representations for Feed-Forward Object Detection