Energy-Aware Imitation Learning for Steering Prediction Using Events and Frames
Authors: Hu Cao, Jiong Liu, Xingzhuo Yan, Rui Song, Yan Xia, Walter Zimmer, Guang Chen, Alois Knoll
Organizations: School of Automation, Southeast University, Sipailou 2, Nanjing, 210096, China. · Chair of Robotics, Artificial Intelligence and Real-time Systems, Technical University of Munich, Boltzmannstraße 15, Munich, 85748, Germany. · School of Computer Science and Technology, Tongji University, Cao’an Road, Shanghai, 200070, China.
In autonomous driving, relying solely on frame-based cameras can lead to inaccuracies caused by factors like long exposure times, high-speed motion, and challenging lighting conditions. To address these issues, we introduce a bio-inspired vision sensor known as the event camera. Unlike conventional cameras, event cameras capture sparse, asynchronous events that provide a complementary modality to mitigate these challenges. In this work, we propose an energy-aware imitation learning framework for steering prediction that leverages both events and frames. Specifically, we design an Energy-driven Cross-modality Fusion Module (ECFM) and an energy-aware decoder to produce reliable and safe predictions. Extensive experiments on two public real-world datasets, DDD20 and DRFuser, demonstrate that our method outperforms existing state-of-the-art (SOTA) approaches. The codes and trained models will be released upon acceptance.
Figures & tables
Figure 1: Performance comparison on the DDD20 and DRFuser datasets. Our proposed method achieves SOTA performance, outperforming previous approaches EyEF Zhou et al. (2025) , CAFR Cao et al. (2024) , DRFuser Munir et al. (2023) , and EFNet Sun et al. (2022) in terms of RMSE and MAE metrics.
Figure 2: The proposed model architecture consists of three main components: a dual-stream backbone network, ECFM modules, and an energy-aware decoder. The backbone network comprises two branches: the event-based ResNet at the bottom and the frame-based ResNet He et al. (2016) at the top. Each ECFM module operates to enhance features across different hierarchical scales. Subsequently, the energy-aware decoder is used to perform steering angle predictions.
Figure 3: The architecture of the proposed ECFM. 3D weights are generated based on energy function. In ECFM, fused features Fffused and Fefused from the frame and event branches are concatenated and then passed through a Conv1×1 layer to produce the final output, Fo . Notably, this Conv1×1 layer is the only learnable component within ECFM.
Architecture
Layers
Input Dimension
Output Dimension
Decoder with ResNet-50
Convolution 2D
2048
1024
Batch normalization
1024
1024
ReLU
1024
1024
Convolution 2D
1024
512
Batch normalization
512
512
Dropout
512
512
Table 1: The detailed architecture of the decoder in the proposed network.
Method
Model Type
Params (M, ↓ )
FLOPs (G, ↓ )
FPS ( ↑ )
RMSE ( ↓ )
MAE ( ↓ )
Bojarski et al. Bojarski (2016)
Frames only
-
-
-
0.1574
-
CNN-LSTM Xu et al. (2017)
-
-
-
0.1429
-
Maqueda et al. Maqueda et al. (2018)
Events Only
-
-
-
0.0716
-
Hu et al. Hu et al. (2020)
Frames + Events
-
-
-
0.0721
-
FAGC Cao et al. (2021)
312.3
88.0
55.5
0.0501
0.0317
EFNet Sun et al. (2022)
139.5
36.8
62.1
0.0430
0.0297
Table 2: Comparison with SOTA methods on the DDD20 dataset. ↑ indicates metrics where higher values signify better performance, and ↓ indicates metrics where lower values are the better.
Method
Model Type
Params (M, ↓ )
FLOPs (G, ↓ )
FPS ( ↑ )
RMSE ( ↓ )
MAE ( ↓ )
FAGC Cao et al. (2021)
Frames + Events
312.3
88.0
56.4
0.1997
0.0240
EFNet Sun et al. (2022)
139.5
36.8
62.3
0.2166
0.0241
DRFuser Munir et al. (2023)
89.3
28.2
35.6
0.2919
0.0396
CAFR Cao et al. (2024)
100.4
24.8
62.5
0.2209
0.0346
EyEF Zhou et al. (2025)
72.7
17.0
46.1
0.1032
0.0130
Ours
76.2
38.4
47.2
0.0801
0.0102
Table 3: Comparison with SOTA methods on the DRFuser dataset. ↑ indicates metrics where higher values signify better performance, and ↓ indicates metrics where lower values are the better.
Figure 4: A quantitative comparison of our proposed method with EyEF Zhou et al. (2025) and DRFuser Munir et al. (2023) on the DDD20 and DRFuser datasets.
Figure 5: The visualization of steering angle prediction in comparison to ground truth on DDD20 (top) and DRFuser (bottom) datasets. Each pair of samples consists of frame data on the left and event data on the right.
Figure 6: Performance vs. model size on the DRFuser dataset. Our proposed method achieves a new SOTA performance while utilizing significantly less number of parameters than previous methods.
Method
RMSE ( ↓ )
MAE ( ↓ )
Events only
0.5102
0.1501
Frames only
0.4609
0.1093
Add directly
0.3499
0.0434
Additive attention
0.2986
0.0403
Ours
0.0801
0.0102
Table 4: Quantitative comparison with different ablation settings. “Add directly” (fusing events and frames by directly using addition operations); “Additive attention” (fusing events and frames by using basic attention operations; see the Appendix for more details).
Integrate
Energy Loss
RMSE
MAE
✗
✗
0.1016
0.0122
✓
✗
0.0909
0.0120
✗
✓
0.0898
0.0108
✓
✓
0.0801
0.0102
Table 5: Quantitative comparison with different decoder designs.
Figure 7: The addition baseline model consists of three main components: a dual-stream backbone network, addition modules, and a prediction decoder.
Figure 8: The architecture of the additive attention.
Figure 9: The additive attention baseline model consists of three main components: a dual-stream backbone network, additive attention modules, and a prediction decoder.
Figure 10: Representative examples of different activation maps on the DRFuser dataset are: (a) event activation map before ECFM; (b) RGB activation map before ECFM; and (c) fused activation map after ECFM.
Autonomous driving develops vehicles that perceive road scenes, predict interactions, and plan and control safe movement with limited or no human input.