Leveraging Complementary Event Frame and Voxel Representations for RGB-Guided Multimodal Pre-training
Authors: Wentao Wu, Xiao Wang, Yi Wang, Chenglong Li, Yonghong Tian, Bin Luo
Organizations: School of Artificial Intelligence, Anhui University, Hefei, China · School of Computer Science and Technology, Anhui University, Hefei, China · Peng Cheng Laboratory, Shenzhen, China, School of Computer Science, Peking University, China, and Shenzhen Graduate School, Peking University, China
In this paper, we propose CM3AE 2.0, a unified RGB Frame and Event-Voxel/-Frame pre-training framework. By jointly learning multimodal spatiotemporal representations and exploiting latent cross-modal dependencies through cross-modal generation, the framework enhances multimodal information interaction and collaborative modeling capabilities. Specifically, we first propose an Event-aware Spatiotemporal Masked Reconstruction (ESMR) strategy to learn unified multimodal spatiotemporal representations. It leverages event responses to guide masking toward dynamically salient regions. Through spatiotemporal masked reconstruction, it learns spatial structures and their temporal variations from continuous RGB-Event observations. In addition, it incorporates multimodal fusion reconstruction and cross-modal contrastive learning to facilitate the modeling of cross-modal complementary information and the learning of shared representations. Building upon this, we further propose a Hierarchical Cross-modal Generation (HCG) strategy to learn latent cross-modal dependencies through visual and language generation. At the visual level, Event-conditioned RGB generation is performed using a pre-trained diffusion model to capture visual associations between the two modalities. At the language level, video-level text description generation based on joint RGB-Event representations establishes connections between multimodal visual features and linguistic semantics, further enriching the learning of cross-modal dependencies. Extensive experiments on four diverse downstream tasks demonstrate that CM3AE 2.0 consistently improves performance, validating the effectiveness and generalization capability of the proposed pre-training framework.
Figures & tables
Fig. 1: Comparison of existing pre-training paradigms and CM3AE 2.0. (a) Representative Event-related pre-training paradigms. (b) Overview of the proposed CM3AE 2.0 framework. (c) Performance comparisons on action recognition and event-guided low-light image enhancement.
Fig. 2: Overview of the proposed CM3AE 2.0 framework, comprising two stages: (1) Event-aware Spatiotemporal Masked Reconstruction (ESMR), which learns unified multimodal spatiotemporal representations by modeling cross-frame dynamics and exploiting complementary information and shared features across modalities; and (2) Hierarchical Cross-modal Generation (HCG), which explores latent cross-modal dependencies through Event-conditioned RGB Image Generation and Video-level Text Generation.
Method
Dataset
Action Recognition
Object Detection
Visual Object Tracking
(HARDVS Dataset [ 2 ] )
(DSEC-Det Dataset [ 59 ] )
(COESOT Dataset [ 58 ] )
Top-1
Top-5
AP[0.5:0.95]
AP0.5
AP0.75
SR
PR
NPR
Scratch
-
41.63
50.66
38.6
58.7
43.5
38.9
44.7
46.4
iBOT [ 10 ]
ImageNet-1K
45.12
51.36
40.9
62.5
45.1
58.3
71.3
70.3
BEiT v2 [ 12 ]
ImageNet-1K
45.01
51.76
41.3
63.3
46.5
43.1
51.6
53.1
EVA [ 60 ]
ImageNet-1K
48.14
55.06
41.1
62.5
45.7
58.1
70.7
70.5
TABLE I: Comparison with existing pre-trained models across three RGB-Event downstream tasks, including action recognition, object detection, and visual object tracking.
Method
Dataset
Action Recognition
Object Detection
Visual Object Tracking
Event-guided Image Enhancement
(HARDVS Dataset [ 2 ] )
(EvDET200K Dataset [ 31 ] )
(EventVOT Dataset [ 32 ] )
(SDE-in Dataset [ 34 ] )
Top-1
Top-5
AP[0.5:0.95]
AP0.5
AP0.75
SR
PR
NPR
SSIM
PSNR
PSNR*
Scratch
-
20.23
37.74
42.2
73.0
44.0
44.8
41.8
57.9
70.2
20.2
23.2
iBOT [ 10 ]
ImageNet-1K
22.99
39.99
46.2
76.3
49.1
55.7
59.7
70.6
74.2
21.6
24.1
BEiT v2 [ 12 ]
ImageNet-1K
26.61
43.52
46.6
78.4
48.4
51.1
52.0
65.0
73.8
21.5
24.0
EVA [ 60 ]
ImageNet-1K
36.44
50.96
47.5
78.7
50.7
56.0
60.3
71.1
74.6
21.5
24.2
TABLE II: Comparison with existing pre-trained models across four event-based downstream tasks, including action recognition, object detection, visual object tracking, and event-guided low-light image enhancement.
Fig. 3: Representative samples in our pre-training dataset.
Method
Stage
Action Recognition
Object Detection
Visual Object Tracking
Event-guided Image Enhancement
(HARDVS Dataset)
(DSEC-Det Dataset)
(COESOT Dataset)
(SDE-in Dataset)
Top-1
Top-5
AP[0.5:0.95]
AP0.5
AP0.75
SR
PR
NPR
SSIM
PSNR
PSNR ∗
Baseline
1
49.29
59.99
41.1
62.7
46.3
58.7
71.4
70.7
72.9
21.5
24.1
+ Multimodal Fusion Reconstruction + and Cross-modal Contrastive Learning
1
50.51
61.39
42.4
63.6
47.8
60.4
72.9
72.6
73.9
21.9
24.2
+ Event-guided Masking
1
51.47
62.41
42.8
64.2
48.1
61.0
73.7
73.4
74.5
22.2
24.3
+ Spatiotemporal Masked Reconstruction
1
52.76
64.86
43.8
65.1
49.0
62.3
75.5
74.7
75.1
22.3
24.6
TABLE III: Ablation results of the proposed pre-training framework on four representative downstream tasks.
Fig. 4: The impact of pre-training epochs on experimental results across different downstream tasks.
Fig. 5: Visualization of masked image reconstruction results using MAE, CM3AE, and CM3AE 2.0.
Fig. 6: Qualitative visualization of video-level text generation.
Fig. 7: Qualitative visualization of RGB images generated from Event-conditioned visual representations by CM3AE 2.0.
Fig. 8: Visualization of the experimental results of each downstream task.
Mask Ratio
Action Recognition
RGB+Event
Event
Top-1
Top-5
Top-1
Top-5
0.50
52.62
64.67
52.28
63.83
0.75
53.95
66.27
53.26
64.58
0.90
51.78
62.83
51.03
61.57
TABLE IV: Effect of different masking ratios on action recognition.
Frames
Action Recognition
RGB-Event
Event
Top-1
Top-5
Top-1
Top-5
2
52.13
64.85
52.27
63.74
4
53.95
66.27
53.26
64.58
8
54.03
66.14
53.19
64.52
TABLE V: Effect of different numbers of input frames on action recognition.
Fig. 9: Visualization of attention maps on different downstream tasks.
Method
Pre-training Data
Action Recognition
Object Detection
Visual Object Tracking
Low-Light Enhancement
(HARDVS Dataset)
(DSEC-Det Dataset)
(COESOT Dataset)
(SDE-in Dataset)
Top-1
Top-5
AP[0.5:0.95]
AP0.5
AP0.75
SR
PR
NPR
SSIM
PSNR
PSNR ∗
Scratch
-
5.63
14.52
17.1
29.2
17.2
26.3
26.9
30.1
59.4
19.9
20.1
MAE [ 9 ]
ImageNet-1K
6.65
18.04
21.1
34.2
21.4
40.1
48.7
50.7
61.3
20.2
21.3
MAE [ 9 ]
REV2M
33.31
49.01
23.1
35.8
23.7
41.5
50.8
52.3
63.2
20.7
21.7
CM3AE [ 25 ]
REV2M
43.68
57.23
26.6
39.8
26.1
43.9
55.5
56.0
63.8
21.1
22.2
TABLE VI: Few-shot transfer results of different pre-trained models on multiple downstream tasks using 10% of the labeled training data for fine-tuning.
Configuration
FLOPs (G)
MACs (G)
Params (M)
MAE
287.42
143.71
335.82
CM3AE
391.56
195.78
350.00
CM3AE 2.0: Stage 1 only
404.81
202.41
357.09
+ text generation
810.14
405.07
8389.46
+ image generation
1682.85
841.43
1300.66
+ both (full model)
2088.18
1044.09
9333.03
TABLE VII: Comparison of computational complexity and model size of different pre-training configurations.
Multimodal pretraining learns reusable representations by training models on paired or mixed data from multiple modalities, such as text, images, audio, and signals.
aLaboratory for Big Data and Decision, National University of Defense Technology, Changsha, Hunan, China · bShanghai AI Laboratory, Shanghai, Shanghai, China
BNRist, THUIBCS, BLBCI, School of Software, Tsinghua University · State Key Laboratory of Heavy Oil Processing, China University of Petroleum (Beijing), Beijing, 102249, China