cs.CVNov 9, 2026

Leveraging Complementary Event Frame and Voxel Representations for RGB-Guided Multimodal Pre-training

Authors: Wentao Wu, Xiao Wang, Yi Wang, Chenglong Li, Yonghong Tian, Bin Luo

Organizations: School of Artificial Intelligence, Anhui University, Hefei, China · School of Computer Science and Technology, Anhui University, Hefei, China · Peng Cheng Laboratory, Shenzhen, China, School of Computer Science, Peking University, China, and Shenzhen Graduate School, Peking University, China

Abstract

In this paper, we propose CM3AE 2.0, a unified RGB Frame and Event-Voxel/-Frame pre-training framework. By jointly learning multimodal spatiotemporal representations and exploiting latent cross-modal dependencies through cross-modal generation, the framework enhances multimodal information interaction and collaborative modeling capabilities. Specifically, we first propose an Event-aware Spatiotemporal Masked Reconstruction (ESMR) strategy to learn unified multimodal spatiotemporal representations. It leverages event responses to guide masking toward dynamically salient regions. Through spatiotemporal masked reconstruction, it learns spatial structures and their temporal variations from continuous RGB-Event observations. In addition, it incorporates multimodal fusion reconstruction and cross-modal contrastive learning to facilitate the modeling of cross-modal complementary information and the learning of shared representations. Building upon this, we further propose a Hierarchical Cross-modal Generation (HCG) strategy to learn latent cross-modal dependencies through visual and language generation. At the visual level, Event-conditioned RGB generation is performed using a pre-trained diffusion model to capture visual associations between the two modalities. At the language level, video-level text description generation based on joint RGB-Event representations establishes connections between multimodal visual features and linguistic semantics, further enriching the learning of cross-modal dependencies. Extensive experiments on four diverse downstream tasks demonstrate that CM3AE 2.0 consistently improves performance, validating the effectiveness and generalization capability of the proposed pre-training framework.

Figures & tables

Explore similar work

CardsList
  1. REALM: An RGB- and Event-Aligned Latent Manifold for Cross-Modal Perception

    Apr 30, 2026Vincenzo Polizzi, David B. Lindell, Jonathan KellyMultimodal Foundation ModelsEvent-Based Vision

  2. CMTFormer: Marrying Transformer with Hierarchical Information Interaction for RGB-Event Object Detection

    Jun 28, 2026Yu Li, Yuenan Hou, Yingmei Wei +2Event-Based VisionEvent-Based Object Detection

  3. Hyper-FEOD: Sparse Hypergraph-Enhanced Frame-Event Object Detection with Fine-Grained MoE

    Apr 13, 2026Wei Bao, Yuehan Wang, Tianhang Zhou +1Event-Based Object DetectionMultimodal Fusion