Generative Event Pretraining with Foundation Model Alignment

Jianwen Cao; Jiaxu Xing; Nico Messikommer; Davide Scaramuzza

cs.CVMar 24, 2026

Generative Event Pretraining with Foundation Model Alignment

Authors: Jianwen Cao, Jiaxu Xing, Nico Messikommer, Davide Scaramuzza

Organizations: Robotics and Perception Group, University of Zurich

Abstract

Event cameras provide robust visual signals under fast motion and challenging illumination conditions thanks to their microsecond latency and high dynamic range. However, their unique sensing characteristics and limited labeled data make it challenging to train event-based visual foundation models (VFMs), which are crucial for learning visual features transferable across tasks. To tackle this problem, we propose GEP (Generative Event Pretraining), a two-stage framework that transfers semantic knowledge learned from internet-scale image datasets to event data while learning event-specific temporal dynamics. First, an event encoder is aligned to a frozen VFM through a joint regression-contrastive objective, grounding event features in image semantics. Second, a transformer backbone is autoregressively pretrained on mixed event-image sequences to capture the temporal structure unique to events. Our approach outperforms state-of-the-art event pretraining methods on a diverse range of downstream tasks, including object recognition, segmentation, and depth estimation. Together, VFM-guided alignment and generative sequence modeling yield a semantically rich, temporally aware event model that generalizes robustly across domains.

Figures & tables

Figure 1: Overall comparison across ten metrics on various datasets. With the same backbone model, our method demonstrates superior and consistent performance across all tasks.

Figure 2: The overall two-stage framework. (a) Alignment stage: Event frames and synchronized images are encoded by an event encoder and a frozen VFM encoder. The event encoder is optimized with regression, contrastive, and preservation terms to match the semantic structure of image features. (b) Autoregressive pretraining stage: Aligned event and image embeddings are interleaved into a single sequence and processed by a causal transformer, which predicts future slices from partial windows, learning long-range temporal structure and cross-modal consistency. Arrows indicate causal dependencies; only a subset is shown for visual clarity.

Training from scratch
Method	Backbone	Dataset	Ep.	N-ImageNet		N-Caltech101
				acc1 $\uparrow$	acc5 $\uparrow$	acc1 $\uparrow$	acc5 $\uparrow$
ViT [ 11 ]	ViT-S/16	N-ImageNet	300	46.70	69.89	55.63	–
Specific trained
EST [ 16 ]	–	–	–	48.93	–	–	–
Self-supervised pretraining

Table 1: Object recognition on N-ImageNet and N-Caltech101. We report top-1 (acc1) and top-5 (acc5) accuracies. The two best-performing methods for each evaluation metric are highlighted in green and orange .

Training from scratch
Method	Backbone	Dataset	Ep.	DDD17		DSEC
				mIoU $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$	mAcc $\uparrow$
ViT [ 11 ]	ViT-S/16	–	–	36.65	46.21	32.66	40.84
Specific trained
ESS [ 52 ]	–	Cityscape	50	61.37	70.87	53.30	62.94
Self-supervised pretraining

Table 2: Semantic segmentation on DDD17 and DSEC. We report mean IoU (mIoU) and mean ACC (mAcc). The two best-performing methods for each evaluation metric are highlighted in green and orange.

Method	Backbone	Dataset	Ep.	Abs $\downarrow$	RMS $\downarrow$
Specific trained
HMNet [ 20 ]	–	–	–	4.61	8.60
Self-supervised pretraining
BeiT [ 1 ]	ViT-B/16	ImageNet-1K	800	4.40	7.56
MAE [ 22 ]	ViT-B/16	ImageNet-1K	800	4.45	7.60
DINOv2 [ 39 ]	ViT-S/16	LVD-142M	–	4.45	7.65

Table 3: Depth estimation on MVSEC. We report absolute error (Abs↓) and root mean squared error (RMS↓). The two best-performing methods for each evaluation metric are highlighted in green and orange.

Figure 3: Visualization of a 16-frame context (blue) and a 32-frame autoregressively generated future (red) on validation interleaved event and image streams. Green boxes highlight consistent motion. Insets enlarge the first context and last generated frames for clarity.

Figure 4: Qualitative results on the DSEC validation set. The RGB images are shown only for visual reference and are not used by the model.

Figure 5: Attention map response on event camera data.

Figure 6: t-SNE visualization of ViT-Base encoder features (20 classes, 50 samples/class from N-ImageNet validation set). Left (a): Before Align ; right (b): After Align . Each color denotes a semantic class.

	ViT-Small		ViT-Base
Metric	Dinov2	Aligned	Dinov2	Aligned
Silhouette [ 49 ] $\uparrow$	-0.03	0.19	0.03	0.27
Davies–Bouldin [ 9 ] $\downarrow$	5.24	2.21	3.75	2.04
Calinski–Harabasz [ 5 ] $\uparrow$	9.39	33.21	14.97	48.56

Table 4: Clustering metrics on encoder features (20 classes, 50 samples/class). Higher is better for Silhouette and Calinski–Harabasz; lower is better for Davies–Bouldin.

Metric	Distance Reduction	ImageNet Acc. (%)
MSE	2.22→0.78 (-64.7%)	31.18→61.45
InfoNCE	2.28→0.57 (-75.0%)	31.18→60.99
Cosine	0.70→0.27 (-61.8%)	31.18→61.80
KL Diver.	0.48→0.15 (-67.5%)	31.18→59.20
Attention	38.76→19.42 (-49.9%)	31.18→56.72
Cosine + InfoNCE	1.40→0.36 (-74.3%)	31.18→ 62.73

Table 5: Feature alignment results on N-ImageNet using 30,000 training steps. Each cell shows Original→Aligned with relative change (%).

Variant	Align	Parad.	DSEC (mIoU / mAcc)
Ours (2 stages)	✓	AR	63.27 / 73.13
Event-only AR (Stage 2)	✓	AR	63.51 / 72.11
Ours - pretrain	✓	None	61.97 / 71.83
Ours - alignment	✗	AR	52.38 / 60.97
Ours - alignment + finetune	✗	AR	58.24 / 66.82
Ours + aggregator [ 30 ]	✓	AR	61.24 / 71.53

Table 6: Ablation on alignment and training paradigm (frozen backbone).

Setting	Value
optimizer	AdamW
Batch size	32
Learning rate	$3\times 10^{-5}$
Weight decay	0
Minimum learning rate	0
Warmup steps	100

Table 7: Hyperparameters for the alignment stage.

Setting	Value
optimizer	AdamW
Batch size	8
Window size	4096
Learning rate	$5\times 10^{-4}$
Weight decay	$1\times 10^{-5}$
Minimum learning rate	0

Table 8: Hyperparameters for the pretraining stage.

Figure 7: Visual comparison of attention maps on N-ImageNet [ 25 ] validation event inputs. We compare the attention responses of DINOv2 [ 39 ] (b, f) and our method (c, g) given input events (a, e), alongside reference RGB images (d, h). Our approach demonstrates more precise localization of semantic structures compared to the diffuse activations of DINOv2.

Method	Back-	Pretrain	Pretrain	Pretrain	Pretrain	Peak
Method	bone	Mode	Res.	EFLOPs (h)	Data Size	GPU Mem.
TESPEC [ 37 ]	Swin-T	Unsup.	$360{\times}640$	31.7 (540h)	$\sim$ 3.5 TB	-
STP [ 28 ]	Swin-T	Sup.	$224{\times}224$	-	0.57 TB	-
GEP (Ours)	ViT-S	Unsup.	$224{\times}224$	3.2 (54h)	0.94 TB	17G

Table 9: Pre-training Efficiency & Data. Comparison of backbone architectures, pre-training paradigm, resolution, compute cost, data scale, and peak memory usage.

Method	Head	Trainable Param.	Finetune (h)	DSEC Seg.
Method				mIoU	mAcc
TESPEC [ 37 ]	UperNet	45M	2.7 (46h)	62.77	70.61
STP [ 28 ]	UperNet	42M	-	62.05	-
GEP (ours)	Linear	22M	0.04 (0.7h)	65.22	74.66

Table 10: Segmentation Performance & Cost.

Pre-training Strategy	Eval. Resolution	DSEC Seg.
(Bins per RGB interval)	(Bins per RGB interval)	mIoU	mAcc
Fixed ( $N=1$ ) (Ours)	$N=1$ (Ours)	42.98	53.09
Dynamic ( $N\in\{1,3,5\}$ )	$N=1$ (Ours)	42.67	52.02
	$N=10$ (Unseen)	42.25	51.39
	$N=10$ (Random)	21.31	28.58

Table 11: Temporal Resolution Ablation. All experiments are transferred based on 10% schedule pretrained pilot runs.

Figure 8: Qualitative comparison of depth estimation on MVSEC [ 63 ] . From left to right: (a) event-based input, (b) ECDDP baseline, (c) our method, (d) ground-truth depth, and (e) corresponding reference RGB images (not used). Our model produces depth maps that more closely match the ground truth, with sharper object boundaries and fewer artifacts on thin structures such as trees and poles.

Event-Based Vision

Event-based vision uses asynchronous camera measurements to perceive motion and scene changes with high temporal resolution and dynamic range.

See all 227 papers

Scaling Dense Event-Stream Pretraining from Visual Foundation Models
Mar 4, 2026Zhiwen Chen, Junhui Hou, Zhiyu Zhu +2Event-Based Vision Representation Learning
City University of Hong Kong · Xidian University
REALM: An RGB- and Event-Aligned Latent Manifold for Cross-Modal Perception
Apr 30, 2026Vincenzo Polizzi, David B. Lindell, Jonathan KellyMultimodal Foundation Models Event-Based Vision
University of Toronto, Robotics Institute · University of Toronto, Department of Computer Science
Hyper-RED: Scalable Event Pre-training via Semantic Hypergraph Distillation
Sep 15, 2026Meisen Wang, Zhiqiang Tian, Wei Bao +3Event-Based Vision Cross-Modal Knowledge Distillation
School of Software Engineering, Xi’an Jiaotong University, Xi’an 710049, China · National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, National Engineering Research Center for Visual Information and Applications, and Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an 710049, China · BNRist, THUIBCS, BLBCI, School of Software, Tsinghua University, Beijing 100084, China +2

Mar 4, 2026J/K move · Enter open · S save

Scaling Dense Event-Stream Pretraining from Visual Foundation Models

Zhiwen Chen, Junhui Hou, Zhiyu Zhu, Jinjian Wu+1

City University of Hong Kong · Xidian University

Event-Based Vision Representation Learning Self-Supervised Pre-Training Cross-Modal Representation Learning Knowledge Distillation

Open paper AskPDF arXiv

Learning versatile, fine-grained representations from irregular event streams is pivotal yet nontrivial, primarily due to the heavy annotation that hinders scalability in dataset size, semantic richness, and application scope. To mitigate this dilemma, we launch a novel self-supervised pretraining method that distills visual foundation models (VFMs) to push the boundaries of event representation at scale. Specifically, we curate an extensive synchronized image-event collection to amplify cross-modal alignment. Nevertheless, due to inherent mismatches in sparsity and granularity between image-event domains, existing distillation paradigms are prone to semantic collapse in event representations, particularly at high resolutions. To bridge this gap, we propose to extend the alignment objective to semantic structures provided off-the-shelf by VFMs, indicating a broader receptive field and stronger supervision. The key ingredient of our method is a structure-aware distillation loss that grounds higher-quality image-event correspondences for alignment, optimizing dense event representations. Extensive experiments demonstrate that our approach takes a great leap in downstream benchmarks, significantly surpassing traditional methods and existing pretraining techniques. This breakthrough manifests in enhanced generalization, superior data efficiency and elevated transferability.

Abstract

Event-Based Vision

Explore similar work

Scaling Dense Event-Stream Pretraining from Visual Foundation Models

REALM: An RGB- and Event-Aligned Latent Manifold for Cross-Modal Perception

Hyper-RED: Scalable Event Pre-training via Semantic Hypergraph Distillation