Scaling Dense Event-Stream Pretraining from Visual Foundation Models

Zhiwen Chen; Junhui Hou; Zhiyu Zhu; Jinjian Wu; Guangming Shi

cs.CVMar 4, 2026

Scaling Dense Event-Stream Pretraining from Visual Foundation Models

Authors: Zhiwen Chen, Junhui Hou, Zhiyu Zhu, Jinjian Wu, Guangming Shi

Organizations: City University of Hong Kong · Xidian University

Abstract

Learning versatile, fine-grained representations from irregular event streams is pivotal yet nontrivial, primarily due to the heavy annotation that hinders scalability in dataset size, semantic richness, and application scope. To mitigate this dilemma, we launch a novel self-supervised pretraining method that distills visual foundation models (VFMs) to push the boundaries of event representation at scale. Specifically, we curate an extensive synchronized image-event collection to amplify cross-modal alignment. Nevertheless, due to inherent mismatches in sparsity and granularity between image-event domains, existing distillation paradigms are prone to semantic collapse in event representations, particularly at high resolutions. To bridge this gap, we propose to extend the alignment objective to semantic structures provided off-the-shelf by VFMs, indicating a broader receptive field and stronger supervision. The key ingredient of our method is a structure-aware distillation loss that grounds higher-quality image-event correspondences for alignment, optimizing dense event representations. Extensive experiments demonstrate that our approach takes a great leap in downstream benchmarks, significantly surpassing traditional methods and existing pretraining techniques. This breakthrough manifests in enhanced generalization, superior data efficiency and elevated transferability.

Figures & tables

Figure 2: Illustration of event-image feature alignment across granularities. Patch-level alignment exacerbates cross-modal mismatches, superpixel grouping is ambiguous, while semantic structure grounds superior event–image correspondences.

Figure 3: Cosine similarity maps obtained with DINOv3 output features (anchored at the distinct white stars). The image features exhibit coherent grouping induced by a strong off-the-shelf semantic structure.

Figure 4: Comparison of dense event features under different distillation strategies . All features are produced by a DINOv3-ViT-B model. Left to right : as spatial resolution increases, event representations degrade to varying degrees. PCA maps become less localized, and similarity maps (anchored at the red dot ) become noisier. Top to bottom : (a) image features; (b) patch-level distillation; (c) superpixel-level distillation; (d) patch-level distillation + event activation mask; (e) patch-level distillation + event activation mask + structure-aware regularization ( Our method ). A more detailed feature analysis is provided in the supplementary materials.

Method	Backbone	DDD17-Seg						DSEC-Semantic
Method	Backbone	LP	1%	5%	10%	20%	Full	LP	1%	5%	10%	20%	Full
MaskCLIP [ 95 ]	ViT-B/16	31.91	53.91	56.27	59.32	59.97	61.27	33.08	33.89	37.03	38.83	42.40	55.01
FC-CLIP [ 91 ]	ConvNeXt-L	54.07	56.38	58.50	60.05	60.85	62.01	43.00	39.12	43.71	44.09	47.77	55.67
OpenESS [ 41 ]	E2VID	55.61	57.58	59.07	61.03	61.78	63.00	44.26	41.41	44.97	46.25	48.28	57.21
Ours	ViT-B/16	57.87	57.23	59.54	61.45	62.06	62.81	58.42	54.37	62.82	63.88	64.15	64.93

Table 1: Comparative study of different semantic segmentation methods under the linear probing (LP), few-shot fine-tuning, and full supervision (Full) settings, respectively, on the DDD17-Seg and DSEC-Semantic datasets. All mIoU scores are in percentage (

\%

). The best mIoU scores from each learning configuration are highlighted in bold .

Method	Present at	Backbone	DDD17		DSEC
Method	Present at	Backbone	Acc $\uparrow$	mIoU $\uparrow$	Acc $\uparrow$	mIoU $\uparrow$
RGB initialization + Fully-Supervised
Ev-SegNet [ 1 ]	CVPRW’19	Xception	89.76	54.81	88.61	51.76
E2VID [ 64 ]	TPAMI’19	ResNet-18	85.84	48.47	80.06	44.08
DTL [ 74 ]	ICCV’21	ResNet-50	-	58.80	-	-
PVT-FPN [ 77 ]	ICCV’21	ResNet-34	94.28	53.89	-	-

Table 2: Quantitative comparison of semantic segmentation on the DDD17-Seg [ 1 ] and DSEC-Semantic dataset [ 71 ] datasets. All scores are in percentage (

\%

). The best are marked with bold , and the second best are marked with underline .

RGB initialization + Fully-Supervised
Method	Present at	Backbone	MVSEC-Depth						DSEC-Depth
Method	Present at	Backbone	$\delta_{1}$ $\uparrow$	$\delta_{2}\uparrow$	$\delta_{3}\uparrow$	AbsRel $\downarrow$	RMSE $\downarrow$	RMSE log $\downarrow$	$\delta_{1}\uparrow$	$\delta_{2}\uparrow$	$\delta_{3}\uparrow$	AbsRel $\downarrow$	RMSE $\downarrow$	RMSE log $\downarrow$
E2Depth [ 30 ]	3DV’20	ResNet-18	0.432	0.717	0.868	0.420	7.268	0.455	0.409	0.719	0.891	0.395	13.258	0.412
EReFormer [ 54 ]	TVSVT’24	Swin-T	0.391	0.652	0.810	0.551	8.373	0.523	0.524	0.824	0.945	0.297	11.608	0.334
Event Pretraining + Fully-Supervised
ECDP [ 89 ]	ICCV’23	ViT-S/16	0.476	0.772	0.863	0.496	7.680	0.506	0.528	0.818	0.938	0.324	11.473	0.376

Table 3: Quantitative comparison of monocular depth estimation on the MVSEC-Depth [ 97 ] and DSEC-Depth [ 23 ] datasets. The best are marked with bold , and the second best are marked with underline .

Method	Backbone	MVSEC-Depth						DSEC-Depth
Method	Backbone	LP	1%	5%	10%	20%	Full	LP	1%	5%	10%	20%	Full
DepthAnyEvent-R [ 4 ]	ViT-S/16	7.473	7.542	7.261	6.794	6.637	6.465	10.584	10.347	9.898	9.534	9.065	8.880
Ours	ViT-S/16	6.756	6.930	6.712	6.477	6.352	6.145	4.861	4.983	4.751	4.728	4.694	4.564

Table 4: Comparative study of different monocular depth estimation methods under the linear probing (LP) and few-shot fine-tuning, and full supervision (Full) settings, respectively, on the MVSEC-Depth and DSEC-Depth datasets. The best RMSE from each learning configuration are highlighted in bold .

Exp.	Distill	Mask	IS Loss	CS Loss	DDD17-Seg		DSEC-Semantic		MVSEC-Depth		DSEC-Depth
Exp.	Distill	Mask	IS Loss	CS Loss	Acc $\uparrow$	mIoU $\uparrow$	Acc $\uparrow$	mIoU $\uparrow$	$\delta_{1}$ $\uparrow$	RMSE $\downarrow$	$\delta_{1}$ $\uparrow$	RMSE $\downarrow$
(a)					91.39	59.60	91.94	64.31	0.593	6.635	0.846	4.424
(b)	$\checkmark$				92.16	62.41	92.74	66.17	0.609	6.114	0.875	4.063
(c)	$\checkmark$	$\checkmark$			92.32	62.67	92.82	66.54	0.611	5.922	0.876	4.025
(d)	$\checkmark$	$\checkmark$	$\checkmark$		92.60	64.84	93.08	69.20	0.620	5.684	0.889	3.792
(e)	$\checkmark$	$\checkmark$		$\checkmark$	92.51	63.62	93.01	68.68	0.614	5.786	0.881	3.870

Table 5: Ablative study results of the proposed key components . Distill denotes the knowledge distillation mark; Mask refers to activation mask regularization; IS Loss denotes the intra-modal structure loss; CS Loss denotes the cross-modal structure loss; (a) indicates the use of a image-domain pretrained model; (b) serves as our baselines; (f) denotes our complete cross-modal distillation framework.

Method	Backbone	indoor flying1		indoor flying2		indoor flying3
Method	Backbone	EPE $\downarrow$	Out $\downarrow$	EPE $\downarrow$	Out $\downarrow$	EPE $\downarrow$	Out $\downarrow$
RGB initialization + Fully-Supervised
EST [ 22 ]	ResNet-18	1.24	5.09	2.05	19.90	1.71	11.67
DCEFlow [ 73 ]	-	0.75	0.60	1.39	8.01	1.13	5.29
Event Pretraining + Fully-Supervised
ECDP [ 89 ]	ResNet-50	0.60	0.35	1.35	8.57	1.12	5.26

Table 6: Quantitative comparison of optical flow estimation on the MVSEC-Flow [ 97 ] dataset. The best are marked with bold , and the second best are marked with underline .

Figure 5: Comparison of dense image features under different visual foundation models through a toy example.

Panel of Table 7 — Table 7: The pretraining dataset configuration and data statistics for the nine real-world event-image datasets used in our experiments.

Panel of Table 8 — Table 8: The pretraining dataset configuration and data statistics for the seven synthetic event-image datasets used in our experiments.

Dataset	Semantic Segmentation		Depth Estimation		Flow Estimation
Dataset	DDD17-Seg	DSEC-Semantic	MVSEC-Depth	DSEC-Depth	MVSEC-Flow
optimizer	AdamW	AdamW	AdamW	AdamW	AdamW
encoder lr	$2\times 10^{-6}$	$2\times 10^{-6}$	$2\times 10^{-6}$	$1\times 10^{-6}$	$1\times 10^{-6}$
decoder lr	$5\times 10^{-6}$	$4\times 10^{-6}$	$4\times 10^{-6}$	$2\times 10^{-6}$	$2\times 10^{-6}$
weight decay	$1\times 10^{-4}$	$1\times 10^{-4}$	$1\times 10^{-4}$	$1\times 10^{-4}$	$1\times 10^{-4}$
batch size	40	12	40	12	24

Table 9: Experimental setup for fine-tuning downstream tasks. lr denotes learning rate. All configurations are based on the ViT-L encoder. Apart from batch size, which depends on model scale, all other settings remain identical across experiments.

Linear Probing
Event Model	mIoU	flat	background	object	vegetation	human	vehicle	Acc
ViT-S/16	55.64	79.61	91.18	15.90	57.51	22.02	67.72	91.27
ViT-B/16	57.87	79.92	91.24	15.87	58.04	34.97	67.05	91.31
ViT-L/16	60.30	81.03	91.49	18.83	57.21	44.18	68.95	91.83
Fine-Tuning (1 $\%$ )
ViT-S/16	53.87	78.61	90.06	10.03	54.59	25.18	64.63	90.41

Table 10: The per-class segmentation results of our methods on the DDD17-Seg dataset. Scores reported are IoUs in percentage.

Event Model	indoor flying1		indoor flying2		indoor flying3
Event Model	EPE $\downarrow$	Out $\downarrow$	EPE $\downarrow$	Out $\downarrow$	EPE $\downarrow$	Out $\downarrow$
ViT-S/16	0.29	0.03	0.38	0.001	0.40	0.001
ViT-B/16	0.28	0.03	0.38	0.001	0.39	0.001
ViT-L/16	0.27	0.03	0.37	0.001	0.39	0.001

Table 11: The optical flow results of our methods on the MVSEC-Flow dataset.

Linear Probing
Event Model	mIoU	background	building	fence	person	pole	road	sidewalk	vegetation	car	wall	traffic-sign	Acc
ViT-S/16	55.46	92.81	81.88	17.45	15.67	24.98	93.20	68.12	78.83	77.72	30.79	43.86	90.12
ViT-B/16	58.42	93.46	83.52	23.88	16.69	27.85	93.72	69.27	80.38	80.13	43.06	43.25	91.44
ViT-L/16	61.29	93.91	85.09	27.66	27.37	33.58	93.34	70.94	82.37	82.27	41.82	48.66	91.69
Fine-Tuning (1 $\%$ )
ViT-S/16	52.97	92.35	81.36	18.04	7.93	18.77	92.55	60.54	78.50	76.22	22.25	37.06	89.56

Table 12: The per-class segmentation results of our methods on the DSEC-Semantic dataset. Scores reported are IoUs in percentage.

Event Model	Metric	MVSEC-Depth						DSEC-Depth
Event Model	Metric	LP	1%	5%	10%	20%	Full	LP	1%	5%	10%	20%	Full
ViT-S/16	$\delta_{1}$ $\uparrow$	0.529	0.526	0.531	0.542	0.560	0.577	0.798	0.795	0.804	0.811	0.816	0.824
ViT-S/16	RMSE $\downarrow$	6.756	6.930	6.712	6.477	6.352	6.145	4.861	4.983	4.751	4.728	4.694	4.564
ViT-B/16	$\delta_{1}$ $\uparrow$	0.571	0.561	0.574	0.587	0.591	0.594	0.845	0.839	0.856	0.863	0.867	0.872
ViT-B/16	RMSE $\downarrow$	6.392	6.546	6.339	6.012	5.908	5.891	4.352	4.471	4.264	4.192	4.154	4.032
ViT-L/16	$\delta_{1}$ $\uparrow$	0.597	0.592	0.601	0.612	0.619	0.625	0.881	0.856	0.883	0.892	0.893	0.896

Table 13: The depth results of our methods on the MVSEC-Depth and DSEC-Depth datasets.

Event Input	DDD17-Seg		DSEC-Depth		MVSEC-Flow
Event Input	Acc $\uparrow$	mIoU $\uparrow$	$\delta_{1}$ $\uparrow$	RMSE $\downarrow$	EPE $\downarrow$	Out $\downarrow$
Color Frame	90.76	56.37	0.784	5.306	1.107	6.720
E2VID	89.25	55.72	0.809	4.928	0.852	3.294
Event Volume	91.39	59.64	0.824	4.564	0.356	0.094

Table 14: Ablative study results of different event aggregation methods.

Time Bin	DDD17-Seg		DSEC-Depth		MVSEC-Flow
Time Bin	Acc $\uparrow$	mIoU $\uparrow$	$\delta_{1}$ $\uparrow$	RMSE $\downarrow$	EPE $\downarrow$	Out $\downarrow$
$B=1$	91.07	58.43	0.819	4.736	0.365	0.104
$\bm{B=3}$	91.39	59.64	0.824	4.564	0.356	0.094
$B=5$	91.20	59.22	0.822	4.613	0.359	0.095

Table 15: Ablative study results of different time bins for event volume aggregation.

Density	DDD17-Seg		DSEC-Depth		MVSEC-Flow
Threshold	Acc $\uparrow$	mIoU $\uparrow$	$\delta_{1}$ $\uparrow$	RMSE $\downarrow$	EPE $\downarrow$	Out $\downarrow$
$\tau=32$	91.33	59.51	0.823	4.538	0.362	0.095
$\bm{\tau=64}$	91.39	59.64	0.824	4.564	0.356	0.094
$\tau=128$	91.25	59.32	0.821	4.640	0.367	0.097

Table 16: Ablative study results of different density thresholds for activation mask constraint.

Alignment	DDD17-Seg		DSEC-Depth		MVSEC-Flow
Objective	Acc $\uparrow$	mIoU $\uparrow$	$\delta_{1}$ $\uparrow$	RMSE $\downarrow$	EPE $\downarrow$	Out $\downarrow$
patch-level (L1)	90.65	56.06	0.785	4.990	0.367	0.098
superpixel-level (L1)	90.88	56.36	0.790	4.937	0.384	0.106
superpixel-level (CL)	90.92	56.72	0.782	5.031	0.435	0.120
ours	91.39	59.64	0.824	4.564	0.356	0.094

Table 17: Ablative study results of different distillation objectives across granularities. CL denotes the contrastive loss.

Alignment	DDD17-Seg		DSEC-Depth		MVSEC-Flow
Objective	Acc $\uparrow$	mIoU $\uparrow$	$\delta_{1}$ $\uparrow$	RMSE $\downarrow$	EPE $\downarrow$	Out $\downarrow$
multi-scale	91.07	58.83	0.816	4.831	0.377	0.102
single-scale	91.39	59.64	0.824	4.564	0.356	0.094

Table 18: Ablative study results of multi-scale distillation.

Figure 6: Cross-modal distillation loss during pretraining of our event-based ViT-S, ViT-B, and ViT-L feature encoders.

	Segment Model		Depth Model		Flow Model
	MParams	GFLOP	MParams	GFLOPs	MParams	GFLOPs
ViT-S	28.23	243.67	18.92	61.23	40.95	349.42
ViT-B	76.74	363.82	74.98	231.93	135.62	962.71
ViT-L	239.09	758.17	257.29	853.16	485.27	3369.48

Table 19: Computational efficiency of our downstream task models, setting an input event volume resolution of

480\times 640

Figure 7: T-SNE plots of learned event features. We sample 20,000 event feature vectors from the DSEC dataset [ 23 ] . We show features from (a) images with pretrained DIVOv3-L; (b) event volume with pretrained DIVOv3-L; (c) event volume with DIVOv3-L after patch-level distillation; (d) event volume with DIVOv3-L using our distillation method.

Figure 8: The learned fine-grained event features (1/2) of our method are primarily presented through cosine similarity maps, with key points anchored at the distinct white stars. Best viewed in color.

Figure 9: The learned fine-grained event features (2/2) of our method are primarily presented through cosine similarity maps, with key points anchored at the distinct white stars. Best viewed in color.

Figure 10: The qualitative comparisons among different event-based semantic segmention approaches on the test set of DSEC-Semantic. Best viewed in color.

Figure 11: The qualitative comparisons among different event-based depth estimation approaches on the test set of DSEC-Depth. Best viewed in color.

Figure 12: The qualitative results of our optical flow estimation approaches on the test set of MVSEC-Flow. Best viewed in color.

Event-Based Vision

Event-based vision uses asynchronous camera measurements to perceive motion and scene changes with high temporal resolution and dynamic range.

See all 227 papers

Hyper-RED: Scalable Event Pre-training via Semantic Hypergraph Distillation
Sep 15, 2026Meisen Wang, Zhiqiang Tian, Wei Bao +3Event-Based Vision Cross-Modal Knowledge Distillation
School of Software Engineering, Xi’an Jiaotong University, Xi’an 710049, China · National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, National Engineering Research Center for Visual Information and Applications, and Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an 710049, China · BNRist, THUIBCS, BLBCI, School of Software, Tsinghua University, Beijing 100084, China +2
Generative Event Pretraining with Foundation Model Alignment
Mar 24, 2026Jianwen Cao, Jiaxu Xing, Nico Messikommer +1Event-Based Vision Vision Foundation Model Adaptation
Robotics and Perception Group, University of Zurich
REALM: An RGB- and Event-Aligned Latent Manifold for Cross-Modal Perception
Apr 30, 2026Vincenzo Polizzi, David B. Lindell, Jonathan KellyMultimodal Foundation Models Event-Based Vision
University of Toronto, Robotics Institute · University of Toronto, Department of Computer Science

Sep 15, 2026J/K move · Enter open · S save

Hyper-RED: Scalable Event Pre-training via Semantic Hypergraph Distillation

Meisen Wang, Zhiqiang Tian, Wei Bao, Chengjie Wang+2

School of Software Engineering, Xi’an Jiaotong University, Xi’an 710049, China · National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, National Engineering Research Center for Visual Information and Applications, and Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an 710049, China · BNRist, THUIBCS, BLBCI, School of Software, Tsinghua University, Beijing 100084, China+2

Event-Based Vision Cross-Modal Knowledge Distillation Representation Learning Cross-Modal Representation Learning

Open paper AskPDF arXiv

Event cameras have shown great potential for robust visual perception, yet scaling event representation learning remains challenging due to the scarcity of large-scale annotated event data. Pretrained image models provide scalable semantic supervision, but existing image-to-event methods rely on rigid pixel-wise or token-wise alignment that overlooks modality discrepancies in texture, density, and appearance, potentially causing semantic collapse and limiting transferability. To address this issue, we propose Hyper-RED, a simple, painless, and scalable image-to-event pretraining framework that transfers high-order semantic structures from images to events. Hyper-RED uses hypergraphs to model and align high-order semantic associations among multiple image and event tokens, enabling cross-modal knowledge transfer while accommodating modality-specific differences rather than enforcing rigid one-to-one correspondence. Specifically, given a paired event--image sample, Hyper-RED leverages DINOv3 to extract spatial token representations and constructs image, event, and cross-modal semantic hypergraphs, where each hyperedge connects multiple semantically correlated tokens. We further introduce a hypergraph relational distillation loss that imposes complementary intra- and cross-modal constraints, enabling the event encoder to inherit image-derived semantic organization while preserving local relational consistency and event-specific characteristics. Experiments on three tasks across five event datasets demonstrate consistent scaling from ViT-S to ViT-L and state-of-the-art performance (Fig.1). The code is available at: https://github.com/meisenwang/Hyper--RED.

Abstract

Event-Based Vision

Explore similar work

Hyper-RED: Scalable Event Pre-training via Semantic Hypergraph Distillation

Generative Event Pretraining with Foundation Model Alignment

REALM: An RGB- and Event-Aligned Latent Manifold for Cross-Modal Perception