Event-based Action Recognition by Pre-Training on Multi-Frame Representations

Emma Norrby; Tomas Wilkinson; Per Nyblom; David Gustafsson

cs.CVOct 9, 2026

Event-based Action Recognition by Pre-Training on Multi-Frame Representations

Authors: Emma Norrby, Tomas Wilkinson, Per Nyblom, David Gustafsson

Organizations: Swedish Defence Research Agency (FOI), Linköping, Sweden

Abstract

Action recognition is a crucial task in domains like robotics, patient monitoring, or surveillance. Recently, event-based action recognition has attracted much attention due to the event camera's ability to capture fast motions and its improved privacy over conventional vision cameras. While frame-based event representations work well for short event sequences, their discriminative ability declines sharply with longer and more complex event sequences. In this work, we introduce a multi-frame representation that greatly improves recognition performance over its single-frame counterpart. We adopt a masked auto-encoding framework to pre-train a Vision Transformer using a sparsity-aware masking algorithm and token selection mechanism to improve downstream performance and control computational complexity. We also introduce multiple mask tokens for the decoder with a position-aware attention mechanism, leading to further improved performance. Our proposed method shows impressive performance for challenging long-duration and complex event sequences, yielding state-of-the-art performance on THU-EACT-50 CHL and DailyDVS-200.

Figures & tables

Figure 1: Overview of the framework. Event streams are converted into multi-frame representations like CSTR or Event Histograms. Each frame is tokenized separately and the activity (Equation 2 ) is used to select the tokens used for further processing. Next the remaining tokens are fed directly to a supervised learning setup or to the sparsity-aware masking algorithm for pre-training. The masking step uses the same activity to bias the visible tokens to have more information. The masked inputs continue on to the masked auto-encoder for pre-training. In the decoder we use multiple mask tokens together with a position-aware attention to allow the decoder to learn how to better initialize the mask tokens depending on their spatial position.

Panel of Figure 2 — Figure 2: Visualization of activity-based token selection and masking strategies using a single input frame. Subfigure (b) shows the input (a) after keeping the 50% most active patches. The bottom row compares random masking with sparsity-aware masking for the same random sampling, i.e., the $\mathbf{u}$ from Eq. 3 is fixed and $c_{s}$ is set to 0 and 0.5 respectively.

Method	SSL	Modality	Params	Top-1 Accuracy (%)
				THU-EACT-50 CHL	DailyDVS-200
HMAX SNN [ 40 ] , from [ 14 ]	$\times$	Event	-	32.7	-
MOTION SNN [ 23 ] , from [ 14 ]	$\times$	Event	-	47.3	-
EV-ACT [ 14 ]	$\times$	Event	21.3M	58.5	-
EventMamba [ 31 ]	$\times$	Event	0.91M	59.4	-
EventBind [ 43 ] , from [ 6 ]	$\times$	Event + Text	635M	61.1	-

Table 1: Comparison of the performance on THU-EACT-50 CHL and DailyDVS-200 datasets. Linear probing (LP) and fine-tuning accuracy (FT) is reported on our results. TS is the percentage of tokens kept after token selection. The second group of experiments are pre-trained on a larger dataset and then finetuned on the smaller dataset. † : pre-trained on DailyDVS-200.

Table 4

Panel of Figure 3 — Figure 3: Linear probing accuracy for CSTR and histogram representations on the THU-EACT-50 CHL dataset using different values of $c_{s}$ and different numbers of frames. Solid and dashed lines indicate CSTR and Event Histograms respectively. Blue , orange , and green colors correspond to 1 , 2 , and 4 frames.

Panel of Figure 4 — Figure 4: Linear probing accuracy for different numbers of frames and token budgets using activity-based token selection on THU-EACT-50 CHL.

Action	Sequence duration (s)	Accuracy (%)
		1 frame	4 frames	8 frames
Sneeze	$2.881\pm 0.647$	19.2	50.0 ( $+$ 30.8)	53.9 ( $+$ 34.7)
Tie shoe laces	$8.729\pm 1.883$	37.5	79.2 ( $+$ 41.7)	75.0 ( $+$ 37.5)
Overall	$4.043\pm 1.607$	33.1	55.6 ( $+$ 22.5)	61.7 ( $+$ 28.6)

Table 4: Performance on two actions with different temporal durations using different numbers of input frames. Sequence duration is reported as mean

\pm

standard deviation.

Explore similar work

TrAction: Action Recognition with Sparse Trajectories
Jun 2, 2026Jan F. Meier, Felix B. Mueller, Alexander Ecker +1Video Action Recognition Video Representation Learning
Institute of Computer Science and Campus Institute Data Science, University Göttingen, Germany · Max Planck Institute for Dynamics and Self-Organization, Göttingen, Germany
Event ActivityNet: A Large-Scale Simulated-Event Benchmark for Untrimmed Action Understanding
Aug 3, 2026Cheng-Yao Hong, Ting-Wei Lin, Yun-Chung Lai +3Event-Based Vision Video Action Recognition
Institute of Information Science, Academia Sinica, Taiwan · Department of Computer Science, National Tsing Hua University, Taiwan
Neural Events: Discrete Asynchronous Autoencoders for Event-Based Vision
Jun 18, 2026Roberto Pellerito, Daniel Gehrig, Shintaro Shiba +1Event-Based Vision Representation Learning
Robotics and Perception Group, University of Zurich · University of Pennsylvania · The University of Tokyo +1

Jun 2, 2026J/K move · Enter open · S save

TrAction: Action Recognition with Sparse Trajectories

Jan F. Meier, Felix B. Mueller, Alexander Ecker, Timo Lüddecke

Institute of Computer Science and Campus Institute Data Science, University Göttingen, Germany · Max Planck Institute for Dynamics and Self-Organization, Göttingen, Germany

Video Action Recognition Video Representation Learning

Open paper AskPDF arXiv

Modern action recognition models operate on memory- and compute-intensive dense RGB video volumes and frequently exploit appearance and background shortcuts, for example, predicting actions from objects or scenes instead of characteristic motion. We investigate an efficient alternative input modality that is largely free of such biases by construction: sparse point trajectories. To this end, we develop a simple transformer architecture for 2.5D trajectory-based recognition together with a masked-trajectory pretraining, which we show to substantially improve downstream action recognition accuracy. Despite using only a fraction of the dense RGB input, our method reaches 45% top-1 on Something-Something V2 and 54% on EPIC-Kitchens-100, and surpasses V-JEPA on time-reversal sensitivity. More importantly, we find trajectory features to be complementary to state-of-the-art appearance-based features. Fusing our pretrained model with DINOv2 and V-JEPA 2 improves top-1 accuracy on Something-Something V2 by 8.7 and 1.6 points, respectively. Code: https://github.com/ecker-lab/TrAction

Abstract

Explore similar work

TrAction: Action Recognition with Sparse Trajectories

Event ActivityNet: A Large-Scale Simulated-Event Benchmark for Untrimmed Action Understanding

Neural Events: Discrete Asynchronous Autoencoders for Event-Based Vision