FLEET: Token-Based Feature Extraction for Event Camera-based Reinforcement Learning

Tristan Gottwald; Maximilian Schier; Melanie Schaller; Bodo Rosenhahn

cs.CVAug 17, 2026

FLEET: Token-Based Feature Extraction for Event Camera-based Reinforcement Learning

Authors: Tristan Gottwald, Maximilian Schier, Melanie Schaller, Bodo Rosenhahn

Organizations: Institute of Information Processing, L3S, Leibniz University Hannover, Germany

Abstract

Event cameras generate asynchronous, high-frequency data streams offering spatially sparse information at lower latency than traditional cameras. In principle, these properties should be ideal for the design of control policies. However, reinforcement learning research in this field remains limited as existing approaches fail to fully exploit the sensor's properties. CNN-based methods negate the sensors benefits by aggregating events into sparse grids. This couples compute cost to sensor resolution and blurs the temporal information. Meanwhile, existing generative baselines rely on the availability of trajectory data to pretrain the model. We propose FLEET (Feature Learning from Events via Efficient Tokenization), a feature extractor that processes event sequences directly. Leveraging random Fourier features and cross-attention, our architecture compresses variable streams into fixed-size latent representations. This decouples inference cost of the feature extractor's backbone from the sensor's resolution, enabling end-to-end learning without auxiliary losses. We validate FLEET on a new, high-throughput benchmark. The results demonstrate that our sequence-based approach surpasses SOTA performance and exhibits superior robustness to variations in observation frequencies.

Figures & tables

Figure 1: Comparison of feature extraction architectures for event camera-based reinforcement learning. Our proposed FLEET architecture (left) processes raw sequence data directly, successfully preserving temporal and spatial resolution. Events are encoded using random Fourier features and mapped to a fixed set of learnable latent vectors using cross-attention. In contrast, standard CNN-based approaches (center) force events into extremely sparse aggregated tensor representations, leading to quantization errors and unnecessary processing overhead due to sparsity. Generative pretraining pipelines like eVAE (right) attempt to solve this but suffer from an objective mismatch: pretraining on an auxiliary reconstruction task forces the representation to encode visual fidelity rather than the features actually relevant to optimal control. Furthermore, these frozen, two-stage approaches can suffer significantly from distribution shifts if the offline pretraining data fails to capture the full scope of dynamics encountered during active deployment.

Figure 2: Overview of our proposed FLEET feature extractor. The number of events within the original sequence

\mathcal{E}_{\tau}

is first reduced or padded to

\mathcal{E}_{\text{batched}}

. The temporal and positional component of each event in

\mathcal{E}_{\text{batched}}

are embedded using shared temporal and spatial random Fourier feature encoders. These embeddings are combined to

\mathbf{z}_{k}

using a temporal gating mechanism. The cross-attention module maps the sequence of embedded events to

N

learnable latent vectors to reduce the dimensionality. The cross-attention is followed by an MLP encoder block. As a last step global average pooling is performed over all latents to form a final representation.

Panel of Figure 3 — Figure 3: Sequence of RGB ( Fig. 3(a) ) and event ( Fig. 3(b) ) camera observations in the Event-CarEnv Racing scenario. Note that the event camera data has been temporarily aggregated into an image format strictly for human interpretability.

Feature Extractor	Racing (Return $\uparrow$ )			Parking (Return $\uparrow$ )			Inv. Dbl. Pendulum (Return $\uparrow$ )
Feature Extractor	IQM	CI95		IQM	CI95		IQM	CI95
NatureCNN (RGB) [ 34 ]	23.89	[12.71,	35.89]	5.01	[4.50,	5.41]	8980.69	[7883.73,	9352.29]
NatureCNN [ 34 ]	6.83	[4.41,	11.44]	3.89	[3.50,	4.82]	560.50	[469.09,	1013.37]
ImpalaCNN [ 12 ]	4.81	[4.39,	5.93]	5.18	[4.77,	5.59]	1175.57	[184.60,	2178.99]
LFF-DS [ 43 ]	5.36	[4.22,	6.46]	0.40	[0.01,	0.81]	848.29	[788.83,	919.08]
eVAE [ 53 ]	1.13	[0.81,	1.43]	0.82	[0.78,	0.90]	429.67	[414.25,	441.36]

Table 1: Performance of various feature extractors in combination with PPO. Interquartile means (IQM) and 95% confidence intervals (CI95) of episodic returns are reported for Event-CarEnv Racing/Parking (

\Delta t=0.05

10^{6}

training steps) and Inverted Double Pendulum (

\Delta t=0.01

5\cdot 10^{6}

steps). Best and second-best results are highlighted. FLEET outperforms all baselines across all environments, including a RGB variant on Racing and Parking.

Panel of Figure 4 — Figure 4: Progression of the IQM and 95% CI of the episodic return in the Event-CarEnv Racing and Parking ( $\Delta t=0.05$ , $10^{6}$ steps) and the Inverted Double Pendulum ( $\Delta t=0.01$ , $5\cdot 10^{6}$ steps) settings. Agents were evaluated for 10 episodes every $10^{5}$ environment steps, reporting the mean episodic return. In the Racing scenario, CNN-based competitors fail to demonstrate learning progress, though they manage to learn suitable policies in the denser Parking setting. The agents using the eVAE and LFF-DS feature extractor fail to learn a viable policy in any environment. Across all tasks, FLEET shows superior performance to all event-based baseline methods.

Feature Extractor	$96\times 64$			$128\times 96$			$192\times 128$			$384\times 256$
	(Return $\uparrow$ )			(Return $\uparrow$ )			(Return $\uparrow$ )			(Return $\uparrow$ )
	IQM	CI95		IQM	CI95		IQM	CI95		IQM	CI95
NatureCNN	6.83	[4.41,	11.44]	11.43	[9.18,	14.70]	16.42	[11.88,	21.25]	13.21	[10.64,	17.55]
DMRCNN	12.94	[9.09,	16.99]	9.55	[6.90,	13.02]	10.46	[8.08,	13.88]	8.57	[6.41 ,	12.62]
FLEET ( ours )	46.96	[43.28,	49.03]	40.48	[36.49,	44.17]	43.65	[36.85,	49.64]	44.32	[39.89,	49.57]

Table 2: Interquartile means and 95% confidence intervals of the episodic return of FLEET for different spatial sensor resolutions, evaluated on the Racing scenarios. FLEET’s performance is near constant despite an increase in spatial resolution.

Figure 5: Episodic return across varying simulation step sizes (

\Delta t=\{0.1,0.05,0.025\}

) in the Racing scenario. FLEET remains robust at higher control frequencies, whereas CNN-based architectures degrade.

Feature Extractor	$0.5\cdot 10^{6}\text{ steps @ }\Delta t=0.1$			$10^{6}\text{ steps @ }\Delta t=0.05$			$2\cdot 10^{6}\text{ steps @ }\Delta t=0.025$
	(Return $\uparrow$ )			(Return $\uparrow$ )			(Return $\uparrow$ )
	IQM	CI95		IQM	CI95		IQM	CI95
NatureCNN [ 34 ]	19.02	[16.82,	21.08]	6.83	[4.41,	11.44]	5.87	[4.10,	10.40]
DMRCNN [ 55 ]	21.00	[19.70,	22.62]	12.94	[9.09,	16.99]	6.78	[5.36,	7.71]
FLEET ( ours )	40.33	[38.43,	42.10]	46.96	[43.28,	49.03]	41.92	[34.18,	46.68]

Table 3: Impact of simulation step size (

\Delta t\in\{0.1,0.05,0.025\}

) on episodic return (IQM and 95% CI) in the Racing scenario. Training steps are scaled to

\{0.5\cdot 10^{6},10^{6},2\cdot 10^{6}\}

respectively to preserve total environment time. FLEET maintains robust performance at higher frequencies compared to CNN-based baseline methods.

Figure 6: Normalized spatial attention weights of the event-to-latent mapping during a Racing episode. The activation patterns demonstrate how distinct latents specialize in different regions of the perceptive field. Latent 7 localizes on the track boundaries and cones immediately in front of the ego-vehicle, while Latent 1 attends to broader, distant dynamics in the upper frame. Because the agent maintains a centered racing line, the spatial variance of attended events naturally scales with distance. Latent 3 highlights the agent’s ability to detect track curvature by jointly attending to diagonally opposed regions ( e.g . upper-left and lower-right), providing context for the steering policy.

Table 10

Hyperparameter	Value
FLEET
Event Sequence Length ( $M$ )	2048
Embedding Dimension ( $d_{\text{embed}}$ )	128
$\sigma_{\text{temporal}}$	5.0
$\sigma_{\text{spatial}}$	20.0
Number of Latent Tokens ( $N$ )	32

Table 6: FLEET hyperparameters used in all of our experiments.

NatureCNN
Hyperparameter	Value
Conv. Architecture (Channels, Kernel, Stride)	$(32,8,1),(64,4,2),(64,3,1)$
Output Dimension	512
ImpalaCNN
Conv. Arch. (Features, Kernel, Stride, ResBlocks)	$(16,3,1,2),(32,3,1,2),(32,3,1,2)$
Output Dimension	256

Table 7: Hyperparameters for the baseline feature extractors used in our experiments.

Figure 7: Course of the ten tracks used during evaluation in the Event-CarEnv Racing setting. The tracks cover different degrees of difficulty from easy (Track 6) to very hard (Track 3).

Panel of Figure 8 — Figure 8: Additional Sequence of RGB ( Fig. 3(a) ) and event ( Fig. 3(b) ) camera observations of a relatively straight section in the Event-CarEnv Racing scenario. Note that the event camera data has been temporarily aggregated into an image format strictly for human interpretability.

Panel of Figure 9 — Figure 9: Sequence of RGB ( Fig. 3(a) ) and event ( Fig. 3(b) ) camera observations in the Event-CarEnv Parking scenario. Note that the event camera data has been temporarily aggregated into an image format strictly for human interpretability.

Panel of Figure 10 — Figure 10: Sequence of RGB ( Fig. 10(a) ) and event ( Fig. 10(b) ) camera observations in the Inverted Double Pendulum. Note that the event camera data has been temporarily aggregated into an image format strictly for human interpretability.

Panel of Figure 11 — Figure 11: Comparison of computational cost ( Fig. 11(a) ) and memory consumption ( Fig. 11(b) ) per minibatch (5120 transitions) for NatureCNN, DMRCNN, and FLEET across increasing spatial resolutions.

Feature Extractor	$96\times 64$		$128\times 96$		$192\times 128$		$384\times 256$
Feature Extractor	GFLOPs( $\downarrow$ )	VRAM GB	GFLOPs( $\downarrow$ )	VRAM GB	GFLOPs( $\downarrow$ )	VRAM GB	GFLOPs( $\downarrow$ )	VRAM GB
NatureCNN	175.11	1.25	353.93	2.49	712.84	4.98	2876.33	19.92
DMRCNN	47.11	0.54	108.05	1.12	236.07	2.28	1053.09	9.37
FLEET ( ours )	83.28	4.09	83.28	4.09	83.28	4.09	83.28	4.09

Table 8: Comparison of computational cost (GFLOPs) and memory consumption (VRAM in GB) per minibatch (5120 transitions) for NatureCNN, DMRCNN, and FLEET across increasing spatial resolutions. Because FLEET directly processes a fixed-length sequence of events rather than sparse grids, its computational load and memory footprint remain perfectly constant, effectively decoupling inference cost from the sensor’s spatial resolution.

Feature Extractor	$96\times 64$	$128\times 96$	$192\times 128$	$384\times 256$
Feature Extractor	Exec. Time $\mathrm{m}\mathrm{s}$	Exec. Time $\mathrm{m}\mathrm{s}$	Exec. Time $\mathrm{m}\mathrm{s}$	Exec. Time $\mathrm{m}\mathrm{s}$
NatureCNN	286.63	577.67	1165.83	4778.59
DMRCNN	81.48	260.72	644.07	3690.64
FLEET ( ours )	11.76	11.76	11.76	11.76

Table 9: Inference execution time per minibatch for NatureCNN, DMRCNN and FLEET across increasing spatial resolutions. FLEET leverages GPU-friendly operations, leading to lower execution times than it’s competitors.

Sequence Sampling Technique	Return $\uparrow$
Sequence Sampling Technique	IQM	CI95	Rel. Change
FLEET ( ours )	46.96	[43.28, 49.03]	-
Temporal Sampling	37.02	[34.68, 39.89]	-21%

Table 10: Interquartile mean, 95% confidence intervals, and relative change in IQM of the episodic return when using temporal sampling (choose last

M

events) when compared to the sampling approach used in FLEET.

Sequence Sampling Technique	Return $\uparrow$
Sequence Sampling Technique	IQM	CI95	Rel. Change
FLEET ( ours )	46.96	[43.28, 49.03]	-
Matched Sequence Baseline	30.06	[26.13, 36.12]	-35%

Table 11: Interquartile mean, 95% confidence intervals, and relative change in IQM of the episodic return comparing the full FLEET architecture to a parameter-matched MLP sequence baseline. This comparison isolates the performance gains provided by the Random Fourier Features and the cross-attention bottleneck from the inherent benefits of the underlying sequence sampling strategy.

Panel of Figure 12 — Figure 12: Hyperparameter study on the Event-CarEnv Racing scenario for the number of latents $N$ in the cross-attention module ( Fig. 12(a) ) and the maximum event sequence length $M$ output by the event sequence sampling module ( Fig. 12(b) ).

Feature Extractor	$N=8$			$N=16$			$N=32$			$N=64$			$N=128$
	(Return $\uparrow$ )			(Return $\uparrow$ )			(Return $\uparrow$ )			(Return $\uparrow$ )			(Return $\uparrow$ )
	IQM	CI95		IQM	CI95		IQM	CI95		IQM	CI95		IQM	CI95
FLEET ( ours )	38.81	[35.99,	41.09]	41.55	[38.94,	44.57]	46.96	[43.28,	49.03]	43.56	[39.51,	47.71]	41.59	[37.11,	48.52]

Table 12: IQM and CI95 of FLEET on Event-CarEnv Racing with varying number of latents

N

in the cross-attention module. Both too many and not enough latents lead to performance degradation.

Feature Extractor	$M=512$			$M=1024$			$M=2048$			$M=4096$
	(Return $\uparrow$ )			(Return $\uparrow$ )			(Return $\uparrow$ )			(Return $\uparrow$ )
	IQM	CI95		IQM	CI95		IQM	CI95		IQM	CI95
FLEET ( ours )	40.05	[37.23,	44.14]	41.39	[36.00,	43.41]	46.96	[43.28,	49.03]	44.04	[40.80,	47.23]

Table 13: Interquartile mean (IQM) and 95% confidence intervals (CI95) of the episodic return for the FLEET architecture on the Event-CarEnv Racing scenario, evaluated across varying event sequence lengths

M

. The results illustrate the impact of the sequence length sampled prior to the cross-attention bottleneck, demonstrating how excessively small values of

M

can constrain the network’s performance, while still performing better than CNN-based approaches.

Aggregation Technique	Return $\uparrow$
Aggregation Technique	IQM	CI95	Rel. Change
Temporal Gating	46.96	[43.28, 49.03]	-
Addition	40.03	[36.47, 44.25]	-14%
Multiplication	31.83	[27.39, 35.80]	-32%
Concatenation	39.74	[34.35, 45.91]	-15%

Table 14: Interquartile mean, 95% confidence intervals, and relative change in IQM of the episodic return when replacing the temporal gating embedding aggregation strategy with element-wise addition or multiplication and concatenation. Temporal gating shows the highest episodic returns.

Representation Learning for RL

Representation learning for reinforcement learning builds state, task, and dynamics features that help agents learn, explore, and transfer.

See all 69 papers

Neural Events: Discrete Asynchronous Autoencoders for Event-Based Vision
Jun 18, 2026Roberto Pellerito, Daniel Gehrig, Shintaro Shiba +1Event-Based Vision Representation Learning
Robotics and Perception Group, University of Zurich · University of Pennsylvania · The University of Tokyo +1
GoStop: Reinforcement Learning for Adaptive Temporal Aggregation in Event-Based Feature Tracking
Jul 17, 2026Youngho Kim, Hoonhee Cho, Jae-Young Kang +1Point Tracking Event-Based Vision
KAIST
Spiking Patches: Asynchronous, Sparse, and Efficient Tokens for Event Cameras
Oct 30, 2025Christoffer Koo Øhrstrøm, Ronja Güldenring, Lazaros NalpantidisEvent-Based Vision Visual Tokenization
Technical University of Denmark

Jun 18, 2026J/K move · Enter open · S save

Neural Events: Discrete Asynchronous Autoencoders for Event-Based Vision

Roberto Pellerito, Daniel Gehrig, Shintaro Shiba, Davide Scaramuzza

Robotics and Perception Group, University of Zurich · University of Pennsylvania · The University of Tokyo+1

Event-Based Vision Representation Learning Autoencoders Event-Based Object Detection

Open paper AskPDF arXiv

Event cameras capture dynamic scenes with exceptional temporal fidelity by representing them as a continuous stream of microsecond resolution events. Each individual event, however, only carries minimal semantic value, merely signaling a localized brightness change. To derive meaningful signals, downstream algorithms need to quickly integrate cues from a potentially massive torrent of low-information events. Current architectures, however, are easily overwhelmed, struggling to balance capturing fine-grained temporal dynamics and maintaining a manageable data throughput. This paper proposes a framework to re-tokenize event streams into a small set of highly informative neural events, each representing a local spatio-temporal context window with a discrete learnable code. Every time this code flips, a neural event is triggered, yielding a highly compressed data stream. We demonstrate that, across object detection and classification, networks trained on neural events are on par or surpass the performance of state-of-the-art approaches while reducing the event rate by a factor of 2.0.

Abstract

Representation Learning for RL

Explore similar work

Neural Events: Discrete Asynchronous Autoencoders for Event-Based Vision

GoStop: Reinforcement Learning for Adaptive Temporal Aggregation in Event-Based Feature Tracking

Spiking Patches: Asynchronous, Sparse, and Efficient Tokens for Event Cameras