Event-Frame Fusion for Inter-Frame Segmentation via Event-Guided Motion

Dalia Hareb; Jean Martinet; Benoit Miramond; Elisabetta Chicca

cs.CVSep 18, 2026

Event-Frame Fusion for Inter-Frame Segmentation via Event-Guided Motion

Authors: Dalia Hareb, Jean Martinet, Benoit Miramond, Elisabetta Chicca

Organizations: I3S, Côte d’Azur University, CNRS, France · LEAT, Côte d’Azur University, France · Zernike Institute, Groningen University, Netherlands

Abstract

Autonomous navigation requires precise and efficient semantic segmentation, yet existing frame-based approaches remain limited by motion blur, glare, latency, and the low temporal resolution (20-30 FPS) of conventional cameras, which leads to information loss between frames. Event cameras have emerged as an alternative sensing modality, capturing intensity changes asynchronously with high temporal resolution, high dynamic range, and sparse outputs. However, event-based algorithms still fall short of frame-based ones in accuracy, as most segmentation methods are designed for dense frame data. To overcome these limitations, we propose a hybrid vision architecture that combines conventional frame-based and event-based cameras. The system integrates two complementary components: (1) a compact Spiking Neural Network (SNN) with 42k parameters for motion estimation, and (2) a lightweight event-driven SNN with 0.84M parameters for frame-based semantic segmentation, which interpolates motion between frames to refine segmentation results. By predicting inter-frame segmentations, the framework achieves segmentation rates of up to 500 Hz with an energy consumption below 1.87 mJ per inference, while maintaining real-time GPU execution at frequencies up to 200 Hz. Additionally, our approach compensates for information loss in frames affected by blur or overexposure, enabling more robust perception in challenging conditions.

Figures & tables

Figure 1: Approach Overview: the first frame at $t_{0}$ is segmented using SegSNet. Events accumulated over the interval $\Delta t$ are processed by the motion estimation network, SegMoSNet, to predict object motion during this period. The estimated motion, illustrated by green arrows on the zoomed-in motion frame, is then used to warp the previous segmentation and generate the updated segmentation at $t_{0}+\Delta t$ .

	w/o motion	w/ motion
	20Hz	100Hz	200Hz	500Hz
MIoU[%]	56.28	56.12	55.76	53.35
MIoU[%]	(-0.00)	(-0.16)	(-0.52)	(-2.93)
Density[%]	-	8.88	5.41	2.67

Table 1: Segmentation performance and event density (percentage of non-zero values in the histogram) across frequencies on DSEC-Semantic

Skipped frames	0 frame	1 frame	2 frames	3 frames
MIoU[%]	56.12	56.71	55.19	54.79
MIoU[%]	(-0.00)	(+0.59)	(-0.93)	(-1.33)

Table 2: Impact of skipping frames on segmentation performance (MIoU) on the DSEC-Semantic dataset. Values in parentheses indicate the decrease in MIoU relative to no frame skipping.

Figure 2: Qualitative results on the DSEC-Semantic dataset: From left to right, we display the following: the RGB image, accumulated events over the $10ms$ interval preceding the RGB capture, the ground-truth segmentation, the segmentation predicted using SegSNet alone, the estimated motion (visualized as green arrows overlaid on the event frame and computed from the $10ms$ events) and finally, the segmentation obtained using our motion-based approach.

Method	Type	Input	Params [M]	MIoU[%]	FPS
Ev-SegSNet [ 2 ]	ANN	E	29.09	51.76	20
ESS [ 31 ]	ANN	E	6.69	51.57	20
ESS [ 31 ]	ANN	E+F	6.69	53.29	20
CMESS [ 38 ]	ANN	E	3.72	57.49	20
CMESS [ 38 ]	ANN	E+F	3.72	59.53	20
HALSIE [ 3 ]	Hybrid	E+F	1.82	52.43	20

Table 3: Comparison of our approach with the state-of-the-art methods using events (E) or events and frames (E+F) for semantic segmentation on DSEC-semantic

	w/o motion	w/ motion
	20Hz	100Hz	200Hz	500Hz
MIoU[%]	67.05	66.75	65.08	59.88
MIoU[%]	(-0.00)	(-0.30)	(-1.97)	(-7.17)
Density[%]	-	1.97	1.07	0.4

Table 4: Segmentation performance and event histogram density across frequencies on DDD17.

Skipped frames	0 frame	1 frame	2 frames	3 frames
MIoU[%]	66.75	62.98	61.38	59.77
MIoU[%]	(-0.00)	(-3.77)	(-5.37)	(-6.98)

Table 5: Impact of skipping frames on segmentation performance (MIoU) on the DDD17 dataset. Values in parentheses indicate the decrease in MIoU relative to no frame skipping.

Method	Type	Input	Params [M]	MIoU[%]	FPS
Ev-SegSNet [ 2 ]	ANN	E	29.09	54.81	20
Ev-SegSNet [ 2 ]	ANN	E+F	29.09	68.36	20
Evdistill [ 34 ]	ANN	E	5.81	58.02	20
ESS [ 31 ]	ANN	E	6.69	61.37	20
ESS [ 31 ]	ANN	E+F	6.69	60.43	20
CMESS [ 38 ]	ANN	E	3.72	58.69	20

Table 6: Comparison of our approach with the state-of-the-art methods on DDD17.

	SegSNet	SegMoSNet	Warping	Real-time
GPU	8.18ms	1.18ms	4.68ms	✓
CPU	34.22ms	13.96ms	31.93ms	✗

Table 7: Runtime comparison on GPU and CPU per inference. Real-time for an inter-frame segmentations predicted at up

200

Hz.

Method	Input	#ACC	#MAC	Energy	FR
ESS [ 31 ] (ANN)	E	46850M	46850M	215.5 1mJ	-
SpikeEDN [ 42 ] (SNN)	E+F	35332M	162M	32.40mJ	0.11
Ours (SegSNet)	F	1586M	60M	1.56mJ	0.29
Ours (SegMoSNet)	E	244M	20M	0.31mJ	0.35
Ours (Total)	E+F	1830M	80M	1.87mJ	-

Table 8: Comparison of #FLOPs between segmentation networks on DSEC-Semantic. FR refers to firing rate of the network.

Method	Input	#ACC	#MAC	Energy	FR
EV-SegSNet [ 2 ] (ANN)	E	9322M	9322M	42.88mJ	-
ESS [ 31 ] (ANN)	E	11700M	11700M	53.82mJ	-
EvDistill [ 34 ] (ANN)	E	29730M	29730M	136.76mJ	-
SpikeEDN [ 42 ] (SNN)	E+F	7211M	66M	6.79mJ	0.091
Ours (SegSNet)	F	355M	15M	0.38mJ	0.31
Ours (SegMoSNet)	E	55M	5M	0.07mJ	0.35

Table 9: Comparison of #FLOPs between segmentation networks on DDD17.

Figure 3: Detailed architecture of SegSNet. "Strd" denotes the stride, H and W represent the input’s height and width and C the number of segmentation classes.

Figure 4: Detailed architecture of SegMoSNet. The upsampling operation is performed using bilinear interpolation.

Figure 5: Recurrent structure representation of the motion estimation network.

Figure 6: Relationship between segmentation accuracy (MIoU) and optical flow accuracy (EPE). The experiments are conducted on a subset of the training set.

	MIoU		Runtime (ms)
	100Hz	200Hz	GPU	CPU
Motion-based (Ours)	56.12%	55.76%	1.18	13.96
OF-based (IDNet)	55.11%	54.13%	122	1350
OF-based (IDNet)	(-0.16)	(-1.63)	( $\times 103$ )	$(\times 96.70)$

Table 10: Comparison of segmentation accuracy at

100

Hz and

200

Hz, along with the average runtime per inference, between our motion estimation network (SegMoSNet) and the optical flow–based network (IDNet) [ 36 ] on the test set. Values between brackets in the MIoU column indicate the performance drop of IDNet relative to SegMoSNet, while those in the runtime column show the speed-up factor of SegMoSNet compared to IDNet.

Explore similar work

Real-time Motion Segmentation with Event-based Normal Flow
Feb 24, 2026Sheng Zhong, Zhongyang Ren, Xiya Zhu +3Event-Based Vision
Neuromorphic Automation and Intelligence Lab (NAIL) at School of Artificial Intelligence and Robotics, Hunan University, Changsha, China. · Capital One. · University of Maryland, College Park.
Motion-aware Event Suppression for Event Cameras
Feb 26, 2026Roberto Pellerito, Nico Messikommer, Giovanni Cioffi +2Event-Based Vision
Robotics and Perception Group, University of Zurich, Switzerland
LiFR-Seg: Anytime High-Frame-Rate Segmentation via Event-Guided Propagation
Mar 22, 2026Xiaoshan Wu, Xiaoyang Lyu, Yifei Yu +3Semantic Segmentation Temporal Video Understanding
The University of Hong Kong · Southern University of Science and Technology

Feb 24, 2026J/K move · Enter open · S save

Real-time Motion Segmentation with Event-based Normal Flow

Sheng Zhong, Zhongyang Ren, Xiya Zhu, Dehao Yuan+2

Neuromorphic Automation and Intelligence Lab (NAIL) at School of Artificial Intelligence and Robotics, Hunan University, Changsha, China. · Capital One. · University of Maryland, College Park.

Event-Based Vision

Open paper AskPDF arXiv

Event-based cameras are bio-inspired sensors with pixels that independently and asynchronously respond to brightness changes at microsecond resolution, offering the potential to handle visual tasks in challenging scenarios. However, due to the sparse information content in individual events, directly processing the raw event data to solve vision tasks is highly inefficient, which severely limits the applicability of state-of-the-art methods in real-time tasks, such as motion segmentation, a fundamental task for dynamic scene understanding. Incorporating normal flow as an intermediate representation to compress motion information from event clusters within a localized region provides a more effective solution. In this work, we propose a normal flow-based motion segmentation framework for event-based vision. Leveraging the dense normal flow directly learned from event neighborhoods as input, we formulate the motion segmentation task as an energy minimization problem solved via graph cuts, and optimize it iteratively with normal flow clustering and motion model fitting. By using a normal flow-based motion model initialization and fitting method, the proposed system is able to efficiently estimate the motion models of independently moving objects with only a limited number of candidate models, which significantly reduces the computational complexity and ensures real-time performance, achieving nearly a 800x speedup in comparison to the open-source state-of-the-art method. Extensive evaluations on multiple public datasets fully demonstrate the accuracy and efficiency of our framework.

Abstract

Explore similar work

Real-time Motion Segmentation with Event-based Normal Flow

Motion-aware Event Suppression for Event Cameras

LiFR-Seg: Anytime High-Frame-Rate Segmentation via Event-Guided Propagation