Hyper-FEOD: Sparse Hypergraph-Enhanced Frame-Event Object Detection with Fine-Grained MoE

Wei Bao; Yuehan Wang; Tianhang Zhou; Siqi Li

cs.CVApr 13, 2026

Hyper-FEOD: Sparse Hypergraph-Enhanced Frame-Event Object Detection with Fine-Grained MoE

Authors: Wei Bao, Yuehan Wang, Tianhang Zhou, Siqi Li

Organizations: BNRist, THUIBCS, BLBCI, School of Software, Tsinghua University · State Key Laboratory of Heavy Oil Processing, China University of Petroleum (Beijing), Beijing, 102249, China

Abstract

The integration of frame-based RGB cameras with event streams constitutes a promising paradigm for robust object detection under challenging dynamic conditions. Nevertheless, effectively modeling intricate multi-modal interactions and reconciling the semantic heterogeneity between RGB and event data remain formidable challenges for high-precision detection. In this paper, we present Hyper-FEOD, a novel high-performance detection framework that synergistically strengthens cross-modal representation learning through two core hypergraph-driven components. Specifically, we first design a Sparse Hypergraph-enhanced Cross-Modal Fusion (SHCF) module that exploits event activity cues to identify motion-critical sparse tokens and performs high-order relational reasoning through hypergraph modeling. This design effectively captures intricate high-order dependencies and rich contextual correlations across modalities. Second, we develop a Fine-Grained Mixture-of-Experts (FG-MoE) module tailored to handle the heterogeneous semantic demands arising from distinct visual regions. By deploying specialized hypergraph experts with varying hyperedge connectivity pattern and incorporating a spatial gating mechanism, FG-MoE adaptively routes features to enable precise enhancement at target regions. Coupled with an auxiliary router loss, the proposed framework ensures stable end-to-end training and optimal feature refinement. Comprehensive experiments conducted on widely-adopted RGB-Event benchmarks show that Hyper-FEOD delivers superior detection performance and consistently outperforms existing state-of-the-art approaches by a notable margin.

Figures & tables

Figure 1: Comparison with existing frame-event fusion paradigms. (a) Prior methods adopt pair-wise interaction with a fixed topology. (b) Our Hyper-FEOD performs high-order multi-to-multi hypergraph interaction, coupled with a Mixture-of-Experts (MoE) for content-adaptive fusion.

Figure 2: Overall architecture of the proposed Hyper-FEOD. A dual-stream backbone first extracts multi-scale features from RGB frames and event streams, which are aligned into middle level via MS-Align. A Fine-Grained Mixture-of-Experts (FG-MoE) then routes the multi-modal aligned features through a dynamic router to a pool of Sparse Hypergraph-enhanced Cross-Modal Fusion (SHCF) experts, each performing event-guided adaptive sampling and sparse hypergraph computation for high-order multi-to-multi cross-modal reasoning. The fused features are scattered back into multi-level features via MS-Scatter and supervised by a base detection loss together with auxiliary load-balancing and router z-losses.

Method	PKU-DAVIS-SOD ( Li et al. 2023 )			DSEC ( Gehrig and Scaramuzza 2024 )			Params (M)
Method	mAP	mAP50	Time (ms)	mAP	mAP50	Time (ms)	Params (M)
JDF ( Li et al. 2019 )	-	44.2	-	-	-	-	$>60^{*}$
FPN-Fusion ( Tomy et al. 2022 )	29.0	57.5	-	-	-	-	-
SFNet ( Liu et al. 2024 )	25.7	50.3	-	30.4	51.4	-	-
EOLO ( Cao et al. 2024 )	19.4	44.8	-	-	-	-	-
NRE-Net ( Liu et al. 2025 )	29.0	57.5	-	-	-	-	-

Table 1: Comparison with state-of-the-art methods on PKU-DAVIS-SOD and DSEC-Detection. Runtime was measured with batch size 1 on an NVIDIA Tesla V100. For Hyper-FEOD, the parameter entry marked by

\dagger

is reported as stored/active under Top-2 inference; Best results are in bold.

Row	SHCF	FG-MoE	mAP	mAP50
1			50.9	71.5
2	✓		51.6	72.3
3		✓	51.9	72.5
4	✓	✓	52.5	73.2

Table 2: Ablation study of Hyper-FEOD on DSEC-Detection. In Row 3, enabling only FG-MoE corresponds to using simple convolutional layers for multimodal fusion.

Row	Sparse Sampling	Hypergraph	mAP	mAP50
1			50.9	71.5
2	✓		51.2	71.9
3		✓	51.4	72.2
4	✓	✓	51.6	72.3

Table 3: Ablation study of SHCF on DSEC-Detection. In Row 2, enabling only Sparse Sampling corresponds to using simple linear layer to refine sparse tokens.

Experts	Top- $k$	Params (M)	mAP	mAP50
4	2	16.07	50.9	71.7
6	2	17.73	51.9	72.6
8	2	19.42	52.5	73.2

Table 4: Ablation study for expert numbers on DSEC-Detection.

Top- $K$	Expert sparsity	mAP	mAP50
2	75%	52.5	73.2
4	50%	52.2	73.0
6	25%	52.2	72.8
8	0%	52.0	72.8

Table 5: Ablation study on Top-

K

selection with 8 experts on DSEC-Detection.

$\mathcal{L}_{\mathrm{bal}}$	$\mathcal{L}_{z}$	mAP	mAP50
		51.6	71.5
✓		52.1	73.0
	✓	52.0	72.8
✓	✓	52.5	73.2

Table 6: Ablation of the MoE auxiliary loss on DSEC-Detection.

Figure 3: Visual comparison of detection results between the baseline and HEDFusion on the DSEC-Detection dataset.

Event-Based Object Detection

Event-based object detection localizes and classifies objects from asynchronous camera events, aiming for fast, robust perception with low latency and power.

See all 36 papers

CMTFormer: Marrying Transformer with Hierarchical Information Interaction for RGB-Event Object Detection
Jun 28, 2026Yu Li, Yuenan Hou, Yingmei Wei +2Event-Based Vision Event-Based Object Detection
aLaboratory for Big Data and Decision, National University of Defense Technology, Changsha, Hunan, China · bShanghai AI Laboratory, Shanghai, Shanghai, China
Event-Adaptive State Transition and Gated Fusion for RGB-Event Object Tracking
Apr 15, 2026Jinlin You, Muyu Li, Xudong ZhaoVisual Object Tracking State Space Models
Dalian University of Technology
M^2C-EvDet: Multi-Domain Multi-Order Cross-Modal Knowledge Distillation for Event-based Object Detection
Jun 23, 2026Wei Bao, Siqi Li, Shouan Pan +2Cross-Modal Knowledge Distillation Event-Based Object Detection
BNRist, THUIBCS, BLBCI, School of Software, Tsinghua University, Beijing 100084, China · School of Economics and Management, Beijing Forestry University, Beijing 100083, China

Jun 28, 2026J/K move · Enter open · S save

CMTFormer: Marrying Transformer with Hierarchical Information Interaction for RGB-Event Object Detection

Yu Li, Yuenan Hou, Yingmei Wei, Jiangming Chen+1

aLaboratory for Big Data and Decision, National University of Defense Technology, Changsha, Hunan, China · bShanghai AI Laboratory, Shanghai, Shanghai, China

Event-Based Vision Event-Based Object Detection Multimodal Fusion Multimodal Object Detection RGB-Event Sensor Fusion

Open paper AskPDF arXiv

Event cameras capture sparse brightness changes with high temporal resolution and high dynamic range, compensating for the deficiencies of the conventional RGB frames. However, previous multi-modal fusion techniques typically fail to handle the inherent heterogeneity between RGB frames and event streams, thus easily leading to noise amplification or redundant feature integration during cross-modal fusion. In this paper, we propose a Cross-Modal information inTeraction transFormer, coined as CMTFormer, which hierarchically integrates RGB and event information to achieve efficient and stable multimodal collaboration. Specifically, we design a shallow-to-deep information interaction scheme. In the shallow stage, we present the Shallow Alignment Module (SAM) to achieve an efficient fusion of RGB and event low-level features, which mitigates attribute disparities and prevents noisy information. In the middle stage, we devise the Cross-modal Enhancement Module (CEM) that utilizes texture and edge information to produce mutually reinforced middle-level features. In the deep stage, we present the Learnable Deep Fusion Module (LDFM) which performs high-level information aggregation through learnable weights, thus enabling the network to adaptively fuse RGB and event clues. A Spatial Prior Module is further designed to utilize global spatial information to enhance localization accuracy. Extensive experiments are conducted on two prevalent event-based object detection benchmarks, i.e., DSEC-Detection and PKU-DAVIS-SOD. Our CMTFormer consistently surpasses the detection counterparts in both uni-modal and multi-modal settings, strongly demonstrating the effectiveness of our paradigm. Codes will be available upon publication.

Abstract

Event-Based Object Detection

Explore similar work

CMTFormer: Marrying Transformer with Hierarchical Information Interaction for RGB-Event Object Detection

Event-Adaptive State Transition and Gated Fusion for RGB-Event Object Tracking

M^2C-EvDet: Multi-Domain Multi-Order Cross-Modal Knowledge Distillation for Event-based Object Detection