Hierarchical Sparse Graph for Multi-dimensional Assignment in Event-based Tracking

Muxi Zha; Yang Shang; Wenlong Zhang; Qifeng Yu; Banglei Guan

cs.CVOct 27, 2026

Hierarchical Sparse Graph for Multi-dimensional Assignment in Event-based Tracking

Authors: Muxi Zha, Yang Shang, Wenlong Zhang, Qifeng Yu, Banglei Guan

Organizations: College of Aerospace Science and Engineering, National University of Defense Technology, Changsha, 410073, Hunan, China · Hunan Provincial Key Laboratory of Image Measurement and Vision Navigation, Changsha, 410073, Hunan, China

Abstract

Existing event-based tracking methods typically operate in an online manner, associating tracklets across adjacent batches. However, this limited temporal window often leads to failures in challenging scenarios, such as long-term occlusions or missed detections, where broader temporal context should be incorporated for high-precision tracking. To achieve robust and accurate tracking, this paper concentrates on the event-based multi-dimensional assignment that associates tracklets across multiple batches simultaneously. Unlike image-based detectors, event-based detectors can generate multiple correct yet spatially distinct tracklets for a single object per batch. This violates the one-to-one rule in classic data association, making event-based association an underdetermined many-to-many matching problem. To the best of our knowledge, this is the first work to tackle this NP-hard problem. To address its underdetermined nature, we formulate the assignment problem via fuzzy logic, where pairwise tracklet correlations are quantified through multiple long-range kinematic metrics. This fuzzy design eliminates the need for hard tracking thresholds and enables multiple heterogeneous metrics to be integrated into a unified correlation. By treating tracklets as nodes and their fuzzy correlations as edges, we construct a Hierarchical Sparse Graph (HSG). Building upon the HSG, we pose assignment as a graph partition task, and employ a heuristic solution that measures vertex affinity for partitioning adaptively. The entire pipeline is training-free. Comprehensive experiments on two challenging event datasets, eTram and Ev-UAV, show that our method outperforms existing event-based and most image-based association models. Particularly, it achieves over 2 $\times$ the performance of OC-SORT across all metrics in the many-to-many association scenario.

Figures & tables

Fig. 1: Illustration of event-based detections and tracklet association. (a) Detection comparison between ground-truth (GT) and the state-of-the-art (SOTA) event-based detector RVT [ 12 ] on eTram [ 13 ] . RVT produces detection boxes at a fixed rate, emulating a frame-like output. (b) 2D and 3D visualizations of tracklets within a 300 ms interval. Distinct colors represent different tracklets. These tracklets are obtained by the SOTA event-based segmentation network EV-SpSegNet [ 14 ] , followed by region growing to cluster spatially connected events for association. (c) Comparisons of two types of event-based data association: image-like one-to-one association, as shown in (a), and event-based many-to-many association, as shown in (b).

Fig. 2: Flowchart of the event-based multi-dimensional assignment optimization. Batches of tracklets from event-based detectors are fed into the fuzzy modeling pipeline. Positional and directional affinities are first computed to evaluate pairwise tracklet relationships. These metrics are integrated via hierarchical and sparse quantization to produce unified correlations, based on which the HSG is constructed. Finally, graph partition is performed on the HSG to identify cohesive partitions as optimal assignments.

Fig. 3: Comparison of quadratic and cubic Bézier curve fitting. Trackets from (a) one object and (b) different objects.

Fig. 4: Two association cases with both low

E_{p}

. (a) Correct and (b) incorrect matching pairs.

Fig. 5: Workflow of our fuzzy modeling pipeline. Compute two kinematic metrics for each pair of tracklets, and an intermediate FCG is generated for each metric. Since these metrics differ in orders and units, we perform hierarchical and sparse quantization on them to generate a unified correlation metric. Finally, the correlations and tracklets are represented as an HSG for subsequent processing.

Fig. 6: Two association cases with direction continuity. (a) Correct matching pairs with big

E_{\theta}

, and (b) incorrect matching pairs with small

E_{\theta}

Fig. 7: Distribution of two metric values. (a) Bézier curve-based position smoothness error $E_{p}$ , where lower values indicate stronger correlation. (b) Direction-displacement-based variation $E_{\theta}$ , where higher values correspond to stronger correlation.

Method	Train	Dataset	F1	MT	ML	MOTA	IDF1
		Split	(%)	(%)	(%)	(%)	(%)
KF+HA	No	Val	99.3	98.9	0.0	1.2	3.7
		Test	99.2	98.9	0.0	1.2	2.9
NS	No	Val	99.3	99.4	0.1	57.7	30.9
		Test	99.2	98.9	0.0	65.0	38.8
IoU	No	Val	99.3	99.4	0.1	87.2	30.7

TABLE I: MOT performance of different event-based association methods on eTram. The best values are highlighted in bold .

Fig. 8: Visualization of GT tracking annotations on eTram. Sequences (from left to right)are val_night_008 and val_night_011, showing GT boxes with few events that limit tracking performance during evaluation.

Fig. 9: Qualitative comparison of OC-SORT and our HSG on two hard eTram sequences under YOLOX detections. Shown case (a) represents small objects with missed detections, and (b) represents dense object distribution and occlusions. Red dashed ellipses highlight the challenging scenarios and result differences between two methods. Two sequences (from left to right) are val_night_001 and val_night_005.

Fig. 10: Qualitative comparison of OC-SORT and our HSG on two hard eTram sequences under RVT detections. Shown case (a) represents objects with partial occlusions, and (b) represents dense object distribution and occlusions. Red dashed ellipses highlight the challenging scenarios and result differences between two methods. Two sequences (from left to right) are test_day_005 and test_day_006.

Method	Tracking	Association	Association	Train	One-to-one	Year	Dataset	F1	MT	ML	MOTA	IDF1
	type	feature type	techniques		constraint		Split	(%)	(%)	(%)	(%)	(%)
							Val	54.7	30.6	18.0	16.5	49.0
Permatrack	MOT-L	M-+A-	HA+ConvGRU [ 59 ]	Yes	Yes	2021	Test	47.9	13.4	21.5	17.1	41.6
								54.7	30.6	18.0	16.9	50.4
							Val	(0.0)	(0.0)	(0.0)	(+0.4)	(+1.4)
								47.9	13.4	21.5	17.6	43.7

TABLE II: Tracking Performance: Our Method vs. SOTA Image-based Trackers under multiple detectors on eTram. Methods sharing the same detection results are grouped in identically colored blocks. In tracking type, -L means linear assignment-based and -M means multi-dimensional assignment-based. In association feature type, M- means motion-based, A- means appearance-based, and Post means post-processing. ‘Train’ column indicates whether the association method utilizes additional cues beyond bounding-box coordinates from a trained network. Regarding performance comparisons, Purple indicates the positive performance gap of our method compared to the baseline , while Dark Green indicates a negative performance gap.

Fig. 11: Visualization of the GT tracking labels and our corrected tracking labels on Ev-UAV. Sequences (from top to bottom): val_16 and test_9.

Fig. 12: Visualization of the object trajectories of GT, proposed HSG, and OC-SORT on Ev-UAV. Sequences feature the challenge of associating tracklets under large spatio-temporal gaps and multiple motion patterns. Sequences (from top to bottom): val_11 $\#$ , val_20 $\#$ , test_6 $*$ , test_10 $*$ , and test_13 $*$ . $\#$ denotes the unmodified sequence, while $*$ denotes the corrected sequence.

Method	Dataset	Track IoU	Track Acc	IDF1
		(%)	(%)	(%)
OC-SORT	Val	21.2	13.1	9.6
	Test	23.3	16.5	8.8
HSG	Val	51.1 (+29.9)	62.3 (+49.2)	66.2 (+56.6)
	Test	47.8 (+24.5)	49.6 (+33.1)	57.3 (+48.5)
OC-SORT	Val $*$	15.3	11.7	4.7

TABLE III: Tracking performance on Ev-UAV.

*

denotes the full set with corrected tracking annotations.

Method	Dataset	FPS	EPS( $\times 10^{6}$ )	Hardware
LTTrack	Val	3.31	–	GPU
	Test	3.68	–
TOPIC	Val	27.56	–
	Test	33.07	–
LPT	Val	38.62	–
	Test	46.48	–

TABLE IV: Running efficiency comparing with association baselines under YOLOX detections on eTram. The best values are in “ bold ”.

Method	Dataset	MT	ML	MOTA	IDF1
		(%)	(%)	(%)	(%)
w/o ➀	Val	42.2	23.2	27.5	52.5
	Test	16.7	39.4	29.0	46.7
w/o ➁	Val	42.3	23.2	27.3	52.7
	Test	16.7	39.4	29.0	46.8
HSG	Val	42.3	23.2	27.3	52.6

TABLE V: Ablation of Bézier curve-based term (➀), and direction-displacement-based term (➁) under YOLOX detections on eTram. “w/o” means without the corresponding component.

Method	Dataset	Track IoU	Track Acc	IDF1
		(%)	(%)	(%)
w/o ➀	Val	41.6	34.9	52.7
	Test	51.6	43.0	59.6
w/o ➁	Val	51.8	61.6	62.5
	Test	46.8	49.6	56.3
HSG	Val	51.1	62.3	66.2

TABLE VI: Ablation of Bézier curve-based term (➀) and direction-displacement-based term (➁) on Ev-UAV.

Method	Dataset	MT	ML	MOTA	IDF1
		(%)	(%)	(%)	(%)
w ➀-l	Val	42.3	23.2	27.2	52.1
	Test	16.7	39.4	29.0	46.9
w ➀-c	Val	42.3	23.2	27.2	52.1
	Test	16.7	39.4	28.9	46.7
HSG	Val	42.3	23.2	27.3	52.6

TABLE VII: Ablation of linear (➀-l), quadratic (➀), and cubic (➀-c) Bézier curve-based term under YOLOX detections on eTram.

Method	Dataset	Track IoU	Track Acc	IDF1
		(%)	(%)	(%)
w ➀-l	Val	54.9	64.9	68.1
	Test	42.4	47.6	59.7
w ➀-c	Val	51.4	67.4	60.5
	Test	40.1	48.3	59.9
HSG	Val	51.1	62.3	66.2

TABLE VIII: Ablation of the linear (➀-l), quadratic (➀), and cubic (➀-c) Bézier curve-based term on Ev-UAV.

Fig. 13: Ablation of the direction-displacement-based variation consistency term (➁) on eTram. Red dashed ellipses highlight cases where multiple tracklets have close positions but different moving directions. Two sequences are presented, separated by a black line: test_day_001 (top) and test_day_005 (bottom).

Method	Detections	Dataset	MOTA (%)	IDF1 (%)
w/o $\lambda$	GT	Val	98.2	80.1
		Test	98.0	77.1
HSG		Val	98.5	95.7
		Test	98.4	95.3
w/o $\lambda$	YOLOX	Val	98.2	80.1
		Test	98.0	77.1

TABLE IX: Ablations of

\lambda

on eTram.

Method	Dataset	MT(%)/p	ML(%)/p	MOTA(%)/p	IDF1(%)/p
w/o ➂	Val	42.3/–	23.2/–	27.2/1.21E-2	52.0/3.99E-2
	Test	16.7/–	39.4/–	29.0/3.56E-4	46.9/2.18E-2
w/o ➃	Val	42.3/–	23.2/–	27.0/4.01E-9	47.2/1.77E-5
	Test	16.7/–	39.4/–	28.8/7.24E-8	42.0/6.25E-6
HSG	Val	42.3	23.2	27.3 ∗‡	52.6 ∗‡
	Test	16.7	39.4	29.0 ∗‡	47.2 ∗‡

TABLE X: Ablation of factors in deriving

\tau

, including node size balancing factor

\gamma

(➂) and denominator

m+1

(➃) under YOLOX detections on eTram.

*

and ‡ denote statistically significant improvement (

p<0.05

) over ➂ and ➃ (paired t-test,

n{=}25

for Val and

n{=}30

for Test), respectively.

Method	Dataset	Track IoU	Track Acc	IDF1
		(%)	(%)	(%)
w/o ➂	Val	51.4	62.5	67.2
	Test	47.8	49.6	57.3
w/o ➃	Val	53.1	64.2	68.7
	Test	47.1	50.3	58.8
HSG	Val	51.1	62.3	66.2

TABLE XI: Ablation of factors in deriving

\tau

, including the balancing factor

\gamma

(➂) and the denominator

m+1

(➃) on Ev-UAV.

Fig. 14: Comparisons of adaptive merging threshold $\tau$ with a series of fixed thresholds. The first row is the results on eTram, and the second row is the results on Ev-UAV.

Multi-Object Tracking

Multi-object tracking (MOT) links detections across video frames to maintain object identities through motion, occlusion, and changing viewpoints.

See all 193 papers

Oct 27, 2026

MvT: Event-based Multi-view Projection for Multiple Object Tracking

Event cameras are increasingly used for Multiple Object Tracking (MOT), but their asynchronous event output often requires specialized methods. Existing processing methods primarily follow two paradigms, pseudo-frames and event-by-event. The former is the prevailing approach since its data format aligns with images, making image-based techniques applicable. However, it suffers from tracking failures when trajectories overlap or are spatially close on pseudo-frames. Facing this challenge, we propose a multi-view pipeline, Multi-view Tracking (MvT), which preserves the 2D data format to leverage image-based techniques directly while introducing additional spatio-temporal views to resolve tracking ambiguities in a single view. MvT comprises a Multi-view Projection (MvP) module and a Multi-view Fusion (MvF) stage. MvP encodes events into three complementary spatio-temporal views while mitigating the pattern discretization. Within MvF, multi-view results are unified into a 3D coordinate system, and tracklets are associated through an optimization model subject to specific criteria combination. Evaluations on four datasets, including our self-collected Small Objects Dataset (SOD), show that MvT seamlessly integrates image-based methods and outperforms existing non-learning and learning trackers in generalized scenarios, and effectively resolves the single-view tracking ambiguities. Being training-free, MvT is applicable when ground-truth annotation is infeasible, thereby highlighting its practical, data-efficient potential. Code is available at https://github.com/zhazhabiu/MvTracking.

Muxi Zha, Banglei Guan, Minzu Liang +4

College of Aerospace Science and Engineering, National University of Defense Technology, Changsha, 410073, Hunan, China · Hunan Provincial Key Laboratory of Image Measurement and Vision Navigation, Changsha, 410073, Hunan, China · College of Science, National University of Defense Technology, Changsha 410073, China +1

Multi-Object Tracking Event-Based Vision

Ask PDF

May 7, 2026

Dynamic Pondering Sparsity-aware Mixture-of-Experts Transformer for Event Stream based Visual Object Tracking

Despite significant progress, RGB-based trackers remain vulnerable to challenging imaging conditions, such as low illumination and fast motion. Event cameras offer a promising alternative by asynchronously capturing pixel-wise brightness changes, providing high dynamic range and high temporal resolution. However, existing event-based trackers often neglect the intrinsic spatial sparsity and temporal density of event data, while relying on a single fixed temporal-window sampling strategy that is suboptimal under varying motion dynamics. In this paper, we propose an event sparsity-aware tracking framework that explicitly models event-density variations across multiple temporal scales. Specifically, the proposed framework progressively injects sparse, medium-density, and dense event search regions into a three-stage Vision Transformer backbone, enabling hierarchical multi-density feature learning. Furthermore, we introduce a sparsity-aware Mixture-of-Experts module to encourage expert specialization under different sparsity patterns, and design a dynamic pondering strategy to adaptively adjust the inference depth according to tracking difficulty. Extensive experiments on FE240hz, COESOT, and EventVOT demonstrate that the proposed approach achieves a favorable trade-off between tracking accuracy and computational efficiency. The source code will be released on https://github.com/Event-AHU/OpenEvTracking.

Shiao Wang, Xiao Wang, Duoqing Yang +5

School of Computer Science and Technology, Anhui University, Hefei 230601, China · Beijing Institute of Technology, Beijing, China · Peng Cheng Laboratory, Shenzhen, China +2

Event-Based Vision Mixture of Experts Visual Object Tracking Vision Transformer Adaptive Computation

Ask PDF

Mar 29, 2026

Spatial Orthogonal Refinement for Robust RGB-Event Visual Object Tracking

Robust visual object tracking (VOT) remains challenging in high-speed motion scenarios, where conventional RGB sensors suffer from severe motion blur and performance degradation. Event cameras, with microsecond temporal resolution and high dynamic range, provide complementary structural cues that can potentially compensate for these limitations. However, existing RGB-Event fusion methods typically treat event data as dense intensity representations and adopt black-box fusion strategies, failing to explicitly leverage the directional geometric priors inherently encoded in event streams to rectify degraded RGB features. To address this limitation, we propose SOR-Track, a streamlined framework for robust RGB-Event tracking based on Spatial Orthogonal Refinement (SOR). The core SOR module employs a set of orthogonal directional filters that are dynamically guided by local motion orientations to extract sharp and motion-consistent structural responses from event streams. These responses serve as geometric anchors to modulate and refine aliased RGB textures through an asymmetric structural modulation mechanism, thereby explicitly bridging structural discrepancies between two modalities. Extensive experiments on the large-scale FE108 benchmark demonstrate that SOR-Track consistently outperforms existing fusion-based trackers, particularly under motion blur and low-light conditions. Despite its simplicity, the proposed method offers a principled and physics-grounded approach to multi-modal feature alignment and texture rectification. The source code of this paper will be released on https://github.com/Event-AHU/OpenEvTracking

Dexing Huang, Shiao Wang, Fan Zhang +1

School of Chemistry & Chemical Engineering, Anhui University, Hefei 230601, China · School of Computer Science and Technology, Anhui University, Hefei 230601, China

RGB-Event Sensor Fusion Visual Object Tracking Multimodal Fusion Event-Based Vision

Ask PDF

Abstract

Multi-Object Tracking

Explore similar work

MvT: Event-based Multi-view Projection for Multiple Object Tracking

Dynamic Pondering Sparsity-aware Mixture-of-Experts Transformer for Event Stream based Visual Object Tracking

Spatial Orthogonal Refinement for Robust RGB-Event Visual Object Tracking