cs.CVMar 29, 2026

Spatial Orthogonal Refinement for Robust RGB-Event Visual Object Tracking

Authors: Dexing Huang, Shiao Wang, Fan Zhang, Xiao Wang

Organizations: School of Chemistry & Chemical Engineering, Anhui University, Hefei 230601, China · School of Computer Science and Technology, Anhui University, Hefei 230601, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Robust visual object tracking (VOT) remains challenging in high-speed motion scenarios, where conventional RGB sensors suffer from severe motion blur and performance degradation. Event cameras, with microsecond temporal resolution and high dynamic range, provide complementary structural cues that can potentially compensate for these limitations. However, existing RGB-Event fusion methods typically treat event data as dense intensity representations and adopt black-box fusion strategies, failing to explicitly leverage the directional geometric priors inherently encoded in event streams to rectify degraded RGB features. To address this limitation, we propose SOR-Track, a streamlined framework for robust RGB-Event tracking based on Spatial Orthogonal Refinement (SOR). The core SOR module employs a set of orthogonal directional filters that are dynamically guided by local motion orientations to extract sharp and motion-consistent structural responses from event streams. These responses serve as geometric anchors to modulate and refine aliased RGB textures through an asymmetric structural modulation mechanism, thereby explicitly bridging structural discrepancies between two modalities. Extensive experiments on the large-scale FE108 benchmark demonstrate that SOR-Track consistently outperforms existing fusion-based trackers, particularly under motion blur and low-light conditions. Despite its simplicity, the proposed method offers a principled and physics-grounded approach to multi-modal feature alignment and texture rectification. The source code of this paper will be released on https://github.com/Event-AHU/OpenEvTracking

Figures & tables

Fig. 1: An overview of our proposed Spatial Orthogonal Refinement for a robust RGB-Event visual object tracking framework.
CategoryTrackerAUC (RSR) ↑\uparrowPR (RPR) ↑\uparrowOP50 ↑\uparrowOP75 ↑\uparrow
RGB-onlySiamRPN++ [ 8 ]21.833.526.17.0
 ATOM [ 17 ]46.571.356.420.1
 DiMP [ 18 ]52.679.165.423.4
 PrDiMP [ 19 ]53.080.565.023.3
RGB-EventATOM+Event [ 14 ]55.581.870.027.4
 DiMP+Event [ 14 ]57.185.171.228.6
TABLE I: Quantitative evaluation on the FE108 dataset. RSR/AUC and RPR/PR represent Success and Precision scores. The best and second best results among fusion-based methods are highlighted accordingly.
VariantAUC ↑\uparrowPR ↑\uparrowNPR ↑\uparrowOP50 ↑\uparrowOP75 ↑\uparrow
SOR w/o StemNet49.3076.1452.9061.9417.48
SOR-Track(No Fine-Tuning)53.2381.4257.2066.5122.14
Gain+3.93+5.28+4.30+4.57+4.66
TABLE II: Quantitative ablation results of the Granular Feature Stem (StemNet) on the FE108 dataset. All metrics are reported in percentage (%). The optimal results are highlighted in bold .
Fig. 2: Qualitative comparison of SOR-Track against the baseline CEUTrack on the FE108 dataset. In this HDR scenario, the baseline (red) drifts to the overexposed background, while our SOR-Track (blue) stays robust.
Fig. 3: Visualization of feature saliency: (a) RGB Input, (b) Event Input, (c) Feature response before SOR refinement, and (d) Concentrated response after SOR refinement. The SOR module effectively suppresses background noise in overexposed regions.
More in this topic

RGB-Event Sensor Fusion

See all 15 papers

Explore similar work

CardsList
Apr 16, 2026

FreqTrack: Frequency Learning based Vision Transformer for RGB-Event Object Tracking

Existing single-modal RGB trackers often face performance bottlenecks in complex dynamic scenes, while the introduction of event sensors offers new potential for enhancing tracking capabilities. However, most current RGB-event fusion methods, primarily designed in the spatial domain using convolutional, Transformer, or Mamba architectures, fail to fully exploit the unique temporal response and high-frequency characteristics of event data. To address this, we1 propose FreqTrack, a frequency-aware RGBE tracking framework that establishes complementary inter-modal correlations through frequency-domain transformations for more robust feature fusion. We design a Spectral Enhancement Transformer (SET) layer that incorporates multi-head dynamic Fourier filtering to adaptively enhance and select frequency-domain features. Additionally, we develop a Wavelet Edge Refinement (WER) module, which leverages learnable wavelet transforms to explicitly extract multi-scale edge structures from event data, effectively improving modeling capability in high-speed and low-light scenarios. Extensive experiments on the COESOT and FE108 datasets demonstrate that FreqTrack achieves highly competitive performance, particularly attaining leading precision of 76.6% on the COESOT benchmark, validating the effectiveness of frequency-domain modeling for RGBE tracking.

Jinlin You, Muyu Li, Xudong Zhao

Dalian University of Technology

Vision TransformerFrequency-Domain Feature LearningRGB-Event Sensor FusionMultimodal Sensor Fusion
AskPDF
Jun 12, 2026

FEMOT: Multi-Object Tracking using Frame and Event Cameras

Conventional RGB cameras have been widely used in multi-object tracking due to their ability to capture rich appearance and semantic information. However, their performance is often degraded under complex real-world challenges, such as motion blur, low illumination, and overexposure. Bio-inspired event cameras offer high temporal resolution and high dynamic range, providing complementary cues under extreme scenarios. Nevertheless, RGB-event multi-object tracking remains underexplored due to the lack of large-scale and well-annotated datasets. To address this issue, we propose FEMOT, a large-scale RGB-event multi-object tracking dataset that covers diverse real-world scenarios and 14 challenging attributes. With both RGB and event data as well as high-quality annotations, FEMOT provides a reliable platform for systematically evaluating RGB-event multi-object tracking methods. Based on FEMOT, we retrain and evaluate over ten strong trackers, thereby establishing a comprehensive benchmark for future research. Furthermore, we propose FEMOTR, a multimodal tracking framework that decouples RGB and event features and fuses them in the frequency domain, thereby effectively exploiting their complementary characteristics for robust object localization and identity association. Extensive experiments on FEMOT and DSEC-MOT datasets demonstrate the effectiveness of the proposed method. The source code and benchmark dataset have been released on https://github.com/Event-AHU/FEMOT.

Shiao Wang, Xiao Wang, Chao Wang +6

School of Computer Science and Technology, Anhui University, Hefei, 230601, China. · 2Peng Cheng Laboratory, Shenzhen, China. · 5Harbin Institute of Technology, Shenzhen, China. +2

Event-Based VisionVisual Object TrackingMultimodal Sensor FusionMulti-Object TrackingMultimodal Visual Object Tracking
AskPDF
May 7, 2026

Dynamic Pondering Sparsity-aware Mixture-of-Experts Transformer for Event Stream based Visual Object Tracking

Despite significant progress, RGB-based trackers remain vulnerable to challenging imaging conditions, such as low illumination and fast motion. Event cameras offer a promising alternative by asynchronously capturing pixel-wise brightness changes, providing high dynamic range and high temporal resolution. However, existing event-based trackers often neglect the intrinsic spatial sparsity and temporal density of event data, while relying on a single fixed temporal-window sampling strategy that is suboptimal under varying motion dynamics. In this paper, we propose an event sparsity-aware tracking framework that explicitly models event-density variations across multiple temporal scales. Specifically, the proposed framework progressively injects sparse, medium-density, and dense event search regions into a three-stage Vision Transformer backbone, enabling hierarchical multi-density feature learning. Furthermore, we introduce a sparsity-aware Mixture-of-Experts module to encourage expert specialization under different sparsity patterns, and design a dynamic pondering strategy to adaptively adjust the inference depth according to tracking difficulty. Extensive experiments on FE240hz, COESOT, and EventVOT demonstrate that the proposed approach achieves a favorable trade-off between tracking accuracy and computational efficiency. The source code will be released on https://github.com/Event-AHU/OpenEvTracking.

Shiao Wang, Xiao Wang, Duoqing Yang +5

School of Computer Science and Technology, Anhui University, Hefei 230601, China · Beijing Institute of Technology, Beijing, China · Peng Cheng Laboratory, Shenzhen, China +2

Event-Based VisionMixture of ExpertsVisual Object TrackingVision TransformerAdaptive Computation
AskPDF