Selective Event Evidence and Persistent Temporal Priors for Event-Based Point Tracking
Authors: Xiaokai Song, Zhang Li, Leqi Liu, Yibin Ye, Kun Wang, Xichao Teng
Organizations: College of Aerospace Science and Engineering, National University of Defense Technology, Changsha 410000, China · Hunan Provincial Key Laboratory of Image Measurement and Vision Navigation, National University of Defense Technology, Changsha 410000, China · Videogrammetry Innovation Center, Hunan Institute of Advanced Technology
Event cameras enable tracking under fast motion and challenging illumination. However, stable long-term point tracking remains challenging: event activity is sparse and varies with scene content and motion, complicating correspondence, while trajectory estimates can drift over long sequences. Together, these challenges motivate a tracker that selectively uses informative event evidence and retains trajectory constraints across temporal windows. We propose Selective Event Evidence and Persistent Temporal Priors (SEPT), an event-only framework for tracking any point (TAP) over long sequences. SEPT introduces two complementary modules: Task-Adaptive Event Evidence Modulation (TEEM) and Persistent Trajectory Prior (PTP). Using only tracking supervision, TEEM learns a bounded reliability field that weights the event tensor across space, time, and channels before feature extraction. PTP carries geometric information across overlapping windows by marginalizing a fixed local quadratic trajectory model with a Schur complement. Its visibility-aware correction refines overlap coordinates, while a clipped terminal velocity initializes coordinates at newly exposed timestamps. SEPT builds on the ETAP architecture. SEPT is trained exclusively on synthetic EventKubric data and evaluated on five datasets across TAP and feature tracking. Results show that SEPT outperforms the retrained ETAP baseline in average Jaccard for TAP and in feature age and expected feature age for feature tracking; it also surpasses several methods combining event and frame inputs on EC and EDS. The source code is publicly available at https://github.com/sssxxxkkkk/SEPT.
Figures & tables
Fig. 1: Bottlenecks and corresponding solutions in event-based point tracking. (a) Sparse, nonuniform event activity yields competing local matches; TEEM learns task-adaptive event modulation before feature extraction. (b) As the tracking window advances, historical constraints are discarded and trajectories may drift; PTP carries a Schur-style trajectory prior into the next window.
Fig. 2: Overview of SEPT. TEEM learns spatiotemporal and channel-wise weights for the event tensor before feature extraction. The tracker iteratively refines query trajectories using local correlations and joint attention. At a window transition, PTP transfers overlap coordinates, visibility logits, and a clipped terminal displacement; its fixed Schur precision couples overlap-coordinate deviations during refinement in the next window. Refined latent descriptors are not reused across windows.
Fig. 3: Schur-style persistent trajectory prior. Expired states are eliminated from a fixed quadratic perturbation model, leaving precision P over the retained overlap. The next window copies overlap coordinates and visibility logits and initializes coordinates at newly exposed timestamps using a clipped terminal displacement. After each transformer update, a validity-masked and visibility-weighted preconditioned gradient softly corrects only overlap coordinates; coordinates at newly exposed timestamps receive no prior correction. State counts are illustrative.
Fig. 4: Synthetic EventKubric examples with varied motion. In each pair (a)–(f), the left panel shows the rendered RGB frame and the right panel its event representation. Colored markers and curves indicate query locations and ground-truth trajectories; an × marks a query point that is occluded at the displayed time.
Dataset
Method
AJ ↑
δavgx↑
OA ↑
EventKubric
ETAP †
0.512
0.633
0.951
SEPT †
0.578
0.693
0.955
EVIMO2
ETAP †
0.635
0.751
0.943
SEPT †
0.673
0.786
0.936
TABLE I: TAP results on EventKubric and EVIMO2. ETAP [ 16 ] is our retrained baseline; SEPT combines TEEM and PTP.
Method
Input
EC
EDS
FA
EFA
FA
EFA
ICP [ 21 ]
E
0.256
0.245
0.060
0.040
EKLT [ 12 ]
E+F
0.811
0.775
0.325
0.205
DDFT [ 14 ]
E+F
0.825
0.818
0.576
0.472
FE-TAP [ 40 ]
E+F
0.844
0.838
0.676
0.589
EM-ICP [ 22 ]
E
0.337
0.334
0.161
0.120
TABLE II: Feature tracking on EC and EDS. E denotes events; E+F denotes events and frames. MATE results are quoted from its original paper. M3ED results are reported by sequence in Table III .
Sequence
HASTE [ 41 ]
AEB-Tracker [ 43 ]
DDFT [ 14 ]
ETAP †
SEPT † (ours)
car_urban_day_ penno_small_loop
0.733/0.669
0.530/0.455
0.813/0.807
0.831 / 0.829
0.824/0.822
car_urban_night_ penno_big_loop
0.693/0.632
0.553/0.504
0.683/0.657
0.836/0.828
0.854 / 0.845
car_urban_night_ penno_small_loop
0.577/0.541
0.457/0.429
0.557/0.540
0.736 / 0.735
0.731/0.729
falcon_indoor_ flight_1
0.576/0.502
0.426/0.387
0.652/0.644
0.740/0.739
0.742 / 0.741
falcon_indoor_ flight_2
0.611/0.558
0.539/0.482
0.750/0.741
0.827 / 0.825
0.802/0.801
falcon_outdoor_day_ fast_flight_2
0.614/0.606
0.434/0.396
0.799/0.704
0.878/0.877
0.881 / 0.881
TABLE III: Per-sequence feature tracking on the nine evaluated M3ED left-camera segments. Entries are FA/EFA, averaged over localization thresholds from 1 to 31 pixels. DDFT [ 14 ] uses its event-only configuration. The final row is the unweighted mean across sequences.
Fig. 5: Qualitative feature-tracking results on M3ED. Rows show four sequences; columns compare SEPT, DDFT [ 14 ] , and HASTE [ 41 ] from left to right. Colored curves show predicted tracks, and × marks the ground-truth point position at each timestamp.
TAP
Feature tracking
EventKubric
EVIMO2
EC
EDS
M3ED
Model
AJ
δavgx
OA
AJ
δavgx
OA
FA
EFA
FA
EFA
FA
EFA
Standard training split (80:15:5)
ETAP
0.524
0.643
0.949
0.622
0.737
0.940
0.839
0.833
0.699
0.599
0.789
0.787
ETAP+TEEM
0.531
0.649
0.944
0.655
0.771
0.937
0.797
0.792
0.719
0.605
0.795
0.795
ETAP+PTP
0.566
0.681
0.946
0.663
0.777
0.943
0.866
0.861
0.726
0.614
0.797
0.794
TABLE IV: Module ablation and training-data sensitivity across TAP (EventKubric and EVIMO2) and feature tracking (EC, EDS, and M3ED). Bold marks the best score for each dataset and metric within a training split.
Fig. 6: TEEM visualizations on real event recordings across datasets and timestamps. Each example shows the input event tensor, the learned event-weight map, and the modulated event evidence. Color bars indicate the weight values.
Fig. 7: TEEM evidence at visible object boundaries on EventKubric. Segmentation-derived ground-truth contours (green) are overlaid on the aligned RGB frame, original event tensor, and TEEM-modulated event evidence.
Fig. 8: EventKubric AJ versus (a) mean inference time per forward pass and (b) peak reserved GPU memory for ETAP and the ablated variants. Peak allocated memory changes little across variants and is not plotted. Results use the expanded training split.
Point tracking follows selected or dense image points through video, enabling motion analysis, 3D reconstruction, and precise spatiotemporal understanding.
Tracking any point (TAP) is a fundamental yet challenging task in computer vision, requiring high precision and long-term motion reasoning. Recent attempts to combine RGB frames and event streams have shown promise, yet they typically rely on synchronous or non-adaptive fusion, leading to temporal misalignment and severe degradation when one modality fails. We introduce TAPFormer, a transformer-based framework that performs asynchronous temporal-consistent fusion of frames and events for robust and high-frequency arbitrary point tracking. Our key innovation is a Transient Asynchronous Fusion (TAF) mechanism, which explicitly models the temporal evolution between discrete frames through continuous event updates, bridging the gap between low-rate frames and high-rate events. In addition, a Cross-modal Locally Weighted Fusion (CLWF) module adaptively adjusts spatial attention according to modality reliability, yielding stable and discriminative features even under blur or low light. To evaluate our approach under realistic conditions, we construct a novel real-world frame-event TAP dataset under diverse illumination and motion conditions. Our method outperforms existing point trackers, achieving a 28.2% improvement in average pixel error within threshold. Moreover, on standard point tracking benchmarks, our tracker consistently achieves the best performance. Project website: tapformer.github.io
Jiaxiong Liu, Zhen Tan, Jinpu Zhang +4
National University of Defense Technology · Hunan University
Event-based 3D tracking enables low-latency and high-speed perception, while existing CNN- and Transformer-based trackers struggle to capture long-range spatiotemporal dependencies in sparse, noisy event streams, especially under real-time and efficiency constraints. To address these challenges, we present E-TraMamba, the first Mamba-based framework for 3D feature tracking on event data. This new framework adopts a linear state-space model for efficient long-range modeling and integrates a lightweight affine-transform predictor to maintain stable tracking under motion blur and occlusion. We also design an effective scheme to fuse multi-scale cues -- local spatiotemporal patches, correlation maps, and positional embeddings -- into a unified representation that enables stable and smooth 3D tracking. We construct a large-scale synthetic dataset, named EvD-PointOdyssey, which is generated with monocular rendering and provides synchronized event streams, depth maps, and accurate 3D trajectories for training and evaluating event-based 3D tracking models. Extensive experiments on event-based benchmarks demonstrate that E-TraMamba achieves state-of-the-art performance, delivering over 2× longer feature lifetimes under strict accuracy thresholds (e.g., 0.1 m), with higher tracked-feature ratios and lower RMSE than all baselines. These results make E-TraMamba a strong candidate for low-latency visual odometry, real-time SLAM, and interactive robotics.
Juwei Shen, Yujie Wu, Changwen Chen
Department of Computing, FCMS The Hong Kong Polytechnic University Hong Kong, China
Feature tracking plays a fundamental role in understanding scene motion and supports various downstream tasks. Event cameras, with their high temporal resolution and asynchronous sensing, enable low-latency and motion-robust perception, making them well-suited for feature tracking under fast and non-linear motion. However, existing event-based feature tracking methods rely on fixed heuristic rules based on hand-tuning for event accumulation. Such strategies fail to adapt to diverse motion dynamics, leading to degraded performance under abrupt motion changes or low-motion scenarios. In this paper, we model event accumulation as a sequential decision-making problem and introduce reinforcement learning (RL) framework to adaptively control the accumulation process for online event-based feature tracking. Our approach trains a RL agent that decides whether to continue accumulating events or to perform tracking inference based on motion cues. The proposed adaptive temporal agent enables dynamic adaptation to varying motion patterns without relying on hand-crafted rules. Furthermore, we introduce a Dynamic Event-based Tracking (DEFT) dataset with dynamic motion distributions to evaluate the robustness of the feature tracking. Extensive experiments demonstrate that integrating our plug-and-play framework to existing feature tracking methods consistently outperforms heuristic-based approaches, improving robustness under dynamic motion while offering a better balance between tracking accuracy and efficiency. Our project codes and datasets are available at https://github.com/kmax2001/GoSTOP