Eye feature extraction from event-based data streams can be performed efficiently and with low energy consumption, offering great utility to real-world eye tracking pipelines. However, few eye feature extractors are designed to handle sudden changes in event density caused by the changes between gaze behaviors that vary in their kinematics, leading to degraded prediction performance. In this work, we address this problem by introducing the adaptive inference state space model (AISSM), a novel architecture for feature extraction that is capable of dynamically adjusting the relative weight placed on current versus recent information. This relative weighting is determined via estimates of the signal-to-noise ratio and event density produced by a complementary dynamic confidence network. Lastly, we craft and evaluate a novel learning technique that improves training efficiency. Experimental results demonstrate that the AISSM system outperforms state-of-the-art models for event-based eye feature extraction.
Figures & tables
Figure 1: Overall architecture of our adaptive inference state space model : (a) The core architecture. The encoder forms a posterior distribution of the current event information q(st−1) while the transition model utilizes a recurrent neural network (RNN) to produce a prior distribution over the next frame p(st∣st−1,ht−1) . The RNN output ( ht−1 ) is also used to update the prior at the next timestep. (b) The head is responsible for the estimation of the eye feature y^ . Its input is an α -weighted linear combination of the prior and posterior distributions over the latent state, i.e., it is a weighted summation of current and prior event information. (c) The dynamic confidence network predicts the weighting term α that is used in (b) , which is an estimate of the current information’s ‘reliability’. (d) The inner workings of the AISSM’s encoder module, which is a combination of CNNs and a multi-layered perceptron (MLP), whose output is reshaped into a 2 D (categorical distribution) matrix that represents the present data state. (e) The inner workings of the AISSM’s transition module, which is a concatenation of the prior recurrent state ( ht−1 ) and the representation state ( st−1 ) sampled from the previous posterior q(st−1) . This input is passed through an MLP, forming the prior (categorical) distribution over past data p(st) .
Model/Metrics
P 5 ↑
P 10 ↑
P 15 ↑
Distance ↓
CNN
17.67±10.88
41.82±20.04
60.17±20.48
0.03±0.01
CNN-GRU
25.33±14.28
48.62±26.66
62.44±28.47
0.03±0.01
CNN-Mamba
16.02±14.38
45.43±39.38
55.61±41.01
0.03±0.01
CNN-BiGRU
21.63±21.48
41.95±41.47
49.33±48.00
0.03±0.01
CB-ConvLSTM
29.86±15.30
57.40±21.25
71.94±18.21
0.03±0.01
MambaPupil
11.35±11.33
32.84±32.37
48.22±45.95
0.13±0.06
Table 1: Test/validation accuracy, P 5 , P 10 , P 15 , and normalized predicted error distance metrics for each model in the 3ET+ dataset at a resolution of 320×320 pixels. ↑Higher mean is better. ↓Lower mean is better. Note that cells highlighted in blue/cyan denote the best performance. All models were implemented under this work’s experimental parameter constraints in order to ensure fair comparison.
Figure 2: The relationship between event density (blue) and successful prediction (hit) within the range of 10 pixels (orange). The x -axis represents the time-steps within an event frame. Top depicts the CNN-GRU; bottom depicts the AISSM.
Figure 3: Our AISSM’s validation accuracy when training with (orange) and without (blue) our proposed long-horizon training technique.
Feature tracking plays a fundamental role in understanding scene motion and supports various downstream tasks. Event cameras, with their high temporal resolution and asynchronous sensing, enable low-latency and motion-robust perception, making them well-suited for feature tracking under fast and non-linear motion. However, existing event-based feature tracking methods rely on fixed heuristic rules based on hand-tuning for event accumulation. Such strategies fail to adapt to diverse motion dynamics, leading to degraded performance under abrupt motion changes or low-motion scenarios. In this paper, we model event accumulation as a sequential decision-making problem and introduce reinforcement learning (RL) framework to adaptively control the accumulation process for online event-based feature tracking. Our approach trains a RL agent that decides whether to continue accumulating events or to perform tracking inference based on motion cues. The proposed adaptive temporal agent enables dynamic adaptation to varying motion patterns without relying on hand-crafted rules. Furthermore, we introduce a Dynamic Event-based Tracking (DEFT) dataset with dynamic motion distributions to evaluate the robustness of the feature tracking. Extensive experiments demonstrate that integrating our plug-and-play framework to existing feature tracking methods consistently outperforms heuristic-based approaches, improving robustness under dynamic motion while offering a better balance between tracking accuracy and efficiency. Our project codes and datasets are available at https://github.com/kmax2001/GoSTOP
Existing Vision Mamba-based RGB-Event(RGBE) tracking methods suffer from using static state transition matrices, which fail to adapt to variations in event sparsity. This rigidity leads to imbalanced modeling-underfitting sparse event streams and overfitting dense ones-thus degrading cross-modal fusion robustness. To address these limitations, we propose MambaTrack, a multimodal and efficient tracking framework built upon a Dynamic State Space Model(DSSM). Our contributions are twofold. First, we introduce an event-adaptive state transition mechanism that dynamically modulates the state transition matrix based on event stream density. A learnable scalar governs the state evolution rate, enabling differentiated modeling of sparse and dense event flows. Second, we develop a Gated Projection Fusion(GPF) module for robust cross-modal integration. This module projects RGB features into the event feature space and generates adaptive gates from event density and RGB confidence scores. These gates precisely control the fusion intensity, suppressing noise while preserving complementary information. Experiments show that MambaTrack achieves state-of-the-art performance on the FE108 and FELT datasets. Its lightweight design suggests potential for real-time embedded deployment.
Gaze event detection is fundamental to vision science, human-computer interaction, and applied analytics. However, current workflows often require specialized programming knowledge and careful handling of heterogeneous raw data formats. Classical detectors such as I-VT and I-DT are effective but highly sensitive to preprocessing and parameterization, limiting their usability outside specialized laboratories. This work introduces a code-free, large language model (LLM)-driven pipeline that converts natural language instructions into an end-to-end analysis. The system (1) inspects raw eye-tracking files to infer structure and metadata, (2) generates executable routines for data cleaning and detector implementation from concise user prompts, (3) applies the generated detector to label fixations and saccades, and (4) returns results and explanatory reports, and allows users to iteratively optimize their code by editing the prompt. Evaluated on public benchmarks, the approach achieves accuracy comparable to traditional methods while substantially reducing technical overhead. The framework lowers barriers to entry for eye-tracking research, providing a flexible and accessible alternative to code-intensive workflows.
Dongyang Guo, Yasmeen Abdrabou, Enkelejda Kasneci
Technical University of Munich, Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · Munich Center for Machine Learning (MCML), Germany