cs.CVMar 30, 2026
SaveBeyond Scanpaths: Graph-Based Gaze Simulation in Dynamic Scenes
Organizations: GlimpseML · Toyota Motor Europe
Abstract
Accurately modelling human attention is essential for numerous computer vision applications, particularly in the domain of automotive safety. Existing methods typically collapse gaze into saliency maps or scanpaths, treating gaze dynamics only implicitly. We instead formulate gaze modelling as an autoregressive dynamical system and explicitly unroll raw gaze trajectories over time, conditioned on both gaze history and the evolving environment. Driving scenes are represented as gaze-centric graphs processed by the Affinity Relation Transformer (ART), a heterogeneous graph transformer that models interactions between driver gaze, traffic objects, and road structure. We further introduce the Object Density Network (ODN) to predict next-step gaze distributions, capturing the stochastic and object-centric nature of attentional shifts in complex environments. We also release Focus100, a new dataset of raw gaze data from 30 participants viewing egocentric driving footage. Trained directly on raw gaze, without fixation filtering, our unified approach produces more natural gaze trajectories, scanpath dynamics, and saliency maps than existing attention models, offering valuable insights for the temporal modelling of human attention in dynamic environments.
Figures & tables
Figure 1: To model driver attention as part of a dynamical system we encode traffic scenes as heterogeneous scene graphs with nodes corresponding to road structure, driving-relevant objects, and egocentric gaze , representing the driver’s foveated field-of-view. The Affinity Relation Transformer processes these graphs to predict next-step gaze probability distributions. Our dynamical systems approach generates state of the art gaze timeseries , scanpaths , and saliency maps from a single model .
Figure 2: Using upstream perception modules and observed gaze position, synchronised driving video and gaze are converted into a spatiotemporal heterogeneous scene graph with nodes for traffic agents, road structure, and driver foveal view. Each node is assigned a feature vector including appearance and depth, while edges represent the spatiotemporal differences and appearance similarities between nodes. Scene graphs are processed by Affinity Relation Transformer (ART) blocks before an Object Density Network (ODN) predicts a Gaussian-mixture distribution for the next gaze position. We train the model with negative log-likelihood of ground-truth gaze under the predicted mixture. For simulation we employ autoregressive rollout by sampling from the mixture, updating the graph with the sampled position, and repeating; simulated gaze sequences can then be post-processed into scanpaths and saliency maps without additional training.
| Node type | Description |
|---|---|
| vehicle | Detections with labels ‘ car ’, ‘ bicycle ’, ‘ motorcycle ’, ‘ bus ’, ‘ truck ’. |
| person | Detections with label ‘ person ’. |
| static | Detections with labels ‘ traffic light ’, ‘ stop sign ’. |
| gaze | Special node representing the gaze state. |
| structure | Special node representing the road structure. |
Table 1: Node type descriptions.
Figure 3: ART computes messages for each pair of connected source and destination nodes and in the input graph, incorporating their relative affinity into each message. Messages are aggregated into updated destination node vectors, . Our novel relative affinity embeddings are highlighted in red .
Figure 4: Violin plots of vehicle (left) and pedestrian (right) count per-frame in the Focus100, MAAD, and DR(eye)VE datasets.
| Sequence | Dynamics | Saliency | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | Model | TC | DTW | LEV | Fix Dur (s) | Fix Rate (fix/s) | AOI TFF (s) | NSS | IG | AUC |
| Focus100 | Human | 0.46 0.09 | 30.93 3.71 | 1.03 0.09 | 0.44 0.56 | 1.61 0.00 | 0.41 0.56 | - | - | - |
| Gaussian | 0.03 0.01 | 66.22 4.78 | 1.55 0.04 | - | - | - | 3.375 | 8.831 | 0.918 | |
| Itti [ 54 ] | 0.03 0.02 | 157.08 3.13 | 2.03 0.01 | - | - | - | 0.321 | 6.455 | 0.606 | |
| GBVS [ 46 ] | 0.03 0.01 | 133.18 2.37 | 1.98 0.01 | - | - | - | 1.333 | 7.220 | 0.849 | |
| GLC [ 74 ] | 0.16 0.05 | 48.72 12.77 | 1.24 0.12 | 0.21 0.15 | 0.57 0.00 | 1.72 1.48 | 4.481 | 9.162 | 0.914 | |
Table 2: Sequence, dynamics and saliency map metrics for different approaches on Focus100 and MAAD. Arrows indicate the direction of improvement; best score per metric is marked in bold , second is underlined . DTW and LEV are in thousands and are sensitive to sequence length, hence the difference in MAAD and Focus100 scores. Dynamics metrics closest to human statistics follow the same notation. Gaze sequences generated by Itti, GBVS and Gaussian produced insufficient fixations to include in the dynamics comparison. Standard deviation across sequences is reported for sequence metrics and across fixations for dynamics metrics.
Figure 5: Gaze sequences and saliency maps generated on a clip of Focus100. The first column shows human gaze sequences, followed by those generated by ART, SCOUT, ViNet and GLC models. Each trace represents a single simulation, with the y-axis indicating time and x-axis showing left-to-right gaze position; blue marks to the left show detected fixations, and average fixation duration (FD) per method is given. On the right, we display observed fixations for humans and model-generated saliency maps for the same video frames, temporally aligned with the gaze sequences for direct comparison. See the Supplementary for further examples.
| Processor | Time | Head | TC | DTW | LEV |
|---|---|---|---|---|---|
| ART | 20 | ODN | 0.22 0.05 | 42.31 4.88 | 1.23 0.10 |
| HGT [ 51 ] | 20 | ODN | 0.21 0.06 | 42.72 5.17 | 1.28 0.09 |
| HEAT [ 88 ] | 20 | ODN | 0.13 0.05 | 59.50 7.17 | 1.47 0.08 |
| ART | 20 | MDN k=10 | 0.14 0.04 | 44.78 4.80 | 1.30 0.10 |
| ART | 20 | MDN k=20 | 0.14 0.04 | 45.69 3.51 | 1.32 0.09 |
| ART | 8 | ODN | 0.17 0.04 | 43.46 4.50 | 1.26 0.09 |
Table 3: Ablation study across temporal window, graph processor, and output distribution head variants. ART with 20 timesteps and ODN head is our reported method elsewhere.
Figure 6: Examples from Focus100. The top row shows diversity in pedestrian traffic, hazardousness, and road type. The bottom row shows the same video frame overlaid with the gaze samples of three separate subjects over the previous window, where higher alpha of the gaze position corresponds to a more recent sample. The example demonstrates the diversity of temporal gaze patterns across subjects for the same stimuli — information which is lost through averaging in traditional saliency map data representations.
| Category | Measure | MAAD [ 43 ] * | Focus100 |
|---|---|---|---|
| Video | # Videos | 8 | 100 |
| Video Length ( ) | 300 | 60 | |
| Anonymised | no | yes | |
| Resolution ( ) | |||
| FoV ( ∘ ) | unknown | 52 | |
| Frequency ( ) | 25 | 10 |
Table 4: Key statistics of MAAD and Focus100 gaze datasets. MAAD collected gaze over a subset of the DR(eye)VE dataset (8 videos in downtown/urban settings). Detections are presented per frame, with mean and standard deviation across the whole dataset. Note that these detections were estimated on downsampled and image resolutions on Focus100 and MAAD, respectively, matching the resolutions used in our methods.
Figure 7: Frame-level distributions of ego-car manoeuvres and road types in the Focus100 dataset.
Figure 8: Human gaze sequences compared to generated sequences by ART, SCOUT, ViNet and GLC on 3 videos from the test set. We plot the and positions of gaze over time separately, including the -axis for completeness as it was not shown in the main text. Each line represents a sampled gaze sequence, with the mean gaze sequence shown in black.
Figure 9: Illustration of the ART block. The block applies LayerNorm, ART attention with a residual connection, another LayerNorm, followed by a two-layer feed-forward network (FFN) with a second residual connection. and denote the node-type-specific learnable parameters controlling the strengths of the residual connections, .
Figure 10: Preceding frames from the same test subsequence show that the largest values in the ART saliency maps are concentrated at positions corresponding to the ground truth gaze points. As shown in the left part of the figure, the gaze dynamics generated by ART closely follow human gaze patterns, whereas the SCOUT, ViNet, and GLC sequences exhibit notably more volatile behaviour.
Figure 11: Another example sequence from the unseen test set is presented. Consistent with the previous examples, the samples generated by ART closely follow the ground-truth human gaze dynamics, while the other methods exhibit unsteady and less realistic gaze behaviour throughout the sequence. The saliency maps produced by ART closely reflect the distribution of ground-truth human gaze points. In contrast, GLC produces low-variance saliency maps concentrated near the centre of the road. SCOUT and ViNet generate saliency maps that are qualitatively similar to those of ART, except in the third row, where they highlight the pedestrians on the pavement on the right.
Figure 12: An example of a failure case for our method. The ART gaze sequence samples shown on the left indicate that the simulated gaze tends to follow the pedestrian crossing the road, in contrast to the ground truth human gaze sequences.
Figure 13: Object salience ranking estimated as the average mixing node weight calculated over 60 ART simulation runs. The node mixing weights are further normalised by the maximum mixing weight in the given frame. Low saliency objects are shown in blue, and the most salient object is shown in red. Note that the colormaps represent rank order and are not consistent across images. See under Object Salience in Section 9 for details.
Figure 14: Evolution of the expected value of the gaze state change direction as a function of time relative to the start of a fixation. In Figs. 14(a) and 14(b) positive values on the y axis denote a higher probability of the gaze state changing from a saccade to a fixation , while negative values indicate a higher probability of the reverse. The start of a fixation (at ) is preceded by an increased probability of a saccade (the dip on the left), and followed by another slight increase of the saccade probability. The gaze state change probability dynamics are very similar when estimated for the ground truth human gaze samples ( Fig. 14(a) ) and using the samples from ART ( Fig. 14(b) ). In Fig. 14(c) we show the model mechanism , i.e . gaze node mixing weight as a function of time relative to a fixation start. High gaze node mixing weight means a high probability of the gaze maintaining the same location in the next timestep, while a lower mixing weight means a higher probability of the gaze changing its location in the next timestep. Note the similarity of this plot to the ones shown in Figs. 14(a) and 14(b) . The shift on the -axis towards the right is due to the gaze node mixing weight affecting the gaze state at the next timestep .
Figure 15: Residual PSD across all shared test sequences. Curves show mean PSD with SEM across all sequences for Human, ART, SCOUT, VINET, and GLC. ART most closely tracks the human spectral variance profile, consistent with the dynamics results presented in the main text.
Figure 16: Negative log-likelihood loss across the test set against the time taken to process each frame and generate a next-step gaze position. The used input video resolution is shown next to each data point.