Video saliency prediction is inherently harder to model than static image saliency due to the additional temporal dimension. Video saliency benchmarks rest on the premise that predicting gaze on video requires utilizing temporal activity distributed across frames. Prior work has challenged this, showing that static baselines recover a significant fraction of the explainable gaze information on LEDOV, a popular video saliency dataset, and that video saliency models fail in the same places as this static baseline. We verify that this diagnosis still stands: under a more capable gold standard than the original analysis, and an updated panel of recent architectures, the strongest temporal architecture in the panel still does not substantially improve over a fine-tuned static baseline. However, it remains unclear whether the marginal gain reflects limitations of current temporal architectures or a lack of temporal patterns in the benchmark itself. We introduce SalTempto, a video saliency benchmark with greater dynamism: 224 clips of highly dynamic content, sourced from the HACS-Segments dataset so that each clip contains an event together with its lead-up and aftermath, with gaze recordings from up to 16 subjects and a training split for adapting pretrained models. On SalTempto, the static baseline recovers only about 13% of the headroom above the centerbias, against more than half on LEDOV. A fine-tuned temporal architecture shows a substantial gain in performance over the static baseline, indicating that it does capture meaningfully more temporal information, which LEDOV fails to measure. Yet, even this SoTA model still leaves nearly half of SalTempto's headroom unexplained, indicating room for improvement in video saliency modelling. Examination of SalTempto also lets us describe human tendencies that models miss. SalTempto link: https://huggingface.co/datasets/bethgelab/video_saliency.
Figures & tables
Figure 1: Static-vs-temporal performance and remaining headroom on LEDOV (left) and SalTempto (right). Each panel shows the dataset’s IG over centerbias LL on the y− axis and the gold-standard IG ceiling (dashed) at the top. Each model’s bar is split into a closed segment (solid; percent of headroom recovered) and a remaining segment (faded; percent unexplained). Both models are scored on the same videos, so error bars on the model markers are 95 % intervals for the paired comparison between them, obtained by removing each video’s mean over the two models ( Cousineau, 2005 ; Morey, 2008 ) ; intervals for absolute IG and for the gold ceiling are in Table 2 . The static-vs-temporal climb (faded line connecting DG-MR to SalFoM) is steeper on SalTempto than on LEDOV, indicating that SalFoM extracts a lot more information than the static baseline on SalTempto than on LEDOV.
Figure 2: (a) Per-frame information-gain residual LL(gold)−LL(SalFoM-Tuned) on LEDOV (green) vs SalTempto (orange). SalTempto is broader and less skewed, with a greater mean performance gap and a heavier tail. (b) Cross-dataset IG (bits) on 10 models evaluated in zero-shot on both LEDOV (x-axis) and SalTempto (y-axis); models trained or fine-tuned on either dataset are excluded to compute zero-shot performance rankings. Error bars are 95 % intervals for paired comparisons between models ( Cousineau, 2005 ; Morey, 2008 ) . Spearman ρ=0.83 , implying model rankings are tightly preserved across datasets; the correlation including the LEDOV-pretrained models (zero-shot SalTempto) is ρ=0.74 (App. A ).
Figure 3: Object permanence despite occlusion. Workshop scene: A hot metal rod in a vice/stand (top-left, red border) gets transiently occluded by a gloved hand sweeping across (top-right, blue border). Gold-standard fixations remain locked at the rod’s position despite occlusion. SalFoM redistributes mass onto the visible motion (the glove); ViNet-E concentrates centrally on the hand region; DeepGaze MR is diffuse across multiple salient elements. Generally, models attend to currently-visible salient content; subjects maintain attention on the (now-occluded) target.
Figure 4: Anticipatory saccades. Badminton smash: shuttle moving right-to-left after the strike (blue arrows mark trajectory across frames, blue boxes locate shuttle). Subject fixations form a smooth trail spanning the shuttle’s trajectory, with mass both trailing the shuttle (pursuit lag) and leading it (anticipatory saccades). The models concentrate mass primarily at the players, missing the entity that viewers find most salient: the moving shuttle.
Figure 5: Scene inertia. The video cuts between different stances of two fencers. The first clip, pre-cut (left, red border), has both fencers at the right of the scene. Post-cut (right, blue border), the scene shows the fencers near the starting positions in a span of just a few frames. The human gaze readings (green dots) are highly consistent between the frames - the Gold Standard shows the highest densities at salient pre-cut regions that are non-salient post-cut. Deep Learning models quickly redistribute attention to the foreground figures based on current-frame salience. The effect is severe enough to perturb the gold standard itself due to its deep learning model component, so the residual-to-gold here underestimates the architectural shortfall by estimating high-variance low-peak densities.
Figure 6: Viewing-onset biases. Karate dojo clip from SalTempto at the onset of the viewing session. Top-left: gold-standard density concentrates tightly at the center, reflecting the centre-anchored gaze that follows clip onset. Top-right and bottom row: SalFoM, ViNet-E, and DeepGaze MR distribute predicted attention across multiple visually salient agents and fail to capture scene-onset behaviour.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Model
LEDOV
SalTempto
val
test
val
test
DG-MR
55.6 %
56.4 %
15.8 %
10.2 %
SalFoM
75.4 %
75.2 %
55.6 %
61.4 %
Appendix
Table 1: IGE (Eq. 1 ) for the static baseline (DG-MR) and the strongest temporal model (SalFoM) on each dataset’s validation and test splits post fine-tuning. SalFoM is fine-tuned by us on each dataset’s training split. DG-MR is fine-tuned by us on SalTempto and uses the released LEDOV-trained checkpoint from Tangemann et al. (2020) on LEDOV.
Dataset
Model
IG (bits)
IGE (%)
mean
95 % jackknife
mean
95 % jackknife
LEDOV
Gold standard
2.359
[2.210, 2.507]
100.0
—
DG-MR (static)
1.321
[1.170, 1.472]
56.0
[51.6, 60.4]
SalFoM (temporal)
1.775
[1.631, 1.918]
75.2
[72.6, 77.9]
Gap (SalFoM − DG-MR)
0.453
[0.371, 0.535]
19.2
[15.8, 22.7]
SalTempto
Gold standard
1.864
[1.613, 2.115]
100.0
—
Appendix
Table 2: Fine-tuned results with leave-one-video-out jackknife intervals. IG is the information gain over the centerbias in bits per fixation and IGE the fraction of the centerbias-to-gold headroom recovered, both pooled over the validation and test videos of each dataset (77 videos on LEDOV, 20 on SalTempto) with every video weighted equally. Intervals are 95 % jackknife intervals over videos. Gap rows give SalFoM minus DG-MR.
Method
FT
LL
IGE
KLD
CC
SIM
AUC
NSS
Best gold standard (adaptive 4-mix + SalFoM)
—
3.1943
100.0 %
0.7102
0.8175
0.5591
0.9755
8.5661
SalFoM
✓
2.3583
55.6 %
1.5343
0.4816
0.3331
0.9147
3.0683
SalFoM
2.2784
51.4 %
1.5807
0.4702
0.3321
0.9140
2.9986
TMFI-Net
2.0917
41.5 %
1.8402
0.4363
0.3091
0.9045
2.7174
ViNet-E
1.9527
34.1 %
1.7652
0.4016
0.2738
0.8955
2.4230
ViNet-A
1.9407
33.4 %
1.7607
0.4011
0.2820
0.8955
2.4431
Appendix
Table 3: SalTempto validation results. FT = per-dataset fine-tuning on SalTempto train. KLD lower-is-better; LL, IGE, CC, SIM, AUC, NSS higher-is-better. Sorted by IGE descending.
Method
FT
LL
IGE
KLD
CC
SIM
AUC
NSS
Best gold standard (adaptive 4-mix + SalFoM)
—
3.1579
100.0 %
0.7496
0.7954
0.5523
0.9697
8.1241
SalFoM
✓
2.4462
61.4 %
1.4629
0.4927
0.3500
0.9158
3.1212
SalFoM
2.3653
57.0 %
1.5091
0.4818
0.3433
0.9125
3.0137
TMFI-Net
2.1345
44.5 %
1.7996
0.4563
0.3228
0.8995
2.8311
TinyHD-single
1.9555
34.8 %
1.7897
0.4082
0.2665
0.8915
2.4952
UniformerSal-Spatial
1.9025
31.9 %
1.5533
0.4657
0.3633
0.9002
2.4965
Appendix
Table 4: SalTempto test results, sorted by IGE descending.
Method
FT
LL
IGE
KLD
CC
SIM
AUC
NSS
Best gold standard (adaptive 4-mix + SalFoM)
—
3.6948
100.0 %
0.4009
0.8861
0.7117
0.9808
9.6380
SalFoM
✓
3.0937
75.4 %
0.9234
0.7289
0.5401
0.9368
4.5045
UniformerSal-SpatioTemporal
✓ †
2.8710
66.2 %
1.0990
0.6630
0.4900
0.9300
4.2670
SalFoM
2.8472
65.3 %
1.0556
0.6919
0.5031
0.9254
4.2242
TMFI-Net
2.7959
63.1 %
1.0804
0.6855
0.4732
0.9244
4.0483
UniformerSal-Spatial
✓ †
2.6471
57.0 %
1.0030
0.6778
0.5353
0.9203
3.9882
Appendix
Table 5: LEDOV validation results. IGE is the fraction of the centerbias-to-gold-standard headroom recovered. FT = per-dataset fine-tuning on LEDOV train; † released checkpoint already trained on LEDOV by the original authors (no further fine-tuning in this work). KLD lower-is-better; LL, IGE, CC, SIM, AUC, NSS higher-is-better. Sorted by IGE descending. Gold-standard centerbias and gold LL: 1.2560 / 3.6948.
Method
FT
LL
IGE
KLD
CC
SIM
AUC
NSS
Best gold standard (adaptive 4-mix + SalFoM)
—
3.6209
100.0 %
0.3955
0.8825
0.7122
0.9803
9.5950
SalFoM
✓
3.0524
75.2 %
0.8969
0.7341
0.5372
0.9378
4.3478
UniformerSal-SpatioTemporal
✓ †
2.9056
68.7 %
0.8218
0.7361
0.5766
0.9329
4.2549
SalFoM
2.8309
65.5 %
1.0163
0.6920
0.5139
0.9288
4.0333
TMFI-Net
2.7860
63.5 %
1.0408
0.6864
0.4780
0.9280
3.8610
UniformerSal-Spatial
✓ †
2.7021
59.8 %
0.9206
0.6886
0.5443
0.9258
3.8798
Appendix
Table 6: LEDOV test results. IGE is the fraction of the centerbias-to-gold-standard headroom recovered. FT = per-dataset fine-tuning on LEDOV train; † released checkpoint already trained on LEDOV by the original authors (no further fine-tuning in this work). KLD lower-is-better; LL, IGE, CC, SIM, AUC, NSS higher-is-better. Sorted by IGE descending. Gold-standard centerbias and gold LL: 1.3329 / 3.6209.
LEDOV
SalTempto
Deep component
ceiling
SalFoM
DG-MR
gap
ceiling
SalFoM
DG-MR
gap
SalFoM (reported)
2.359
75.2 %
56.0 %
19.2
1.864
58.5 %
13.0 %
45.5
UniformerSal
2.328
76.2 %
56.7 %
19.5
1.831
59.5 %
13.3 %
46.3
DeepGaze MR
2.283
77.7 %
57.9 %
19.9
1.808
60.3 %
13.4 %
46.8
TMFI-Net
2.252
78.8 %
58.7 %
20.2
1.848
59.0 %
13.1 %
45.8
Appendix
Table 7: Gold-standard sensitivity to the deep mixture component. Ceiling is the gold-standard IG over the centerbias in bits, pooled over validation and test videos; SalFoM and DG-MR columns give the IGE of the fine-tuned models under that ceiling, and gap is their difference in percentage points.
Dataset
Row
IG (bits)
95 % interval
IGE (%)
LEDOV
DG-MR (fine-tuned)
1.321
[1.254, 1.389]
56.0
SalFoM, frame repeated
1.515
[1.452, 1.579]
64.3
SalFoM (fine-tuned)
1.775
[1.725, 1.824]
75.2
Temporal context (SalFoM − frame repeated)
0.259
[0.185, 0.333]
11.0
Static features (frame repeated − DG-MR)
0.194
[0.095, 0.293]
8.2
SalTempto
DG-MR (fine-tuned)
0.243
[0.072, 0.414]
13.0
Appendix
Table 8: SalFoM with and without temporal context. The frame-repeated row feeds the fine-tuned SalFoM the target frame at every position of its context window. IG is over the centerbias in bits per fixation, pooled over the released evaluation videos of each dataset (77 on LEDOV, 20 on SalTempto); the IGE column gives the share of the centerbias-to-gold headroom (Eq. 1 ), or IGE points for the two difference rows. The three model rows within a dataset are scored on the same videos and carry 95 % intervals for paired comparisons among them, obtained by removing each video’s mean over the three rows ( Cousineau, 2005 ; Morey, 2008 ) ; the difference rows are paired over the same videos with 95 % t -intervals. Intervals for absolute IG are in Table 2 .
Split
KL (bits)
LL (bits)
Optical flow (px/frame)
LEDOV val
2.378±1.059
−1.057±0.887
0.995±1.619
LEDOV test
2.170±0.997
−1.212±0.808
1.213±2.131
SalTempto val
3.106±1.059
−0.055±0.499
2.529±1.413
SalTempto test
2.979±0.804
−0.093±0.611
3.080±2.226
Appendix
Table 9: Dataset-level dynamism. KL is the pairwise symmetrised divergence (in bits) between per-frame gold-standard densities, averaged over within-video frame pairs (diagonal excluded) and across videos. LL is the cross-frame transfer log-likelihood: one frame’s gold-standard density evaluated on another frame’s fixations, in bits above a uniform spatial baseline, averaged over within-video frame pairs and across videos. RAFT optical-flow magnitudes are at matched evaluation resolution (long edge 512).
This paper presents an overview of the NTIRE 2026 Challenge on Video Saliency Prediction. The goal of the challenge participants was to develop automatic saliency map prediction methods for the provided video sequences. The novel dataset of 2,000 diverse videos with an open license was prepared for this challenge. The fixations and corresponding saliency maps were collected using crowdsourced mouse tracking and contain viewing data from over 5,000 assessors. Evaluation was performed on a subset of 800 test videos using generally accepted quality metrics. The challenge attracted over 20 teams making submissions, and 7 teams passed the final phase with code review. All data used in this challenge is made publicly available - https://github.com/msu-video-group/NTIRE26_Saliency_Prediction.
Andrey Moskalenko, Alexey Bryncev, Ivan Kosmynin +40
In this report, we present our champion solution for the NTIRE 2026 Challenge on Video Saliency Prediction held in conjunction with CVPR 2026. To exploit complementary inductive biases for video saliency, we propose Video Saliency with Adaptive Gated Experts (ViSAGE), a multi-expert ensemble framework. Each specialized decoder performs adaptive gating and modulation to refine spatio-temporal features. The complementary predictions from different experts are then fused at inference. ViSAGE thereby aggregates diverse inductive biases to capture complex spatio-temporal saliency cues in videos. On the Private Test set, ViSAGE ranked first on two out of four evaluation metrics, and outperformed most competing solutions on the other two metrics, demonstrating its effectiveness and generalization ability. Our code has been released at https://github.com/iLearn-Lab/CVPRW26-ViSAGE.
Kun Wang, Yupeng Hu, Zhiran Li +3
Shandong University · Harbin Institute of Technology (Shenzhen) · City University of Hong Kong +1
We present BIAS, a fast, biologically inspired model for dynamic visual saliency detection in continuous video streams. Building on the Itti--Koch framework, BIAS incorporates a retina-inspired motion detector to extract temporal features, enabling the generation of saliency maps that integrate both static and motion information. Foci of attention (FOAs) are identified using a greedy multi-Gaussian peak-fitting algorithm that balances winner-take-all competition with information maximization. BIAS detects salient regions with millisecond-scale latency and outperforms heuristic-based approaches and several deep-learning models on the DHF1K dataset, particularly in videos dominated by bottom-up attention. Applied to traffic accident analysis, BIAS demonstrates strong real-world utility, achieving state-of-the-art performance in cause-effect recognition and anticipating accidents up to 0.72 seconds before manual annotation with reliable accuracy. Overall, BIAS bridges biological plausibility and computational efficiency to achieve interpretable, high-speed dynamic saliency detection.
Zhao-ji Zhang, Ya-tang Li
Academy for Advanced Interdisciplinary Studies, Peking University, Beijing 100871, China · Chinese Institute for Brain Research, Beijing (CIBR), Beijing 102206, China