Moving Forward with Video Saliency: A New Dataset and Benchmark where Motion Matters

Susmit Agrawal; Rebecca Wanner; Juliane Verwiebe; Matthias Tangemann; Matthias Bethge; Matthias Kümmerer

cs.CVOct 2, 2026

Moving Forward with Video Saliency: A New Dataset and Benchmark where Motion Matters

Authors: Susmit Agrawal, Rebecca Wanner, Juliane Verwiebe, Matthias Tangemann, Matthias Bethge, Matthias Kümmerer

Organizations: Tübingen AI Center, University of Tübingen · IMPRS-IS · Optocycle · University of Toronto ·

Abstract

Video saliency prediction is inherently harder to model than static image saliency due to the additional temporal dimension. Video saliency benchmarks rest on the premise that predicting gaze on video requires utilizing temporal activity distributed across frames. Prior work has challenged this, showing that static baselines recover a significant fraction of the explainable gaze information on LEDOV, a popular video saliency dataset, and that video saliency models fail in the same places as this static baseline. We verify that this diagnosis still stands: under a more capable gold standard than the original analysis, and an updated panel of recent architectures, the strongest temporal architecture in the panel still does not substantially improve over a fine-tuned static baseline. However, it remains unclear whether the marginal gain reflects limitations of current temporal architectures or a lack of temporal patterns in the benchmark itself. We introduce SalTempto, a video saliency benchmark with greater dynamism: 224 clips of highly dynamic content, sourced from the HACS-Segments dataset so that each clip contains an event together with its lead-up and aftermath, with gaze recordings from up to 16 subjects and a training split for adapting pretrained models. On SalTempto, the static baseline recovers only about 13% of the headroom above the centerbias, against more than half on LEDOV. A fine-tuned temporal architecture shows a substantial gain in performance over the static baseline, indicating that it does capture meaningfully more temporal information, which LEDOV fails to measure. Yet, even this SoTA model still leaves nearly half of SalTempto's headroom unexplained, indicating room for improvement in video saliency modelling. Examination of SalTempto also lets us describe human tendencies that models miss. SalTempto link: https://huggingface.co/datasets/bethgelab/video_saliency.

Figures & tables

Figure 1: Static-vs-temporal performance and remaining headroom on LEDOV (left) and SalTempto (right). Each panel shows the dataset’s IG over centerbias LL on the

y-

axis and the gold-standard IG ceiling (dashed) at the top. Each model’s bar is split into a closed segment (solid; percent of headroom recovered) and a remaining segment (faded; percent unexplained). Both models are scored on the same videos, so error bars on the model markers are 95 % intervals for the paired comparison between them, obtained by removing each video’s mean over the two models ( Cousineau, 2005 ; Morey, 2008 ) ; intervals for absolute IG and for the gold ceiling are in Table 2 . The static-vs-temporal climb (faded line connecting DG-MR to SalFoM) is steeper on SalTempto than on LEDOV, indicating that SalFoM extracts a lot more information than the static baseline on SalTempto than on LEDOV.

Panel of Figure 2 — Figure 2: (a) Per-frame information-gain residual $\mathrm{LL}(gold)-\mathrm{LL}(SalFoM\text{-}Tuned)$ on LEDOV (green) vs SalTempto (orange). SalTempto is broader and less skewed, with a greater mean performance gap and a heavier tail. (b) Cross-dataset IG (bits) on 10 models evaluated in zero-shot on both LEDOV (x-axis) and SalTempto (y-axis); models trained or fine-tuned on either dataset are excluded to compute zero-shot performance rankings. Error bars are 95 % intervals for paired comparisons between models ( Cousineau, 2005 ; Morey, 2008 ) . Spearman $\rho=0.83{}$ , implying model rankings are tightly preserved across datasets; the correlation including the LEDOV-pretrained models (zero-shot SalTempto) is $\rho=0.74{}$ (App. A ).

Figure 3: Object permanence despite occlusion. Workshop scene: A hot metal rod in a vice/stand (top-left, red border) gets transiently occluded by a gloved hand sweeping across (top-right, blue border). Gold-standard fixations remain locked at the rod’s position despite occlusion. SalFoM redistributes mass onto the visible motion (the glove); ViNet-E concentrates centrally on the hand region; DeepGaze MR is diffuse across multiple salient elements. Generally, models attend to currently-visible salient content; subjects maintain attention on the (now-occluded) target.

Figure 4: Anticipatory saccades. Badminton smash: shuttle moving right-to-left after the strike (blue arrows mark trajectory across frames, blue boxes locate shuttle). Subject fixations form a smooth trail spanning the shuttle’s trajectory, with mass both trailing the shuttle (pursuit lag) and leading it (anticipatory saccades). The models concentrate mass primarily at the players, missing the entity that viewers find most salient: the moving shuttle.

Figure 5: Scene inertia. The video cuts between different stances of two fencers. The first clip, pre-cut (left, red border), has both fencers at the right of the scene. Post-cut (right, blue border), the scene shows the fencers near the starting positions in a span of just a few frames. The human gaze readings (green dots) are highly consistent between the frames - the Gold Standard shows the highest densities at salient pre-cut regions that are non-salient post-cut. Deep Learning models quickly redistribute attention to the foreground figures based on current-frame salience. The effect is severe enough to perturb the gold standard itself due to its deep learning model component, so the residual-to-gold here underestimates the architectural shortfall by estimating high-variance low-peak densities.

Figure 6: Viewing-onset biases. Karate dojo clip from SalTempto at the onset of the viewing session. Top-left: gold-standard density concentrates tightly at the center, reflecting the centre-anchored gaze that follows clip onset. Top-right and bottom row: SalFoM, ViNet-E, and DeepGaze MR distribute predicted attention across multiple visually salient agents and fail to capture scene-onset behaviour.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Model	LEDOV		SalTempto
	val	test	val	test
DG-MR	55.6 %	56.4 %	15.8 %	10.2 %
SalFoM	75.4 %	75.2 %	55.6 %	61.4 %

Appendix

Table 1: IGE (Eq. 1 ) for the static baseline (DG-MR) and the strongest temporal model (SalFoM) on each dataset’s validation and test splits post fine-tuning. SalFoM is fine-tuned by us on each dataset’s training split. DG-MR is fine-tuned by us on SalTempto and uses the released LEDOV-trained checkpoint from Tangemann et al. (2020) on LEDOV.

Dataset	Model	IG (bits)		IGE (%)
		mean	95 % jackknife	mean	95 % jackknife
LEDOV	Gold standard	2.359	[2.210, 2.507]	100.0	—
	DG-MR (static)	1.321	[1.170, 1.472]	56.0	[51.6, 60.4]
	SalFoM (temporal)	1.775	[1.631, 1.918]	75.2	[72.6, 77.9]
	Gap (SalFoM $-$ DG-MR)	0.453	[0.371, 0.535]	19.2	[15.8, 22.7]
SalTempto	Gold standard	1.864	[1.613, 2.115]	100.0	—

Appendix

Table 2: Fine-tuned results with leave-one-video-out jackknife intervals. IG is the information gain over the centerbias in bits per fixation and IGE the fraction of the centerbias-to-gold headroom recovered, both pooled over the validation and test videos of each dataset (77 videos on LEDOV, 20 on SalTempto) with every video weighted equally. Intervals are 95 % jackknife intervals over videos. Gap rows give SalFoM minus DG-MR.

Method	FT	LL	IGE	KLD	CC	SIM	AUC	NSS
Best gold standard (adaptive 4-mix $+$ SalFoM)	—	3.1943	100.0 %	0.7102	0.8175	0.5591	0.9755	8.5661
SalFoM	✓	2.3583	55.6 %	1.5343	0.4816	0.3331	0.9147	3.0683
SalFoM		2.2784	51.4 %	1.5807	0.4702	0.3321	0.9140	2.9986
TMFI-Net		2.0917	41.5 %	1.8402	0.4363	0.3091	0.9045	2.7174
ViNet-E		1.9527	34.1 %	1.7652	0.4016	0.2738	0.8955	2.4230
ViNet-A		1.9407	33.4 %	1.7607	0.4011	0.2820	0.8955	2.4431

Appendix

Table 3: SalTempto validation results. FT

=

per-dataset fine-tuning on SalTempto train. KLD lower-is-better; LL, IGE, CC, SIM, AUC, NSS higher-is-better. Sorted by IGE descending.

Method	FT	LL	IGE	KLD	CC	SIM	AUC	NSS
Best gold standard (adaptive 4-mix $+$ SalFoM)	—	3.1579	100.0 %	0.7496	0.7954	0.5523	0.9697	8.1241
SalFoM	✓	2.4462	61.4 %	1.4629	0.4927	0.3500	0.9158	3.1212
SalFoM		2.3653	57.0 %	1.5091	0.4818	0.3433	0.9125	3.0137
TMFI-Net		2.1345	44.5 %	1.7996	0.4563	0.3228	0.8995	2.8311
TinyHD-single		1.9555	34.8 %	1.7897	0.4082	0.2665	0.8915	2.4952
UniformerSal-Spatial		1.9025	31.9 %	1.5533	0.4657	0.3633	0.9002	2.4965

Appendix

Table 4: SalTempto test results, sorted by IGE descending.

Method	FT	LL	IGE	KLD	CC	SIM	AUC	NSS
Best gold standard (adaptive 4-mix $+$ SalFoM)	—	3.6948	100.0 %	0.4009	0.8861	0.7117	0.9808	9.6380
SalFoM	✓	3.0937	75.4 %	0.9234	0.7289	0.5401	0.9368	4.5045
UniformerSal-SpatioTemporal	✓ †	2.8710	66.2 %	1.0990	0.6630	0.4900	0.9300	4.2670
SalFoM		2.8472	65.3 %	1.0556	0.6919	0.5031	0.9254	4.2242
TMFI-Net		2.7959	63.1 %	1.0804	0.6855	0.4732	0.9244	4.0483
UniformerSal-Spatial	✓ †	2.6471	57.0 %	1.0030	0.6778	0.5353	0.9203	3.9882

Appendix

Table 5: LEDOV validation results. IGE is the fraction of the centerbias-to-gold-standard headroom recovered. FT

=

per-dataset fine-tuning on LEDOV train;

\dagger

released checkpoint already trained on LEDOV by the original authors (no further fine-tuning in this work). KLD lower-is-better; LL, IGE, CC, SIM, AUC, NSS higher-is-better. Sorted by IGE descending. Gold-standard centerbias and gold LL: 1.2560 / 3.6948.

Method	FT	LL	IGE	KLD	CC	SIM	AUC	NSS
Best gold standard (adaptive 4-mix $+$ SalFoM)	—	3.6209	100.0 %	0.3955	0.8825	0.7122	0.9803	9.5950
SalFoM	✓	3.0524	75.2 %	0.8969	0.7341	0.5372	0.9378	4.3478
UniformerSal-SpatioTemporal	✓ †	2.9056	68.7 %	0.8218	0.7361	0.5766	0.9329	4.2549
SalFoM		2.8309	65.5 %	1.0163	0.6920	0.5139	0.9288	4.0333
TMFI-Net		2.7860	63.5 %	1.0408	0.6864	0.4780	0.9280	3.8610
UniformerSal-Spatial	✓ †	2.7021	59.8 %	0.9206	0.6886	0.5443	0.9258	3.8798

Appendix

Table 6: LEDOV test results. IGE is the fraction of the centerbias-to-gold-standard headroom recovered. FT

=

per-dataset fine-tuning on LEDOV train;

\dagger

	LEDOV				SalTempto
Deep component	ceiling	SalFoM	DG-MR	gap	ceiling	SalFoM	DG-MR	gap
SalFoM (reported)	2.359	75.2 %	56.0 %	19.2	1.864	58.5 %	13.0 %	45.5
UniformerSal	2.328	76.2 %	56.7 %	19.5	1.831	59.5 %	13.3 %	46.3
DeepGaze MR	2.283	77.7 %	57.9 %	19.9	1.808	60.3 %	13.4 %	46.8
TMFI-Net	2.252	78.8 %	58.7 %	20.2	1.848	59.0 %	13.1 %	45.8

Appendix

Table 7: Gold-standard sensitivity to the deep mixture component. Ceiling is the gold-standard IG over the centerbias in bits, pooled over validation and test videos; SalFoM and DG-MR columns give the IGE of the fine-tuned models under that ceiling, and gap is their difference in percentage points.

Dataset	Row	IG (bits)	95 % interval	IGE (%)
LEDOV	DG-MR (fine-tuned)	1.321	[1.254, 1.389]	56.0
	SalFoM, frame repeated	1.515	[1.452, 1.579]	64.3
	SalFoM (fine-tuned)	1.775	[1.725, 1.824]	75.2
	Temporal context (SalFoM $-$ frame repeated)	0.259	[0.185, 0.333]	11.0
	Static features (frame repeated $-$ DG-MR)	0.194	[0.095, 0.293]	8.2
SalTempto	DG-MR (fine-tuned)	0.243	[0.072, 0.414]	13.0

Appendix

Table 8: SalFoM with and without temporal context. The frame-repeated row feeds the fine-tuned SalFoM the target frame at every position of its context window. IG is over the centerbias in bits per fixation, pooled over the released evaluation videos of each dataset (77 on LEDOV, 20 on SalTempto); the IGE column gives the share of the centerbias-to-gold headroom (Eq. 1 ), or IGE points for the two difference rows. The three model rows within a dataset are scored on the same videos and carry 95 % intervals for paired comparisons among them, obtained by removing each video’s mean over the three rows ( Cousineau, 2005 ; Morey, 2008 ) ; the difference rows are paired over the same videos with 95 %

t

-intervals. Intervals for absolute IG are in Table 2 .

Split	KL (bits)	LL (bits)	Optical flow (px/frame)
LEDOV val	$2.378\pm 1.059$	$\phantom{-}1.057\pm 0.887$	$0.995\pm 1.619$
LEDOV test	$2.170\pm 0.997$	$\phantom{-}1.212\pm 0.808$	$1.213\pm 2.131$
SalTempto val	$3.106\pm 1.059$	$-0.055\pm 0.499$	$2.529\pm 1.413$
SalTempto test	$2.979\pm 0.804$	$\phantom{-}0.093\pm 0.611$	$3.080\pm 2.226$

Appendix

Table 9: Dataset-level dynamism. KL is the pairwise symmetrised divergence (in bits) between per-frame gold-standard densities, averaged over within-video frame pairs (diagonal excluded) and across videos. LL is the cross-frame transfer log-likelihood: one frame’s gold-standard density evaluated on another frame’s fixations, in bits above a uniform spatial baseline, averaged over within-video frame pairs and across videos. RAFT optical-flow magnitudes are at matched evaluation resolution (long edge 512).

Video Saliency Prediction

See all 12 papers

NTIRE 2026 Challenge on Video Saliency Prediction: Methods and Results
Apr 16, 2026Andrey Moskalenko, Alexey Bryncev, Ivan Kosmynin +40Video Saliency Prediction Visual Saliency
ViSAGE @ NTIRE 2026 Challenge on Video Saliency Prediction
Apr 9, 2026Kun Wang, Yupeng Hu, Zhiran Li +3Video Saliency Prediction Visual Saliency
Shandong University · Harbin Institute of Technology (Shenzhen) · City University of Hong Kong +1
BIAS: A Biologically Inspired Algorithm for Video Saliency Detection
Apr 10, 2026Zhao-ji Zhang, Ya-tang LiVideo Saliency Prediction Visual Saliency
Academy for Advanced Interdisciplinary Studies, Peking University, Beijing 100871, China · Chinese Institute for Brain Research, Beijing (CIBR), Beijing 102206, China

Apr 16, 2026J/K move · Enter open · S save

NTIRE 2026 Challenge on Video Saliency Prediction: Methods and Results

Andrey Moskalenko, Alexey Bryncev, Ivan Kosmynin, Kira Shilovskaya+39

Video Saliency Prediction Visual Saliency

Open paper AskPDF arXiv

This paper presents an overview of the NTIRE 2026 Challenge on Video Saliency Prediction. The goal of the challenge participants was to develop automatic saliency map prediction methods for the provided video sequences. The novel dataset of 2,000 diverse videos with an open license was prepared for this challenge. The fixations and corresponding saliency maps were collected using crowdsourced mouse tracking and contain viewing data from over 5,000 assessors. Evaluation was performed on a subset of 800 test videos using generally accepted quality metrics. The challenge attracted over 20 teams making submissions, and 7 teams passed the final phase with code review. All data used in this challenge is made publicly available - https://github.com/msu-video-group/NTIRE26_Saliency_Prediction.

Abstract

Video Saliency Prediction

Explore similar work

NTIRE 2026 Challenge on Video Saliency Prediction: Methods and Results

ViSAGE @ NTIRE 2026 Challenge on Video Saliency Prediction

BIAS: A Biologically Inspired Algorithm for Video Saliency Detection