Neural video codecs quality assessment dataset and benchmark

Nikolay Safonov; Nikita Gornostaev; Alexandra Dubonos; Dmitriy Vatolin

cs.CVAug 29, 2026

Neural video codecs quality assessment dataset and benchmark

Authors: Nikolay Safonov, Nikita Gornostaev, Alexandra Dubonos, Dmitriy Vatolin

Organizations: AI Center, Lomonosov Moscow State University Moscow, Russia · MSU Institute for Artificial Intelligence Moscow, Russia · Lomonosov Moscow State University Moscow, Russia

Abstract

Video traffic constitutes a significant share of global web traffic. To reduce its volume, video codecs have been developed and continuously improved. While the industry has achieved substantial progress in traditional video coding, neural video codecs (NVCs) have recently emerged as a new approach that applies deep learning to video compression. This creates new challenges for compression quality assessment, which is essential for the further development and improvement of such codecs. In particular, it is important to evaluate the novel temporal compression paradigms introduced by NVCs. In this work, we present a large-scale subjective dataset of videos compressed with both neural and traditional video codecs. The subjective scores were collected through crowd-sourced pairwise comparisons. The proposed dataset provides a valuable resource for the development and benchmarking of video quality metrics tailored to neural video codecs. The dataset is available at the following link: https://videoprocessing.github.io/nvc-dataset-benchmark

Figures & tables

	Dataset	Source	Orig.	Dist.	Subjective Framework	Subj.	Ans.
General	MCL-JCV (2016) Wang et al.,2016	Raw	30	1,560	In-lab	150	78K
	VideoSet (2017) Wang et al.,2017	Raw	220	45,760	In-lab	800	-
	SJTU-4K (2017) Zhu et al.,2016	Raw	20	200	In-lab	30	6K
	GamingVSET (2018) Barman et al.,2018	Raw	24	576	In-lab	25	-
	NFLX (2016) Li et al.,2016	Raw	12	300	In-lab	54	9K
	KUGVD (2019) Barman et al.,2019	Raw	6	144	In-lab	17	-

Table 1. Summary of subjective compressed video quality datasets including neurally compressed and the proposed dataset.

Figure 1. Statistics (Range, Uniformity) of various complexity metrics for proposed and other datasets.

		NVC			Traditional			General
Type	Metric	PLCC $\uparrow$	SROCC $\uparrow$	KROCC $\uparrow$	PLCC $\uparrow$	SROCC $\uparrow$	KROCC $\uparrow$	PLCC $\uparrow$	SROCC $\uparrow$	KROCC $\uparrow$
NR	KONIQ++ Su et al.,2021	0.230	0.209	0.146	0.006	0.008	0.003	0.153	0.137	0.096
	EONSS Wang et al.,2019	0.248	0.240	0.177	0.124	0.107	0.075	0.201	0.189	0.138
	RankIQA Liu et al.,2017	0.368	0.347	0.260	0.575	0.619	0.467	0.472	0.483	0.364
	DOVER Wu et al.,2023	0.371	0.349	0.245	0.898	0.874	0.724	0.642	0.616	0.498
	LINEARITY Li et al.,2020a	0.395	0.310	0.225	0.780	0.790	0.636	0.577	0.566	0.428

Table 2. Performance of IQA/VQA metrics on the NVC, Traditional, and the whole (General) sets. Best results are bold , second-best are underlined . NR stands for the no-reference quality metrics and FR for the full-reference ones.

Figure 2. Distribution of metric scores. Each metric appears on a separate axis

Figure 3. (a) Pairwise Spearman correlation between objective quality metrics computed per video. (b)-(d) Pearson, Spearman and Kendall correlation respectively of objective quality metrics with subjective scores on the proposed dataset NVC and traditional codecs parts. The colormap indicates whether a metric performs better (green area) or worse (red area) on NVC compared to traditional approaches. Most metrics appear above the main diagonal, meaning that they generally perform better for traditional codecs.

Video Quality Assessment

Video quality assessment measures perceived visual and temporal fidelity in captured, compressed, streamed, and generated videos.

See all 52 papers

FDIM: A Feature-distance-based Generic Video Quality Metric for Versatile Codecs
Apr 27, 2026Jiayi Wang, Lichun Zhang, Xiaoqi Zhuang +3Video Quality Assessment Video Compression
College of Information Science and Electronic Engineering, Zhejiang University, and also with the Zhejiang Key Laboratory of Multimodal Communication Networks and Intelligent Information Processing, Hangzhou 310058, China · Central Media Technology Institute, Huawei, Hangzhou 310058, China
CodecArena: Codec Quality Assessment via Visual Reinforcement Learning
Aug 10, 2026Jiaye Fu, Weiqi Li, Qiankun Gao +5Video Quality Assessment RL for VLMs
School of Electronic and Computer Engineering, Peking University · School of Computer Science, Peking University · Pengcheng Laboratory
GameScope: A Multi-Attribute, Multi-Codec Benchmark Dataset for Gaming Video Quality Assessment
May 2, 2026Rajesh Sureddi, Shreshth Saini, Avinab Saha +1Video Quality Assessment
The University of Texas at Austin∗ · The University of Texas at Austin∗§ · Google Research +1

Apr 27, 2026J/K move · Enter open · S save

FDIM: A Feature-distance-based Generic Video Quality Metric for Versatile Codecs

Jiayi Wang, Lichun Zhang, Xiaoqi Zhuang, Jiaqi Zhang+2

College of Information Science and Electronic Engineering, Zhejiang University, and also with the Zhejiang Key Laboratory of Multimodal Communication Networks and Intelligent Information Processing, Hangzhou 310058, China · Central Media Technology Institute, Huawei, Hangzhou 310058, China

Video Quality Assessment Video Compression

Open paper AskPDF arXiv

Video technology is advancing toward Ultra High Definition (UHD) and High Dynamic Range (HDR), which intensifies the need for higher compression efficiency for these high-specification videos. Beyond advances in traditional codecs, neural video codecs (NVCs) have attracted significant research attention and have evolved rapidly over the past few years. The coding artifacts of NVCs often exhibit content-varying and generative characteristics, which differ from those of conventional codecs and are challenging for traditional video quality assessment (VQA) methods to capture. Therefore, VQA metrics are required to generalize across different codecs, content types, and dynamic ranges to better support video codec research and evaluation. In this paper, we propose FDIM, a feature-distance-based generic video quality metric for both traditional and neural video codecs across SDR and HDR formats. FDIM employs a hybrid architecture that integrates deep and hand-crafted features. The deep feature component learns multi-scale representations to capture distortions ranging from structural and textural fidelity degradation to high-level semantic deviations, while the hand-crafted feature component provides stable complementary cues to improve overall generalization. We trained FDIM on a large-scale subjective quality assessment dataset (DCVQA) consisting of over 16k video sequences encoded by traditional block-based hybrid video codecs and end-to-end perceptually optimized neural video codecs. Extensive experiments on ten SDR/HDR VQA datasets containing diverse, previously unseen codecs demonstrate that FDIM achieves strong generalization and high correlation with subjective assessment. The source code for FDIM and the DCVQA validation set will be released at https://github.com/MCL-ZJU/FDIM.

Abstract

Video Quality Assessment

Explore similar work

FDIM: A Feature-distance-based Generic Video Quality Metric for Versatile Codecs

CodecArena: Codec Quality Assessment via Visual Reinforcement Learning

GameScope: A Multi-Attribute, Multi-Codec Benchmark Dataset for Gaming Video Quality Assessment