Explicit Regional Representations For Video Quality Assessment
Authors: Fuyi Yang, Lei Yang, Andrew Liu, Chenyu Wei, Xiaoyi Guo, Ming Kai Hsu
Organizations: University of California, Los Angeles · Transtreams Technologies, Inc. · University of Southern California · University of Wisconsin–Madison · BASIS Independent Fremont
End-to-end video quality assessment (VQA) methods leave the regional evidence underlying their predictions largely implicit. We introduce an explicit pixel-level regional representation for VQA, instantiated as a Quality-Relevance Heatmap (QRH) that combines localized degradation evidence with perceptual priors under vision-language model guidance, without requiring pixel-level annotations. We apply QRH to existing VQA models through two deterministic fragment-sampling strategies: distribution-preserving sampling approximates the original sampler's QRH-score distribution without retraining, while quality-aware sampling prioritizes regions with high QRH scores and is paired with fine-tuning. Experiments with DOVER on multiple benchmarks show that the former closely approximates predictions averaged over random samples, while the latter improves most evaluated metrics across popular VQA benchmarks. These results support explicit regional representations as a useful interface for downstream VQA.
Figures & tables
Figure 1: Quality-Relevance Heatmap (QRH) construction. Quality evidence maps (appearance degradation, temporal inconsistency, and compression artifacts) are adaptively fused under VLM guidance, while motion vulnerability and saliency are combined into a perceptual-prior representation. Their element-wise product yields QRH, an explicit pixel-level regional representation that combines localized quality evidence with perceptual importance.
Method
LSVQ-test
LSVQ-1080p
KoNViD-1k
LIVE-VQC
SROCC ↑
PLCC ↑
SROCC ↑
PLCC ↑
SROCC ↑
PLCC ↑
SROCC ↑
PLCC ↑
BRISQUE [ 6 ]
.569
.576
.497
.531
.646
.647
.524
.536
TLVQM [ 9 ]
.772
.774
.589
.616
.732
.724
.670
.691
VIDEVAL [ 10 ]
.794
.783
.545
.554
.751
.741
.630
.640
VSFA [ 11 ]
.801
.796
.675
.704
.784
.794
.734
.772
Patch-VQ w/o patches [ 18 ]
.814
.816
.686
.708
.781
.781
.747
.776
Table 1: Performance of LSVQ-trained VQA methods. QRH+DOVER uses QAS adaptation; DOVER is re-evaluated under our pipeline. The best result in each column is shown in bold.
Figure 2: Illustration and evaluation of QRH-guided sampling. (a) Representative frame. (b) Normalized QRH-score distribution and DPS targets, with ten highlighted. (c) Regions matched to the highlighted DPS targets on the normalized QRH. (d) Per-video residuals relative to the ten-run DOVER mean on LSVQ-test. (e) QAS region selection on the same QRH.
Target dataset
LIVE-VQC
KoNViD-1k
YouTube-UGC
Method
Pre-training dataset
SROCC ↑
PLCC ↑
SROCC ↑
PLCC ↑
SROCC ↑
PLCC ↑
TLVQM [ 9 ]
N/A (hand-crafted)
.799
.803
.773
.768
.669
.659
VIDEVAL [ 10 ]
N/A (hand-crafted)
.752
.751
.783
.780
.779
.773
VSFA [ 11 ]
None
.773
.795
.773
.775
.724
.743
CoINVQ [ 13 ]
Self-collected
.727
.795
.767
.764
.816
.802
Patch-VQ [ 18 ]
PaQ-2-PiQ
.827
.837
.791
.786
–
–
Table 2: Transfer-learning performance on smaller VQA benchmarks. Our results are averaged over ten random 80%/20% train/test splits; baseline values are taken from the literature. Since the fully labeled YouTube-UGC dataset is unavailable, our results use a matched 992/1380 video subset, reported for reference only.
Large Multimodal Models (LMMs) have shown promise for video quality assessment, but most methods still predict an absolute score for each video. Such pointwise supervision often mixes perceptual quality with dataset-specific calibration, including annotation protocols, rating habits, and score distributions. As a result, the learned scoring rule may work well within a benchmark but transfer poorly across unseen domains. We argue that relative comparisons alleviate the absolute-scale calibration bias by focusing purely on perceptual differences rather than dataset-specific rating habits. Consequently, we propose VersusQ, a pairwise margin reasoning framework driven entirely by direct comparisons. Specifically, VersusQ performs LMM-based comparison between two videos, reasons about their visual and temporal quality differences, and predicts a signed continuous margin that captures both the preferred choice and the degree of difference. Furthermore, to align interpretable comparison rationales with fine-grained numerical differences, we introduce Margin-Coupled GRPO, which jointly optimizes rollout-based relational reasoning and continuous margin regression. Extensive experiments on multiple public VQA benchmarks demonstrate that VersusQ achieves state-of-the-art performance, strong cross-domain generalization, and reliable fine-grained ranking under heterogeneous evaluation scenarios.
Shibei Meng, Binxin Yang, Yuan Liu +4
The Chinese University of Hong Kong · Beijing Normal University · WeChat Vision, Tencent Inc. +2
No-reference video quality assessment (NR VQA) has recently seen promising progress with deep learning. However, video data is inherently large, and processing them with deep models incurs high computational cost. This challenge is particularly acute in VQA, where preserving original-resolution cues and dense temporal information is critical for accuracy. Existing efficiency-driven preprocessing strategies, such as fragmenting, reduce computation but alter the input data distribution, limiting effective reuse of pretrained video foundation models (ViFMs). To address these challenges, we propose High-Fidelity Video Quality Assessment (HFVQA), a framework built on fixed-size spatio-temporal (ST) patches that is fully compatible with pretrained ViFMs. HFVQA samples ST patches across multiple scales, including the original resolution, with minimal temporal subsampling to preserve low-level quality cues and semantic context. To limit computation, HFVQA introduces a lightweight auxiliary network trained end-to-end with the ViFM encoder to learn VQA-specific saliency. Distilled directly from quality supervision, this saliency captures task-specific importance patterns, reflecting that video quality perception is dominated by a small subset of spatio-temporal regions. By combining high-fidelity spatio-temporal cues with learned, task-specific saliency, HFVQA achieves SOTA performance on standard NR VQA benchmarks while processing as little as 12% of candidate ST patches, making high-fidelity ViFM-based VQA computationally tractable.
Hakan Emre Gedik, Shashank Gupta, Alan Bovik
The University of Texas at Austin · University of Colorado Boulder
Recent multimodal large language models (MLLMs) have shown promising performance on video quality assessment (VQA) tasks. However, adapting them to new scenarios remains expensive due to large-scale retraining and costly mean opinion score (MOS) annotations. In this paper, we argue that a pretrained MLLM already provides a useful perceptual prior for VQA, and that the main challenge is to efficiently calibrate this prior to the target MOS space. Based on this insight, we propose DPC-VQA, a decoupling perception and calibration framework for video quality assessment. Specifically, DPC-VQA uses a frozen MLLM to provide a base quality estimate and perceptual prior, and employs a lightweight calibration branch to predict a residual correction for target-scenario adaptation. This design avoids costly end-to-end retraining while maintaining reliable performance with lower training and data costs. Extensive experiments on both user-generated content (UGC) and AI-generated content (AIGC) benchmarks show that DPC-VQA achieves competitive performance against representative baselines, while using less than 2% of the trainable parameters of conventional MLLM-based VQA methods and remaining effective with only 20% of MOS labels. The code will be released upon publication.
Xinyue Li, Shubo Xu, Zhichao Zhang +3
Shanghai Jiao Tong University, Shanghai, China · Baidu Inc., Beijing, China · Xinjiang University, Urumqi, China