HiVE-VQA: Hierarchical Interaction via Volumetric Embedding for Video Quality Assessment
Organizations: Visual Information Laboratory, University of Bristol, Bristol, UK
Abstract
The proliferation of user-generated video content has made accurate and robust no-reference video quality assessment (NR-VQA) a highly important challenge. Existing research increasingly relies on architecturally complex multi-branch pipelines or computationally intensive fine-tuning of single-branch backbones. In this paper, we investigate whether a single frozen video foundation model can serve as an effective NR-VQA backbone, and propose HiVE-VQA, the first framework (to our knowledge) to apply a JEPA-based video encoder to this task. HiVE-VQA extracts token grids from multiple depths of a pre-trained V-JEPA 2.1 encoder and stacks them into a hierarchical spatio-temporal volume, which a compact 3D convolutional head fuses across its layer, spatial, and temporal axes, combining low-level distortion cues from shallow layers with the semantic and temporal context captured at greater depths. The encoder remains frozen throughout, so only the head is trained: 5.2M parameters (5.7% of the total), with no auxiliary multi-modal models and an inference cost that is invariant to video resolution. Trained solely on LSVQ and evaluated without any fine-tuning, HiVE-VQA outperforms 17 state-of-the-art (SOTA) methods in SROCC and PLCC on both intra-dataset (LSVQ-test, LSVQ-1080p) and cross-dataset (KoNViD-1k, LIVE-VQC) benchmarks. These results indicate that hierarchical, depth-wise fusion of features from a single frozen video foundation model provides a more effective and efficient basis for NR-VQA than task-specific multi-network fusion. Code and model checkpoints are available at https://github.com/RobHam99/HiVE-VQA.
Figures & tables
| Testing Type | Intra-Dataset | Cross-Dataset | Average | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Testing Datasets | LSVQ test | LSVQ 1080p | KoNViD-1k | LIVE-VQC | Overall | |||||||
| Type | Methods | Source | SROCC | PLCC | SROCC | PLCC | SROCC | PLCC | SROCC | PLCC | SROCC | PLCC |
| Classical | BRISQUE [ 27 ] | TIP, 2012 | 0.569 | 0.576 | 0.497 | 0.531 | 0.646 | 0.647 | 0.524 | 0.536 | 0.559 | 0.573 |
| TLVQM [ 13 ] | TIP, 2019 | 0.772 | 0.774 | 0.589 | 0.616 | 0.732 | 0.724 | 0.670 | 0.691 | 0.691 | 0.701 | |
| VIDEVAL [ 37 ] | TIP, 2021 | 0.794 | 0.783 | 0.545 | 0.554 | 0.751 | 0.741 | 0.630 | 0.640 | 0.680 | 0.680 | |
| Deep (Multi-branch) | PVQ w/o patch [ 50 ] | CVPR, 2021 | 0.814 | 0.816 | 0.686 | 0.708 | 0.782 | 0.781 | 0.747 | 0.776 | 0.757 | 0.770 |
| Type | Methods | Source | LIVE-VQC | KoNViD-1k | YouTube-UGC | Average | ||||
|---|---|---|---|---|---|---|---|---|---|---|
| SROCC | PLCC | SROCC | PLCC | SROCC | PLCC | SROCC | PLCC | |||
| Classical | TLVQM [ 13 ] | TIP, 2019 | 0.799 | 0.803 | 0.773 | 0.768 | 0.669 | 0.659 | 0.732 | 0.726 |
| VIDEVAL [ 37 ] | TIP, 2021 | 0.752 | 0.751 | 0.783 | 0.780 | 0.779 | 0.773 | 0.772 | 0.772 | |
| RAPIQUE [ 38 ] | OJSP, 2021 | 0.755 | 0.786 | 0.803 | 0.817 | 0.759 | 0.768 | 0.774 | 0.790 | |
| Classical + Deep | CNN+TLVQM [ 12 ] | ACMMM, 2020 | 0.825 | 0.834 | 0.816 | 0.818 | N/A | N/A | 0.821 | 0.826 |
| CNN+VIDEVAL [ 37 ] | TIP, 2021 | 0.785 | 0.810 | 0.815 | 0.817 | 0.808 | 0.803 | 0.806 | 0.810 | |
| Models | Input Size | Depth | Dim | Param | FLOPs |
|---|---|---|---|---|---|
| HiVE-VQA-M | 12 | 768 | 91M | 188G | |
| HiVE-VQA | 12 | 768 | 91M | 2406G |
| Methods | Params (M) | 540p | 720p | 1080p | ||||
| Trainable | Total | GFLOPs | Time | GFLOPs | Time | GFLOPs | Time | |
| VSFA [ 16 ] | 0.5 | 24.1 | 6440 | 1.506 | 11426 | 2.556 | 25712 | 5.291 |
| PVQ [ 50 ] | 54.2 | 54.2 | 9203 | 1.792 | 13842 | 2.968 | 36760 | 6.556 |
| BVQA [ 15 ] | 57.8 | 57.8 | 17705 | 3.145 | 31533 | 7.813 | 70714 | 14.34 |
| FAST-VQA [ 45 ] | 27.7 | 27.7 | 284 | 0.056 | 284 | 0.056 | 284 | 0.056 |
| FasterVQA [ 46 ] | 27.7 | 27.7 | 70 | 0.032 | 70 | 0.032 | 70 | 0.032 |
| Encoder Layers | Efficiency | LSVQ | LSVQ | KoNViD-1k | LIVE-VQC | Average | ||||||||
| L3 | L6 | L9 | L12 | GFLOPs | SROCC | PLCC | SROCC | PLCC | SROCC | PLCC | SROCC | PLCC | SROCC | PLCC |
| ✓ | ✓ | ✓ | 2403 | 0.905 | 0.905 | 0.830 | 0.861 | 0.897 | 0.900 | 0.856 | 0.878 | 0.872 | 0.886 | |
| ✓ | ✓ | ✓ | 2403 | 0.903 | 0.903 | 0.833 | 0.862 | 0.895 | 0.897 | 0.853 | 0.873 | 0.871 | 0.884 | |
| ✓ | ✓ | ✓ | 2403 | 0.903 | 0.903 | 0.830 | 0.860 | 0.894 | 0.896 | 0.860 | 0.879 | 0.872 | 0.885 | |
| ✓ | ✓ | ✓ | 2403 | 0.901 | 0.901 | 0.827 | 0.858 | 0.891 | 0.891 | 0.860 | 0.876 | 0.870 | 0.882 | |
| ✓ | 2395 | 0.853 | 0.852 | 0.738 | 0.770 | 0.837 | 0.834 | 0.780 | 0.796 | 0.802 | 0.813 | |||
| Quality Network (Backbone) | LSVQ | LSVQ | KoNViD-1k | LIVE-VQC | Average | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| SROCC | PLCC | SROCC | PLCC | SROCC | PLCC | SROCC | PLCC | SROCC | PLCC | |
| Ridge Regressor (V-JEPA 2.1 ViT-B) | 0.878 | 0.875 | 0.779 | 0.821 | 0.871 | 0.873 | 0.818 | 0.847 | 0.837 | 0.854 |
| HiVE-VQA (ViT-B) | 0.850 | 0.851 | 0.741 | 0.772 | 0.835 | 0.839 | 0.763 | 0.810 | 0.797 | 0.818 |
| HiVE-VQA (VideoMAE ViT-B) | 0.815 | 0.813 | 0.699 | 0.727 | 0.793 | 0.794 | 0.782 | 0.806 | 0.772 | 0.785 |
| HiVE-VQA (V-JEPA 2.1 ViT-B) | 0.906 | 0.906 | 0.835 | 0.867 | 0.902 | 0.903 | 0.864 | 0.882 | 0.877 | 0.890 |