Contrastive learning-based video quality assessment-jointed video vision transformer for video recognition

Jian Sun; Mohammad H. Mahoor

cs.CVMar 11, 2026

Contrastive learning-based video quality assessment-jointed video vision transformer for video recognition

Authors: Jian Sun, Mohammad H. Mahoor

Organizations: Department Of Computer Science, University of Denver, 2155 E Wesley Ave, Denver, 80210, CO, United States of America. · Department Of Electrical and Computer Engineering, University of Denver, 2155 E Wesley Ave, Denver, 80210, CO, United States of America.

Abstract

Video quality significantly affects video classification. We found this problem when we classified Mild Cognitive Impairment well from clear videos, but worse from blurred ones. From then, we realized that referring to Video Quality Assessment (VQA) may improve video classification. This paper proposed Self-Supervised Learning-based Video Vision Transformer combined with No-reference VQA for video classification (SSL-V3) to fulfill the goal. SSL-V3 leverages Combined-SSL mechanism to join VQA into video classification and address the label shortage of VQA, which commonly occurs in video datasets, making it impossible to provide an accurate Video Quality Score. In brief, Combined-SSL takes video quality score as a factor to directly tune the feature map of the video classification. Then, the score, as an intersected point, links VQA and classification, using the supervised classification task to tune the parameters of VQA. SSL-V3 achieved robust experimental results on two datasets. For example, it reached an accuracy of 94.87% on some interview videos in the I-CONECT (a facial video-involved healthcare dataset), verifying SSL-V3's effectiveness.

Figures & tables

Figure 1: The structure of the new proposed SSL-V3. In the upper branch, ViViT(FE) loads $\textbf{X}_{1}$ and extracts Spatio-Temporal features $f_{\text{S}_{1}}$ . The VQA head and classification (CLS) head then process $f_{\text{S}_{1}}$ to compute $\text{VQS}_{1}$ and $\text{CLS}_{1}$ , respectively - output features from each head. VQA head contains Sequence Score Regressor and Video Score Regressor, while CLS head is Multi-branch Classifier. The Tune-CLS module updates $\text{CLS}_{1}$ using $\text{VQS}_{1}$ to produce the prediction score $\hat{\text{Y}}_{1}$ . Simultaneously, the lower branch follows the same procedure to generate $f_{\text{S}_{2}}$ , $\text{VQS}_{2}$ , $\text{CLS}_{2}$ , and $\hat{\text{Y}}_{2}$ .

Figure 2: The structure of Sequence Score Regressor. It has two channels. The first one, Sequence Feature Processor, is to drop the $z_{cls1}$ and reshape $f_{S_{1}}$ from [bs, $n_{t}$ , d] to [bs, $n_{t}$ , d, 1]. The other one, Sequence Weight Estimate Network, is to generate weights, $\tilde{w}_{f}$ for corresponding $f_{S_{1}}$ . Finally, it conducts weighted sum operation to get $SQS_{1}$ . $C_{i,j}$ is the $j^{th}$ cube from the $i^{th}$ sequence in a certain clip.

Figure 3: The structure of VSR. It has three channels. The central one is the $SQS_{1}$ , denoted by $s_{2}$ . The upper one does 1-D convolutional operation on $SQS_{1}$ and captures the temporal motion effect, $m_{j}$ , of the $SQS_{1,j}$ . Then, we deploy the Softmax operation to normalize the weight vector $m$ and output $\hat{m}$ . Multiplying $SQS_{1}$ and $\hat{m}$ returns the temporal motion effect score $s_{1}$ . Following the same pattern, we get the normalized temporal hysteresis effect $\hat{h}$ and the temporal hysteresis effect score $s_{3}$ . Finally, the Temporal Memory-Based Fusion Module generates $VQS_{1}$ by concatenating $s_{1}$ , $s_{2}$ , and $s_{3}$ and loading them into a two-layers FC module.

	Crafts Hobbies	Day Time TV Shows	Movie Genres	School Subjects
Subject Number	32	41	35	39
Male/Female	11/21	11/30	10/25	11/28
MCI/NC	20/12	20/21	21/14	22/17
Frame Number	691872	859872	770656	797968

Table 1: The details of researched themes. Subject Number is the total number of videos in the corresponding theme. Male/Female and MCI/NC show the gender distribution and category distribution. Frame Number is the total image number of each theme after converting videos into frames.

Panel of Figure 4 — Figure 4: Two sample frames from the I-CONECT dataset. In (a), the window of the interviewee is bigger than that of the interviewer because the interviewer was speaking. Conversely, in (b), the interviewer was talking so that her window was bigger than the interviewee’s.

Figure 5: More subjects from Hockey Fight Detection dataset. The upper line represents high-quality frames. The bottom line stands for poor-quality frames.

Panel of Figure 6 — Figure 6: Two samples from HF dataset. (a) is the scenario of fighting action. (b) presents the normal non-fight situation.

Test Theme	Crafts Hobbies	Day Time TV Shows	Movie Genres	School Subjects
Fold Num	11	14	11	13
Accuracy	30/32	37/41	31/35	37/39
Accuracy	93.75%	90.24%	88.57%	94.87%
F1	95.00%	90.00%	91.30%	95.65%
AUC	93.33%	90.24%	88.90%	94.12%
MCI/NC	20/12	20/21	21/14	22/17

Table 2: The prediction accuracy of detecting MCI on 4 themes using K-fold evaluation.

	Data Modality	Accuracy	F1
Chen et al. Chen et al. (2020a)	Text	79.15%	-
Liu et al. Liu et al. (2022)	Text, MRI	87.00%	89.00%
Alsuhaibani et al. Alsuhaibani et al. (2024)	Video	87.50%	89.00%
MC-ViViT ( Sun et al., 2023a )	Video	90.63%	93.03%
SSL-V3 (No VQA)	Video	87.80%	86.49%
SSL-V3 (VQA)	Video	94.87%	95.65%

Table 3: Compare the performance of SSL-V3 with other models on the I-CONECT dataset. No VQA means to classify the videos without referring to the video quality, while VQA represents using VQA to classify videos.

Accuracy	F1	AUC	Sensitivity	Specificity
98.6 $\pm$ 0.8%	98.59 $\pm$ 0.81%	98.6 $\pm$ 0.8%	98.0 $\pm$ 1.26%	99.2 $\pm$ 0.99%

Table 4: The prediction accuracy of detecting violence on HF dataset. The report format is mean

\pm

STD%

	Accuracy	F1
CNN+SVM ( Eitta et al., 2021 )	93.7%	-
2D CNN ( Serrano et al., 2018 )	94.60 $\pm$ 0.6%	-
MobileNet ( Irfanullah et al., 2022 )	96.66%	-
C3D+SVM ( Sernani et al., 2021 )	97.86%	97.87%
Computationally Intelligent VD ( Ullah et al., 2022 )	98%	98.10%
Spatial Encoder ( Hanson et al., 2018 )	98.1 $\pm$ 0.58%	-

Table 5: Compare the performance of other models with SSL-V3 on the HF dataset.

SSR	VSR	Crafts Hobbies	Day Time TV Shows	Movie Genres	School Subjects	Hockey Fight
✗	✗	84.38%	87.80%	80.00%	82.05%	(95 $\pm$ 1.41)%
✓		87.50%	78.05%	88.57%	84.62%	(95.6 $\pm$ 1.36)%
	✓	87.50%	87.80%	82.86%	76.92%	(95.6 $\pm$ 2.5)%
✓	✓	93.75%	90.24%	88.57%	94.87%	(98.6 $\pm$ 0.8)%

Table 6: The prediction accuracy on 4 I-CONECT themes and HF dataset with different regressors. SSR and VSR denotes Sequence Score Regressor and Video Score Regressor respectively.

Figure 7: The prediction accuracy on 4 themes (I-CONECT dataset) with different regressors.

FL	CL Loss	BCE Loss	Crafts Hobbies	Day Time TV Shows	Movie Genres	School Subjects	Hockey Fight
✓			81.25%	65.85%	54.29%	51.28%	(92.8 $\pm$ 1.6)%
✓	✓		84.38%	85.37%	71.43%	79.49%	(94.4 $\pm$ 1.96)%
✓		✓	81.25%	68.29%	71.43%	69.23%	(94.2 $\pm$ 1.47)%
✓	✓	✓	93.75%	90.24%	8 8.57%	94.87%	(98.6 $\pm$ 0.8)%

Table 7: The prediction accuracy on 4 I-CONECT themes and HF dataset with different loss functions. CL means Contrastive Learning.

Figure 8: The prediction accuracy on 4 themes (I-CONECT dataset) with different loss functions. CL means Contrastive Learning.

Panel of Figure 9 — Figure 9: The extracted spatio-temporal feature maps on 16 consecutive frames from two datasets. SSL-V3 focuses more on the bright areas and less on the dark ones. Given that there are $n_{t}$ tubelets, where $n_{t}$ = $\frac{T}{t}$ = $\frac{16}{4}$ = $4$ , there are four feature maps. To simplify the visualization, we pick the first frame of each tubelet to represent it. Fig. 9a is from the I-CONECT dataset, while Fig. 9b is from HF one.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Test Themes	Crafts Hobbies			Day Time TV Shows
Test Themes	FL	FL + BCE	FL + CL	FL	FL + BCE	FL + CL
Accuracy	26/32	26/32	27/32	27/41	28/41	28/41
Accuracy	81.25%	81.25%	84.38%	65.85%	68.29%	85.37%
F1	82.35%	84.21%	85.71%	50.00%	68.29%	85.00%
AUC	85.00%	81.67%	87.50%	65.12%	68.33%	85.36%
MCI/NC	20/12			20/21

Appendix

Table 8: Specific results on 4 themes with different loss functions. CL means Contrastive Learning.

Test Themes	Crafts Hobbies			Day Time TV Shows
Test Themes	No VQA	VSR	SSR	No VQA	VSR	SSR
Accuracy	27/32	28/32	28/32	36/41	36/41	32/41
Accuracy	84.38%	87.50%	87.50%	87.80%	87.80%	78.05%
F1	87.18%	90.00%	90.00%	86.49%	87.18%	78.05%
AUC	86.67%	86.67%	86.67%	87.62%	87.74%	78.10%
MCI/NC	20/12			20/21

Appendix

Table 9: Specific results on 4 themes with different regression head.

Test Themes	Study Loss Function
Test Themes	FL	FL + CL	FL + BCE
Accuracy	(92.80 $\pm$ 1.60)%	(94.40 $\pm$ 1.96)%	(94.20 $\pm$ 1.47)%
F1	(92.92 $\pm$ 1.74)%	(94.38 $\pm$ 1.96)%	(94.19 $\pm$ 1.47)%
AUC	(92.80 $\pm$ 1.60)%	(94.40 $\pm$ 1.96)%	(94.20 $\pm$ 1.47)%
Sensitivity/	(90.80 $\pm$ 3.71)%/	(94.00 $\pm$ 4)%/	(94.00 $\pm$ 2.53)% /
Specificity	(94.80 $\pm$ 2.04)%	(94.80 $\pm$ 4.31)%	(94.40 $\pm$ 2.93)%

Appendix

Table 10: Specific results on HF dataset with different loss functions and regressors. CL means Contrastive Learning.

Initials	Vocabulary
AD	Alzheimer’s Disease
ADRD	Alzheimer’s Disease Related Dementia
BEEF	Bidirectional Effect-Based Fusion
CBS Loss	Combined Batch- and Subject-level Loss
CLS	Classification
HF	Hockey Fight Detection

Appendix

Table 11: Glossary

Explore similar work

FGSVQA: Frequency-Guided Short-form Video Quality Assessment
May 19, 2026Xinyi Wang, Angeliki Katsenou, Junxiao Shen +1Video Quality Assessment
School of Computer Science, University of Bristol, Bristol BS1 8UB, UK
Analytic Score Optimization for Multi Dimension Video Quality Assessment
Feb 18, 2026Boda Lin, Yongjie Zhu, Wenyu Qin +2Video Quality Assessment
Kling Team, Kuaishou Technology, Beijing, China.
DPC-VQA: Decoupling Quality Perception and Residual Calibration for Video Quality Assessment
Apr 14, 2026Xinyue Li, Shubo Xu, Zhichao Zhang +3Video Quality Assessment VLM Calibration
Shanghai Jiao Tong University, Shanghai, China · Baidu Inc., Beijing, China · Xinjiang University, Urumqi, China

May 19, 2026J/K move · Enter open · S save

FGSVQA: Frequency-Guided Short-form Video Quality Assessment

Xinyi Wang, Angeliki Katsenou, Junxiao Shen, David Bull

School of Computer Science, University of Bristol, Bristol BS1 8UB, UK

Video Quality Assessment

Open paper AskPDF arXiv

Short-form video poses new challenges to the quality assessment of user-generated content (UGC) due to its complex generation pipeline, rapid content variation, and mixed distortions. To address this challenge, we propose an end-to-end video quality assessment (VQA) framework that employs a dense visual encoder based on CLIP, and incorporates compression priors derived from the frequency domain to generate artifact- and structure-aware weight maps for feature aggregation. By explicitly decomposing artifact, structure, and original visual feature branches and adaptively fusing them over time through a learned gating module, the proposed method achieves accurate and efficient quality prediction. Experimental results show that our method achieves strong performance on short-form video datasets in terms of average rank and linear correlation (SRCC: 0.736, PLCC: 0.787), while maintaining efficient inference runtime. The code and additional results are available at: https://github.com/xinyiW915/FGSVQA.

Abstract

Explore similar work

FGSVQA: Frequency-Guided Short-form Video Quality Assessment

Analytic Score Optimization for Multi Dimension Video Quality Assessment

DPC-VQA: Decoupling Quality Perception and Residual Calibration for Video Quality Assessment