Scaling Audio-Visual Quality Assessment Dataset via Crowdsourcing

Renyu Yang; Jian Jin; Lili Meng; Meiqin Liu; Yilin Wang; Balu Adsumilli; Weisi Lin

cs.CVFeb 26, 2026

Scaling Audio-Visual Quality Assessment Dataset via Crowdsourcing

Authors: Renyu Yang, Jian Jin, Lili Meng, Meiqin Liu, Yilin Wang, Balu Adsumilli, Weisi Lin

Organizations: College of Computing and Data Science, Nanyang Technological University, 639798, Singapore. · School of Information Science and Engineering, Shandong Normal University, 250014, China · Institute of Information Science, Beijing Jiao Tong University, 100044, China. · YouTube Media Algorithms team, Google Inc., Mountain View, CA, 94043, USA

Abstract

Audio-visual quality assessment (AVQA) research has been stalled by limitations of existing datasets: they are typically small in scale, with insufficient diversity in content and quality, and annotated only with overall scores. These shortcomings provide limited support for model development and multimodal perception research. We propose a practical approach for AVQA dataset construction. First, we design a crowdsourced subjective experiment framework for AVQA, breaks the constraints of in-lab settings and achieves reliable annotation across varied environments. Second, a systematic data preparation strategy is further employed to ensure broad coverage of both quality levels and semantic scenarios. Third, we extend the dataset with additional annotations, enabling research on multimodal perception mechanisms and their relation to content. Finally, we validate this approach through YT-NTU-AVQ, the largest and most diverse AVQA dataset to date, consisting of 1,620 user-generated audio and video (A/V) sequences. The dataset and platform code are available at https://github.com/renyu12/YT-NTU-AVQ

Figures & tables

Dataset	Source	Collection	Unique AV	Total AV	Distortion	Resolution	Length	Label	Rating Method	Year
UnB-AVQ	CDVL	Manual	6	72	Synthetic	720p	8s	Multi-MOS	In-Lab	2013
LIVE-SJTU	CDVL	Manual	14	336	Synthetic	1080p	8s	AVQA MOS	In-Lab	2019
SJTU-UAV	YFCC100M	Manual	520	520	In-the-wild	720p–1080p	8s	AVQA MOS	In-Lab	2022
Ours	YouTube CC	Sampling + Manual	1620	1620	In-the-wild	$\leq$ 1080p	10s †	Multi-label	Crowdsourcing	2025

Table 1: Comparison of representative AVQA datasets. ( † A small fraction of YT-NTU-AVQ sequences are shorter than 10s)

Figure 1: Distributions of MOS scores in our AVQA dataset. From left to right: overall audio-visual quality (AVQA), video-only quality (AV_VQA), and audio-only quality (AV_AQA). (Mean and standard deviation: AVQA:

\mu=3.47

\sigma=0.72

; AV_VQA:

\mu=3.49

\sigma=0.77

; AV_AQA:

\mu=3.44

\sigma=0.64

)

Figure 2: Distribution of average SROCC values across submissions in different crowdsourcing stages. Submission quality improves progressively from pretest to formal evaluation, validating the effectiveness of our multi-stage filtering pipeline.

Panel of Figure 3 — Figure 3: Scatter plots of MOS comparisons: (a) AVQA vs AV_VQA, (b) AVQA vs AV_AQA, and (c) AV_VQA vs AV_AQA. All three pairs show high correlations, revealing strong monotonic relationships among the score types.

Type	Method	SROCC	PLCC
AQA	PAM	0.4217	0.4358
	AudioBox (CE)	0.4328	0.4335
	AudioBox (PQ)	0.5145	0.5335
VQA	FasterVQA	0.7868	0.7865
	FastVQA	0.8503	0.8514
	Q-Align	0.9342	0.9329

Table 2: Performance (SROCC/PLCC) of baseline models on YT-NTU-AVQ dataset.

Group	$n$	Subj. A%	Diff (V)	Diff (A)
A $<<$ V	87	56.95	-0.06	+0.37
A $<$ V	284	54.20	+0.01	+0.18
A $\approx$ V	489	51.97	+0.08	+0.09
A $>$ V	291	48.52	+0.20	+0.01
A $>>$ V	469	45.08	+0.34	-0.17
Overall	1620	50.01	-0.02	+0.05

Table 3: Subjective audio attention and AVQA–modality score differences across groups defined by the relative quality difference

AV\_AQA-AV\_VQA

(thresholds:

\pm 0.1

slight,

\pm 0.3

large). Subj. A% : mean subjective audio attention. Diff (V/A) : mean difference AVQA minus video/audio scores (positive indicates AVQA score is higher).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Screenshot of the Environment Preparation Confirmation Page

Figure 5: Screenshot of the A/V Sequence Rating Interface

Figure 6: Screenshot of the Additional Questions

Figure 7: Qualification requirements for workers to participate in the Formal Test on AMT.

Figure 8: Distribution of audio semantic label combinations after sampling.

Figure 9: Distributions of pseudo labels before and after sampling. From top to bottom: audio quality (AudioBox PQ), audio quality (AudioBox CE), and video quality (FasterVQA). Weighted sampling and manual additions improve label coverage across the score range.

Panel of Figure 10 — Figure 10: Distributions of normalized low-level audio-visual attributes across four datasets: UnB-AVQ, LIVE-SJTU, SJTU-UAV, and our YT-NTU-AVQ. Our dataset demonstrates broader coverage in both video (TI, SI) and audio (SC, SE) domains.

Figure 11: Distribution of spatial resolutions in YT-NTU-AVQ. The most frequent resolutions are shown individually, while the remaining resolutions are aggregated into an “Others” category.

Figure 12: Distribution of videos across predefined coarse semantic content categories in YT-NTU-AVQ.

Explore similar work

Multimodal Confidence Modeling in Audio-Visual Quality Assessment
May 2, 2026Mayesha Maliha R. Mithila, Mylene C. Q. FariasVideo Quality Assessment Uncertainty-Aware Multimodal Fusion
Texas State University Department of Computer Science
Analytic Score Optimization for Multi Dimension Video Quality Assessment
Feb 18, 2026Boda Lin, Yongjie Zhu, Wenyu Qin +2Video Quality Assessment
Kling Team, Kuaishou Technology, Beijing, China.
LLaVA-Assessor: Building the Foundation LMM For Visual Quality Assessment
Aug 11, 2026Ziheng Jia, Zicheng Zhang, Jiaying Qian +2Image Quality Assessment Video Quality Assessment
Shanghai Artificial Intelligence Laboratory, Shanghai 200032, China

May 2, 2026J/K move · Enter open · S save

Multimodal Confidence Modeling in Audio-Visual Quality Assessment

Mayesha Maliha R. Mithila, Mylene C. Q. Farias

Texas State University Department of Computer Science

Video Quality Assessment Uncertainty-Aware Multimodal Fusion Cross-Modal Attention Multimodal Fusion

Open paper AskPDF arXiv

Audio-visual quality assessment (AVQA) is essential for streaming, teleconferencing, and immersive media. In realistic streaming scenarios, distortions are often asymmetric, where one modality may be severely degraded while the other remains clean. Still, most contemporary AVQA metrics treat audio and video as equally reliable, causing confidence-unaware fusion to emphasize unreliable signals. This paper proposes MCM-AVQA, a multimodal confidence-aware AVQA framework that explicitly estimates modality-specific confidence and injects it into a dedicated audio-visual mixer for cross-modal attention. The Audio-Visual Mixer utilizes frame-level, confidence-guided channel attention to gate fusion, modulating feature interaction between modalities so that high-confidence streams dominate while unreliable inputs are suppressed, preserving temporal degradation patterns. A multi-head visual confidence estimator turns frame-level artifact probabilities into temporally smoothed, clip-level visual confidence scores, while an audio confidence module derives confidence from speech-quality cues without requiring a clean reference. Experiments on multiple AVQA benchmarks show that MCM-AVQA, and specifically its confidence-guided Audio-Visual Mixer, improve correlation with human mean opinion scores and yield more interpretable behavior under real-world asymmetric audio-visual distortions.

Abstract

Explore similar work

Multimodal Confidence Modeling in Audio-Visual Quality Assessment

Analytic Score Optimization for Multi Dimension Video Quality Assessment

LLaVA-Assessor: Building the Foundation LMM For Visual Quality Assessment