When AVSR Meets Video Conferencing: Dataset, Degradation, and the Hidden Mechanism Behind Performance Collapse
Authors: Yihuan Huang, Jun Xue, Liu Jiajun, Daixian Li, Tong Zhang, Zhuolin Yi, Yanzhen Ren, Kai Li
Organizations: Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education · School of Cyber Science and Engineering, Wuhan University · Tsinghua University
Audio-Visual Speech Recognition (AVSR) has achieved remarkable progress in offline conditions, yet its robustness in real-world video conferencing (VC) remains largely unexplored. This paper presents the first systematic evaluation of state-of-the-art AVSR models across mainstream VC platforms, revealing severe performance degradation caused by transmission distortions and spontaneous human hyper-expression. To address this gap, we construct MLD-VC, the first multimodal dataset tailored for VC, comprising 31 speakers, 22.79 hours of audio-visual data, and explicit use of the Lombard effect to enhance human hyper-expression. Through comprehensive analysis, we find that speech enhancement algorithms are the primary source of distribution shift, which alters the first and second formants of audio. Interestingly, we find that the distribution shift induced by the Lombard effect closely resembles that introduced by speech enhancement, which explains why models trained on Lombard data exhibit greater robustness in VC. Fine-tuning AVSR models on MLD-VC mitigates this issue, achieving an average 17.5% reduction in CER across several VC platforms. Our findings and dataset provide a foundation for developing more robust and generalizable AVSR systems in real-world video conferencing. MLD-VC is available at https://huggingface.co/datasets/nccm2p2/MLD-VC.
Figures & tables
Figure 1: Compared to the face-to-face scenario, we identify two key factors affecting AVSR in video conferencing: transmission distortions in online and hyper-expression in a hindered communication environment.
Model
Dataset
Modal
Language
Platform
WER(%)↓
Δ WER(%)↓
CER(%)↓
Δ CER(%)↓
mWhisper-Flamingo [ 30 ]
LRS3 [ 31 ]
A
En
Offline
0.68
Zoom
10.38
9.70
Lark
19.55
18.87
Tencent Meeting
11.89
11.21
AV
En
Offline
0.73
Zoom
9.22
8.49
Table 1: Performance of AVSR models on video conferencing platforms. “Offline” refers to the original dataset. “*” indicates that the corresponding metric is not applicable or not reported. “ Δ ” refers to the absolute difference between VC platforms and Offline. Bold indicates the least change for each platform.
Dataset
Speakers
Duration(h)
Language
VC Num
Year
Lombard-Grid
54
3.81
En
2018
RoomReader
118
8
En
1
2022
MLD-VC (Ours)
31
22.79
En & Zh
4
2025
Table 2: Comparison of the proposed MLD-VC with existing multimodal Lombard and video conferencing datasets. “VC Num” refers to the number of video conferencing platforms included. “*” indicates that the dataset was collected offline.
Figure 2: Probability density curves of five acoustic features (F0, F1, F2, Loudness, and AlphaRatio) across five subsets in the proposed MLD-VC under Plain (left column) and Lombard (right column). Across features(column-wise), both Plain and Lombard show noticeable high-frequency shifts in F1 and F2 under video conferencing conditions. Across conditions (row-wise), Lombard also exhibits overall higher F1 and F2 than Plain .
Figure 3: Illustration of how video conferencing platforms affect speech formants. We simulate platform processing by applying OPUS compression and three typical speech enhancement algorithms. The four subplots show F1 and F2 distributions before and after processing. Only the enhancement stages introduce noticeable spectral shifts and formant distortions, which explain the frequency bias observed in real video conferencing recordings.
Acoustic Feature
Platform
Offline
Tencent Meeting
Lark
DingTalk
Zoom
F0 ( Plain )
37.28
37.66
37.51
37.66
37.39
F0 ( Lombard )
37.58
37.85
37.62
37.64
37.55
F1 ( Plain )
606.90
674.43
704.19
774.61
687.88
F1 ( Lombard )
660.06
756.11
770.17
825.44
712.73
F2 ( Plain )
1655.66
1680.78
1727.37
1783.51
1727.45
Table 3: Peak abscissas of probability density curves for different acoustic features.
Test Dataset
Platform
Finetune
CER(%)↓
Reduction(%)
Chinese-Lips [ 32 ]
Tencent Meeting
×
10.97
Tencent Meeting
✓
9.65
12.0
Lark
×
18.53
Lark
✓
13.64
26.4
Zoom
×
9.22
Zoom
✓
7.93
14.0
Table 4: Results of fine-tuning with MLD-VC dataset on AVSR performance under video conferencing conditions. “*” indicates that the corresponding metric is not applicable or not reported.
Online
Hyper-expression
Tencent Meeting
Lark
Zoom
✓
✓
9.65
13.64
7.93
×
✓
10.15
15.52
10.53
✓
×
10.01
14.48
9.61
Table 5: Ablation study on the effects of online data and hyper-expression in MLD-VC fine-tuning. The results show that both online recording conditions and hyper-expression samples significantly contribute to lowering the CER.
Name
Verb
Classifier
Adjective
Noun
旭峰 (Xufeng)
买 (Mai)
零个 (Lingge)
大 (Da)
沙发 (Shafa)
青木 (Qingmu)
乘 (Cheng)
一架 (Yijia)
小 (Xiao)
飞机 (Feiji)
林俊 (Linjun)
坐 (Zuo)
两只 (Liangzhi)
旧 (Jiu)
火车 (Huoche)
建树 (Jianshu)
去 (Qu)
三条 (Santiao)
快 (Man)
菠萝 (Boluo)
郭浩 (Guohao)
拿 (Na)
四段 (Siduan)
慢 (Man)
枕头 (Zhentou)
南月 (Nanyue)
养 (Yang)
五棵 (Wuke)
新 (Xin)
床 (Chuang)
Table 6: The candidate phoneme-balanced word and the corresponding Pinyin.
Figure 4: Picture of the recording environment.
Figure 5: Duration distribution across subsets of the proposed MLD-VC dataset. “Offline” refers to the subset of recorded content that was captured before video conferencing.
Figure 6: The illustration of face landmarks.
Figure 7: Probability density curves of three visual features (lip width, lip height, and lip roundness) across five subsets in the proposed MLD-VC under Plain (left column) and Lombard (right column).
Visual Feature
Platform
Offline
Tencent Meeting
Lark
DingTalk
Zoom
Lip Width ( clean )
0.73
0.74
0.74
0.74
0.75
Lip Width ( 80 dB )
0.74
0.74
0.74
0.74
0.75
Lip Height ( clean )
0.29
0.28
0.28
0.29
0.28
Lip Height ( 80 dB )
0.27
0.26
0.26
0.27
0.27
Lip Roundness ( clean )
0.41
0.40
0.40
0.41
0.37
Table 7: Peak abscissas of probability density curves for different visual features.
Figure 8: The static page MLD-VC demo in the supplementary material.