Bootstrapping Audiovisual Speech Recognition in Zero-AV-Resource Scenarios

Pol Buitrago; Javier Hernando

eess.ASMar 9, 2026

Bootstrapping Audiovisual Speech Recognition in Zero-AV-Resource Scenarios

Authors: Pol Buitrago, Javier Hernando

Organizations: Barcelona Supercomputing Center (BSC), Spain · Universitat Polit`ecnica de Catalunya (UPC), Spain

Abstract

Audiovisual speech recognition (AVSR) combines acoustic and visual cues to improve transcription robustness under challenging conditions but remains out of reach for most under-resourced languages due to the lack of labeled video corpora for training. Synthetic visual data have been shown to be an effective augmentation strategy for addressing AV data scarcity. However, a more challenging scenario arises for languages such as Catalan, where no real audiovisual data are available for training. In this study, we investigate whether AVSR can be bootstrapped in such a zero-AV-resource setting, using synthetic visual data as the sole source of visual supervision. We synthesize over 700 hours of talking-head video and fine-tune a pre-trained AV-HuBERT model. On a manually annotated Catalan benchmark, our model achieves near state-of-the-art (SOTA) performance with much fewer parameters and training data than SOTA ASR systems such as Whisper-large-v3, outperforms an identically trained audio-only baseline, and preserves multimodal advantages under acoustic degradation. Scalable synthetic video thus offers a viable substitute for real recordings in zero-AV-resource AVSR.

Figures & tables

Figure 1 : Pipeline for generating synthetic audiovisual data from speech audio and facial images. Still face images are selected and animated with lip movements synchronized to the input speech using a pre-trained Wav2Lip+GAN model Prajwal et al. (2020) ; Goodfellow et al. (2014) , producing talking-head videos aligned with the original audio.

Figure 2 : The 68-point facial landmark template used for morphological face filtering. Facial landmarks are extracted using dlib King (2009) .

Figure 3 : Semi-automatic pipeline for annotating the Catalan AV benchmark. The process combines segment extraction, morphological filtering, automatic pseudo-labeling Radford et al. (2022) , and manual verification and refinement via a custom graphical interface (GUI).

Figure 4 : Label-Inspector GUI used for human verification and refinement of automatically generated audiovisual annotations. The interface provides audio and video playback, precise temporal boundary adjustment, clip cutting and invalidation controls, editable transcription, and reference frames from different points of each clip to facilitate rapid inspection of temporal alignment and visual quality.

Model	AV	A	V
Catalan AVSR (Our model)	19.6%	23.1%	105%

Table 1 : WER (%) on the real Catalan audiovisual benchmark for three separately trained AV-HuBERT variants: audiovisual (AV), audio-only (A), and video-only (V).

Model	WER	Catalan Data	Parameters
Our model (AV)	19.6%	723h	325M
Whisper-large	31.4%	1,883h	1,550M
Whisper-large-v3	18.3%	>1,883h	1,550M

Table 2 : WER comparison on our manually annotated Catalan AV test set. Results compare our AVSR model with two state-of-the-art Whisper variants.

Panel of Figure 5 — Figure 5 : WER (%) comparison under additive white Gaussian noise at different SNR levels.

Dataset	$\rho_{0}$	$\rho^{*}$	$\tau^{*}$ (ms)	$S$
LIP-RTVE (real)	$-0.024$	$0.288$	$-160$	$0.234$
TV3Parla (synth.)	$0.007$	$0.314$	$-120$	$0.320$
ParlamentParla (synth.)	$-0.050$	$0.280$	$-120$	$0.258$

Table 3 : Audiovisual temporal coupling between mouth opening and acoustic energy.

\rho_{0}

, zero-lag correlation;

\rho^{*}

, best-lag correlation;

\tau^{*}

, lag at peak correlation;

S

, synchrony excess (correlation beyond chance). All values are dataset medians.

Comparison	$\delta$	Effect
Real vs. TV3Parla	$-0.326$	Small
Real vs. ParlamentParla	$-0.135$	Negligible

Table 4 : Effect sizes (

\delta

) for real-vs-synthetic differences in synchrony excess. Negligible effect sizes indicate comparable audiovisual coupling.

Dataset	$\|\Delta\hat{o}(t)\|$	Optical Flow
LIP-RTVE (real)	0.010	0.352
TV3Parla (synth.)	0.007	0.516
ParlamentParla (synth.)	0.007	0.512
Synthetic vs. Real	$\delta=-0.601$	$\delta=0.447$
Effect	Large	Medium

Table 5 : Articulatory motion: frame-to-frame landmark opening change and optical-flow magnitude in the mouth region (medians). Cliff’s

\delta

corresponds to the pooled synthetic-versus-real comparison.

Dataset	$\hat{I}(\hat{o};a)$	Effect vs. Real
LIP-RTVE (real)	0.0563	Baseline
TV3Parla (synth.)	0.0574	$\delta=-0.012$ (negligible)
ParlamentParla (synth.)	0.0465	$\delta=-0.067$ (negligible)

Table 6 : Audiovisual energy correlation (mutual information estimate between mouth opening and acoustic energy envelope,

k=5

Explore similar work

M2S-AVSR: Modality-aware Multi-view Self-supervised Representation for Robust Audio-Visual Speech Recognition
Jun 4, 2026Fei Su, Cancan Li, Ming Li +1Audio-Visual Reasoning Multi-View
School of Artificial Intelligence and the School of Computer Science, Wuhan University, China · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China · School of Artificial Intelligence, Wuhan University, China
Candor-LR: A Dyadic Conversational Dataset for Audio-Visual Speech Recognition
Sep 9, 2026Rishabh Jain, Aristeidis Papadopoulos, Zhaofeng Lin +1Automatic Speech Recognition Evaluation Conversational Datasets
Sigmedia Group, School of Engineering, Trinity College Dublin, Ireland
LRS-VoxMM: A benchmark for in-the-wild audio-visual speech recognition
Apr 30, 2026Doyeop Kwak, Jeongsoo Choi, Suyeon Lee +1Automatic Speech Recognition Evaluation Audio-Visual Reasoning
Korea Advanced Institute of Science and Technology, South Korea

Jun 4, 2026, eess.ASJ/K move · Enter open · S save

M2S-AVSR: Modality-aware Multi-view Self-supervised Representation for Robust Audio-Visual Speech Recognition

Fei Su, Cancan Li, Ming Li, Juan Liu

School of Artificial Intelligence and the School of Computer Science, Wuhan University, China · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China · School of Artificial Intelligence, Wuhan University, China

Audio-Visual Reasoning Multi-View

Open paper AskPDF arXiv

Audio-Visual Speech Recognition (AVSR) enhances speech recognition robustness by leveraging visual cues, while real-world scenarios remain challenging due to viewpoint variation, audio distortion, and visual occlusion, which degrade modality quality and increase audio-visual asynchrony. In this paper, we propose a novel Modality-aware Multi-view Self-supervised representation framework for robust Audio-Visual Speech Recognition (M2S-AVSR). First, we introduce a multi-view representation learning encoder to learn view-invariant visual speech representations. Next, we employ a modality-aware module that explicitly models modality quality and cross-modal synchrony to perform fine-grained modality-aware fusion, enabling fine-grained visual information injection during decoding. In addition, we release AISHELL8-RealScene, a public multi-scenario, multi-view conversational audio-visual dataset recorded in real-world environments, and establish a speech recognition benchmark on it. Experiments on English and Mandarin benchmarks demonstrate the effectiveness of the proposed method under challenging conditions. On LRS3, M2S-AVSR achieves up to 29.4% relative improvement under viewpoint perturbation and visual degradation settings. Our method also achieves new state-of-the-art performance on the MISP2021-AVSR test set. On AISHELL8-RealScene, it achieves the best result in outdoor scenes. The proposed method and dataset provide useful support for future research on robust speech and multimodal tasks under realistic conditions.

Abstract

Explore similar work

M2S-AVSR: Modality-aware Multi-view Self-supervised Representation for Robust Audio-Visual Speech Recognition

Candor-LR: A Dyadic Conversational Dataset for Audio-Visual Speech Recognition

LRS-VoxMM: A benchmark for in-the-wild audio-visual speech recognition