Bootstrapping Audiovisual Speech Recognition in Zero-AV-Resource Scenarios

Pol Buitrago; Javier Hernando

eess.ASMar 9, 2026

Bootstrapping Audiovisual Speech Recognition in Zero-AV-Resource Scenarios

Authors: Pol Buitrago, Javier Hernando

Organizations: Barcelona Supercomputing Center (BSC), Spain · Universitat Polit`ecnica de Catalunya (UPC), Spain

Abstract

Audiovisual speech recognition (AVSR) combines acoustic and visual cues to improve transcription robustness under challenging conditions but remains out of reach for most under-resourced languages due to the lack of labeled video corpora for training. Synthetic visual data have been shown to be an effective augmentation strategy for addressing AV data scarcity. However, a more challenging scenario arises for languages such as Catalan, where no real audiovisual data are available for training. In this study, we investigate whether AVSR can be bootstrapped in such a zero-AV-resource setting, using synthetic visual data as the sole source of visual supervision. We synthesize over 700 hours of talking-head video and fine-tune a pre-trained AV-HuBERT model. On a manually annotated Catalan benchmark, our model achieves near state-of-the-art (SOTA) performance with much fewer parameters and training data than SOTA ASR systems such as Whisper-large-v3, outperforms an identically trained audio-only baseline, and preserves multimodal advantages under acoustic degradation. Scalable synthetic video thus offers a viable substitute for real recordings in zero-AV-resource AVSR.

Figures & tables

Figure 1 : Pipeline for generating synthetic audiovisual data from speech audio and facial images. Still face images are selected and animated with lip movements synchronized to the input speech using a pre-trained Wav2Lip+GAN model Prajwal et al. (2020) ; Goodfellow et al. (2014) , producing talking-head videos aligned with the original audio.

Figure 2 : The 68-point facial landmark template used for morphological face filtering. Facial landmarks are extracted using dlib King (2009) .

Figure 3 : Semi-automatic pipeline for annotating the Catalan AV benchmark. The process combines segment extraction, morphological filtering, automatic pseudo-labeling Radford et al. (2022) , and manual verification and refinement via a custom graphical interface (GUI).

Figure 4 : Label-Inspector GUI used for human verification and refinement of automatically generated audiovisual annotations. The interface provides audio and video playback, precise temporal boundary adjustment, clip cutting and invalidation controls, editable transcription, and reference frames from different points of each clip to facilitate rapid inspection of temporal alignment and visual quality.

Model	AV	A	V
Catalan AVSR (Our model)	19.6%	23.1%	105%

Table 1 : WER (%) on the real Catalan audiovisual benchmark for three separately trained AV-HuBERT variants: audiovisual (AV), audio-only (A), and video-only (V).

Model	WER	Catalan Data	Parameters
Our model (AV)	19.6%	723h	325M
Whisper-large	31.4%	1,883h	1,550M
Whisper-large-v3	18.3%	>1,883h	1,550M

Table 2 : WER comparison on our manually annotated Catalan AV test set. Results compare our AVSR model with two state-of-the-art Whisper variants.

Panel of Figure 5 — Figure 5 : WER (%) comparison under additive white Gaussian noise at different SNR levels.

Dataset	$\rho_{0}$	$\rho^{*}$	$\tau^{*}$ (ms)	$S$
LIP-RTVE (real)	$-0.024$	$0.288$	$-160$	$0.234$
TV3Parla (synth.)	$0.007$	$0.314$	$-120$	$0.320$
ParlamentParla (synth.)	$-0.050$	$0.280$	$-120$	$0.258$

Table 3 : Audiovisual temporal coupling between mouth opening and acoustic energy.

\rho_{0}

, zero-lag correlation;

\rho^{*}

, best-lag correlation;

\tau^{*}

, lag at peak correlation;

S

, synchrony excess (correlation beyond chance). All values are dataset medians.

Comparison	$\delta$	Effect
Real vs. TV3Parla	$-0.326$	Small
Real vs. ParlamentParla	$-0.135$	Negligible

Table 4 : Effect sizes (

\delta

) for real-vs-synthetic differences in synchrony excess. Negligible effect sizes indicate comparable audiovisual coupling.

Dataset	$\|\Delta\hat{o}(t)\|$	Optical Flow
LIP-RTVE (real)	0.010	0.352
TV3Parla (synth.)	0.007	0.516
ParlamentParla (synth.)	0.007	0.512
Synthetic vs. Real	$\delta=-0.601$	$\delta=0.447$
Effect	Large	Medium

Table 5 : Articulatory motion: frame-to-frame landmark opening change and optical-flow magnitude in the mouth region (medians). Cliff’s

\delta

corresponds to the pooled synthetic-versus-real comparison.

Dataset	$\hat{I}(\hat{o};a)$	Effect vs. Real
LIP-RTVE (real)	0.0563	Baseline
TV3Parla (synth.)	0.0574	$\delta=-0.012$ (negligible)
ParlamentParla (synth.)	0.0465	$\delta=-0.067$ (negligible)

Table 6 : Audiovisual energy correlation (mutual information estimate between mouth opening and acoustic energy envelope,

k=5

Explore similar work

Jun 4, 2026eess.AS

M2S-AVSR: Modality-aware Multi-view Self-supervised Representation for Robust Audio-Visual Speech Recognition

Audio-Visual Speech Recognition (AVSR) enhances speech recognition robustness by leveraging visual cues, while real-world scenarios remain challenging due to viewpoint variation, audio distortion, and visual occlusion, which degrade modality quality and increase audio-visual asynchrony. In this paper, we propose a novel Modality-aware Multi-view Self-supervised representation framework for robust Audio-Visual Speech Recognition (M2S-AVSR). First, we introduce a multi-view representation learning encoder to learn view-invariant visual speech representations. Next, we employ a modality-aware module that explicitly models modality quality and cross-modal synchrony to perform fine-grained modality-aware fusion, enabling fine-grained visual information injection during decoding. In addition, we release AISHELL8-RealScene, a public multi-scenario, multi-view conversational audio-visual dataset recorded in real-world environments, and establish a speech recognition benchmark on it. Experiments on English and Mandarin benchmarks demonstrate the effectiveness of the proposed method under challenging conditions. On LRS3, M2S-AVSR achieves up to 29.4% relative improvement under viewpoint perturbation and visual degradation settings. Our method also achieves new state-of-the-art performance on the MISP2021-AVSR test set. On AISHELL8-RealScene, it achieves the best result in outdoor scenes. The proposed method and dataset provide useful support for future research on robust speech and multimodal tasks under realistic conditions.

Fei Su, Cancan Li, Ming Li +1

School of Artificial Intelligence and the School of Computer Science, Wuhan University, China · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China · School of Artificial Intelligence, Wuhan University, China

Audio-Visual Reasoning Multi-View

Ask PDF

Sep 9, 2026eess.AS

Candor-LR: A Dyadic Conversational Dataset for Audio-Visual Speech Recognition

Current audio-visual speech recognition (AVSR) benchmarks, like LRS3, rely heavily on clean, scripted and rehearsed speech. They fail to reflect the complexity of natural conversation, which involves overlapping speech, spontaneous turn-taking, unscripted vocabulary and variable acoustic conditions. To shift the field toward realistic dialogue, we introduce Candor-LR, a conversational benchmark derived from the CANDOR corpus of 1,656 natural dyadic videoconferences. Our custom data preparation pipeline yields 713.5, 10.1, and 60.1 hours of training, validation, and test data, respectively. Evaluating pretrained AVSR models on Candor-LR reveals that audio-only accuracy drops sharply compared to LRS3, but visual cues compensate effectively, driving much larger performance gains on Candor-LR than on LRS3. Furthermore, training on this corpus significantly improves cross-domain robustness under both clean and noisy conditions, as its realistic conversational data captures broader audio-video features. We open-source our pipeline to ensure reproducibility, establishing Candor-LR as a challenging benchmark for conversational AVSR.

Rishabh Jain, Aristeidis Papadopoulos, Zhaofeng Lin +1

Sigmedia Group, School of Engineering, Trinity College Dublin, Ireland

Automatic Speech Recognition Evaluation Conversational Datasets Dialogue Benchmarks

Ask PDF

Apr 30, 2026eess.AS

LRS-VoxMM: A benchmark for in-the-wild audio-visual speech recognition

We introduce LRS-VoxMM, an in-the-wild benchmark for audio-visual speech recognition (AVSR). The benchmark is derived from VoxMM, a dataset of diverse real-world spoken conversations with human-annotated transcriptions. We select AVSR-suitable samples and preprocess them in an LRS-style format for direct use in existing AVSR pipelines. Compared with commonly used benchmarks, LRS-VoxMM covers a more diverse range of scenarios and acoustic conditions. We also release distorted evaluation sets with additive noise, reverberation, and bandwidth limitation to support evaluation under severe acoustic degradation. Experimental results show that LRS-VoxMM is considerably harder than LRS3 and that the contribution of visual information becomes more evident as the audio signal degrades. LRS-VoxMM supports more realistic AVSR benchmarking and encourages further research on the role of visual information in challenging real-world conditions.

Doyeop Kwak, Jeongsoo Choi, Suyeon Lee +1

Korea Advanced Institute of Science and Technology, South Korea

Automatic Speech Recognition Evaluation Audio-Visual Reasoning Transcript Voxel

Ask PDF