Improving Audiovisual Speech Recognition through Synthetic Visual Data Augmentation
Authors: Pol Buitrago, Pol Gàlvez, Javier Hernando
Organizations: Universitat Politècnica de Catalunya (UPC), Carrer de Jordi Girona, 1–3, Barcelona, 08034, Spain · Barcelona Supercomputing Center (BSC), Carrer de Jordi Girona, 29, Barcelona, 08034, Spain
Audiovisual Speech Recognition (AVSR) is a multimodal approach to speech recognition that incorporates visual information from lip movements to enhance model performance. Despite its advantages, its development remains constrained by the limited availability of labeled audiovisual (AV) datasets. This work explores the use of synthetic visual data as a solution, using an audio-driven talking-head pipeline to generate lip-synchronized visual content from existing audio data. We evaluate the effectiveness of synthetic visual data both as an augmentation strategy and as a standalone training resource, applying our approach to Spanish and Catalan. Our results show that augmenting real AV data with synthetic samples yields relative Word Error Rate (WER) reductions of up to 16.2%, demonstrating the potential of this approach. Moreover, we demonstrate that synthetic data alone can serve as a baseline for AVSR training in languages lacking AV datasets. These findings provide evidence that synthetic visual data can serve as a scalable solution to AVSR data scarcity, enabling broader language coverage.
Figures & tables
Figure 1: Audio-Visual Synthetic Data Generator ( AVSynthGen ) pipeline scheme. The system generates synchronized audiovisual data by applying lip synchronization to static images using speech audio inputs.
Figure 2: Visualization of the 68 facial landmark template used in the morphological filtering stage. The facial landmarks are extracted using dlib ( King, 2009 ) .
Figure 3: Morphological detection stage of the AVSynthGen pipeline. Facial landmarks are extracted using dlib ( King, 2009 ) , and the mouth region is defined from the corresponding keypoints. Inter-landmark distances are used to evaluate mouth visibility through geometric thresholds.
Figure 4: Synthetic visual examples generated by the AVSynthGen pipeline. The images show lip-synchronized faces produced by animating static images using real speech audio.
Figure 5: Semi-automatic pipeline for annotating the Catalan AV benchmark. Raw video material is segmented into short clips, filtered using the morphological detection stage (Section 2.1 ) to ensure mouth visibility, automatically pseudo-labeled with an ASR system ( Radford et al., 2022 ) , normalized, and manually reviewed and corrected via a custom graphical interface ( Label-Inspector ).
Name
Language
Duration
— Real AV Datasets —
LIP-RTVE
Spanish
≈ 13h
CMU-MOSEAS es
Spanish
≈ 13h
MuAViC es
Spanish
≈ 180h
AV-CAT (custom test)
Catalan
≈ 51min
— Synthetic AV Datasets —
Table 1: AV datasets employed in our experiments.
Configuration
Training Datasets
RealVisual es
LIP-RTVE, CMU-MOSEAS es , MuAViC es ( ≈ 206h)
SynthVisual es
SynthAV-CV ( ≈ 512h)
MixedVisual es
LIP-RTVE, CMU-MOSEAS es , MuAViC es , SynthAV-CV ( ≈ 718h)
SynthVisual cat
TV3ParlAV, ParlAVment ( ≈ 723h)
Table 2: Overview of the training datasets used for each AVSR training configuration. Configuration names indicate the source of the visual training data and the target language.
Model (V)
LIP-RTVE
CMU-MOSEAS es
MuAViC es
RealVisual es
90.4%
90.7%
102%
SynthVisual es
97.4%
97%
101.9%
MixedVisual es
68%( ↓ 24.7%)
67.1%( ↓ 26%)
98%( ↓ 3.9%)
Table 3: WER comparison of the trained Spanish models in the video-only modality on different benchmarks. The values in parentheses indicate the relative improvement over the RealVisual es (video-only) model.
Figure 6: WER comparison for the video-only modality on different datasets. The “Average” bar represents the mean WER across the three datasets.
Model (AV)
LIP-RTVE
CMU-MOSEAS es
MuAViC es
RealVisual es
9.3%
15.4%
16.6%
SynthVisual es
21.1%
35.2%
39.6%
MixedVisual es
8.1%( ↓ 12.9%)
12.9%( ↓ 16.2%)
15.7%( ↓ 5.4%)
Table 4: WER comparison of the trained Spanish AVSR models on different benchmarks. The values in parentheses indicate the relative improvement over the RealVisual es model.
Figure 7: WER comparison for the audiovisual modality of AVSR models on different datasets. The “Average” bar represents the mean WER across the three datasets. The results for the SynthVisual es model are omitted to focus on the relative improvement achieved through the data augmentation strategy.
Model
LIP-RTVE
CMU-MOSEAS es
MuAViC es
Anwar et al. (multilingual)
24.8%
25.5%
16.2%
Anwar et al. (monolingual)
17.6%
16.7%
15.9%
RealVisual es [Ours]
9.3%
15.4%
16.6%
MixedVisual es [Ours]
8.1%
12.9%
15.7 %
Table 5: WER comparison of our models against the multilingual and monolingual Spanish AVSR baselines from Anwar et al. (2023) on different benchmarks.
Figure 8: Split violin plot of mean Integrated Gradients for the video modality, comparing RealVisual es (real video) and SynthVisual es (synthetic video).
Figure 9: WER performance of MixedVisual es under different acoustic noise conditions. Results are shown for audiovisual (AV), audio-only (A), and video-only (V) modalities across varying SNR levels.
Model / Modality
Audiovisual
Audio-only
Video-only
SynthVisual cat
19.6%
23.1%
105%
Table 7: WER of the SynthVisual cat model, evaluated on the manually labeled AV-CAT benchmark. Results are shown for audiovisual (AV), audio-only (A), and video-only (V) trained variants.
School of Artificial Intelligence and the School of Computer Science, Wuhan University, China · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China · School of Artificial Intelligence, Wuhan University, China