cs.CVSep 6, 2026
SaveWhen Speech Meets Lips: Interpretable Audio-Visual Synchronization for L2 Pronunciation Assessment
Organizations: Wenzhou-Kean University Wenzhou, China
Abstract
Automatic Pronunciation Assessment (APA) systems have achieved strong performance with transformer-based models and self-supervised speech representations. However, most methods rely only on acoustic signals and overlook temporal synchronization between speech and articulatory movements, limiting diagnostic feedback on timing mismatches important for L2 pronunciation training. We propose an interpretable audio-visual synchronization framework that explicitly models speech-lip temporal alignment through feature encoding, cross-attention fusion, lag estimation, stability quantification, and visualization. The framework introduces frame-level lag trajectories and a Lag Stability Index (LSI) to quantify synchronization robustness. We also interviewed 30 participants, including 10 instructors and 20 students with diverse first-language backgrounds, to assess its effectiveness. By transforming implicit alignment into interpretable representations, the framework connects automatic scoring with actionable Computer-Aided Pronunciation Training feedback. Datasets and supplemental materials are available at https://www.robots.ox.ac.uk/~vgg/data/lip_reading/.
Figures & tables
Figure 1. Overview of the proposed audio–visual synchronization analysis framework. The system models cross-modal temporal alignment between speech and lip movements through feature encoding, cross-attention fusion, lag estimation, and stability quantification, enabling interpretable pronunciation feedback. Overview of the proposed audio–visual synchronization analysis framework.
Figure 2. Lip geometry modeling and contour representation. (a) Key geometric measurements of lip articulation, including upper lip width, lower lip width, and vertical lip height, which characterize mouth openness and articulation extent during speech production. (b) Extracted facial landmark configuration used to capture detailed lip motion and structural variations across frames. (c) Lip contour reconstruction from detected landmark points to obtain a continuous representation of the lip boundary. (d) Bézier curve fitting for curvature aware lip shape modeling, enabling smooth contour approximation and robust geometric feature extraction. These representations provide stable visual descriptors for downstream audio–visual synchronization analysis and temporal alignment modeling. Lip geometry modeling and contour representation.
| Dataset | Duration Range | #Speakers | Recording Condition | Avg Frames | Evaluation Focus |
|---|---|---|---|---|---|
| LabSpeech-A | 2–5 s | 10 | Controlled studio | 120 | Stable articulation |
| LabSpeech-B | 5–10 s | 8 | Controlled studio | 240 | Lag consistency |
| Conversational-1 | 3–15 s | 12 | Natural conversation | 360 | Local misalignment |
| Conversational-2 | 10–30 s | 15 | Natural conversation | 720 | Dynamic variability |
| Benchmark-AV | 2–8 s | 20 | Benchmark corpus | 180 | Cross-speaker comparison |
| Noisy-AV | 5–12 s | 10 | Background noise | 300 | Robustness |
Table 1. Datasets and Their Characteristics
Figure 3. Cross-speaker synchronization comparison using embedding distributions and lag statistics. (a) Synchronization embedding distribution projected into a two-dimensional space, showing clustering patterns across speakers from different linguistic backgrounds. Distinct clusters indicate variations in articulation timing and synchronization dynamics. (b) Lag Stability Index (LSI) across speaker groups, illustrating differences in synchronization consistency during speech production. Higher LSI values indicate more stable audio–visual temporal alignment. (c) Mean lag comparison across speakers, highlighting systematic differences in temporal offsets between speech signals and lip movements across speaker groups. Cross-speaker synchronization comparison using embedding distributions and lag statistics.
Figure 4. Dataset characteristics and temporal synchronization analysis across recording conditions. (a) Normalized structural feature distribution across datasets, summarizing differences in duration range, number of speakers, frame density, and recording conditions. These variations highlight the diversity of evaluation scenarios used in synchronization analysis. (b) Temporal lag smoothing effect showing the comparison between raw lag trajectories and the smoothed lag signal. Smoothing reduces high-frequency fluctuations while preserving the overall synchronization trend, enabling more stable lag estimation. (c) Multi-speaker synchronization comparison illustrating lag displacement patterns across speakers. Individual speaker trajectories are shown alongside the mean synchronization curve and standard deviation band, revealing variability in temporal alignment dynamics. Dataset characteristics and temporal synchronization analysis across recording conditions.
Figure 5. Participant diversity and recording conditions in the educational study. The study includes participants from diverse linguistic backgrounds, including East Asian (Mandarin L1), South Asian (Hindi L1), African (Yoruba/Swahili L1), and European (Spanish L1) speakers. Recordings were collected under multiple real world conditions, including studio microphones, laptop microphones, smartphone recordings, and classroom ambient recorders, reflecting a range of acoustic environments typical in educational settings. For privacy protection, all participant faces shown in this figure are anonymized and represented using illustrative avatars rather than real identities. Participant diversity and recording conditions in the educational study.
Figure 6. Survey evaluation results across eight instructional dimensions. The user study evaluates the proposed system across eight instructional dimensions: visualization interpret-ability, LSI usefulness, educational applicability, interface transparency, instructional clarity, feedback efficiency, user confidence, and adoption intent. Bars indicate the mean participant ratings on a 5-point Likert scale, while the red line shows the normalized score proportion. The dashed reference line corresponds to a score of 4.5, illustrating that most evaluation dimensions received consistently high ratings, indicating strong perceived usability and instructional value of the system. Survey evaluation results across eight instructional dimensions.