Automatic Pronunciation Assessment (APA) systems have achieved strong performance with transformer-based models and self-supervised speech representations. However, most methods rely only on acoustic signals and overlook temporal synchronization between speech and articulatory movements, limiting diagnostic feedback on timing mismatches important for L2 pronunciation training. We propose an interpretable audio-visual synchronization framework that explicitly models speech-lip temporal alignment through feature encoding, cross-attention fusion, lag estimation, stability quantification, and visualization. The framework introduces frame-level lag trajectories and a Lag Stability Index (LSI) to quantify synchronization robustness. We also interviewed 30 participants, including 10 instructors and 20 students with diverse first-language backgrounds, to assess its effectiveness. By transforming implicit alignment into interpretable representations, the framework connects automatic scoring with actionable Computer-Aided Pronunciation Training feedback. Datasets and supplemental materials are available at https://www.robots.ox.ac.uk/~vgg/data/lip_reading/.
Figures & tables
Figure 1. Overview of the proposed audio–visual synchronization analysis framework. The system models cross-modal temporal alignment between speech and lip movements through feature encoding, cross-attention fusion, lag estimation, and stability quantification, enabling interpretable pronunciation feedback. Overview of the proposed audio–visual synchronization analysis framework.
Figure 2. Lip geometry modeling and contour representation. (a) Key geometric measurements of lip articulation, including upper lip width, lower lip width, and vertical lip height, which characterize mouth openness and articulation extent during speech production. (b) Extracted facial landmark configuration used to capture detailed lip motion and structural variations across frames. (c) Lip contour reconstruction from detected landmark points to obtain a continuous representation of the lip boundary. (d) Bézier curve fitting for curvature aware lip shape modeling, enabling smooth contour approximation and robust geometric feature extraction. These representations provide stable visual descriptors for downstream audio–visual synchronization analysis and temporal alignment modeling. Lip geometry modeling and contour representation.
Dataset
Duration Range
#Speakers
Recording Condition
Avg Frames
Evaluation Focus
LabSpeech-A
2–5 s
10
Controlled studio
120
Stable articulation
LabSpeech-B
5–10 s
8
Controlled studio
240
Lag consistency
Conversational-1
3–15 s
12
Natural conversation
360
Local misalignment
Conversational-2
10–30 s
15
Natural conversation
720
Dynamic variability
Benchmark-AV
2–8 s
20
Benchmark corpus
180
Cross-speaker comparison
Noisy-AV
5–12 s
10
Background noise
300
Robustness
Table 1. Datasets and Their Characteristics
Figure 3. Cross-speaker synchronization comparison using embedding distributions and lag statistics. (a) Synchronization embedding distribution projected into a two-dimensional space, showing clustering patterns across speakers from different linguistic backgrounds. Distinct clusters indicate variations in articulation timing and synchronization dynamics. (b) Lag Stability Index (LSI) across speaker groups, illustrating differences in synchronization consistency during speech production. Higher LSI values indicate more stable audio–visual temporal alignment. (c) Mean lag comparison across speakers, highlighting systematic differences in temporal offsets between speech signals and lip movements across speaker groups. Cross-speaker synchronization comparison using embedding distributions and lag statistics.
Figure 4. Dataset characteristics and temporal synchronization analysis across recording conditions. (a) Normalized structural feature distribution across datasets, summarizing differences in duration range, number of speakers, frame density, and recording conditions. These variations highlight the diversity of evaluation scenarios used in synchronization analysis. (b) Temporal lag smoothing effect showing the comparison between raw lag trajectories and the smoothed lag signal. Smoothing reduces high-frequency fluctuations while preserving the overall synchronization trend, enabling more stable lag estimation. (c) Multi-speaker synchronization comparison illustrating lag displacement patterns across speakers. Individual speaker trajectories are shown alongside the mean synchronization curve and standard deviation band, revealing variability in temporal alignment dynamics. Dataset characteristics and temporal synchronization analysis across recording conditions.
Figure 5. Participant diversity and recording conditions in the educational study. The study includes participants from diverse linguistic backgrounds, including East Asian (Mandarin L1), South Asian (Hindi L1), African (Yoruba/Swahili L1), and European (Spanish L1) speakers. Recordings were collected under multiple real world conditions, including studio microphones, laptop microphones, smartphone recordings, and classroom ambient recorders, reflecting a range of acoustic environments typical in educational settings. For privacy protection, all participant faces shown in this figure are anonymized and represented using illustrative avatars rather than real identities. Participant diversity and recording conditions in the educational study.
Figure 6. Survey evaluation results across eight instructional dimensions. The user study evaluates the proposed system across eight instructional dimensions: visualization interpret-ability, LSI usefulness, educational applicability, interface transparency, instructional clarity, feedback efficiency, user confidence, and adoption intent. Bars indicate the mean participant ratings on a 5-point Likert scale, while the red line shows the normalized score proportion. The dashed reference line corresponds to a score of 4.5, illustrating that most evaluation dimensions received consistently high ratings, indicating strong perceived usability and instructional value of the system. Survey evaluation results across eight instructional dimensions.
Recent advances in automatic pronunciation assessment (APA) have driven a shift from scripted to free-form assessment. However, existing methods often rely on multi-stage pipelines with forced alignment and separate feature extractors, limiting their practicality across response settings. In light of this, we propose OpenPA, a novel APA framework that leverages Whisper's cross-attention mechanism to jointly predict word- and utterance-level pronunciation scores without forced alignment. Through mixed-transcript adaptation and adaptive substitution weighting, OpenPA accommodates heterogeneous transcripts across closed- and open-response settings within a single model. Trained solely on speechocean762, OpenPA achieves the strongest overall performance among evaluated systems under in-domain closed and simulated open-response conditions, while maintaining its performance zero-shot on MultiPA. An additional cross-task evaluation on oral-cancer speech suggests its potential applicability beyond non-native speech.
Lip movements can match the timing of speech without matching the spoken sounds. We introduce PVSync, a unified model for audio-visual offset estimation and phoneme-level articulation scoring. PVSync combines window-level contrastive learning for synchronisation with a phoneme-level articulation objective that aligns audio and video embeddings of the same viseme class across clips. Visemes group phonemes with similar visible articulation. Viseme labels are derived automatically from forced-aligned transcripts, without manual annotations. On offset-corrected videos from 13 talking-head video generation models, PVSync matches human rankings of lip-sync quality more closely than LSE-C, achieving a Spearman correlation of 0.83 versus 0.34. On an automatically constructed benchmark from held-out speech, PVSync distinguishes viseme-matched from mismatched audio-visual pairs with an ROC AUC of 0.91. PVSync also outperforms SyncNet and MTD-VocaLiST in temporal offset recovery on held-out in-the-wild clips. Code and benchmark data will be released upon acceptance.
Training automated pronunciation assessment often relies on labeled learner errors or non-native corpora that are costly to collect. We propose a lightweight framework trained only on native speech resources, operating unsupervised or lightly calibrated with a small set of scored utterances. At inference, learner speech is discretized with an SSL encoder and a K-means codebook. A token language model trained on native sequences computes surprisal where higher surprisal indicates phonotactic deviation. We add a transcript-guided Text2DUnit--DTW module that predicts native token sequences from reference text and aligns them to acoustic tokens to derive error-sensitive features. Surprisal and alignment features are fused via simple regression. On SpeechOcean762, PCC improves from 0.60 to 0.66 with transcript guidance, near supervised baselines. Cross-dataset evaluation on L2-ARCTIC shows consistent gains.