Pronunciation assessment requires acoustic evidence that is temporally precise, diagnostically meaningful, and faithful to the learner's actual production. However, existing acoustic models often struggle to provide recognition and segmentation evidence simultaneously. CTC-based phone recognizers can predict phone sequences flexibly, but their sparse and peaky posteriors often miss phone boundaries and fine-grained pronunciation cues. In contrast, text-dependent forced aligners provide reliable temporal information when transcripts are available, but are not directly applicable to reference-free pronunciation analysis. In this work, we propose a topology-aware frame-wise acoustic model that learns dense ordered state posteriors within each phone. The key idea is to recover phone-internal state structure in a neural acoustic model by combining ordered subphonetic states with optimal temporal transport classification (OTTC). This combination encourages dense monotonic frame-level state discrimination while preserving phone recognition ability. Experiments on read, spontaneous, and L2 speech show improved segmentation over neural baselines with competitive recognition performance. Downstream evaluations further show gains in mispronunciation detection and automatic pronunciation assessment. Probing analysis suggests that the learned states capture phoneme-dependent acoustic structure rather than arbitrary frame-level distributions.
Figures & tables
Fig. 1 : Illustration of proposed subphonetic model.
Fig. 2 : Illustration of subphonetic topologies.
Fig. 3 : Comparison of frame-level probability distributions across training criteria and topologies. Different topology states are shown in different colors. CTC produces sparse spikes, while OTTC produces a dense and complete subphonetic trace.
TABLE I : Datasets for training, validation, and evaluation.
Model
Segmentation
Recognition / Acoustic Fidelity
Text-dependent FA
Text-independent
PER ↓
MDD F1 ↑
TIMIT Dev
TIMIT Test
R-value ↑
TIMIT
Buckeye
L2-ARCTIC
SO762
L2-ARCTIC
SO762
TSE ↓
R-value ↑
TSE ↓
R-value ↑
TIMIT Test
Buckeye
L2-ARCTIC
Dev.
Test
Cano.
Perc.
Cano.
Perc.
Kaldi TDNN-F † [ 31 ]
45.39
63.59
45.41
63.53
-
-
-
20.91
24.30
34.36
25.05
27.27
54.73
54.44
0.474
0.156
ZIPA ‡ [ 32 ]
79.21
55.16
78.93
56.51
55.23
53.71
53.72
23.32
26.42
32.62
12.03
19.79
44.47
44.11
0.377
0.147
Charsiu FC–20ms [ 18 ]
40.72
76.55
40.36
76.38
76.28
70.02
75.99
22.04
24.17
31.83
14.68
21.73
43.93
43.49
0.431
0.188
TABLE II : Baselines and proposed acoustic model’s segmentation and recognition performance. Cano. and Perc. denote the canonical and perceived phone labels used in L2 corpora, respectively.
TABLE III : Ablation on L2-ARCTIC. All models were trained from scratch on the L2-ARCTIC training set, following common practice for MDD. Notably, the CTC conditioning head in Fig. 1 was not applied in this ablation.
Model
Phone
Word PCC
Utt. PCC
MSE ↓
PCC ↑
Acc. ↑
Str. ↑
Tot. ↑
Acc. ↑
Flu. ↑
Pro. ↑
Tot. ↑
Kaldi TDNN-F [ 5 ]
0.085
0.612
0.533
0.291
0.549
0.714
0.753
0.760
0.742
w/ Charsiu’s LogP
0.126
0.293
0.308
0.094
0.317
0.585
0.661
0.660
0.614
w/ Ours LogP
0.130
0.272
0.297
0.040
0.306
0.580
0.626
0.635
0.596
w/ Charsiu’s TS
0.124
0.300
0.306
0.189
0.312
0.517
0.596
0.583
0.536
w/ Ours TS
0.123
0.302
0.307
0.277
0.310
0.513
0.592
0.580
0.531
TABLE IV : Comparison with representative APA systems GOPT [ 5 ] and ConPCO [ 41 ] . Indented rows denote controlled variants that use the same scoring pipeline but replace either the frame-level log posteriors or the phone-level timestamps. Acc., Str., Flu., Pro., and Tot. denote accuracy, stress, fluency, prosody, and total score, respectively.
Fig. 4 : CTC peaks as acoustic landmarks for probing subphonetic states
Fig. 5 : Subphonetic topology and CTC-probe analysis for pronunciation modeling.
Training automated pronunciation assessment often relies on labeled learner errors or non-native corpora that are costly to collect. We propose a lightweight framework trained only on native speech resources, operating unsupervised or lightly calibrated with a small set of scored utterances. At inference, learner speech is discretized with an SSL encoder and a K-means codebook. A token language model trained on native sequences computes surprisal where higher surprisal indicates phonotactic deviation. We add a transcript-guided Text2DUnit--DTW module that predicts native token sequences from reference text and aligns them to acoustic tokens to derive error-sensitive features. Surprisal and alignment features are fused via simple regression. On SpeechOcean762, PCC improves from 0.60 to 0.66 with transcript guidance, near supervised baselines. Cross-dataset evaluation on L2-ARCTIC shows consistent gains.
In speech evaluation, an Automatic Speech Recognition (ASR) model often computes time boundaries and phoneme posteriors for input features. However, limited data for ASR training hinders expansion of speech evaluation to low-resource languages. Open-source weakly-supervised models are capable of ASR over many languages, but they are frame-asynchronous and not phonemic, hindering feature extraction for speech evaluation. This paper proposes to overcome incompatibilities for feature extraction with weakly-supervised models, easing expansion of speech evaluation to low-resource languages. Phoneme posteriors are computed by mapping ASR hypotheses to a phoneme confusion network. Word instead of phoneme-level speaking rate and duration are used. Phoneme and frame-level features are combined using a cross-attention architecture, obviating phoneme time alignment. This performs comparably with standard frame-synchronous features on English speechocean762 and low-resource Tamil datasets.
Jeremy H. M. Wong, Nancy F. Chen
Institute for Infocomm Research (I2R), A⋆STAR, Singapore
Mispronunciation Detection and Diagnosis (MDD) requires modeling fine-grained acoustic deviations. However, current ASR-derived MDD systems often face inherent limitations. In particular, CTC-based models favor sequence-level alignments that neglect transient mispronunciation cues, while explicit canonical priors bias predictions toward intended targets. To address these bottlenecks, we propose a prompt-free framework decoupling acoustic fidelity from canonical guidance. First, we introduce CROTTC, an acoustic model enforcing monotonic, frame-level alignment to accurately capture pronunciation deviations. Second, we implicitly inject mispronunciation information via the IF strategy under the knowledge transfer principle. Experiments show CROTTC-IF achieves a 71.77% F1-score on L2-ARCTIC and 71.70% F1-score on the Iqra'Eval2 leaderboard. With empirical analysis, we demonstrate that decoupling acoustics from explicit priors provides highly robust MDD.
Haopeng Geng, Longfei Yang, Xi Chen +3
Graduate School of Engineering, The University of Tokyo, Japan