X-MDD: LLM-Based Mispronunciation Detection and Diagnosis for Unseen Languages with Canonical References

Xintong Wang; Junchuan Zhao; Ye Wang

cs.SDOct 16, 2026

X-MDD: LLM-Based Mispronunciation Detection and Diagnosis for Unseen Languages with Canonical References

Authors: Xintong Wang, Junchuan Zhao, Ye Wang

Organizations: School of Computing, National University of Singapore

Abstract

Mispronunciation detection and diagnosis (MDD) provides phone-level feedback to language learners, but extending it to new languages typically requires collecting and annotating learner speech for each target language. This repeated annotation effort is costly and limits the language coverage of pronunciation feedback. We study whether MDD capabilities learned from annotated learner speech can be applied to languages unseen during MDD training when canonical pronunciation references are available. Our multilingual LLM-based model, X-MDD, uses canonical speech and phones as references to predict the phone sequence actually produced by a learner. It incorporates in-context demonstrations for inference-time adaptation and is further optimized with group relative policy optimization (GRPO) using detection F1 and phone error rate. We further analyze how canonical inputs affect prediction and find that (1) canonical speech and phones jointly contribute to actual-phone prediction, and (2) conflicting speech evidence reduces the tendency to copy canonical phones. Trained only on Mandarin and English, X-MDD achieves a detection F1 of 38.07 on Arabic Qur'anic recitation, compared with 44.14 for a supervised challenge baseline trained on Arabic. It also achieves F1 scores of 78.68 and 58.78 on Mandarin and English, respectively, demonstrating effective multilingual MDD within a single model.

Figures & tables

Figure 1: Overview of X-MDD , built on Qwen3-ASR-1.7B. Canonical speech is synthesized from canonical phones, using learner speech as the reference for voice cloning. Conditioned on these three inputs and optional in-context demonstrations, the model generates the learner’s actual phone sequence for MDD.

Model	Mandarin (ZH)					English (EN)
Model	PER $\downarrow$	P $\uparrow$	R $\uparrow$	F1 $\uparrow$	DER $\downarrow$	PER $\downarrow$	P $\uparrow$	R $\uparrow$	F1 $\uparrow$	DER $\downarrow$
Monolingual baselines
w2v-CTC ( Peng et al., 2021 )	23.04	74.17	78.58	76.31	40.39	15.97	55.19	48.15	51.43	27.82
Pitch-aware RNN-T ( Wang et al., 2024b )	28.08	68.33	37.98	48.83	49.20	16.56	51.63	49.25	50.41	29.98
ATP-LLM ( Wu et al., 2024 )	20.53	82.37	69.23	75.23	35.36	14.20	63.67	20.42	30.92	21.23
WHU-IASP ( Yang et al., 2026 )	23.00	73.51	78.59	75.97	39.21	14.96	55.42	52.75	54.05	27.85

Table 1: MDD performance on Mandarin (ZH) and English (EN). All values are percentages; P and R denote detection precision and recall. For PER, F1, and DER, the best and second-best results across all models are shown in bold and underline , respectively.

Setting	ZH			EN
Setting	PER $\downarrow$	F1 $\uparrow$	DER $\downarrow$	PER $\downarrow$	F1 $\uparrow$	DER $\downarrow$
No canonical reference
Base	17.75	79.49	31.67	15.64	54.89	25.39
+ ICL	17.86	80.21	32.38	15.39	55.10	24.93
+ GRPO	18.53	79.42	32.94	16.69	54.14	25.62
+ ICL + GRPO	18.32	80.35	32.98	16.60	55.82	24.26

Table 2: Effects of canonical references, ICL, and GRPO on seen languages (%). Base excludes ICL and GRPO; ICL uses four test-time demonstrations. Bold / underline mark the best/second-best F1 within each block and language.

Setting	PER $\downarrow$	F1 $\uparrow$	DER $\downarrow$
IqraEval baseline ∗	16.15	44.14	38.8
No canonical reference
Base	606.10	12.72	92.51
+ ICL + GRPO	84.64	18.80	58.93
Canonical phones
Base	10.46	17.40	46.11

Table 3: Results on unseen Arabic (%). Base excludes ICL and GRPO; ICL uses 128 test-time demonstrations. Bold / underline mark the best/second-best results for each metric, excluding the Arabic-trained baseline ( ∗ ).

$N$	PER $\downarrow$	P $\uparrow$	R $\uparrow$	F1 $\uparrow$	DER $\downarrow$
0	10.29	38.39	32.54	35.22	44.55
2	9.43	36.30	32.15	34.10	43.51
8	8.45	40.85	32.54	36.22	42.53
32	8.14	45.92	31.52	37.38	38.88
64	7.38	49.68	30.09	37.48	37.86
128	7.75	49.74	30.84	38.07	40.33

Table 4: Effect of the number of ICL demonstrations (

N

) on Arabic MDD. Metrics are percentages; P/R denote precision/recall. Bold / underline mark the best/second-best results, including ties.

Figure 2: Mean attention shares across the three inputs on the Arabic test set. Shading marks layers 19–28, where canonical phones receive over 50%. The vertical guide marks layer 5 (Figure 3 ).

Figure 3: Layer-5 decoder attention of X-MDD for three Arabic utterances. Insets magnify the gray-boxed regions, with red boxes highlighting mispronounced output phones and red x-axis labels marking their canonical counterparts. Input segments are horizontally rescaled, with a separate color scale for each utterance.

Figure 4: Canonical phone intervention across three languages, showing how often the model follows the replaced phone at correctly pronounced and mispronounced positions. Error bars indicate 95% confidence intervals.

Figure 5: Logit-lens outcomes on Arabic substitution errors under teacher forcing. At layer 28, the actual-phone share increases from 16.0% (SFT) to 24.4% (GRPO), a gain of 8.5 percentage points.

Figure 6: SFT ranks of actual phones newly corrected by GRPO under teacher forcing. Solid/light bars denote new CD/TA, normalized within each language and group. The 2nd bin includes rank-1 entries from the original grouping.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Method	CP Input	CS Input	Multilingual
ERN ( Lee et al., 2016 )	✓	$\times$	$\triangle$
CNN-RNN-CTC ( Leung et al., 2019 )	$\times$	$\times$	$\triangle$
w2v-CTC ( Peng et al., 2021 )	$\times$	$\times$	$\triangle$
VietMDD ( Huu et al., 2023 )	✓	$\times$	$\triangle$
L1-MultiMDD ( Kheir et al., 2024 )	✓	$\times$	✓
Pitch-Aware RNN-T ( Wang et al., 2024b )	$\triangle$	$\times$	$\triangle$

Appendix

Table 5: Comparison with representative MDD systems. CP and CS denote canonical phones and canonical speech, respectively.

\checkmark

: demonstrated;

\triangle

: supported but not evaluated;

\times

: not supported.

Reference	Prediction
Reference	Correct	Incorrect
Correct	TA	FR
Incorrect	FA	$\mathrm{TR}=\mathrm{CD}+\mathrm{DE}$

Appendix

Table 7: MDD confusion matrix. True rejections (TR) comprise correct diagnoses (CD) and diagnosis errors (DE).

Setting	$N$	Mandarin (ZH)					English (EN)
Setting	$N$	PER $\downarrow$	P $\uparrow$	R $\uparrow$	F1 $\uparrow$	DER $\downarrow$	PER $\downarrow$	P $\uparrow$	R $\uparrow$	F1 $\uparrow$	DER $\downarrow$
No canonical reference (learner speech only)
Base	0	17.75	81.85	77.25	79.49	31.67	15.64	51.08	59.31	54.89	25.39
+ ICL	2	17.88	81.78	78.54	80.12	32.53	15.48	51.50	59.03	55.01	25.04
+ ICL	4	17.86	81.97	78.54	80.21	32.38	15.39	51.60	59.11	55.10	24.93
+ GRPO	0	18.53	80.22	78.64	79.42	32.94	16.69	48.69	60.96	54.14	25.62

Appendix

Table 8: Full ablation results on Mandarin (ZH) and English (EN), with

N

test-time demonstrations. All values are percentages; P/R denote detection precision/recall. Bold / underline mark the best/second-best results within each group and language, including ties.

Canonical references	Setting	$N$	PER $\downarrow$	P $\uparrow$	R $\uparrow$	F1 $\uparrow$	DER $\downarrow$
IqraEval baseline ∗		–	16.15	30.39	77.07	44.14	38.80
None (learner speech only)	Base	0	606.10	6.82	94.33	12.72	92.51
	+ ICL	128	143.33	9.89	77.21	17.54	62.77
	+ GRPO	0	371.74	7.27	88.91	13.44	87.29
	+ ICL + GRPO	128	84.64	10.76	74.29	18.80	58.93
Phones	Base	0	10.47	20.84	14.94	17.40	46.11

Appendix

Table 9: Full ablation results on unseen Arabic with

N

test-time demonstrations. All metrics are percentages; P/R denote detection precision/recall. Bold / underline mark the best/second-best results, excluding the Arabic-trained baseline ( ∗ ) and rows with PER

>100\%

Figure 7: Decoder attention at layers 5, 15, and 27 (columns) for three Arabic utterances (rows). Insets enlarge matched regions across layers. Red boxes highlight mispronounced positions, and red ticks mark their canonical phones. Input segments are horizontally rescaled, with a separate color scale for each panel.

Decoding	PER $\downarrow$	P $\uparrow$	R $\uparrow$	F1 $\uparrow$	DER $\downarrow$	OOV $\downarrow$
Canonical phones
Constrained	10.47	27.50	23.02	25.06	41.58	–
Unconstrained	10.96	25.68	23.98	24.80	43.78	1.02
Canonical speech + phones
Constrained	7.75	49.74	30.84	38.07	40.33	–
Unconstrained	7.82	49.62	31.49	38.53	42.05	0.12

Appendix

Table 10: Phone-constrained decoding on Arabic with ICL + GRPO and

N=128

demonstrations. All values are percentages; OOV denotes out-of-inventory symbols. Bold marks the better PER, F1, and DER within each group.

Demonstrations	Mandarin (ZH)					English (EN)					Arabic (AR)
Demonstrations	PER $\downarrow$	P $\uparrow$	R $\uparrow$	F1 $\uparrow$	DER $\downarrow$	PER $\downarrow$	P $\uparrow$	R $\uparrow$	F1 $\uparrow$	DER $\downarrow$	PER $\downarrow$	P $\uparrow$	R $\uparrow$	F1 $\uparrow$	DER $\downarrow$
Same language	17.56	82.36	76.29	79.21	31.23	12.97	59.81	57.62	58.69	20.13	8.61	37.55	30.81	33.85	43.08
3 same + 1 other	17.55	82.37	75.85	78.98	30.88	13.01	59.67	55.67	57.60	19.75	8.82	36.53	31.17	33.63	43.73
2 same + 2 other	17.61	82.70	75.36	78.86	31.37	12.85	60.72	54.74	57.58	20.18	10.38	34.47	31.37	32.85	44.87
1 same + 3 other	17.45	83.11	75.13	78.92	31.15	12.87	60.72	52.51	56.32	19.92	10.45	34.21	32.63	33.40	42.05
All-language mix	17.56	82.76	75.11	78.75	31.18	12.78	60.94	52.67	56.50	19.29	10.67	33.70	31.70	32.67	44.59

Appendix

Table 11: Demonstration-language ablation with inference-only ICL (

N=4

), using demonstrations from other speakers. Same/other refer to the query language; Other 1/2 denote EN/AR for ZH, ZH/AR for EN, and ZH/EN for AR. Mixed-language sampling is uniform subject to the stated counts. All metrics are percentages; bold / underline mark the best/second-best PER, F1, and DER within each query language.

Figure 8: Rates of copying replacement phones at intervention ratios of 10%, 50%, and 70%, grouped by original pronunciation status. Both speech inputs remain unchanged. Error bars indicate 95% confidence intervals.

Figure 9: Effect of same-speaker versus other-speaker demonstrations on Mandarin, English, and Arabic. We report

\Delta\mathrm{F1}=\mathrm{F1}_{\mathrm{same}}-\mathrm{F1}_{\mathrm{other}}

\Delta\mathrm{PER}=\mathrm{PER}_{\mathrm{other}}-\mathrm{PER}_{\mathrm{same}}

, and

\Delta\mathrm{DER}=\mathrm{DER}_{\mathrm{other}}-\mathrm{DER}_{\mathrm{same}}

, so that positive

\Delta

consistently indicates better performance with same-speaker demonstrations. All differences are in percentage points (pp).

Demonstrations	Mandarin (ZH)					English (EN)					Arabic (AR)
Demonstrations	PER $\downarrow$	P $\uparrow$	R $\uparrow$	F1 $\uparrow$	DER $\downarrow$	PER $\downarrow$	P $\uparrow$	R $\uparrow$	F1 $\uparrow$	DER $\downarrow$	PER $\downarrow$	P $\uparrow$	R $\uparrow$	F1 $\uparrow$	DER $\downarrow$
4 real	17.72	82.36	75.40	78.72	31.18	12.82	60.42	57.44	58.89	19.75	9.12	38.13	32.18	34.90	43.84
3 real + 1 synthetic	17.57	82.57	75.55	78.90	30.91	12.79	60.43	56.20	58.24	19.20	9.27	37.76	32.96	35.20	43.62
2 real + 2 synthetic	17.66	82.34	75.46	78.75	31.17	12.87	60.38	54.49	57.28	19.53	10.08	35.81	33.82	34.79	43.30
1 real + 3 synthetic	17.57	82.69	75.28	78.81	31.02	12.96	60.33	52.62	56.21	20.32	10.28	35.58	34.48	35.02	43.77
4 synthetic	17.76	82.59	74.28	78.22	31.19	13.03	60.10	51.00	55.17	20.27	9.67	34.99	35.25	35.12	43.15

Appendix

Table 12: Real versus synthetic demonstrations for inference-only ICL with X-MDD (

N=4

). Real demonstrations come from the query speaker. Synthetic demonstrations use synthesized canonical and learner speech: CosyVoice error augmentation for ZH/EN and Iqra_TTS for AR. All metrics are percentages; bold / underline mark the best/second-best PER, F1, and DER within each language, including ties.

Figure 10: Qualitative examples of substitutions recovered by GRPO on unseen Arabic and seen English. Base and ICL falsely accept the canonical phone, whereas ICL + GRPO predicts the annotated actual phone and correctly detects the substitution. Blue indicates phones matching the target, and orange indicates phones differing from the target.

Figure 11: Qualitative examples of prediction errors corrected by increasing the number of demonstrations on unseen Arabic and seen English. With more demonstrations, the model recovers the annotated actual phone sequence, which matches the canonical pronunciation in both examples.

Figure 12: Qualitative examples of substitutions recovered by adding canonical speech on unseen Arabic and seen English. With canonical phones alone, the model falsely accepts the canonical pronunciation, whereas adding canonical speech recovers the annotated actual phone and correctly detects the substitution.

Speech Processing

Speech processing develops computational methods for recognizing, synthesizing, analyzing, and transforming spoken audio.

See all 401 papers

Jun 4, 2026

Domain-Aware Mispronunciation Detection and Diagnosis Using Language-Specific Statistical Graphs

Mispronunciation Detection and Diagnosis (MDD) has gained increasing importance in computer-assisted language learning and speech technology in recent years. In this paper, we propose a method for constructing statistical graphs that enable models to learn phoneme confusion patterns represented as directed graphs. Furthermore, we introduce a language-specific strategy to capture systematic pronunciation differences across various native language (L1) backgrounds. The effectiveness of our approach is demonstrated through extensive experiments on the L2-ARCTIC benchmark, where it achieves an F1-score of 59.52%, outperforming several competitive baselines.

Huu Tuong Tu, Hanh Nguyen, Thien Van Luong +3

1Hanoi University of Science and Technology · 2VNPT AI, VNPT Group · 3National Economics University

Speech Processing

Ask PDF

Apr 24, 2026

Beyond Acoustic Sparsity and Linguistic Bias: A Prompt-Free Paradigm for Mispronunciation Detection and Diagnosis

Mispronunciation Detection and Diagnosis (MDD) requires modeling fine-grained acoustic deviations. However, current ASR-derived MDD systems often face inherent limitations. In particular, CTC-based models favor sequence-level alignments that neglect transient mispronunciation cues, while explicit canonical priors bias predictions toward intended targets. To address these bottlenecks, we propose a prompt-free framework decoupling acoustic fidelity from canonical guidance. First, we introduce CROTTC, an acoustic model enforcing monotonic, frame-level alignment to accurately capture pronunciation deviations. Second, we implicitly inject mispronunciation information via the IF strategy under the knowledge transfer principle. Experiments show CROTTC-IF achieves a 71.77% F1-score on L2-ARCTIC and 71.70% F1-score on the Iqra'Eval2 leaderboard. With empirical analysis, we demonstrate that decoupling acoustics from explicit priors provides highly robust MDD.

Haopeng Geng, Longfei Yang, Xi Chen +3

Graduate School of Engineering, The University of Tokyo, Japan

Speech Processing

Ask PDF

Jun 20, 2026

Using Phonological-Level Wav2Vec2 for Mandarin Automatic Mispronunciation Detection and Diagnosis

Automatic mispronunciation detection and diagnosis (MDD) plays a crucial role in L2 Mandarin pronunciation learning. While end-to-end (E2E) based MDD methods have substantially improved phoneme-level detection accuracy, diagnostic feedback remains limited, as segmental and tonal errors are not explicitly separated. In this paper, we propose a phonological feature-based MDD framework that models both segmental and tonal attributes within a unified Wav2Vec2 CTC architecture. Experimental results show that the proposed method reduces the False Acceptance Rate (FAR) by 10.1% and the Diagnostic Error Rate (DER) by 23.6% compared with the phoneme-only baseline system. By decomposing phonemes into low-level phonological components, the proposed approach enables more detailed and interpretable diagnostic feedback for L2 learners.

Jinghao Chen, Mostafa Shahin, Beena Ahmed

School of Electrical Engineering and Telecommunications, UNSW, Australia

Speech Processing

Ask PDF

Abstract

Speech Processing

Explore similar work

Domain-Aware Mispronunciation Detection and Diagnosis Using Language-Specific Statistical Graphs

Beyond Acoustic Sparsity and Linguistic Bias: A Prompt-Free Paradigm for Mispronunciation Detection and Diagnosis

Using Phonological-Level Wav2Vec2 for Mandarin Automatic Mispronunciation Detection and Diagnosis