Accurate and efficient pronunciation transcription is essential for preparing text-to-speech training data at scale. Existing approaches have different limitations: grapheme-to-pronunciation (G2P) and speech-to-pronunciation (S2P) methods each capture only partial information, using only text or only speech, while speech-and-text-to-pronunciation (ST2P) methods use both but require costly pronunciation-annotated data. To address this problem, we propose a training-free ST2P pipeline that integrates both lexical and acoustic information at inference time. Lexical resources and G2P tools generate text-constrained candidates, and a left-to-right greedy search selects the best one using whole-sequence negative log-likelihoods from frozen pretrained S2P models. On three Japanese corpora, our method reduces Character Error Rate (CER) from 0.60--1.40% (text-only baseline) to 0.04--0.17% with reference transcripts, and 0.64--1.58% with ASR transcripts. It outperforms all baselines, including a trained ST2P model and commercial multimodal LLMs. Our greedy search method is 3--3.5× faster than beam search at similar CER, and the cascade is 2× faster than direct decoding ensuring the efficiency and accuracy. In Spanish, French, and preliminary English, it also surpasses four open multimodal LLMs and the best traditional methods.
Figures & tables
Figure 1: Processing cost versus pronunciation error rates on JVS-dev. The horizontal axis is the real-time factor (log scale; lower is faster) and the vertical axis is character error rate (CER) (log scale; lower is better). The proposed methods (ours) achieves the lowest CER at low computational cost.
Figure 2: Training-free pronunciation transcription. (a) Dictionaries, G2P, and rules generate readings for transcript spans. Complete pronunciations y(r) are converted to model-specific tokens and evaluated by a frozen S2P model given speech x , combining lexical constraints and acoustic evidence through negative log-likelihood (NLL). (b) Left-to-right greedy search for “read the record” ( P=0 ; illustrative NLLs). Each node represents a complete pronunciation with earlier choices fixed and later spans at their defaults. The lowest-NLL candidate (green) is committed; gray nodes are rejected, and dashed outlines mark span defaults. The final node is the search output; cascade rescoring and the subsequent margin check are not shown.
Reference text
ASR text
System
dev
JSUT
par
dev
JSUT
par
Commercial models (reference only): audio + text
Gemini 2.5 Flash
0.62
0.84
1.29
0.98
1.14
1.95
Gemini 3 Flash
0.64
0.52
0.74
0.88
0.68
1.72
Gemini 3.5 Flash
0.50
0.59
0.84
0.82
0.69
1.67
Gemini 3.6 Flash
1.19
1.69
1.17
1.95
2.55
1.87
Table 1: Japanese CER (%, ↓ ) on 512 common utterances per corpus. dev/par: JVS-dev/JVS-par. † : dictionary-constrained decoding inspired by [ 11 ] . Furigana Whisper’s JSUT cells are omitted (training data). Gray: commercial references. Bold: column minima excluding commercial models.
Configuration
JVS-dev
JSUT
JVS-par
Components
w/o gate
0.154
0.193
0.054
w/o cascade
0.161
0.182
0.058
w/ beam
0.167
0.160
0.042
Audio substitution
Proposed w/ other audio
15.329
16.192
12.440
Table 2: Japanese ablations (CER %, reference text, 512 utterances). Proposed: cascade, gate ( τ=0.6 ), greedy search. w/o gate: τ=0 ; w/o cascade: fusion every utterance; w/ beam: B=5 . Other/silent audio: use unrelated or silent audio, keeping text/candidates.
System
Spanish
French
English
n=200
n=192
n=200
Commercial models (reference only): audio + text
Gemini 2.5 Flash
2.23
4.41
13.84
Gemini 3 Flash
2.13
2.98
13.05
Gemini 3.5 Flash
2.01
4.05
13.85
Gemini 3.6 Flash
2.53
3.82
13.70
Table 3: PFER (%, ↓ ) on shared subsets with candidate variation. (A): reference text only; others: text and audio. Gray: commercial references. Bold: best (non-commercial).
Mispronunciation Detection and Diagnosis (MDD) requires modeling fine-grained acoustic deviations. However, current ASR-derived MDD systems often face inherent limitations. In particular, CTC-based models favor sequence-level alignments that neglect transient mispronunciation cues, while explicit canonical priors bias predictions toward intended targets. To address these bottlenecks, we propose a prompt-free framework decoupling acoustic fidelity from canonical guidance. First, we introduce CROTTC, an acoustic model enforcing monotonic, frame-level alignment to accurately capture pronunciation deviations. Second, we implicitly inject mispronunciation information via the IF strategy under the knowledge transfer principle. Experiments show CROTTC-IF achieves a 71.77% F1-score on L2-ARCTIC and 71.70% F1-score on the Iqra'Eval2 leaderboard. With empirical analysis, we demonstrate that decoupling acoustics from explicit priors provides highly robust MDD.
Haopeng Geng, Longfei Yang, Xi Chen +3
Graduate School of Engineering, The University of Tokyo, Japan
Training automated pronunciation assessment often relies on labeled learner errors or non-native corpora that are costly to collect. We propose a lightweight framework trained only on native speech resources, operating unsupervised or lightly calibrated with a small set of scored utterances. At inference, learner speech is discretized with an SSL encoder and a K-means codebook. A token language model trained on native sequences computes surprisal where higher surprisal indicates phonotactic deviation. We add a transcript-guided Text2DUnit--DTW module that predicts native token sequences from reference text and aligns them to acoustic tokens to derive error-sensitive features. Surprisal and alignment features are fused via simple regression. On SpeechOcean762, PCC improves from 0.60 to 0.66 with transcript guidance, near supervised baselines. Cross-dataset evaluation on L2-ARCTIC shows consistent gains.
We present GRAFT, a per-word pronunciation conditioning mechanism for text-to-speech neural codec language modeling. Existing systems reach high intelligibility and naturalness but inherit the ambiguity of text and mispronounce rare proper nouns, loanwords and technical terms. Even phoneme-conditioned models offer no direct acoustic handle for per-word pronunciation. GRAFT controls the pronunciation of a chosen word from a short spoken sample of it, encoded with the model's own speech tokenizer and bound to the word's position in the prompt. Voice conversion during training-data construction disentangles the hint speaker from the target speaker, so the hint may come from any voice while the output stays in the target voice. In a blind English listening study, human raters rank GRAFT first by a clear margin, judging its rendering of the difficult word closest to a reference recording of that word. On a five-language objective benchmark, GRAFT reduces target-word phoneme error rate by 22-39% over the identical text-only backbone and outperforms competitive open-source zero-shot systems, both phoneme- and text-conditioned, on target-word pronunciation, while preserving speaker similarity and naturalness.
Antonis Asonitis, Francesco Verdini, Aref Farhadipour +4