Training-Free Pronunciation Transcription via Text-Constrained Acoustic Rescoring
Organizations: The University of Tokyo · Sakana AI
Abstract
Accurate and efficient pronunciation transcription is essential for preparing text-to-speech training data at scale. Existing approaches have different limitations: grapheme-to-pronunciation (G2P) and speech-to-pronunciation (S2P) methods each capture only partial information, using only text or only speech, while speech-and-text-to-pronunciation (ST2P) methods use both but require costly pronunciation-annotated data. To address this problem, we propose a training-free ST2P pipeline that integrates both lexical and acoustic information at inference time. Lexical resources and G2P tools generate text-constrained candidates, and a left-to-right greedy search selects the best one using whole-sequence negative log-likelihoods from frozen pretrained S2P models. On three Japanese corpora, our method reduces Character Error Rate (CER) from 0.60--1.40% (text-only baseline) to 0.04--0.17% with reference transcripts, and 0.64--1.58% with ASR transcripts. It outperforms all baselines, including a trained ST2P model and commercial multimodal LLMs. Our greedy search method is 3--3.5 faster than beam search at similar CER, and the cascade is 2 faster than direct decoding ensuring the efficiency and accuracy. In Spanish, French, and preliminary English, it also surpasses four open multimodal LLMs and the best traditional methods.
Figures & tables
| Reference text | ASR text | |||||
| System | dev | JSUT | par | dev | JSUT | par |
| Commercial models (reference only): audio + text | ||||||
| Gemini 2.5 Flash | 0.62 | 0.84 | 1.29 | 0.98 | 1.14 | 1.95 |
| Gemini 3 Flash | 0.64 | 0.52 | 0.74 | 0.88 | 0.68 | 1.72 |
| Gemini 3.5 Flash | 0.50 | 0.59 | 0.84 | 0.82 | 0.69 | 1.67 |
| Gemini 3.6 Flash | 1.19 | 1.69 | 1.17 | 1.95 | 2.55 | 1.87 |
| Configuration | JVS-dev | JSUT | JVS-par |
|---|---|---|---|
| Components | |||
| w/o gate | 0.154 | 0.193 | 0.054 |
| w/o cascade | 0.161 | 0.182 | 0.058 |
| w/ beam | 0.167 | 0.160 | 0.042 |
| Audio substitution | |||
| Proposed w/ other audio | 15.329 | 16.192 | 12.440 |
| System | Spanish | French | English |
|---|---|---|---|
| Commercial models (reference only): audio + text | |||
| Gemini 2.5 Flash | 2.23 | 4.41 | 13.84 |
| Gemini 3 Flash | 2.13 | 2.98 | 13.05 |
| Gemini 3.5 Flash | 2.01 | 4.05 | 13.85 |
| Gemini 3.6 Flash | 2.53 | 3.82 | 13.70 |
Speech Processing
Speech processing develops computational methods for recognizing, synthesizing, analyzing, and transforming spoken audio.
See all 413 papers