Voice agents built as cascaded automatic speech recognition (ASR)->large language model (LLM)-> text-to-speech (TTS) pipelines increasingly rely on conversational TTS: streaming, context conditioned models that adapt delivery to dialogue history. Such models are natural and expressive but offer little explicit control over how rare, trust-critical drug names are pronounced, because these entities are too infrequent, and their vocabulary too fast-growing, to be covered by any amount of pre-, mid-, or post-training. We retrofit phoneme controllability onto a pretrained conversational TTS model without any pre/mid-training, using an ARPAbet vocabulary exten-sion, continued phoneme-augmented supervised fine-tuning (SFT), and a training-free inference-time acoustic-biasing step. On a 7,500 clip clinical drug-pronunciation evaluation, our model raises strict Gemini-judged correctness from 54.6% (no phoneme augmentation) to 90.7%, outperforming every conversational and stateless TTS baseline on the automatic metric. Because no training stage sees a drug name, the entire evaluation is out-of-domain, so the gains reflect learned pronunciation rather than memorization. In a blinded, randomized human study, adult listeners judge 84% of our clips correct, versus 82% for the strongest baseline, with overlapping confidence intervals on a stratified subset. Across all 250 matched clips, Gemini agrees with the human majority on 84.8% of binary labels (k{appa}=0.57).
Figures & tables
Figure 1: Overview of phoneme-aware continued SFT. (1) We assemble a multi-corpus speech mixture. (2) Stochastic augmentation retains a word’s graphemes with probability 1−p or replaces them with ARPAbet special tokens with probability p (illustrated for Mounjaro ; illustrative only, as no drug name appears in SFT). (3) Grapheme or phoneme tokens are interleaved with 16-codebook acoustic frames. (4) The FireRedTTS-2 backbone predicts the first acoustic codebook, and its depth decoder predicts the remaining codebooks. (5) Starting from the pretrained checkpoint, continued SFT jointly optimizes backbone, depth-decoder, and text losses.
Model
#Params
Drug-pron ↑
PER ↓
UTMOSv2 ↑
WER ↓
SIM ↑
phon_0
1.7B
54.6 ± 0.3
25.61
3.49
2.01
67.1
phon_20
1.7B
86.7 ± 0.7
22.39
3.48
1.81
67.2
phon_40
1.7B
90.9 ± 1.0
21.72
3.48
1.87
67.1
phon_60 (ours)
1.7B
90.7 ± 0.6
22.20
3.49
1.80
67.2
phon_80
1.7B
91.5 ± 0.3
21.41
3.49
1.92
66.6
phon_100
1.7B
91.8 ± 0.4
21.13
3.50
1.81
65.6
Table 1: Pronunciation, acoustic quality, and Seed-TTS-Eval results. Baselines are grouped into conversational (context-conditioned) and stateless (single-utterance) TTS. #Params is total published/inferred parameter count (backbone + decoder where applicable; — denotes an undisclosed proprietary API). Drug-pron is Gemini-judged correctness (%), mean ± 5-run σ ; PER is Wav2Vec2 entity-span phoneme error rate (%); phon_0 and FireRedTTS-2 base receive plain graphemes; the remaining FireRed variants receive ARPAbet and FireRed-specific reference biasing. CosyVoice3 receives CMU-ARPAbet; ElevenLabs and Inworld rows marked IPA receive inline IPA; rows marked plain, VoxCPM2, and MOSS receive graphemes. External systems receive no FireRed biasing. Baseline WER/SIM values are reported Seed-TTS-Eval Test-EN results from [ 32 ] ; values for our variants use the same evaluation protocol. Best value per column in bold .
Table 4: Adult-listener drug-pronunciation correctness (%) on a subset of drug_ref , stratified across systems and Gemini labels ( 50 clips per system; three or four ratings per clip). A clip is correct when at least two of three or three of four listeners assign correct . Gemini scores the same clips. Brackets give 95% Wilson CIs.
We present GRAFT, a per-word pronunciation conditioning mechanism for text-to-speech neural codec language modeling. Existing systems reach high intelligibility and naturalness but inherit the ambiguity of text and mispronounce rare proper nouns, loanwords and technical terms. Even phoneme-conditioned models offer no direct acoustic handle for per-word pronunciation. GRAFT controls the pronunciation of a chosen word from a short spoken sample of it, encoded with the model's own speech tokenizer and bound to the word's position in the prompt. Voice conversion during training-data construction disentangles the hint speaker from the target speaker, so the hint may come from any voice while the output stays in the target voice. In a blind English listening study, human raters rank GRAFT first by a clear margin, judging its rendering of the difficult word closest to a reference recording of that word. On a five-language objective benchmark, GRAFT reduces target-word phoneme error rate by 22-39% over the identical text-only backbone and outperforms competitive open-source zero-shot systems, both phoneme- and text-conditioned, on target-word pronunciation, while preserving speaker similarity and naturalness.
Antonis Asonitis, Francesco Verdini, Aref Farhadipour +4
Transfer learning is widely used for low-resource text-to-speech. When the target corpus contains phonemes unseen in pre-training, the model must expand its phoneme inventory during fine-tuning; we call the process "phoneme addition." However, it remains unclear whether the pre-trained ability to generate seen phonemes contributes to this process. This study investigates phoneme addition in two settings: (1) a simulation setup using LLM-generated phoneme-controlled corpora that enables investigation without considering confounding factors, and (2) a real-speech cross-lingual transfer setup (English to Japanese) to validate whether the findings hold in practice. Experiments in both settings showed that while fine-tuning achieved higher naturalness than training from scratch, it required as much or more data to achieve comparable PER for new phonemes. These results indicate that pre-training mainly contributes to naturalness improvement, but offers limited benefit for phoneme addition.
We propose UtterTune, a lightweight method for adapting a multilingual text-to-speech (TTS) system built on a large language model (LLM). It improves control of pronunciation in the target language while preserving performance in the others. Although LLM architectures have enabled TTS models to achieve remarkable naturalness, accurately modeling grapheme-to-phoneme (G2P) mapping and prosody remains challenging, especially when the model omits an explicit G2P module and directly processes minimally encoded text (e.g., byte-pair encoding). UtterTune leverages low-rank adaptation to enable the control of segmental pronunciation and pitch accent at the phoneme level for Japanese speech, the target language in this paper, while maintaining naturalness and speaker similarity in a zero-shot setting. Objective and subjective evaluations confirm its effectiveness.