Voice agents built as cascaded automatic speech recognition (ASR)->large language model (LLM)-> text-to-speech (TTS) pipelines increasingly rely on conversational TTS: streaming, context conditioned models that adapt delivery to dialogue history. Such models are natural and expressive but offer little explicit control over how rare, trust-critical drug names are pronounced, because these entities are too infrequent, and their vocabulary too fast-growing, to be covered by any amount of pre-, mid-, or post-training. We retrofit phoneme controllability onto a pretrained conversational TTS model without any pre/mid-training, using an ARPAbet vocabulary exten-sion, continued phoneme-augmented supervised fine-tuning (SFT), and a training-free inference-time acoustic-biasing step. On a 7,500 clip clinical drug-pronunciation evaluation, our model raises strict Gemini-judged correctness from 54.6% (no phoneme augmentation) to 90.7%, outperforming every conversational and stateless TTS baseline on the automatic metric. Because no training stage sees a drug name, the entire evaluation is out-of-domain, so the gains reflect learned pronunciation rather than memorization. In a blinded, randomized human study, adult listeners judge 84% of our clips correct, versus 82% for the strongest baseline, with overlapping confidence intervals on a stratified subset. Across all 250 matched clips, Gemini agrees with the human majority on 84.8% of binary labels (k{appa}=0.57).
Figures & tables
Figure 1: Overview of phoneme-aware continued SFT. (1) We assemble a multi-corpus speech mixture. (2) Stochastic augmentation retains a word’s graphemes with probability 1−p or replaces them with ARPAbet special tokens with probability p (illustrated for Mounjaro ; illustrative only, as no drug name appears in SFT). (3) Grapheme or phoneme tokens are interleaved with 16-codebook acoustic frames. (4) The FireRedTTS-2 backbone predicts the first acoustic codebook, and its depth decoder predicts the remaining codebooks. (5) Starting from the pretrained checkpoint, continued SFT jointly optimizes backbone, depth-decoder, and text losses.
Model
#Params
Drug-pron ↑
PER ↓
UTMOSv2 ↑
WER ↓
SIM ↑
phon_0
1.7B
54.6 ± 0.3
25.61
3.49
2.01
67.1
phon_20
1.7B
86.7 ± 0.7
22.39
3.48
1.81
67.2
phon_40
1.7B
90.9 ± 1.0
21.72
3.48
1.87
67.1
phon_60 (ours)
1.7B
90.7 ± 0.6
22.20
3.49
1.80
67.2
phon_80
1.7B
91.5 ± 0.3
21.41
3.49
1.92
66.6
phon_100
1.7B
91.8 ± 0.4
21.13
3.50
1.81
65.6
Table 1: Pronunciation, acoustic quality, and Seed-TTS-Eval results. Baselines are grouped into conversational (context-conditioned) and stateless (single-utterance) TTS. #Params is total published/inferred parameter count (backbone + decoder where applicable; — denotes an undisclosed proprietary API). Drug-pron is Gemini-judged correctness (%), mean ± 5-run σ ; PER is Wav2Vec2 entity-span phoneme error rate (%); phon_0 and FireRedTTS-2 base receive plain graphemes; the remaining FireRed variants receive ARPAbet and FireRed-specific reference biasing. CosyVoice3 receives CMU-ARPAbet; ElevenLabs and Inworld rows marked IPA receive inline IPA; rows marked plain, VoxCPM2, and MOSS receive graphemes. External systems receive no FireRed biasing. Baseline WER/SIM values are reported Seed-TTS-Eval Test-EN results from [ 32 ] ; values for our variants use the same evaluation protocol. Best value per column in bold .
Table 4: Adult-listener drug-pronunciation correctness (%) on a subset of drug_ref , stratified across systems and Gemini labels ( 50 clips per system; three or four ratings per clip). A clip is correct when at least two of three or three of four listeners assign correct . Gemini scores the same clips. Brackets give 95% Wilson CIs.