This paper addresses fine-grained prosody control for large-language-model (LLM) based text-to-speech (TTS) synthesis of Mandarin Chinese, motivated by multimedia content-creation scenarios---film dubbing, audiobook production, and interactive voice interfaces---in which prosodic adjustments must be placed on specific characters rather than applied uniformly to whole utterances. We present a data-driven framework in which fine-grained prosodic markers (pitch, duration, energy, and pause) are automatically extracted from raw audio at scale and embedded into the text stream as out-of-vocabulary Unicode symbols. The LLM of an existing TTS system is then fine-tuned to interpret these markers while its architecture remains unchanged, yielding character-level temporal control that complements existing utterance-level style control. An optional text-LLM front-end can translate natural-language instructions into marker sequences. Experiments on 2.15M Mandarin utterances (5,858 hours) with CosyVoice2 as the backbone demonstrate reliable control of character duration, pitch, pause duration, and global speaking rate, with perceptual quality essentially preserved. A cross-tokenizer analysis further indicates that the ASR-optimized tokenizer of the backbone attenuates prosodic variation relative to reconstruction-oriented alternatives---an observation we report as hypothesis-generating evidence for prosody-sensitive system design.
Figures & tables
Figure 1. Inference-time view of the framework. At inference, marked text is authored directly or generated by the optional instruction front-end; the offline extraction pipeline (Figure 2 ) supplies the marked training data. The base model architecture is unchanged. Block diagram of the inference path: marked text enters the fine-tuned TTS language model, which outputs acoustic tokens decoded to speech.
Figure 2. Character-level prosody extraction. Forced alignment provides temporal boundaries for feature extraction and marker insertion. Flowchart of the extraction pipeline: audio and text are force aligned by two systems whose boundary estimates are fused; per-character pitch, duration, energy, and inter-phrase pause features are then computed and converted to Unicode markers inserted around characters.
Table 4. Pause controllability: target ranges vs. measured means (2,000 samples per marker).
Figure 3. Mean per-utterance character duration conditioned on the five global rate prompts (group means across speakers). Grouped bar or line chart of mean character duration for the five speaking-rate prompts, showing monotone separation of group means.
Attribute
Base
Ours
Naturalness
4.46
4.44
Realism
4.10
4.15
Speaker similarity
3.95
4.00
Table 5. Perceptual evaluation (mean opinion scores, 24 listeners; descriptive comparison on emotionally charged content; Base = unmodified CosyVoice2).
Figure 4. Tokenizer sensitivity to acoustic perturbations (NED between original and perturbed token sequences; higher = more sensitive). CosyVoice2 (ASR-optimized) is least sensitive overall; reconstruction-oriented tokenizers are more sensitive, with DualCodec lying in between, suggesting a continuum rather than a strict dichotomy. Bar chart of normalized edit distance between original and perturbed token sequences for four tokenizers across volume, F0, and noise perturbations.
Personalized text-to-speech (TTS) aims to clone the target speaker in the synthesized speech, imitating both the voice and speaking style. Current large language model (LLM)-based TTS methods ignore the style-specific prosodic patterns in generated speech, resulting in deficient style learning and thus limiting speaker similarity in synthesized speech. To this end, we investigate the prosody learning conditioned on the synthesized speech, and propose to predict the prosody of the current syllable based on previously predicted speech. Experimental results obtained on three datasets demonstrated the efficacy of the proposed dynamic prosody prediction method in enhancing the prosody learning capability, thereby improving the speaker similarity of the generated speech. Audio samples are available at https://muzw.github.io/dynapros/.
Zhenwei Mou, Liping Chen, Yajun Hu +3
University of Science and Technology of China, Anhui, China · iFLYTEK, Anhui, China
While recent Large Language Model (LLM)-based Text-to-Speech (TTS) systems have achieved remarkable naturalness, they predominantly rely on implicit end-to-end generation paradigms, resulting in coarse-grained control. In scenarios demanding precise stylistic interventions and strict temporal alignment, such as audiobook narration and video dubbing, the inability to explicitly manipulate word-level acoustic attributes remains a critical bottleneck. This limitation is primarily amplified by the severe scarcity of fine-grained annotated datasets and the architectural challenge of integrating multi-dimensional control signals into discrete autoregressive generation. To address this, we propose a unified framework for highly precise word-level control. First, we construct WordVoice-5A, a massive 4.7k-hour bilingual dataset featuring five-dimensional word-level annotations (duration, boundary, energy, pitch and tone) developed through a rigorous linguistically-guided pipeline. Second, we introduce WordVoice to transform the implicit generation process into an explicit, highly controllable paradigm. Specifically, we introduce a bound-token mechanism within the LLM to formulate an explicit ``acoustic planning'' process, enabling adaptive multi-task prosodic planning and flexible manual intervention. Furthermore, we augment the token-to-waveform stage with a fine-grained acoustic modulation module, bridging the resolution gap to strictly align word-level attributes between highly compressed discrete tokens and continuous waveforms. Extensive experiments demonstrate that WordVoice achieves superior, decoupled control over multiple acoustic dimensions while maintaining competitive zero-shot synthesis stability. The code and audio samples are publicly available at https://xxh333.github.io/wordvoice-demo/.
Sihang Nie, Jinxin Ji, Xiaofen Xing +4
South China University of Technology · Huya Inc. · Tongji University +2
We propose UtterTune, a lightweight method for adapting a multilingual text-to-speech (TTS) system built on a large language model (LLM). It improves control of pronunciation in the target language while preserving performance in the others. Although LLM architectures have enabled TTS models to achieve remarkable naturalness, accurately modeling grapheme-to-phoneme (G2P) mapping and prosody remains challenging, especially when the model omits an explicit G2P module and directly processes minimally encoded text (e.g., byte-pair encoding). UtterTune leverages low-rank adaptation to enable the control of segmental pronunciation and pitch accent at the phoneme level for Japanese speech, the target language in this paper, while maintaining naturalness and speaker similarity in a zero-shot setting. Objective and subjective evaluations confirm its effectiveness.