This paper addresses fine-grained prosody control for large-language-model (LLM) based text-to-speech (TTS) synthesis of Mandarin Chinese, motivated by multimedia content-creation scenarios---film dubbing, audiobook production, and interactive voice interfaces---in which prosodic adjustments must be placed on specific characters rather than applied uniformly to whole utterances. We present a data-driven framework in which fine-grained prosodic markers (pitch, duration, energy, and pause) are automatically extracted from raw audio at scale and embedded into the text stream as out-of-vocabulary Unicode symbols. The LLM of an existing TTS system is then fine-tuned to interpret these markers while its architecture remains unchanged, yielding character-level temporal control that complements existing utterance-level style control. An optional text-LLM front-end can translate natural-language instructions into marker sequences. Experiments on 2.15M Mandarin utterances (5,858 hours) with CosyVoice2 as the backbone demonstrate reliable control of character duration, pitch, pause duration, and global speaking rate, with perceptual quality essentially preserved. A cross-tokenizer analysis further indicates that the ASR-optimized tokenizer of the backbone attenuates prosodic variation relative to reconstruction-oriented alternatives---an observation we report as hypothesis-generating evidence for prosody-sensitive system design.
Figures & tables
Figure 1. Inference-time view of the framework. At inference, marked text is authored directly or generated by the optional instruction front-end; the offline extraction pipeline (Figure 2 ) supplies the marked training data. The base model architecture is unchanged. Block diagram of the inference path: marked text enters the fine-tuned TTS language model, which outputs acoustic tokens decoded to speech.
Figure 2. Character-level prosody extraction. Forced alignment provides temporal boundaries for feature extraction and marker insertion. Flowchart of the extraction pipeline: audio and text are force aligned by two systems whose boundary estimates are fused; per-character pitch, duration, energy, and inter-phrase pause features are then computed and converted to Unicode markers inserted around characters.
Table 4. Pause controllability: target ranges vs. measured means (2,000 samples per marker).
Figure 3. Mean per-utterance character duration conditioned on the five global rate prompts (group means across speakers). Grouped bar or line chart of mean character duration for the five speaking-rate prompts, showing monotone separation of group means.
Attribute
Base
Ours
Naturalness
4.46
4.44
Realism
4.10
4.15
Speaker similarity
3.95
4.00
Table 5. Perceptual evaluation (mean opinion scores, 24 listeners; descriptive comparison on emotionally charged content; Base = unmodified CosyVoice2).
Figure 4. Tokenizer sensitivity to acoustic perturbations (NED between original and perturbed token sequences; higher = more sensitive). CosyVoice2 (ASR-optimized) is least sensitive overall; reconstruction-oriented tokenizers are more sensitive, with DualCodec lying in between, suggesting a continuum rather than a strict dichotomy. Bar chart of normalized edit distance between original and perturbed token sequences for four tokenizers across volume, F0, and noise perturbations.