Diffusion-based TTS has demonstrated strong capability in emotional speech generation. However, existing emotion alignment methods primarily reinforce target emotional expressions while overlooking modeling of competing emotional modes, limiting fine-grained controllability. This issue becomes more pronounced under classifier-free guidance (CFG), which relies on contrasting conditional trajectories to shape generation. We propose Emotion Bidirectional Preference Optimization (Emo-BPO), a simple effective framework that jointly learns emotion-aligned and emotion-contrastive score functions from reordered same-text emotional pairs. Our method requires neither auxiliary reward models nor additional annotations, and can be seamlessly integrated into existing diffusion-based TTS systems to improve emotional controllability and expressiveness.
Figures & tables
Models
Emo SIM
Prosody SIM
Intelligibility
UTMOS
Speech Emotion Recognition
Neutral
Angry
Happy
Sad
Surprise
Mean
emospeech [ 8 ]
96.35
3.39
7.13
4.24
0.71
0.64
0.58
0.60
0.52
0.61
cosyvoice [ 10 ]
97.07
3.64
4.32
4.41
0.69
0.76
0.61
0.74
0.65
0.69
cosyvoice2 [ 11 ]
98.47
3.78
3.77
4.43
0.88
0.87
0.72
0.83
0.78
0.82
Emosphere++ [ 21 ]
98.16
3.69
4.94
4.35
0.79
0.81
0.67
0.77
0.69
0.75
EmoVoice [ 22 ]
98.59
3.67
4.16
4.39
0.87
0.85
0.76
0.80
0.79
0.81
Table 1: Objective evaluation: Comparison of Emo-BPO with representative emotional TTS baselines in terms of emotion similarity (Emo SIM), prosody similarity (Prosody SIM), intelligibility, naturalness (UTMOS), and speech emotion recognition accuracy. Emo-BPO achieves the best performance on Emo SIM, Prosody SIM, UTMOS, and mean emotion recognition accuracy while maintaining competitive intelligibility. Results are averaged over three independent runs.
Table 2
Figure 1: Overview of Emo-BPO. Training constructs inverted same-text emotional pairs to jointly learn emotion-aligned (EA) and emotion-contrastive (EC) diffusion branches while partially fine-tuning a pretrained backbone. At inference, EA and EC predictions are combined via contrastive guidance under classifier-free guidance, reinforcing target emotional trajectories and suppressing competing emotional modes.
Diffusion and flow-matching based text-to-speech (TTS) models excel in naturalness but often lack explicit emotion control, as emotional signals remain entangled with speaker identity. We discover that emotion embedding emerges as a linearly decodable direction of frozen hidden states, nearly orthogonal to the direction embedding speaker identity. This inspires a plug-and-play framework DUET for emotion control over pretrained diffusion and flow-matching based TTS models. During generation, DUET unifies dual-space control to achieve fine-grained emotion intervention in a single per-step update: hidden space steering shifts generation along the target emotion direction, while mel-space guidance refines spectral details through gradients backpropagated from a differentiable vocoder. We validate DUET on five architecturally diverse pretrained TTS backbones across three datasets, where it outperforms 10 supervised state-of-the-art emotional TTS baselines across paradigms and achieves the highest human-rated emotion appropriateness. To further showcase its qualitative behavior, we deploy DUET on an Ameca humanoid robot, where it produces richly expressive emotional speech on the humanoid, demonstrating the strong potential for plug-and-play affective interaction for embodied agents.
Xu Zhang, Longbing Cao, Zhangkai Wu
Frontier AI Research Centre, Macquarie University, Sydney, NSW, Australia
Large language model (LLM)-based text-to-speech (TTS) systems enable prompt-conditioned emotional control but struggle with fine-grained emotion intensity due to the semantic -- acoustic gap between text and speech. To address this challenge, we formulate emotion intensity control in LLM-based TTS as a learning-to-rank problem and propose Emo-LiPO, a listwise preference optimization framework that aligns prompt-conditioned speech generation with relative emotion intensity expressed in text. Emo-LiPO explicitly models global intensity ordering within each emotion under fixed transcripts, enabling more faithful and continuous emotional expression. We further construct ESD-plus, a multi-speaker dataset with explicit emotion intensity variations, to support fine-grained emotion modeling and evaluation. Experiments on ESD-plus demonstrate that Emo-LiPO significantly improves emotion accuracy and intensity controllability over both supervised- and DPO-based LLM TTS baselines, with particularly pronounced gains at high intensity levels.
Yihang Lin, Li Zhou, Congwei Cao +4
The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute · Agency for Science, Technology and Research +2
Instruction-based controllable speech synthesis enables users to specify emotions through natural language. However, existing approaches often rely on coarse emotion labels and lack explicit modeling of fine-grained intensity. We propose EmoInstruct-TTS, a dual-path instruction-guided framework for emotional speech synthesis. We introduce Emotion2embed, a supervised semantic-acoustic emotion embedding covering 48 emotional states, including fine-grained categories and intensity levels. To infer embeddings from free-form instructions, we design an Instruction-Conditioned Emotion Flow Model (ICE-Flow) that generates acoustically grounded emotion representations. The inferred embeddings are integrated into an LLM-based synthesis pipeline to provide explicit emotional control while preserving semantic planning. Experiments show improved emotional controllability and speech naturalness over strong baselines.
Minghui Wu, Ganjun Liu, Zikun Fang +6
University of Science and Technology of China, China · iFLYTEK Research, China · Huawei Technologies Co., Ltd., China