Emotional Speech Synthesis

Follow this topic Ask about this topic

Emotional speech synthesis studies how to make generated speech convey feelings, intensity, and changing affect while keeping the words clear and the voice consistent. Systems use emotion labels or natural-language instructions, prosody controls such as pitch and duration, and newer autoregressive, diffusion, and flow-matching models; preference learning and activation steering offer ways to refine control. Some work also uses dialogue context or visual cues to plan an emotionally appropriate response. A central challenge is balancing expressive, fine-grained control with naturalness, intelligibility, and speaker identity, while ensuring automatic emotion measures reflect human judgments.

Latest papers

47 papers9 papers in the last four weeks, level with the four weeks before. 0.1% of all new papers.

When the Persona Contradicts the Line: Diagnosing Plan-to-Synthesis Failures in Chain-of-Thought TTS
Oct 26, 2026Joonyong Park, Jerry LiTTS Synthesis Emotional Speech Synthesis
Spellbrush, CA, USA
EmoFI-TTS: Face-Conditioned Expressive Voice Synthesis with Intensity Modulation
Oct 21, 2026Yejin Jeon, Luel Hagos Beyene, Marie Maltais +2Expressive Speech Synthesis Emotional Speech Synthesis
Mila-Québec AI Institute · McGill University · NM-AIST +3
Too Emotional to Be Fake? Intra-Utterance Emotion Shifts Expose Failure Modes in Speech Deepfake Detectors
Oct 16, 2026Jingyi Chen, Surya KoppisettiAudio Deepfake Detection Emotional Speech Synthesis
Reality Defender
Prime-EVC: One-Step Expressive Emotional Voice Conversion via Emotion-Primed Flow Matching
Oct 12, 2026Yu Pan, Xiaojuan Cai, Liang Zhang +7Voice Conversion Emotional Speech Synthesis
Kyushu University, Japan · Donghua University, China · National Institute of Informatics, Japan +3
E3-System: Expressive EmoData, Explicitly Controllable, Easy-to-Use TTS
Oct 9, 2026Yiqiao Huang, Xiaolong Lin, Zhenpeng Zhan +1TTS Synthesis Expressive Speech Synthesis
Harvard University, Cambridge, MA, USA · Baidu Inc., Shenzhen, China · The Chinese University of Hong Kong, Shenzhen, China
Emo-BPO: Emotion Bidirectional Preference Optimization for Diffusion-based Emotional TTS
Oct 9, 2026Jiacheng Shi, Hongfei Du, Xinyuan Song +3Preference Optimization for Diffusion Models Emotional Speech Synthesis
College of William & Mary, USA · Emory University, USA · George Mason University, USA
Steerspeech: Activation Steering For Emotion Control In Generated Speech
Oct 7, 2026Afsara Benazir, Darius Pétermann, Felix Xiaozhu Lin +1Controllable Speech Generation Emotional Speech Synthesis
University of Virginia · Netflix, Inc.
Tracing a Sparse Emotion-Control Circuit in LLM-Based Text-to-Speech
Oct 4, 2026Hongfei Du, Jiacheng Shi, Yanfu Zhang +1Mechanistic Interpretability Controllable Speech Generation
Department of Computer Science, William & Mary, Williamsburg, USA
VoiceWeaver: Staged Learning of Structured Controls for Expressive Speech and Sound-Event Generation
Oct 3, 2026Xiaosu Su, Yun Cao, Yiping Ni +1Controllable Speech Generation TTS Synthesis
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
EmoRES-TTS: Residual-Enhanced Vector Steering for Emotional Speech Generation
Sep 29, 2026Kuan-Po Huang, Haohe Liu, Puyuan Peng +5Controllable Speech Generation Emotional Speech Synthesis
Reality Labs at Meta · National Taiwan University · FAIR at Meta
Toward Human-Aligned Judgement of Speech Emotion Similarity
Sep 26, 2026Yun-Shao Tsai, Yi-Cheng Lin, Chih-Kai Yang +7Speech Generation Evaluation Emotional Speech Synthesis
Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · Independent Researcher · National Taiwan University, Taiwan +2
ReaFlow-TTS: Realization-Conditioned Flow Matching for High-Quality and Controllable Speech Synthesis
Sep 24, 2026Junyi Zhao, Yihao Qin, Changsheng MaFlow Matching TTS Synthesis
School of Information Science and Technology, Lanzhou University, China
COT-TTS: Audio Context-Aware Text-to-Speech with Chain-of-Thought Reasoning
Sep 19, 2026Weizhen Bian, Sitong Cheng, Rongxiu Zhong +9TTS Synthesis Expressive Speech Synthesis
The Hong Kong University of Science and Technology, Hong Kong SAR, China · JIUTIAN Research, China Mobile, Beijing, China · The State Key Laboratory of Multimedia Information Processing, Peking University, Beijing, China +2
Listen Before You Speak: Response Planning from Listener Facial Reactions for Conversational Speech Generation
Sep 18, 2026Yunji ChuControllable Speech Generation Emotional Speech Synthesis
Department of Artificial Intelligence, Sogang University, Seoul, Republic of Korea
The VoiceMOS Challenge 2026: Evaluating Speech Enhancement, Emotional TTS and Accented TTS Systems
Sep 12, 2026Wen-Chin Huang, Wei Wang, Marvin Sach +4Speech Quality Assessment TTS Evaluation
Nagoya University, Japan · Shanghai Jiao Tong University, China · Technische Universität Braunschweig, Germany +3
Continuous-Time Acoustic Modelling with Neural Controlled Differential Equations
Sep 10, 2026Mattias Cross, Minghui Zhao, Anton RagniNeural Controlled Differential Equations TTS Synthesis
Speech and Hearing Group University of Sheffield Sheffield, United Kingdom
Post-Training Zero-Shot TTS for Fine-Grained Emotion and Duration Control via Natural Language
Sep 10, 2026Lianru Gao, Yujie Guo, Yong QinTTS Synthesis Zero-Shot TTS
TMCC, College of Computer Science, Nankai University, Tianjin, China
Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesis
Sep 3, 2026Sanyuan Chen, Min-Jae Hwang, Sho Inoue +12TTS Synthesis Audio Diffusion Models
FAIR at Meta
Sequential Trajectories and Simultaneous Blending: Multi-Emotion Modeling for Instruction-Following TTS
Aug 31, 2026Yan Zhou, Yun Hong, Yang FengControllable Speech Generation Emotional Speech Synthesis
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
EmoTra-TTS: Smooth Intra-Utterance Emotion Transitions for Speech Synthesis
Aug 24, 2026Tianchi Liu, Zeyang Song, Tianrui Wang +3Emotional Speech Synthesis Speech Prosody
LIGHTSPEED · National University of Singapore · Nanyang Technological University +1
Iterative Self-Learning for Expressive Text-to-Speech Synthesis
Aug 16, 2026Nicholas Sanders, Gustav Eje Henter, Simon King +1Expressive Speech Synthesis TTS Synthesis
University of Edinburgh
SceneTTS-Bench: A Benchmark for Scene-Level TTS in Drama Dubbing
Aug 13, 2026Yizhong Geng, Yanliang Li, Jinghan Yang +3Long-Form TTS Synthesis TTS Synthesis
Beijing University of Posts and Telecommunications Beijing, China · Beijing DeepLogic Intelligence Technology Co., Ltd. Beijing, China · University of California USA
Luna-TTS Family Technical Report
Aug 12, 2026Feng Yin, Shuai Shi, Junjie Zheng +19Non-Autoregressive Text Generation TTS Synthesis
VUI Labs Research
Let Me Look at You: Advanced Facial Expression Modeling for Conversational Speech Synthesis
Jul 27, 2026Yifan Hu, Shuwei He, Rui Liu +1TTS Synthesis Audio-Driven Facial Animation
Inner Mongolia University Hohhot, China · SRIBD, School of Artificial Intelligence The Chinese University of Hong Kong Shenzhen, China
AuEmoChat: Authentic Emotion Understanding and Rendering for Conversational Speech Synthesis
Jul 17, 2026Zhenqi Jia, Yuan Zhao, Aruukhan +2TTS Synthesis Emotional Speech Synthesis
Inner Mongolia University · The Chinese University of Hong Kong
A Geometric Perspective on Composable Emotion Steering in Text-to-Speech Models
Jul 1, 2026Siyi Wang, James Bailey, Ting DangLanguage Model Steering Conditional Flow Matching
The University of Melbourne, Australia · Monash University, Australia
LuxEmo: Expressive Text-to-Speech Corpus for Luxembourgish
Jun 30, 2026Nina Hosseini-Kivanani, Sandipana DowerahSpeech Processing Low-Resource TTS Synthesis
Radio T´el´evisioun L¨etzebuerg (RTL) & University of Luxembourg · Esch-sur-Alzette, Luxembourg · Tallinn University of Technology +1
HPRO: Hierarchical Progressive Reward Optimization via Preference Extraction for Emotional Text-to-Speech
Jun 26, 2026Sihang Nie, Xiaofen Xing, Rui Xing +5Reward Modeling Speech Prosody
South China University of Technology, China · Huya Inc., China · Tongyi Fun Team, Alibaba Group, China +1
An Empirical Study on Learning Latent Representations for Emotional Speech Synthesis
Jun 12, 2026Vinh Dang Quang, Huy Ngo QuangRepresentation Learning TTS Synthesis
Aimesoft JSC Hanoi, Vietnam
Emo-LiPO: Listwise Preference Optimization for Fine-Grained Emotion Intensity Control in LLM-based Text-to-Speech
Jun 11, 2026Yihang Lin, Li Zhou, Congwei Cao +4TTS Synthesis Preference Optimization
The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute · Agency for Science, Technology and Research +2

Oct 26, 2026J/K move · Enter open · S save

When the Persona Contradicts the Line: Diagnosing Plan-to-Synthesis Failures in Chain-of-Thought TTS

Joonyong Park, Jerry Li

Spellbrush, CA, USA

TTS Synthesis Emotional Speech Synthesis TTS Evaluation Multilingual TTS Synthesis

Open paper AskPDF arXiv

Chain-of-thought TTS systems rely on language-model planners to translate persona and context into vocal delivery, but it is unclear where this intent is lost when the requested affect conflicts with lexical sentiment. We introduce a bilingual Japanese--English persona--line conflict benchmark in which each item pairs one line with congruent and conflicting personas and pre-declares the expected acoustic direction. We probe this direction in the planner-generated delivery phrase, the planner-written emotion vector, the QwenEmotion-generated vector, and the rendered waveform. Planner outputs are usually direction-correct, while downstream agreement can be substantially lower. Bypassing QwenEmotion provides only modest gains over the full planner-controlled Base. Supervising QwenEmotion raises the direction rate to 75% in Japanese and 84% in English. A learned text-to-latent interface reaches the highest English rate of 89% and the highest Japanese arousal rate of 88%. These results show that learned repair of the plan-to-synthesis interface can recover counter-textual delivery, although its effect differs across languages and acoustic axes.