Expressive Speech Synthesis

Follow this topic Ask about this topic

Latest papers

25 papers5 papers in the last four weeks, up 25% on the four weeks before. 0.0% of all new papers.

EmoFI-TTS: Face-Conditioned Expressive Voice Synthesis with Intensity Modulation
Oct 21, 2026Yejin Jeon, Luel Hagos Beyene, Marie Maltais +2Expressive Speech Synthesis Emotional Speech Synthesis
Mila-Québec AI Institute · McGill University · NM-AIST +3
Steering intonational tunes in text-to-speech synthesis
Oct 19, 2026Juan Pablo Vigneaux, Jennifer ColeTTS Synthesis Speech Prosody
Northwestern University
Speech LLMs That Breathe: Modeling Non-Verbal Vocalizations in Dialogue
Oct 19, 2026Heyang Liu, Lixin Zhang, Jiayi Huang +6Nonverbal Vocalization Generation Speech Language Models
Shanghai Jiao Tong University · Ant Group
E3-System: Expressive EmoData, Explicitly Controllable, Easy-to-Use TTS
Oct 9, 2026Yiqiao Huang, Xiaolong Lin, Zhenpeng Zhan +1TTS Synthesis Expressive Speech Synthesis
Harvard University, Cambridge, MA, USA · Baidu Inc., Shenzhen, China · The Chinese University of Hong Kong, Shenzhen, China
Edit Who Speaks, Control How They Speak: Global Timbre Editing and Local Instruction Control for TTS
Oct 8, 2026Junchuan Zhao, Chenglin Xu, Wei Zeng +3TTS Synthesis Controllable Speech Generation
National University of Singapore · LIGHTSPEED · Nanyang Technological University +1
VoiceWeaver: Staged Learning of Structured Controls for Expressive Speech and Sound-Event Generation
Oct 3, 2026Xiaosu Su, Yun Cao, Yiping Ni +1Controllable Speech Generation TTS Synthesis
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
InstCharVoice: Grounding Natural-Language Instructions for Character-Level Control in Text-to-Speech
Sep 28, 2026Sihang Nie, Xueru Li, Xiaofen Xing +4TTS Synthesis Language Model-Based Control
South China University of Technology · Huya Inc. · The Hongkong Polytechnic University
Interactive TTS: Dynamic Speaking Style Adaptation for Expressive Speech Synthesis
Sep 22, 2026Wenjie Tian, Kangxiang Xia, Jingbin Hu +7TTS Synthesis Direct Preference Optimization
Alibaba ATH Token Foundry
COT-TTS: Audio Context-Aware Text-to-Speech with Chain-of-Thought Reasoning
Sep 19, 2026Weizhen Bian, Sitong Cheng, Rongxiu Zhong +9TTS Synthesis Expressive Speech Synthesis
The Hong Kong University of Science and Technology, Hong Kong SAR, China · JIUTIAN Research, China Mobile, Beijing, China · The State Key Laboratory of Multimedia Information Processing, Peking University, Beijing, China +2
Scalable Direction-Following TTS via Voice Impression-Guided Pseudo Triplet Construction
Sep 2, 2026Kenichi Fujita, Yusuke IjimaExpressive Speech Synthesis Text-to-Speech
NTT, Inc., Japan
Iterative Self-Learning for Expressive Text-to-Speech Synthesis
Aug 16, 2026Nicholas Sanders, Gustav Eje Henter, Simon King +1Expressive Speech Synthesis TTS Synthesis
University of Edinburgh
CtrlSpeech: Coarse-to-Fine Control for Expressive Speech Synthesis
Aug 8, 2026Zhisheng Zheng, Xiaohang Sun, Zhu Liu +5TTS Synthesis Zero-Shot TTS
The University of Texas at Austin · Amazon
Harness TTS: Towards Context-Aware Expressive Speech Synthesis with Harness Layer
Jul 20, 2026Shengfan Shen, Di Wu, Xingchen Song +5TTS Synthesis Language Model-Based Control
Nanjing University, China · MiLM Plus, Xiaomi Inc., China · WeNet Open Source Community
AutoSIFT: Automatic Style Sifting for Controllable Speech Generation with Arbitrary Style Infilling
Jul 14, 2026Haowei Lou, Junda Wu, Chengkai Huang +4TTS Synthesis Representation Disentanglement
UNSW Sydney · University of California San Diego · Macquarie University +1
VoiceTTA: Enhancing Zero-Shot Text-to-Speech via Reinforcement Learning-Based Test-Time Adaptation
Jun 25, 2026Tianxin Xie, Chenxing Li, Dong Yu +1Test-Time Adaptation Reinforcement Learning
The Hong Kong University of Science and Technology (Guangzhou) · Tencent
Adaptive Oscillatory Inductive Bias for Modeling Sharp Prosodic Dynamics in Diffusion-Based TTS
Jun 24, 2026Sandipan Dhar, Nirmesh J. Shah, Ashishkumar P. Gudmalwar +1TTS Synthesis Audio Diffusion Models
Media Analysis, Sony Research India
EmoInstruct-TTS: Dual-Path Instruction-Guided Emotional Speech Synthesis
Jun 8, 2026Minghui Wu, Ganjun Liu, Zikun Fang +6Representation Learning TTS Synthesis
University of Science and Technology of China, China · iFLYTEK Research, China · Huawei Technologies Co., Ltd., China
AgentSteerTTS: A Multi-Agent Closed-Loop Framework for Composite-Instruction Text-to-Speech
May 14, 2026Bin Kang, Shaoguo Wen, Yang Fan +6TTS Synthesis Expressive Speech Synthesis
University of Chinese Academy of Sciences · Shenzhen Loop Area Institute · Tencent Turinglab +2
AuDirector: A Self-Reflective Closed-Loop Framework for Immersive Audio Storytelling
May 12, 2026Yiming Ren, Xuenan Xu, Ziyang Zhang +3TTS Synthesis Human-in-the-Loop AI
Shanghai Artificial Intelligence Laboratory · Tsinghua University
Bridging What the Model Thinks and How It Speaks: Expressive Speech Generation via Self-Aware Intent-Realization Alignment
Apr 13, 2026Kuang Wang, Lai Wei, Ping Lin +8Expressive Speech Synthesis Speech Language Models
The Chinese University of Hong Kong, Shenzhen · Tencent Ethereal Audio Lab · Southeast University +2
MambaVoiceCloning: Efficient and Expressive Text-to-Speech via State-Space Modeling and Diffusion Control
Mar 31, 2026Sahil Kumar, Namrataben Patel, Honggang Wang +1Expressive Speech Synthesis TTS Synthesis
PhD Program in Mathematics, Yeshiva University, New York, NY 10033, USA · Department of Computer Science & Engineering, Yeshiva University, New York, NY 10033, USA · School of Artificial Intelligence, Chuzhou University, Anhui, 239000, China
Voxtral TTS
Mar 26, 2026Mistral-AI, :, Alexander H. Liu +186Multilingual TTS Synthesis TTS Synthesis
NV-Bench: Benchmark of Nonverbal Vocalization Synthesis for Expressive Text-to-Speech Generation
Mar 16, 2026Qinke Ni, Huan Liao, Dekun Chen +2Nonverbal Vocalization Generation Expressive Speech Synthesis
The Chinese University of Hong Kong, Shenzhen
DiFlowDubber: Discrete Flow Matching for Automated Video Dubbing via Cross-Modal Alignment and Synchronization
Mar 15, 2026Ngoc-Son Nguyen, Thanh V. T. Tran, Jeongsoo Choi +3Expressive Speech Synthesis TTS Synthesis
FPT Software AI Center, Vietnam · KAIST, South Korea · Independent researcher, work done in part while author was at the FPT Software AI Center, Vietnam. +1
Hello-Chat: Towards Realistic Social Audio Interactions
Feb 16, 2026Yueran Hou, Peilei Jia, Zihan Sun +5Expressive Speech Synthesis Audio-Language Models

Oct 21, 2026J/K move · Enter open · S save

EmoFI-TTS: Face-Conditioned Expressive Voice Synthesis with Intensity Modulation

Yejin Jeon, Luel Hagos Beyene, Marie Maltais, Hyounghun Kim+1

Mila-Québec AI Institute · McGill University · NM-AIST+3

Expressive Speech Synthesis Emotional Speech Synthesis

Open paper AskPDF arXiv

Controlling emotional intensity remains challenging in expressive text-to-speech synthesis, particularly when emotion must be inferred without explicit labels. We propose EmoFI-TTS, a face-conditioned framework that derives emotion category and intensity from a single facial image while extracting speaker characteristics from reference audio. Modality-specific representations are fused to condition a pretrained speech synthesizer, with cross-speaker training encouraging separation of affect and vocal identity. An intensity-conditioned refinement module enhances the generated mel-spectrogram, while interpolation of learned intensity embeddings enables continuous control without additional training. On speaker-disjoint MEAD evaluation, EmoFI-TTS achieves on average 64.4% and 75.2% objective and subjective emotion intensity accuracies, while retaining low word error rate and high naturalness.