Expressive Speech Synthesis

Follow this topic Ask about this topic

Latest papers

23 papers5 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Speech LLMs That Breathe: Modeling Non-Verbal Vocalizations in Dialogue
Oct 19, 2026Heyang Liu, Lixin Zhang, Jiayi Huang +6Speech Language Models Spoken Dialogue Systems
Shanghai Jiao Tong University · Ant Group
E3-System: Expressive EmoData, Explicitly Controllable, Easy-to-Use TTS
Oct 9, 2026Yiqiao Huang, Xiaolong Lin, Zhenpeng Zhan +1TTS Synthesis Expressive Speech Synthesis
Harvard University, Cambridge, MA, USA · Baidu Inc., Shenzhen, China · The Chinese University of Hong Kong, Shenzhen, China
Edit Who Speaks, Control How They Speak: Global Timbre Editing and Local Instruction Control for TTS
Oct 8, 2026Junchuan Zhao, Chenglin Xu, Wei Zeng +3TTS Synthesis Controllable Speech Generation
National University of Singapore · LIGHTSPEED · Nanyang Technological University +1
VoiceWeaver: Staged Learning of Structured Controls for Expressive Speech and Sound-Event Generation
Oct 3, 2026Xiaosu Su, Yun Cao, Yiping Ni +1Controllable Speech Generation TTS Synthesis
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
InstCharVoice: Grounding Natural-Language Instructions for Character-Level Control in Text-to-Speech
Sep 28, 2026Sihang Nie, Xueru Li, Xiaofen Xing +4TTS Synthesis Language Model-Based Control
South China University of Technology · Huya Inc. · The Hongkong Polytechnic University
Interactive TTS: Dynamic Speaking Style Adaptation for Expressive Speech Synthesis
Sep 22, 2026Wenjie Tian, Kangxiang Xia, Jingbin Hu +7TTS Synthesis Direct Preference Optimization
Alibaba ATH Token Foundry
COT-TTS: Audio Context-Aware Text-to-Speech with Chain-of-Thought Reasoning
Sep 19, 2026Weizhen Bian, Sitong Cheng, Rongxiu Zhong +9TTS Synthesis Expressive Speech Synthesis
The Hong Kong University of Science and Technology, Hong Kong SAR, China · JIUTIAN Research, China Mobile, Beijing, China · The State Key Laboratory of Multimedia Information Processing, Peking University, Beijing, China +2
Scalable Direction-Following TTS via Voice Impression-Guided Pseudo Triplet Construction
Sep 2, 2026Kenichi Fujita, Yusuke IjimaExpressive Speech Synthesis Text-to-Speech
NTT, Inc., Japan
Iterative Self-Learning for Expressive Text-to-Speech Synthesis
Aug 16, 2026Nicholas Sanders, Gustav Eje Henter, Simon King +1Expressive Speech Synthesis TTS Synthesis
University of Edinburgh
CtrlSpeech: Coarse-to-Fine Control for Expressive Speech Synthesis
Aug 8, 2026Zhisheng Zheng, Xiaohang Sun, Zhu Liu +5TTS Synthesis Zero-Shot TTS
The University of Texas at Austin · Amazon
Harness TTS: Towards Context-Aware Expressive Speech Synthesis with Harness Layer
Jul 20, 2026Shengfan Shen, Di Wu, Xingchen Song +5TTS Synthesis Language Model-Based Control
Nanjing University, China · MiLM Plus, Xiaomi Inc., China · WeNet Open Source Community
AutoSIFT: Automatic Style Sifting for Controllable Speech Generation with Arbitrary Style Infilling
Jul 14, 2026Haowei Lou, Junda Wu, Chengkai Huang +4TTS Synthesis Representation Disentanglement
UNSW Sydney · University of California San Diego · Macquarie University +1
VoiceTTA: Enhancing Zero-Shot Text-to-Speech via Reinforcement Learning-Based Test-Time Adaptation
Jun 25, 2026Tianxin Xie, Chenxing Li, Dong Yu +1Test-Time Adaptation Reinforcement Learning
The Hong Kong University of Science and Technology (Guangzhou) · Tencent
Adaptive Oscillatory Inductive Bias for Modeling Sharp Prosodic Dynamics in Diffusion-Based TTS
Jun 24, 2026Sandipan Dhar, Nirmesh J. Shah, Ashishkumar P. Gudmalwar +1TTS Synthesis Audio Diffusion Models
Media Analysis, Sony Research India
EmoInstruct-TTS: Dual-Path Instruction-Guided Emotional Speech Synthesis
Jun 8, 2026Minghui Wu, Ganjun Liu, Zikun Fang +6Representation Learning TTS Synthesis
University of Science and Technology of China, China · iFLYTEK Research, China · Huawei Technologies Co., Ltd., China
AgentSteerTTS: A Multi-Agent Closed-Loop Framework for Composite-Instruction Text-to-Speech
May 14, 2026Bin Kang, Shaoguo Wen, Yang Fan +6TTS Synthesis Expressive Speech Synthesis
University of Chinese Academy of Sciences · Shenzhen Loop Area Institute · Tencent Turinglab +2
AuDirector: A Self-Reflective Closed-Loop Framework for Immersive Audio Storytelling
May 12, 2026Yiming Ren, Xuenan Xu, Ziyang Zhang +3TTS Synthesis Human-in-the-Loop AI
Shanghai Artificial Intelligence Laboratory · Tsinghua University
Bridging What the Model Thinks and How It Speaks: Expressive Speech Generation via Self-Aware Intent-Realization Alignment
Apr 13, 2026Kuang Wang, Lai Wei, Ping Lin +8Expressive Speech Synthesis Speech Language Models
The Chinese University of Hong Kong, Shenzhen · Tencent Ethereal Audio Lab · Southeast University +2
MambaVoiceCloning: Efficient and Expressive Text-to-Speech via State-Space Modeling and Diffusion Control
Mar 31, 2026Sahil Kumar, Namrataben Patel, Honggang Wang +1Expressive Speech Synthesis TTS Synthesis
PhD Program in Mathematics, Yeshiva University, New York, NY 10033, USA · Department of Computer Science & Engineering, Yeshiva University, New York, NY 10033, USA · School of Artificial Intelligence, Chuzhou University, Anhui, 239000, China
Voxtral TTS
Mar 26, 2026Mistral-AI, :, Alexander H. Liu +186Multilingual TTS Synthesis TTS Synthesis
NV-Bench: Benchmark of Nonverbal Vocalization Synthesis for Expressive Text-to-Speech Generation
Mar 16, 2026Qinke Ni, Huan Liao, Dekun Chen +2Expressive Speech Synthesis TTS Synthesis
The Chinese University of Hong Kong, Shenzhen
DiFlowDubber: Discrete Flow Matching for Automated Video Dubbing via Cross-Modal Alignment and Synchronization
Mar 15, 2026Ngoc-Son Nguyen, Thanh V. T. Tran, Jeongsoo Choi +3Expressive Speech Synthesis TTS Synthesis
FPT Software AI Center, Vietnam · KAIST, South Korea · Independent researcher, work done in part while author was at the FPT Software AI Center, Vietnam. +1
Hello-Chat: Towards Realistic Social Audio Interactions
Feb 16, 2026Yueran Hou, Peilei Jia, Zihan Sun +5Expressive Speech Synthesis Audio-Language Models

Oct 19, 2026J/K move · Enter open · S save

Speech LLMs That Breathe: Modeling Non-Verbal Vocalizations in Dialogue

Heyang Liu, Lixin Zhang, Jiayi Huang, Ziyang Cheng+5

Shanghai Jiao Tong University · Ant Group

Speech Language Models Spoken Dialogue Systems Audio Generation Expressive Speech Synthesis

Open paper AskPDF arXiv

The user experience of speech interaction models has improved, benefiting from the modeling of paralinguistic attributes and advances in speech reconstruction paradigms. However, human communication follows extensive norms regarding non-verbal vocalizations (NVVs) spanning respiratory, affective, and discourse. The omission of these events imposes inherent limitations on the naturalness of current approaches. In this work, we cover 13 major types of NVVs and construct a natural dialogue dataset that conforms to human conversational norms through an automated synthesis pipeline. We enhance the ability of speech interaction models to generate contextually appropriate NVV events through multi-stage fine-tuning and NVV-aware GRPO (Group Relative Policy Optimization). Based on distinct modeling paradigms, we substantially improve the expressiveness of VocalNet and VITA-Qinyu in empathy interactions and role-play dialogues, and further discuss the architectural necessity for achieving high-quality natural speech generation.