Nonverbal Vocalization Generation

Latest papers

8 papers2 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Speech LLMs That Breathe: Modeling Non-Verbal Vocalizations in Dialogue
Oct 19, 2026Heyang Liu, Lixin Zhang, Jiayi Huang +6Nonverbal Vocalization Generation Speech Language Models
Shanghai Jiao Tong University · Ant Group
The NVVSpeech Challenge 2026: Overview and Lessons for Speech Understanding and Generation with Nonverbal Vocalizations
Oct 12, 2026Liumeng Xue, Yuang Cao, Jingbin Hu +12Nonverbal Vocalization Generation Audio Understanding
School of Intelligence Science and Technology, Nanjing University · State Key Laboratory of Novel Software Technology, Nanjing University · Shenzhen Pimei Technology +5
Modeling, Scaling, and Decoding: Optimizing Controllable Speech Generation with Nonverbal Vocalizations
Sep 13, 2026Ziyu Zhang, Yun Chen, Taihui Wang +5Nonverbal Vocalization Generation Controllable Speech Generation
Tencent HY Speech Team
Preference Optimization with LALM Feedback for Continuous Autoregressive Non-Verbal Vocalization Generation
Sep 10, 2026Jingbin Hu, Qirui Zhan, Yuang Cao +7Nonverbal Vocalization Generation Direct Preference Optimization
Shenzhen Pimei Technology Co., Ltd., Guangdong, China · Yutu Zhineng, Beijing, China · State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China +1
Preference Optimization for Non-Verbal Vocalization Synthesis
Aug 25, 2026Haoyang Li, Chenglin Xu, Junchuan Zhao +4Nonverbal Vocalization Generation Preference Optimization
Nanyang Technological University, Singapore · LIGHTSPEED, Singapore · National University of Singapore, Singapore +2
NVV-SuperBench: Beyond Words, Beyond Quality-Benchmarking Nonverbal Vocalizations in Speech Generation
Apr 17, 2026Liumeng Xue, Weizhen Bian, Jiahao Pan +9Nonverbal Vocalization Generation Speech Generation Evaluation
Nanjing University · The Hong Kong University of Science and Technology · The Chinese University of Hong Kong +4
NV-Bench: Benchmark of Nonverbal Vocalization Synthesis for Expressive Text-to-Speech Generation
Mar 16, 2026Qinke Ni, Huan Liao, Dekun Chen +2Nonverbal Vocalization Generation Expressive Speech Synthesis
The Chinese University of Hong Kong, Shenzhen
Affectron: Emotional Speech Synthesis with Affective and Contextually Aligned Nonverbal Vocalizations
Mar 15, 2026Deok-Hyeon Cho, Hyung-Seok Oh, Seung-Bin Kim +1Nonverbal Vocalization Generation Emotional Speech Synthesis
Department of Artificial Intelligence, Korea University, Seoul, Korea

Oct 19, 2026J/K move · Enter open · S save

Speech LLMs That Breathe: Modeling Non-Verbal Vocalizations in Dialogue

Heyang Liu, Lixin Zhang, Jiayi Huang, Ziyang Cheng+5

Shanghai Jiao Tong University · Ant Group

Nonverbal Vocalization Generation Speech Language Models Spoken Dialogue Systems Audio Generation Expressive Speech Synthesis

Open paper AskPDF arXiv

The user experience of speech interaction models has improved, benefiting from the modeling of paralinguistic attributes and advances in speech reconstruction paradigms. However, human communication follows extensive norms regarding non-verbal vocalizations (NVVs) spanning respiratory, affective, and discourse. The omission of these events imposes inherent limitations on the naturalness of current approaches. In this work, we cover 13 major types of NVVs and construct a natural dialogue dataset that conforms to human conversational norms through an automated synthesis pipeline. We enhance the ability of speech interaction models to generate contextually appropriate NVV events through multi-stage fine-tuning and NVV-aware GRPO (Group Relative Policy Optimization). Based on distinct modeling paradigms, we substantially improve the expressiveness of VocalNet and VITA-Qinyu in empathy interactions and role-play dialogues, and further discuss the architectural necessity for achieving high-quality natural speech generation.