Modeling, Scaling, and Decoding: Optimizing Controllable Speech Generation with Nonverbal Vocalizations

Ziyu Zhang; Yun Chen; Taihui Wang; Hanzhao Li; Qicong Xie; Rilin Chen; Zhixian Zhao; Lei Xie

eess.ASSep 13, 2026

Modeling, Scaling, and Decoding: Optimizing Controllable Speech Generation with Nonverbal Vocalizations

Authors: Ziyu Zhang, Yun Chen, Taihui Wang, Hanzhao Li, Qicong Xie, Rilin Chen, Zhixian Zhao, Lei Xie

Organizations: Tencent HY Speech Team

Abstract

Controllable synthesis of nonverbal vocalizations (NVVs) is es- sential for natural and expressive speech, but remains challeng- ing due to their acoustic diversity and imbalanced distribution in existing corpora. To address these challenges, we develop an NVV-aware DiTAR system that models continuous speech latents, encodes the 16 target NVV categories as dedicated to- kens, and adapts stop prediction to distinguish mid-utterance vocalizations from utterance boundaries. Training begins with large-scale bilingual pre-training on diverse NVV speech, fol- lowed by continued supervised fine-tuning on a corpus en- hanced through targeted synthetic augmentation and frequency- aware rebalancing. At inference time, we select the acoustic prompt, tune the LM-guidance and noise-injection scales, and apply Best-of-N sampling with multi-metric selection to re- duce generation failures. The final system achieves an official weighted bilingual score of 62.786, ranking first in Mandarin, second in English, and first overall among participating systems in Track 2 of the ISCSLP 2026 NVVSpeech Challenge. Ab- lation studies show that targeted augmentation benefits under- represented NVV categories the most, while robust candidate selection requires balancing NVV correctness, lexical fidelity, and perceptual quality.

Figures & tables

Figure 1: Overview of our NVV-aware speech generation system. (a) NVV-aware DiTAR, where dedicated NVV tokens condition patch-wise continuous speech generation through causal self-attention. (b) The training and inference pipeline, including bilingual pre-training, long-tail augmentation, continued SFT, decoding-configuration search, and Best-of- $N$ sampling.

System	CER $\downarrow$	WER $\downarrow$	DNS $\uparrow$	P $\uparrow$	R $\uparrow$	F1 $\uparrow$	NTD $\downarrow$
Official baseline	6.267	3.159	3.026	0.769	0.251	0.569	0.073
NVV-aware DiTAR	5.603	2.949	3.126	0.816	0.263	0.570	0.071
+ Long-tail SFT	5.211	2.711	3.172	0.878	0.275	0.574	0.069
+ Decoding search	5.098	2.597	3.183	0.863	0.265	0.582	0.068
+ BoN (final)	5.012	2.194	3.221	0.870	0.271	0.589	0.069

Table 1: Objective metrics for progressive system variants. CER and WER are reported for Mandarin and English, respectively. DNS denotes DNSMOS, and P, R, and F1 denote precision, recall, and CA-F1.

Tag	ZH utterances		EN utterances		$\Delta S_{\mathrm{ZH/EN}}$
Tag	Original	Added	Original	Added	$\Delta S_{\mathrm{ZH/EN}}$
cry	5,424	500	37	1,000	$+4.63/+23.30$
snore	3,728	200	1,023	400	$+1.50/+0.78$
sneeze	3,657	500	1,024	400	$-1.15/-1.98$
yawn	252	1,000	17	1,000	$+3.35/+2.95$
burp	4,328	400	1,726	400	$+7.30/+2.30$

Table 2: Targeted augmentation for weak NVV categories. Original/Added denote the numbers of initial/synthesized utterances, and

\Delta S

denotes the change from NVV-aware DiTAR to long-tail-aware SFT.

$w$	$\tau$	CER ${}_{\mathrm{ZH}}\downarrow$	WER ${}_{\mathrm{EN}}\downarrow$	DNSMOS $\uparrow$	CA-F1 $\uparrow$	NTD $\downarrow$
2.5	0.7	6.217	3.017	3.161	0.569	0.058
1.5	0.7	5.998	2.614	3.177	0.543	0.062
2.0	0.7	5.741	2.661	3.188	0.558	0.065
2.0	0.9	5.098	2.597	3.183	0.582	0.068

Table 3: Decoding-configuration ablation on the development subset. All rows use the same checkpoint.

Selection strategy	CER ${}_{\mathrm{ZH}}\downarrow$	WER ${}_{\mathrm{EN}}\downarrow$	DNSMOS $\uparrow$	CA-F1 $\uparrow$	NTD $\downarrow$
Single sample	5.743	3.148	3.157	0.542	0.059
Random selection	6.311	3.277	3.161	0.581	0.067
DNSMOS only	6.012	3.480	3.229	0.543	0.061
NVV detector only	5.977	2.792	3.102	0.591	0.068
Multi-metric	5.012	2.194	3.221	0.589	0.069

Table 4: BoN selection ablation with five candidates.

System	LALM ZH / EN / Bi.	Subjective ZH / EN / Bi.	Final ZH / EN / Bi.
Ours	76.22 69.66 72.94	2.838 $\pm$ 0.096 3.156 $\pm$ 0.099 2.997 $\pm$ 0.070	61.775 63.797 62.786
Official baseline	77.62 70.30 73.96	2.714 $\pm$ 0.089 3.072 $\pm$ 0.091 2.893 $\pm$ 0.066	59.700 63.163 61.431

Table 5: Official Track 2 results.

Controllable Speech Generation

Controllable speech generation steers synthesized speech's voice, prosody, emotion, timing, pronunciation, and conversational delivery.

See all 115 papers

NVV-SuperBench: Beyond Words, Beyond Quality-Benchmarking Nonverbal Vocalizations in Speech Generation
Apr 17, 2026Liumeng Xue, Weizhen Bian, Jiahao Pan +9Speech Generation Evaluation Speech Quality Assessment
Nanjing University · The Hong Kong University of Science and Technology · The Chinese University of Hong Kong +4
Speech LLMs That Breathe: Modeling Non-Verbal Vocalizations in Dialogue
Oct 19, 2026Heyang Liu, Lixin Zhang, Jiayi Huang +6Speech Language Models Spoken Dialogue Systems
Shanghai Jiao Tong University · Ant Group
The NVVSpeech Challenge 2026: Overview and Lessons for Speech Understanding and Generation with Nonverbal Vocalizations
Oct 12, 2026Liumeng Xue, Yuang Cao, Jingbin Hu +12Audio Understanding Audio Generation
School of Intelligence Science and Technology, Nanjing University · State Key Laboratory of Novel Software Technology, Nanjing University · Shenzhen Pimei Technology +5

Apr 17, 2026J/K move · Enter open · S save

NVV-SuperBench: Beyond Words, Beyond Quality-Benchmarking Nonverbal Vocalizations in Speech Generation

Liumeng Xue, Weizhen Bian, Jiahao Pan, Wenxuan Wu+8

Nanjing University · The Hong Kong University of Science and Technology · The Chinese University of Hong Kong+4

Speech Generation Evaluation Speech Quality Assessment Controllable Speech Generation Audio Generation

Open paper AskPDF arXiv

Nonverbal vocalizations (NVVs), such as laughing, sighing, and sobbing, are essential for human-like speech, yet standardized evaluation rarely jointly assesses whether systems generate the intended NVVs, place them correctly, and keep them salient without harming speech. We present NVV-SuperBench, a bilingual English/Chinese benchmark for speech generation with NVVs. It provides a unified 45-type taxonomy and a multi-axis protocol beyond conventional speech quality assessment, evaluating NVV-specific controllability, placement, and perceptual salience. We benchmark 15 speech generation systems spanning prompt-based and tag-based control paradigms, using objective metrics, human listening tests, and LLM-based multi-rater evaluation. Results show that NVV controllability often decouples from speech quality, while low-SNR oral cues and long-duration affective NVVs remain bottlenecks. NVV-SuperBench highlights current gaps and supports progress toward more human-like speech generation.

Abstract

Controllable Speech Generation

Explore similar work

NVV-SuperBench: Beyond Words, Beyond Quality-Benchmarking Nonverbal Vocalizations in Speech Generation

Speech LLMs That Breathe: Modeling Non-Verbal Vocalizations in Dialogue

The NVVSpeech Challenge 2026: Overview and Lessons for Speech Understanding and Generation with Nonverbal Vocalizations