Controllable synthesis of nonverbal vocalizations (NVVs) is es- sential for natural and expressive speech, but remains challeng- ing due to their acoustic diversity and imbalanced distribution in existing corpora. To address these challenges, we develop an NVV-aware DiTAR system that models continuous speech latents, encodes the 16 target NVV categories as dedicated to- kens, and adapts stop prediction to distinguish mid-utterance vocalizations from utterance boundaries. Training begins with large-scale bilingual pre-training on diverse NVV speech, fol- lowed by continued supervised fine-tuning on a corpus en- hanced through targeted synthetic augmentation and frequency- aware rebalancing. At inference time, we select the acoustic prompt, tune the LM-guidance and noise-injection scales, and apply Best-of-N sampling with multi-metric selection to re- duce generation failures. The final system achieves an official weighted bilingual score of 62.786, ranking first in Mandarin, second in English, and first overall among participating systems in Track 2 of the ISCSLP 2026 NVVSpeech Challenge. Ab- lation studies show that targeted augmentation benefits under- represented NVV categories the most, while robust candidate selection requires balancing NVV correctness, lexical fidelity, and perceptual quality.
Figures & tables
Figure 1: Overview of our NVV-aware speech generation system. (a) NVV-aware DiTAR, where dedicated NVV tokens condition patch-wise continuous speech generation through causal self-attention. (b) The training and inference pipeline, including bilingual pre-training, long-tail augmentation, continued SFT, decoding-configuration search, and Best-of- N sampling.
System
CER ↓
WER ↓
DNS ↑
P ↑
R ↑
F1 ↑
NTD ↓
Official baseline
6.267
3.159
3.026
0.769
0.251
0.569
0.073
NVV-aware DiTAR
5.603
2.949
3.126
0.816
0.263
0.570
0.071
+ Long-tail SFT
5.211
2.711
3.172
0.878
0.275
0.574
0.069
+ Decoding search
5.098
2.597
3.183
0.863
0.265
0.582
0.068
+ BoN (final)
5.012
2.194
3.221
0.870
0.271
0.589
0.069
Table 1: Objective metrics for progressive system variants. CER and WER are reported for Mandarin and English, respectively. DNS denotes DNSMOS, and P, R, and F1 denote precision, recall, and CA-F1.
Tag
ZH utterances
EN utterances
ΔSZH/EN
Original
Added
Original
Added
cry
5,424
500
37
1,000
+4.63/+23.30
snore
3,728
200
1,023
400
+1.50/+0.78
sneeze
3,657
500
1,024
400
−1.15/−1.98
yawn
252
1,000
17
1,000
+3.35/+2.95
burp
4,328
400
1,726
400
+7.30/+2.30
Table 2: Targeted augmentation for weak NVV categories. Original/Added denote the numbers of initial/synthesized utterances, and ΔS denotes the change from NVV-aware DiTAR to long-tail-aware SFT.
w
τ
CER ZH↓
WER EN↓
DNSMOS ↑
CA-F1 ↑
NTD ↓
2.5
0.7
6.217
3.017
3.161
0.569
0.058
1.5
0.7
5.998
2.614
3.177
0.543
0.062
2.0
0.7
5.741
2.661
3.188
0.558
0.065
2.0
0.9
5.098
2.597
3.183
0.582
0.068
Table 3: Decoding-configuration ablation on the development subset. All rows use the same checkpoint.
Selection strategy
CER ZH↓
WER EN↓
DNSMOS ↑
CA-F1 ↑
NTD ↓
Single sample
5.743
3.148
3.157
0.542
0.059
Random selection
6.311
3.277
3.161
0.581
0.067
DNSMOS only
6.012
3.480
3.229
0.543
0.061
NVV detector only
5.977
2.792
3.102
0.591
0.068
Multi-metric
5.012
2.194
3.221
0.589
0.069
Table 4: BoN selection ablation with five candidates.
Nonverbal vocalizations (NVVs), such as laughing, sighing, and sobbing, are essential for human-like speech, yet standardized evaluation rarely jointly assesses whether systems generate the intended NVVs, place them correctly, and keep them salient without harming speech. We present NVV-SuperBench, a bilingual English/Chinese benchmark for speech generation with NVVs. It provides a unified 45-type taxonomy and a multi-axis protocol beyond conventional speech quality assessment, evaluating NVV-specific controllability, placement, and perceptual salience. We benchmark 15 speech generation systems spanning prompt-based and tag-based control paradigms, using objective metrics, human listening tests, and LLM-based multi-rater evaluation. Results show that NVV controllability often decouples from speech quality, while low-SNR oral cues and long-duration affective NVVs remain bottlenecks. NVV-SuperBench highlights current gaps and supports progress toward more human-like speech generation.
Liumeng Xue, Weizhen Bian, Jiahao Pan +9
Nanjing University · The Hong Kong University of Science and Technology · The Chinese University of Hong Kong +4
The user experience of speech interaction models has improved, benefiting from the modeling of paralinguistic attributes and advances in speech reconstruction paradigms. However, human communication follows extensive norms regarding non-verbal vocalizations (NVVs) spanning respiratory, affective, and discourse. The omission of these events imposes inherent limitations on the naturalness of current approaches. In this work, we cover 13 major types of NVVs and construct a natural dialogue dataset that conforms to human conversational norms through an automated synthesis pipeline. We enhance the ability of speech interaction models to generate contextually appropriate NVV events through multi-stage fine-tuning and NVV-aware GRPO (Group Relative Policy Optimization). Based on distinct modeling paradigms, we substantially improve the expressiveness of VocalNet and VITA-Qinyu in empathy interactions and role-play dialogues, and further discuss the architectural necessity for achieving high-quality natural speech generation.
Nonverbal vocalizations (NVVs), such as laughter and sighs, convey meaning alongside words. NVV-aware speech systems must recognize and generate these events while preserving lexical accuracy and perceptual quality. Yet differences in annotation, resources, and evaluation complicate comparisons, and aggregate scores can obscure specific weaknesses. We review ten system descriptions from the NVVSpeech Challenge 2026, covering understanding and generation in Chinese and English across 16 event categories. Our analysis examines supervision construction, task adaptation, and inference-time or preference-based feedback. Results reveal persistent recognition difficulties with breath and gasp and trade-offs between event recognition and lexical accuracy, as well as between generation intelligibility and acoustic quality, with human listening providing complementary evaluation evidence. These findings identify priorities for advancing NVV-aware speech understanding, controllable generation, and comparable evaluation.
Liumeng Xue, Yuang Cao, Jingbin Hu +12
School of Intelligence Science and Technology, Nanjing University · State Key Laboratory of Novel Software Technology, Nanjing University · Shenzhen Pimei Technology +5