Affectron: Emotional Speech Synthesis with Affective and Contextually Aligned Nonverbal Vocalizations

Deok-Hyeon Cho; Hyung-Seok Oh; Seung-Bin Kim; Seong-Whan Lee

cs.SDMar 15, 2026

Affectron: Emotional Speech Synthesis with Affective and Contextually Aligned Nonverbal Vocalizations

Authors: Deok-Hyeon Cho, Hyung-Seok Oh, Seung-Bin Kim, Seong-Whan Lee

Organizations: Department of Artificial Intelligence, Korea University, Seoul, Korea

Abstract

Nonverbal vocalizations (NVs), such as laughter and sighs, are central to the expression of affective cues in emotional speech synthesis. However, learning diverse and contextually aligned NVs remains challenging in open settings due to limited NV data and the lack of explicit supervision. Motivated by this challenge, we propose Affectron as a framework for affective and contextually aligned NV generation. Built on a small-scale open and decoupled corpus, Affectron introduces an NV-augmented training strategy that expands the distribution of NV types and insertion locations. We further incorporate NV structural masking into a speech backbone pre-trained on purely verbal speech to enable diverse and natural NV synthesis. Experimental results demonstrate that Affectron produces more expressive and diverse NVs than baseline systems while preserving the naturalness of the verbal speech stream.

Figures & tables

Figure 1: Analysis of emotional attribute change patterns across temporal gaps (up to a gap of 10) for the EARS Richter et al. (2024) dataset (verbal only) and the NonverbalTTS Borisov et al. (2025) dataset (verbal-nonverbal combined speech).

Figure 2: Overview of the Affectron training framework. NV candidates are selected and routed to contextually appropriate locations to construct NV-augmented training samples, which are then used to fine-tune the VoiceCraft backbone for affect-aware NV synthesis.

Figure 3: AB preference test comparing the proposed NV augmentation with rule-guided randomized strategies following CapSpeech Wang et al. (2025a) .

Method	DA	EDNM	EAR	NSM	Nonverbal Metrics				Verbal Metrics
Method	DA				NV-Acc ( $\uparrow$ )	NV-Sim ( $\uparrow$ )	EECS ( $\uparrow$ )	SECS ( $\uparrow$ )	WER ( $\downarrow$ )	EECS ( $\uparrow$ )	SECS ( $\uparrow$ )
Seen Speakers
Augmented GT	-	-	-	-	85.96	-	0.5796	0.9231	1.13	0.6186	0.9163
VoiceCraft Peng et al. (2024)	✗	✗	✗	✗	10.49	0.5898	0.6149	0.8950	9.05	0.6212	0.8927
Affectron (Proposed)	✓	✗	✗	✗	58.78	0.5988	0.5455	0.8927	6.06	0.6190	0.8950
	✓	✓	✗	✗	35.83	0.6085	0.5648	0.8897	6.02	0.6126	0.8892

Table 1: Experimental results of the proposed method for both seen and unseen speakers. DA, EDNM, EAR, and NSM denote data augmentation, emotion-driven top-

K

NV matching, emotion-aware top-

K

routing, and NV structural masking, respectively. Augmented GT applies our NV augmentation to the ground truth.

Figure 4: Comparison of our proposed method in terms of NTN-MOS and NEC-MOS. Augmented GT applies our NV augmentation to the ground truth. Vertical lines illustrate the 95% confidence intervals.

Figure 5: Comparison of the diversity of generated fine-grained filler variations across models.

Method	Acc@1 ( $\uparrow$ )		Acc@3 ( $\uparrow$ )		Acc@5 ( $\uparrow$ )		JSD ( $\downarrow$ )		HD ( $\downarrow$ )
Method	Location	Type	Location	Type	Location	Type	Location	Type	Location	Type
Qwen 2.5-7B Hui et al. (2024)	7.48	25.65	14.13	47.03	25.30	69.39	0.1425	0.2139	0.3970	0.5027
LLaMA 3.1-8B Grattafiori et al. (2024)	5.46	16.03	18.76	49.88	21.85	77.32	0.4399	0.3020	0.3884	0.5741
GPT-oss-20B Agarwal et al. (2025)	26.84	16.98	50.24	51.19	62.47	72.68	0.1278	0.1130	0.4328	0.3523
Vicuna-7B Chiang et al. (2023)	13.78	11.28	23.63	41.57	39.07	69.48	0.1535	0.2431	0.3847	0.5176
Affectron-330M (Proposed)	12.59	75.77	29.69	85.99	44.06	91.69	0.0523	0.0051	0.2414	0.0723

Table 2: Comparison of zero-shot NV type and location prediction across LLMs and the Affectron model. Prediction accuracy is measured using top-

K

accuracy (Acc@1/3/5) and distributional alignment is assessed using Jensen-Shannon Divergence (JSD) and Hellinger Distance (HD).

Figure 6: Comparison of emotional attribute change patterns across temporal gaps using angular distance, Cartesian distance, and spherical radial differences.

Method	Acc@1 ( $\uparrow$ )	Acc@3 ( $\uparrow$ )	Acc@5 ( $\uparrow$ )	JSD ( $\downarrow$ )	HD ( $\downarrow$ )
Cartesian Distance	11.05	28.86	41.69	0.0568	0.2517
Radial Difference	11.88	28.98	41.45	0.0544	0.2495
Angular Distance	12.59	29.69	44.06	0.0523	0.2414

Table 3: Comparison of zero-shot NV type and location prediction across distance metrics. Prediction accuracy is measured using top-

K

accuracy (Acc@1/3/5) and distributional alignment is assessed using Jensen-Shannon Divergence (JSD) and Hellinger Distance (HD).

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Acc@1 ( $\uparrow$ )	Acc@3 ( $\uparrow$ )	Acc@5 ( $\uparrow$ )	JSD ( $\downarrow$ )	HD ( $\downarrow$ )
Wav2Vec2-Base	75.15	85.04	86.94	0.0074	0.0881
HuBERT-Base	74.60	85.15	88.72	0.0104	0.1041
WavLM-Base	75.10	85.27	87.29	0.0071	0.0869
CLAP	40.93	58.84	73.43	0.1288	0.3870
Emotion2Vec	75.77	85.99	91.69	0.0051	0.0723

Appendix

Table 4: Comparison of alternative embedding choices for NV type prediction within the top-

K

NV matching framework.

NV Category	Angry	Disgusted	Fearful	Happy	Neutral	Sad	Surprised
Agreement	236	30	3	297	45	20	148
Anger	571	98	4	115	44	36	80
Congratulations	339	150	8	429	61	50	470
Filler	72	89	–	91	185	56	144
Greetings	151	30	2	254	27	15	313
Cheering	32	16	11	181	7	10	83

Appendix

Table 5: Distribution of Emotion2Vec-based pseudo-label predictions across NV categories in the EARS dataset.

Figure 7: Overview of the emotion-aware top- $K$ routing process. (a) Word-level emotional attributes are extracted from verbal speech using MFA and an emotional attribute predictor. (b) Affective distance computation between an NV candidate and verbal segments via angular distance in spherical coordinates.

Method	Nonverbal Metrics				Verbal Metrics
Method	NV-Acc ( $\uparrow$ )	NV-Sim ( $\uparrow$ )	EECS ( $\uparrow$ )	SECS ( $\uparrow$ )	WER ( $\downarrow$ )	EECS ( $\uparrow$ )	SECS ( $\uparrow$ )
Augmented GT	89.29	-	0.5487	0.9092	2.93	0.6733	0.8995
VoiceCraft-330M ♠ Peng et al. (2024)	-	-	-	-	8.89	0.5410	0.8483
CosyVoice2-0.5B ♠ Du et al. (2024)	25.00	0.4097	0.4991	0.8751	1.97	0.4876	0.8739
Fun-CosyVoice3-0.5B ♠ Du et al. (2025)	27.38	0.4222	0.4792	0.8952	1.65	0.4904	0.8916
Dia-1.6B ♠ Nari-labs (2025)	13.10	0.4894	0.4549	0.8484	11.5	0.5204	0.8690

Appendix

Table 6: Comparison of NV-capable zero-shot TTS models on unseen speakers in the EARS dataset Richter et al. (2024) zero-shot evaluation setting.

\spadesuit

denotes a pre-trained model using the official implementation without any training or fine-tuning on the EARS dataset.

\clubsuit

denotes a model fine-tuned on the EARS dataset based on the official pre-trained checkpoint.

Method	Nonverbal Metrics				Verbal Metrics
Method	NV-Acc ( $\uparrow$ )	NV-Sim ( $\uparrow$ )	EECS ( $\uparrow$ )	SECS ( $\uparrow$ )	WER ( $\downarrow$ )	EECS ( $\uparrow$ )	SECS ( $\uparrow$ )
Augmented GT	89.29	-	0.5487	0.9092	2.93	0.6733	0.8995
Cross-speaker NV mixing	14.29	0.4939	0.4731	0.8571	9.40	0.5520	0.8604
Same-speaker NV mixing	16.67	0.4819	0.4979	0.8675	8.89	0.5518	0.8660

Appendix

Table 7: Ablation study on cross-speaker and same-speaker NV mixing in unseen-speaker zero-shot TTS synthesis.

Method	$K_{\text{EDNM}}$	$K_{\text{EAR}}$	Nonverbal Metrics				Verbal Metrics
Method			NV-Acc ( $\uparrow$ )	NV-Sim ( $\uparrow$ )	EECS ( $\uparrow$ )	SECS ( $\uparrow$ )	WER ( $\downarrow$ )	EECS ( $\uparrow$ )	SECS ( $\uparrow$ )
Seen Speakers
Augmented GT	-	-	85.96	-	0.5796	0.9231	1.13	0.6186	0.9163
Affectron (Proposed)	1	1	27.92	0.5965	0.5684	0.8887	8.35	0.6148	0.8885
	3	3	32.18	0.5961	0.5673	0.8891	6.59	0.6216	0.8886
	5	5	36.16	0.5970	0.5656	0.8905	6.19	0.6155	0.8878

Appendix

Table 8: Grid search over the top-

K

hyperparameters for emotion-driven NV matching (EDNM) and emotion-aware routing (EAR). Augmented GT applies our NV augmentation to the ground truth.

Figure 8: Word-cloud visualization of generated fine-grained filler variations for VoiceCraft Peng et al. (2024) and Affectron. The size of each keyword corresponds to its relative frequency among the realized filler variants.

Figure 9: Sample prompt templates for NV type and location prediction.

Method	Acc@1 ( $\uparrow$ )		Acc@3 ( $\uparrow$ )		Acc@5 ( $\uparrow$ )		JSD ( $\downarrow$ )		HD ( $\downarrow$ )
Method	Location	Type	Location	Type	Location	Type	Location	Type	Location	Type
Qwen2-Audio-7B (Text only) Chu et al. (2024)	6.68	12.00	16.39	53.56	18.68	76.96	0.2862	0.4346	0.5704	0.7150
Qwen2-Audio-7B (Text and Speech) Chu et al. (2024)	7.36	12.83	18.05	57.60	18.41	80.17	0.2353	0.4338	0.5188	0.7023
Qwen2.5-Omni-7B (Text only) Xu et al. (2025)	6.53	12.47	12.11	43.94	27.32	69.90	0.2213	0.3334	0.4967	0.6138
Qwen2.5-Omni-7B (Text and Speech) Xu et al. (2025)	7.36	20.07	12.23	56.41	28.86	75.30	0.2155	0.3307	0.4961	0.6096
Affectron-330M (Proposed)	12.59	75.77	29.69	85.99	44.06	91.69	0.0523	0.0051	0.2414	0.0723

Appendix

Table 9: Comparison of zero-shot NV type and location prediction across LLMs and the Affectron model. Prediction accuracy is measured using top-

K

accuracy (Acc@1/3/5) and distributional alignment is assessed using Jensen-Shannon Divergence (JSD) and Hellinger Distance (HD).

Figure 10: Comparison of Mel-spectrograms generated by different models for the same utterance. Results are shown for a seen speaker and an unseen speaker. NV segments are highlighted in red, and NV - Sim scores are annotated in blue. EDNM, EAR, and NSM denote emotion-driven top- $K$ NV matching, emotion-aware top- $K$ routing, and NV structural masking, respectively. Augmented GT applies our affectively aligned NV augmentation to the ground truth.

Figure 11: Detailed information on listener requirements and the AB preference test interfaces.

Figure 12: Detailed information on listener requirements and NV-context emotional congruence (NEC-MOS) evaluation interfaces.

Figure 13: Detailed information on listener requirements and NV-type naturalness (NTN-MOS) evaluation interfaces.

Nonverbal Vocalization Generation

See all 8 papers

Speech LLMs That Breathe: Modeling Non-Verbal Vocalizations in Dialogue
Oct 19, 2026Heyang Liu, Lixin Zhang, Jiayi Huang +6Nonverbal Vocalization Generation Speech Language Models
Shanghai Jiao Tong University · Ant Group
Preference Optimization for Non-Verbal Vocalization Synthesis
Aug 25, 2026Haoyang Li, Chenglin Xu, Junchuan Zhao +4Nonverbal Vocalization Generation Preference Optimization
Nanyang Technological University, Singapore · LIGHTSPEED, Singapore · National University of Singapore, Singapore +2
Modeling, Scaling, and Decoding: Optimizing Controllable Speech Generation with Nonverbal Vocalizations
Sep 13, 2026Ziyu Zhang, Yun Chen, Taihui Wang +5Nonverbal Vocalization Generation Controllable Speech Generation
Tencent HY Speech Team

Oct 19, 2026J/K move · Enter open · S save

Speech LLMs That Breathe: Modeling Non-Verbal Vocalizations in Dialogue

Heyang Liu, Lixin Zhang, Jiayi Huang, Ziyang Cheng+5

Shanghai Jiao Tong University · Ant Group

Nonverbal Vocalization Generation Speech Language Models Spoken Dialogue Systems Audio Generation Expressive Speech Synthesis

Open paper AskPDF arXiv

The user experience of speech interaction models has improved, benefiting from the modeling of paralinguistic attributes and advances in speech reconstruction paradigms. However, human communication follows extensive norms regarding non-verbal vocalizations (NVVs) spanning respiratory, affective, and discourse. The omission of these events imposes inherent limitations on the naturalness of current approaches. In this work, we cover 13 major types of NVVs and construct a natural dialogue dataset that conforms to human conversational norms through an automated synthesis pipeline. We enhance the ability of speech interaction models to generate contextually appropriate NVV events through multi-stage fine-tuning and NVV-aware GRPO (Group Relative Policy Optimization). Based on distinct modeling paradigms, we substantially improve the expressiveness of VocalNet and VITA-Qinyu in empathy interactions and role-play dialogues, and further discuss the architectural necessity for achieving high-quality natural speech generation.

Abstract

Nonverbal Vocalization Generation

Explore similar work

Speech LLMs That Breathe: Modeling Non-Verbal Vocalizations in Dialogue

Preference Optimization for Non-Verbal Vocalization Synthesis

Modeling, Scaling, and Decoding: Optimizing Controllable Speech Generation with Nonverbal Vocalizations