cs.CLNov 14, 2026

A Systematic Study of Cross-Token Transfer for Data-Efficient Non-Verbal Text-to-Speech

Authors: Jing Pan, Haibin Wu, Minxue Niu, Florian Metze, Yuzong Liu

Organizations: Meta, USA

Abstract

Non-verbal vocalizations (NVs), such as laughter and breath, carry affective cues that words alone do not convey. Yet NV data remains scarce because speech annotation pipelines often omit or normalize these events. Existing work has improved NV text-to-speech through model design and data curation, but whether one NV token can help a model learn another has received little systematic study. We investigate this cross-token transferability in a controlled CosyVoice 2 setup. We first trace a laughter-only scaling curve, then hold laughter at 200 or 1,600 examples while adding breath, sigh, cough, lip-smack, or neutral speech. With only 200 laughter examples, adding 3,200 cough examples raises end-to-end laughter F1 from 0.170 to 0.529, surpassing the 0.495 obtained from 1,600 laughter examples alone. At the 1,600-example anchor, cough reaches 0.695, nearly matching 6,400 laughter examples (0.697). Waveform-level Whisper-D scoring and a semantic-token transcriber show consistent trends. Cross-token transfer therefore offers a complementary route to low-resource NV generation.

Figures & tables

Explore similar work

CardsList
  1. Preference Optimization for Non-Verbal Vocalization Synthesis

    Aug 25, 2026Haoyang Li, Chenglin Xu, Junchuan Zhao +4Nonverbal Vocalization GenerationPreference Optimization

  2. Affectron: Emotional Speech Synthesis with Affective and Contextually Aligned Nonverbal Vocalizations

    Mar 15, 2026Deok-Hyeon Cho, Hyung-Seok Oh, Seung-Bin Kim +1Nonverbal Vocalization GenerationEmotional Speech Synthesis

  3. Speech LLMs That Breathe: Modeling Non-Verbal Vocalizations in Dialogue

    Oct 19, 2026Heyang Liu, Lixin Zhang, Jiayi Huang +6Nonverbal Vocalization GenerationSpeech Language Models