VoiceWeaver: Staged Learning of Structured Controls for Expressive Speech and Sound-Event Generation

Xiaosu Su; Yun Cao; Yiping Ni; Xiaowei Yi

cs.SDOct 3, 2026

VoiceWeaver: Staged Learning of Structured Controls for Expressive Speech and Sound-Event Generation

Authors: Xiaosu Su, Yun Cao, Yiping Ni, Xiaowei Yi

Organizations: Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China

Abstract

Adding expressive and environmental controls to a speech generator requires learning heterogeneous attributes without losing earlier capabilities. VoiceWeaver addresses this problem with structured label prefixes and staged emotion-tone-event training in a shared text-audio model. Replay-based distillation retains earlier predictions, while embedding decorrelation and attribute dropout regularize conditioning. Evaluation separates single-attribute correctness, joint emotion-event generation, and three-attribute correctness. Rounded to whole percentage points, emotion accuracy is 86% in Chinese and 83% in English. Chinese emotion-event joint accuracy is about 70%, versus 56% for mixed-task training and 48% for Ming-omni-tts; English joint accuracy is 66%. Three-attribute accuracy is 59% in Chinese and 55% in English on 500 samples each (57% pooled). Text error rates exceed those of the external TTS baselines. Audio samples are available at https://anonymous.4open.science/api/repo/VoiceWeaver1-4D88/file/index.html.

Figures & tables

Figure 1: VoiceWeaver architecture and token types. The autoregressive backbone predicts codebook 0; the lightweight decoder predicts codebooks 1–15 before codec reconstruction.

Figure 2: Three-stage training. Emotion, tone, and sound-event controls are introduced in sequence; replay and frozen teachers retain earlier capabilities.

Model	ZH	EN	ZH	ZH	EN
	Emotion $\uparrow$		Tone $\uparrow$	CER $\downarrow$	WER $\downarrow$
No labels	19	17	23	4.18	4.53
IndexTTS2	83	80	–	1.88	2.34
CosyVoice 2	75	72	–	1.52	2.57
Ming-omni-tts	77	75	–	2.14	2.82
VoiceWeaver	86	83	70	4.31	4.68

Table 1: Expressive control and intelligibility (%). Classification accuracies are rounded to whole points; CER/WER retain two decimals. Tone is evaluated in Chinese; “–” denotes no matched tone result.

Lang.	Model	Event	Joint	MC	CN
ZH	Ming-omni-tts	63	48	46	3.74
	VoiceWeaver	79	70	67	4.21
EN	Ming-omni-tts	62	46	43	3.68
	VoiceWeaver	76	66	64	4.14

Table 2: Emotion–event joint generation. Event, Joint, and MC are rounded percentages; CN is rated from 1 to 5. All columns: higher is better. Joint and MC do not include tone correctness.

Training / variant	Emo.	Tone	Event	Joint
Full (E–T–V)	86	70	79	70
E–V–T	84	67	75	63
T–E–V	84	65	74	59
Mixed tasks	82	62	72	56
w/o retention distillation	83	67	76	63
w/o decorrelation	83	68	75	64

Table 3: Training orders and ablations (rounded %). E/T/V: emotion/tone/event. Joint is Chinese emotion–event accuracy; single-attribute emotion and tone use their respective test sets.

Controllable Speech Generation

Controllable speech generation steers synthesized speech's voice, prosody, emotion, timing, pronunciation, and conversational delivery.

See all 115 papers

Steerspeech: Activation Steering For Emotion Control In Generated Speech
Oct 7, 2026Afsara Benazir, Darius Pétermann, Felix Xiaozhu Lin +1Controllable Speech Generation Emotional Speech Synthesis
University of Virginia · Netflix, Inc.
JoyAI-Voice 2.0: A Full-Continuous Autoregressive Speech Generation Model with Semantic-Acoustic Joint Representation
Sep 29, 2026Yafeng Chen, Boya Dong, Yankun Huang +17Audio Generation TTS Synthesis
Intern · Joy Future Academy, JD.com
Post-Training Zero-Shot TTS for Fine-Grained Emotion and Duration Control via Natural Language
Sep 10, 2026Lianru Gao, Yujie Guo, Yong QinTTS Synthesis Zero-Shot TTS
TMCC, College of Computer Science, Nankai University, Tianjin, China

Oct 7, 2026J/K move · Enter open · S save

Steerspeech: Activation Steering For Emotion Control In Generated Speech

Afsara Benazir, Darius Pétermann, Felix Xiaozhu Lin, Salar Rahili

University of Virginia · Netflix, Inc.

Controllable Speech Generation Emotional Speech Synthesis TTS Synthesis Activation Steering

Open paper AskPDF arXiv

Pretrained text-to-speech (TTS) models can generate expressive speech, but reliable inference-time emotion control remains challenging: prompts and reference audio offer coarse, inconsistent control, whereas specialized conditioning and model adaptation require costly training. We present SteerSpeech, a lightweight activation-steering framework that controls emotion by injecting steering vectors into hidden activations. For each target emotion we train a lightweight low-rank transform, using a multi-expert objective that encourages monotonic emotion control while preserving speaker identity and linguistic content, constraining steering drift, and keeping the TTS backbone frozen. To optimize through discrete speech tokens, we introduce a two-pass generation-and-replay pipeline using a straight-through estimator to backpropagate expert supervision through sampled tokens. At inference, a target-emotion steering direction is optimized with its respective transform and injected into the base TTS model. Objective and subjective evaluations with Qwen3-TTS across seen, unseen, and accented speakers show stronger continuous emotion control with limited speaker and content degradation. SteerSpeech achieves 1.08x-7.12x baseline target-emotion scores and for a representative emotion subjectively, it receives 78.1%-96.8% intensity preference and 1.43x-1.46x speaker-identity preservation at high steering strengths.

Abstract

Controllable Speech Generation

Explore similar work

Steerspeech: Activation Steering For Emotion Control In Generated Speech

JoyAI-Voice 2.0: A Full-Continuous Autoregressive Speech Generation Model with Semantic-Acoustic Joint Representation

Post-Training Zero-Shot TTS for Fine-Grained Emotion and Duration Control via Natural Language