EmoRES-TTS: Residual-Enhanced Vector Steering for Emotional Speech Generation

Kuan-Po Huang; Haohe Liu; Puyuan Peng; Haibin Wu; Zhaoheng Ni; Hung-yi Lee; Jinwon Lee; Neha Chachra

cs.SDSep 29, 2026

EmoRES-TTS: Residual-Enhanced Vector Steering for Emotional Speech Generation

Authors: Kuan-Po Huang, Haohe Liu, Puyuan Peng, Haibin Wu, Zhaoheng Ni, Hung-yi Lee, Jinwon Lee, Neha Chachra

Organizations: Reality Labs at Meta · National Taiwan University · FAIR at Meta

Abstract

Emotion-conditioned text-to-speech (TTS) models may fail to express the requested emotion reliably, and improving controllability by additional training is costly in both computation and emotion-labeled speech training data. We therefore study vector steering, a training-free approach that modifies the internal representations of a frozen model. CoCoEmo, a conventional vector steering method for emotion TTS, treats each emotion vector as an indivisible direction controlled by a single global strength, limiting adherence to the requested emotion. In this work, we first discover that an emotion vector can be decomposed into a shared component that moves speech away from neutral expression and a residual component that directs generation toward the requested emotion. Building on this finding, we propose Emotion Residual-Enhanced Steering for TTS (EmoRES), a novel method that controls the two components without retraining the backbone. On IEMOCAP, EmoRES outperforms CoCoEmo across all four objective emotion metrics on the IndexTTS-2 and CosyVoice2 backbones. Rank correlation improves by 26.13 and 12.97 percentage points, corresponding to relative gains of 118.8% and 33.1%, while emotion hit rate improves by 12.95 and 6.92 points, corresponding to relative gains of 20.1% and 9.8%. Human evaluation further shows a relative improvement up to 35.0% in the rate at which listeners correctly identified the dominant requested emotion and up to a 17.3% improvement in fidelity, while listeners prefer EmoRES for naturalness in up to 63.8% of pairwise comparisons. Component ablations further demonstrate that effective control benefits from preserving the shared component while strengthening the residual of the emotion steering vectors.

Figures & tables

Figure 1: Residual weight against steering strength

\alpha

on IEMOCAP (OOD) with IndexTTS-2 for each objective metric. Each curve is fixed at

\lambda_{c}=1

with

\lambda_{r}

ablated. Every arm is norm-matched, so at a given

\alpha

, all arms inject the same steering vector norm and differ only in the direction.

\lambda_{r}=1

is CoCoEmo’s steering method. Shaded bands are

\pm

one standard deviation over five different runs.

In-distribution evaluation on CREMA-D (dev)
Method	E-SIM $\uparrow$	TEP $\uparrow$	$\rho\uparrow$	H-Rate $\uparrow$	S-SIM $\uparrow$	WER $\downarrow$
Ground Truth	72.56	55.51	–	–	87.86	1.08
IndexTTS-2
No-steer	55.00 $\pm$ 0.36	16.83 $\pm$ 1.39	–	–	87.18 $\pm$ 0.16	6.30 $\pm$ 0.05
CoCoEmo	64.46 $\pm$ 0.40	38.69 $\pm$ 1.21	25.24 $\pm$ 4.52	76.00 $\pm$ 1.78	85.77 $\pm$ 0.22	7.60 $\pm$ 0.14
EmoRES, $\lambda_{r}=3$	64.80 $\pm$ 0.25	39.98 $\pm$ 0.88	45.87 $\pm$ 6.99	82.49 $\pm$ 2.34	86.90 $\pm$ 0.21	6.78 $\pm$ 0.16

Table 1: Mixed-emotion evaluation on CREMA-D (dev) and IEMOCAP (OOD). All steered rows use

\alpha=5

and

\lambda_{c}=1

. CoCoEmo is the

\lambda_{r}=1

special case. Values are percentages, reported as mean

\pm

standard deviation over five replicates. Best results among the synthesis rows are bold , second-best are underlined .

	IndexTTS-2		CosyVoice2
Method	Dom-hit $\uparrow$	Fidelity $\uparrow$	Dom-hit $\uparrow$	Fidelity $\uparrow$
No-steer	17.78 %	25.04 %	32.42 %	33.28 %
CoCoEmo	54.72 %	60.05 %	58.18 %	50.80 %
EmoRES	73.89 %	66.32 %	77.88 %	59.60 %
$\Delta$	$+$ 19.17 ∗	$+$ 6.27 ∗	$+$ 19.70 ∗	$+$ 8.80 ∗
95% CI of $\Delta$	[13.06, 25.56]	[3.44, 8.94]	[13.33, 26.07]	[5.91, 11.81]

Table 2: Subjective evaluation on IEMOCAP, comparing CoCoEmo (

\lambda_{r}=1

) against EmoRES (

\lambda_{r}=3

\Delta

is the difference between EmoRES and CoCoEmo. ∗ denotes statistical significance.

Method	$\lambda_{c}$	$\lambda_{r}$	E-SIM $\uparrow$	TEP $\uparrow$	$\rho$ $\uparrow$	H-Rate $\uparrow$	S-SIM $\uparrow$	WER $\downarrow$
Ground Truth	–	–	72.56	55.51	–	–	87.86	1.08
IndexTTS-2
No-steer	–	–	55.00 $\pm$ 0.36	16.83 $\pm$ 1.39	–	–	87.18 $\pm$ 0.16	6.30 $\pm$ 0.05
Residual only	$0$	$1$	63.05 $\pm$ 0.29	35.88 $\pm$ 0.77	45.16 $\pm$ 5.92	82.16 $\pm$ 2.06	87.64 $\pm$ 0.18	6.36 $\pm$ 0.06
Shared & Residual	$1$	$1$	64.46 $\pm$ 0.40	38.69 $\pm$ 1.21	25.24 $\pm$ 4.52	76.00 $\pm$ 1.78	85.77 $\pm$ 0.22	7.60 $\pm$ 0.14
Shared only	$1$	$0$	59.51 $\pm$ 0.33	26.64 $\pm$ 1.18	5.30 $\pm$ 7.86	69.62 $\pm$ 2.21	85.13 $\pm$ 0.22	6.69 $\pm$ 0.26

Table 3: Component ablation on CREMA-D in the mixed setting. Every steered row uses steering strength

\alpha=5

, norm-matched to the same steering vector norm and differs from the others only in how that norm is split between the shared and residual components and in the direction each component points. Mean

\pm

one standard deviation over five speaker-prompt runs. Best per column and backbone in bold. See Appendix D for the design and for the row contrasts.

Figure 2: Emotion2vec embedding t-SNE plots for utterances generated by IndexTTS-2 on the single-emotion CREMA-D subset (

n=486

). Each point is colored by its requested emotion. Values above each panel report the accuracy of probing and the mean cosine silhouette score.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Geometric illustration of EmoRES in a two-dimensional projection with an example of two emotions.

Figure 4: Layer-wise emotion separability and causal steering performance for CosyVoice2 on CREMA-D. The left plot shows the accuracy of logistic regression and nearest-centroid emotion probes. The right plot shows TEP when EmoRES is applied to one layer at a time using

\lambda_{c}=1

\lambda_{r}=3

, and

\alpha=5

. Stars mark the two layers with the highest probe accuracy, while shading marks the two layers with the highest TEP.

IndexTTS-2
	Method	$\lambda_{c}$	$\lambda_{r}$	E-SIM $\uparrow$	TEP $\uparrow$	$\rho$ $\uparrow$	H-Rate $\uparrow$	S-SIM $\uparrow$	WER $\downarrow$
(a)	Ground Truth	–	–	72.56	55.51	–	–	87.86	1.08
(b)	No-steer	–	–	55.00 $\pm$ 0.36	16.83 $\pm$ 1.39	–	–	87.18 $\pm$ 0.16	6.30 $\pm$ 0.05
(c)	Residual only	$0$	$1$	63.05 $\pm$ 0.29	35.88 $\pm$ 0.77	45.16 $\pm$ 5.92	82.16 $\pm$ 2.06	87.64 $\pm$ 0.18	6.36 $\pm$ 0.06
(d)	Shared randomized	$1\,\mathrm{rand}$	$1$	59.55 $\pm$ 0.42	28.50 $\pm$ 1.21	21.52 $\pm$ 5.27	74.49 $\pm$ 2.04	87.74 $\pm$ 0.15	6.62 $\pm$ 0.09
(e)	Shared & Residual	$1$	$1$	64.46 $\pm$ 0.40	38.69 $\pm$ 1.21	25.24 $\pm$ 4.52	76.00 $\pm$ 1.78	85.77 $\pm$ 0.22	7.60 $\pm$ 0.14

Appendix

Table 4: Component ablation on CREMA-D in the mixed-emotion setting with steering strength

\alpha=5

. Every steered row is norm-matched to the same steering vector magnitude and differs from the others only in how that norm is split between the shared and residual components and in the direction each component points. A coefficient written “

b\,\mathrm{rand}

” marks a component replaced by a Gaussian at its own norm. Such rows are controls and are excluded from the bold comparison. Mean

\pm

one standard deviation over five runs. Best per column and backbone among the unrandomized settings in bold. See Section D for the design and for the row contrasts.

In-distribution evaluation on CREMA-D (dev)
Method	E-SIM $\uparrow$	TEP $\uparrow$	S-SIM $\uparrow$	WER $\downarrow$
IndexTTS-2
neutral prompt
No-steer	59.21 $\pm$ 0.38	30.16 $\pm$ 1.37	88.63 $\pm$ 0.16	6.54 $\pm$ 0.10
CoCoEmo	68.08 $\pm$ 0.45	50.95 $\pm$ 1.78	84.82 $\pm$ 0.09	6.73 $\pm$ 0.12
EmoRES, $\lambda_{r}=3$	71.86 $\pm$ 0.40	62.27 $\pm$ 1.39	85.09 $\pm$ 0.22	8.47 $\pm$ 0.47

Appendix

Table 5: Single-emotion evaluation with neutral and target-emotion speaker prompts . All steered rows use

\alpha=5

and

\lambda_{c}=1

. CoCoEmo is the

\lambda_{r}=1

special case. Best among the synthesis rows is bold , second-best underlined .

In-distribution evaluation on CREMA-D (dev)
Method	E-SIM $\uparrow$	TEP $\uparrow$	$\rho\uparrow$	H-Rate $\uparrow$	S-SIM $\uparrow$	WER $\downarrow$
Ground Truth	72.56	55.51	–	–	87.86	1.08
IndexTTS-2
No-steer	47.84 $\pm$ 0.10	2.58 $\pm$ 0.27	–	–	89.38 $\pm$ 0.16	6.44 $\pm$ 0.02
CoCoEmo	52.77 $\pm$ 0.29	12.87 $\pm$ 0.95	11.43 $\pm$ 5.87	70.92 $\pm$ 1.93	89.85 $\pm$ 0.14	6.32 $\pm$ 0.08
EmoRES, $\lambda_{r}=3$	56.18 $\pm$ 0.23	20.51 $\pm$ 0.67	32.37 $\pm$ 6.39	78.27 $\pm$ 2.37	90.12 $\pm$ 0.13	7.13 $\pm$ 0.10

Appendix

Table 6: Mixed-emotion evaluation without native emotion conditioning : IndexTTS-2 runs with no emotion embeddings and CosyVoice2 without the instructions, so the injected steering vectors are the only explicit emotion-control signal. CoCoEmo is the

\lambda_{r}=1

special case. Mean

\pm

standard deviation over five replicates. Best among the synthesis rows is bold , second-best underlined .

IndexTTS-2
SER judge	Method	TEP $\uparrow$	$\rho\uparrow$	H-Rate $\uparrow$
emotion2vec+ large	No-steer	30.76 $\pm$ 0.44	–	–
	CoCoEmo	36.22 $\pm$ 0.80	22.00 $\pm$ 4.83	64.34 $\pm$ 2.39
	EmoRES, $\lambda_{r}=3$	41.11 $\pm$ 0.39	48.13 $\pm$ 2.44	77.29 $\pm$ 1.67
Qwen3-Omni	No-steer	37.08 $\pm$ 0.22	–	–
	CoCoEmo	39.79 $\pm$ 0.43	31.60 $\pm$ 3.24	66.33 $\pm$ 1.63

Appendix

Table 7: Mixed-emotion evaluation on IEMOCAP (OOD) under three different speech emotion recognizers. All steered rows use

\lambda_{c}=1

. Values are percentages, mean

\pm

standard deviation over five replicates. Recognizers differ in calibration, so absolute levels are not comparable across blocks. Best values per block are marked bold .

Explore similar work

Jun 3, 2026cs.SD

Task-Vector Arithmetic for Emotional Expressivity Control in Language-Model-Based Text-to-Speech

We investigate whether task-vector arithmetic, successful for cross-speaker emotional intensity control in modular text-to-speech (TTS), transfers to large-scale TTS systems built on language-model backbones with in-context learning (LM-TTS). Through a systematic elimination study over four progressively narrower operands on Qwen3-TTS-12Hz-1.7B - model weights via LoRA fine-tuning, continuous codec embeddings, discrete codec tokens, and the speaker embedding (x-vector) produced by an ECAPA-TDNN encoder jointly trained with the synthesis backbone - we localize the dominant carrier of emotional prosody to the x-vector. Building on this finding, we propose a training-free method based on centroid arithmetic in x-vector space: an emotion direction

τ= \mathbb{E}_i[x(s_i,\text{emo})] -\mathbb{E}_i[x(s_i,\text{neutral})]

applied to an unseen target speaker as

x_{\text{new}} = x(\text{target},\text{neutral}) + α\cdotτ

. Using ESD (English) as the

τ

source and emoUERJ (Brazilian Portuguese) as a cross-lingual ground-truth target, we observe average gains of

+0.29

in emotion2vec cosine over the ICL baseline on English held-out speakers and

+0.09

on Brazilian Portuguese held-out speakers, while largely preserving identity (WavLM SECS

\gtrsim 0.88

for the multi-speaker

τ

variant) and intelligibility (WER

\approx 0

in PT-BR). These results offer initial evidence that the dominant carrier of emotional prosody in this class of models is localizable, by elimination, to the co-trained speaker embedding, where training-free centroid arithmetic remains effective even under cross-lingual transfer.

Daniel Oliveira de Brito, Arnaldo Candido Junior

Instituto de Biociências, Letras e Ciências Exatas Universidade Estadual Paulista “Júlio de Mesquita Filho” (UNESP) São José do Rio Preto, SP, Brazil

Flow-Matching Text-To-Speech Text-To-Speech Synthesis Cross-Lingual Transfer Cosine Similarity

Ask PDF

Oct 4, 2026cs.SD

Tracing a Sparse Emotion-Control Circuit in LLM-Based Text-to-Speech

LLM-based text-to-speech (TTS) models can generate emotionally expressive speech, but how reference emotion is routed through the model and realized in decoded speech remains unclear. We introduce two emotion-sensitive metrics for matched neutral and emotional syntheses---a codec trajectory score and a late residual direction score---and use them to score activation-patching interventions. Under controlled matched-reference conditions, this analysis identifies a sparse source-to-readout component-level circuit: 23--27 attention heads and MLPs per emotion, roughly 5% of the components considered, recover or suppress 74--88% of the late emotion-readout shift on held-out cases. The circuit combines a shared component backbone with emotion-specific components; cross-emotion activation swaps reduce the target readout in 47 of 48 cases. In decoded speech, the same intervention produces consistent changes in pitch, energy, and spectral brightness over 24 matched pairs per emotion. A readout-matched residual-direction baseline produces only 17--27% of the intervention's pitch effect, showing that internal readout movement alone does not explain the decoded acoustic changes. These results trace a compact causal route from reference-derived prefix information to emotion-relevant properties of generated speech.

Hongfei Du, Jiacheng Shi, Yanfu Zhang +1

Department of Computer Science, William & Mary, Williamsburg, USA

Topic pending

Ask PDF

Jun 11, 2026cs.SD

Emo-LiPO: Listwise Preference Optimization for Fine-Grained Emotion Intensity Control in LLM-based Text-to-Speech

Large language model (LLM)-based text-to-speech (TTS) systems enable prompt-conditioned emotional control but struggle with fine-grained emotion intensity due to the semantic -- acoustic gap between text and speech. To address this challenge, we formulate emotion intensity control in LLM-based TTS as a learning-to-rank problem and propose Emo-LiPO, a listwise preference optimization framework that aligns prompt-conditioned speech generation with relative emotion intensity expressed in text. Emo-LiPO explicitly models global intensity ordering within each emotion under fixed transcripts, enabling more faithful and continuous emotional expression. We further construct ESD-plus, a multi-speaker dataset with explicit emotion intensity variations, to support fine-grained emotion modeling and evaluation. Experiments on ESD-plus demonstrate that Emo-LiPO significantly improves emotion accuracy and intensity controllability over both supervised- and DPO-based LLM TTS baselines, with particularly pronounced gains at high intensity levels.

Yihang Lin, Li Zhou, Congwei Cao +4

The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute · Agency for Science, Technology and Research +2

Emotion Recognition

Ask PDF