Steering Follows Geometry, Not Labels: Emotion Directions in a Full-Duplex Speech Model

Pulak Kuli

cs.CLOct 6, 2026

Steering Follows Geometry, Not Labels: Emotion Directions in a Full-Duplex Speech Model

Authors: Pulak Kuli

Organizations: Independent Researcher

Abstract

Full-duplex voice agents need to modulate emotion and delivery during real-time conversations, when de-escalating a complaint, carrying urgency in dispatch, softening a clinical result. Emotion and delivery control is well studied for TTS and turn based models through prompt-conditioned synthesis, reference-conditioned synthesis and activation steering; PersonaPlex controls identity in a duplex model but not affect. We study emotion steering in Moshi, a fully open sourced full-duplex speech language model, across four emotions, using mean-difference activation steering, which costs only a few vector additions per frame and no retraining. We show that emotion is linearly decodable from Moshi's residual stream, but activation steering is only partially achievable, and unevenly so; as happy, angry and surprise steer towards a shared direction while sad is distinctly steerable. We also show that the shared component across the three emotions cannot simply be projected away from all the emotions equally.

Figures & tables

Panel of Figure 1 — Figure 1: Where the vector enters. Moshi is frozen; the only change is one direction added to the output of the three layers marked in red - 0, 8 and 24 - every frame from speech onset. The injection sits upstream of the split into the text head and the audio depformer, so it can shape delivery without touching the codec.

contrast	chance	L0	L8	L24	best (layer)
angry vs happy	0.500	0.567	0.692	0.750	0.767 (27)
angry vs surprise	0.500	0.717	0.908	0.875	0.925 (11)
happy vs surprise	0.500	0.717	0.758	0.742	0.792 (17)
angry vs sad	0.500	0.767	0.867	0.850	0.892 (31)
happy vs sad	0.500	0.800	0.892	0.892	0.933 (13)
sad vs surprise	0.500	0.783	0.942	0.933	0.967 (13)

Table 1: Probe accuracy on the same states the directions are built from, L0, L8 and L24 are the layers the steering vectors are injected into; the last column gives the best of all 32 layers, with that layer in brackets. Bold is the strongest of the three injected layers; shaded cells sit within 30% of the headroom above chance.

condition	P	arg%	P(tgt) (% c.)	arg%	rival	WER	P(tgt) (% c.)	arg%	rival	WER
	real speech		forced text				free duplex
unsteered	0.924	92.2	0.635	65.7	—	24.3	0.505	48.6	—	10.7
angry steered	0.717	71.9	0.177 (25%)	18.3	0.328 happ	32.2	0.189 (26%)	19.2	0.370 happ	13.5
happy steered	0.633	64.1	0.481 (76%)	49.3	0.276 surp	61.8	0.331 (52%)	31.4	0.150 surp	20.2
sad steered	0.448	45.3	0.650 (145%)	68.6	0.037 happ	51.4	0.313 (70%)	32.1	0.199 happ	23.5
surprise steered	0.458	45.3	0.136 (30%)	13.6	0.463 happ	71.5	0.143 (31%)	13.3	0.464 happ	35.6

Table 2: Steering with the text forced (720 sentences per condition) and in free duplex dialogue (50 per input emotion). % c. is the percentage of the real-speech ceiling reached, arg% the share of clips whose top label is the requested emotion. Shading on a rival marks where it beats that emotion. See Appendix A .

condition	P(neutral)	P(happy)	WER%	voiced s
unsteered	0.599	0.181	24.7	1.98
random 0	0.569	0.159	26.1	1.93
random 1	0.526	0.197	21.8	1.98
random 2	0.601	0.189	21.3	1.98
random 3	0.537	0.237	28.7	1.96
angry steered	0.042	0.328	32.2	2.32

Table 3: Four random directions injected at the same norm, through the same hooks and schedule as the emotion directions, on 240 forced-text sentences each.

condition	angry	happy	sad	surpr.	neutral
unsteered	0.001	0.163	0.035	0.004	0.635
angry	0.177	0.328	0.032	0.084	0.042
real	0.717	0.030	0.001	0.001	0.154
happy	0.007	0.481	0.057	0.276	0.018
real	0.005	0.633	0.002	0.006	0.160
sad	0.008	0.037	0.650	0.015	0.134

Table 4: emotion2vec probability by label: forced text on the left, free duplex on the right. Each steered row is followed by real ESD speech of that emotion, the most any clip could score. Cell depth is proportional to the value in that emotion’s colour; underline marks the emotion requested and bold the largest in the row.

condition	P(tgt)	rival ( $n$ )	P(tgt)	rival ( $n$ )	P(tgt)	rival
	first 1.2 s		first 2.0 s		whole clip
angry steered	0.122	0.241 happ (710)	0.154	0.248 happ (555)	0.177	0.328 happ
happy steered	0.409	0.080 sad (718)	0.433	0.086 surp (676)	0.481	0.276 surp
sad steered	0.361	0.084 happ (717)	0.450	0.051 surp (699)	0.650	0.037 happ
surprise steered	0.125	0.376 happ (714)	0.104	0.333 happ (600)	0.136	0.463 happ

Table 5: Forced-text clips scored over a growing prefix rather than in full, against the strongest rival in each window. Bold is the larger of the two; shading marks the windows where a rival beats the requested emotion.

n

falls with the window because not every clip is that long.

condition	angry	happy	sad	surpr.	neutral
unsteered	-0.363	0.118	0.305	-0.414	0.579
angry steered	-0.023	0.284	-0.043	-0.048	-0.118
real angry	0.677	-0.199	-0.075	-0.353	0.011
happy steered	-0.261	0.394	-0.197	0.330	-0.339
real happy	-0.182	0.563	0.036	-0.287	0.067
sad steered	-0.262	-0.018	0.665	-0.422	0.228

Table 6: Centred cosine to the centroid of real ESD speech of each emotion, unsteered baseline in the first row; shading, underline and bold as in Table 4 . For angry and surprise the nearest centroid is not the requested one.

Panel of Figure 2 — Figure 2: Cosine between emotion directions at layer 24. Left: directions built from all ten speakers, so each pair shares voices. Right: built from disjoint halves, so none do; the bold diagonal is an emotion against itself and margin its lead over the nearest other.

condition	$\beta$	P(tgt)	% ceil.	strongest rival	WER%
angry steered	1.0	0.184	26	0.334 (happ)	32.2
	0.5	0.288	40	0.162 (happ)	22.6
	0.0	0.016	2	0.040 (happ)	23.0
happy steered	1.0	0.503	79	0.279 (surp)	62.4
	0.5	0.510	80	0.175 (surp)	65.2
	0.0	0.205	32	0.115 (sad)	73.5

Table 7: Keeping a fraction

\beta

of the direction the emotions share. Bold marks the settings where the requested emotion beats its strongest rival. clips with

\geq 1.5

s of voiced audio. The intermediate

\beta=0.25

is in Appendix C .

emotion	recording ( $n$ )	prime ( $n$ )	continuation ( $n$ )	prime/rec.%	gains most	silent%
angry	0.791 (67)	0.315 (59)	0.073 (15)	40	neutral 0.06 $\rightarrow$ 0.27	85
happy	0.663 (64)	0.563 (52)	0.234 (16)	85	neutral 0.03 $\rightarrow$ 0.12	84
sad	0.720 (68)	0.470 (63)	0.314 (15)	65	neutral 0.17 $\rightarrow$ 0.33	85
surprise	0.593 (75)	0.459 (66)	0.151 (14)	77	happy 0.08 $\rightarrow$ 0.20	86

Table 8: Prime and release, 100 recordings per emotion, over clips clearing the 1.5 s voiced threshold (

n

in brackets). gains most is the label taking the largest share of what the re-rendering loses. See Appendix A .

condition	P(tgt)	real speech	% real	through codec	% codec
angry	0.177	0.717	25	0.315	56
happy	0.481	0.633	76	0.563	85
sad	0.650	0.448	145	0.470	138
surprise	0.136	0.458	30	0.459	30

Table 9: Steering against two references: real ESD speech, and the model’s own re-rendering of it. The real speech column is measured differently from Table 8 ’s recording column; Appendix A says how.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

run	condition	kept	total	dropped%
forced text	unsteered	662	720	8.1
forced text	angry	710	720	1.4
forced text	happy	718	720	0.3
forced text	sad	717	720	0.4
forced text	surprise	714	720	0.8
duplex	unsteered	72	250	71.2

Appendix

Table 10: Clips reaching the 1.5 s voiced-audio threshold.

condition	$\beta$	P(tgt)	% ceil.	strongest rival	WER%
angry steered	1.0	0.184	26	0.334 (happ)	32.2
	0.5	0.288	40	0.162 (happ)	22.6
	0.25	0.119	17	0.081 (happ)	15.3
	0.0	0.016	2	0.040 (happ)	23.0
happy steered	1.0	0.503	79	0.279 (surp)	62.4
	0.5	0.510	80	0.175 (surp)	65.2

Appendix

Table 11: All four settings of

\beta

Explore similar work

Steerspeech: Activation Steering For Emotion Control In Generated Speech
Oct 7, 2026Afsara Benazir, Darius Pétermann, Felix Xiaozhu Lin +1
University of Virginia · Netflix, Inc.
SteerDuplex: Steerable Duplex Speech Dialogue Models
Sep 14, 2026Utkarsh Tyagi, Ramaneswaran Selvakumar, Advait Gosai +13Full-Duplex Speech Models Turn-Taking
Scale AI · University of Maryland
A Geometric Perspective on Composable Emotion Steering in Text-to-Speech Models
Jul 1, 2026Siyi Wang, James Bailey, Ting DangSpeech Language Models Emotion Recognition
The University of Melbourne, Australia · Monash University, Australia

Oct 7, 2026, cs.SDJ/K move · Enter open · S save

Steerspeech: Activation Steering For Emotion Control In Generated Speech

Afsara Benazir, Darius Pétermann, Felix Xiaozhu Lin, Salar Rahili

University of Virginia · Netflix, Inc.

Open paper AskPDF arXiv

Pretrained text-to-speech (TTS) models can generate expressive speech, but reliable inference-time emotion control remains challenging: prompts and reference audio offer coarse, inconsistent control, whereas specialized conditioning and model adaptation require costly training. We present SteerSpeech, a lightweight activation-steering framework that controls emotion by injecting steering vectors into hidden activations. For each target emotion we train a lightweight low-rank transform, using a multi-expert objective that encourages monotonic emotion control while preserving speaker identity and linguistic content, constraining steering drift, and keeping the TTS backbone frozen. To optimize through discrete speech tokens, we introduce a two-pass generation-and-replay pipeline using a straight-through estimator to backpropagate expert supervision through sampled tokens. At inference, a target-emotion steering direction is optimized with its respective transform and injected into the base TTS model. Objective and subjective evaluations with Qwen3-TTS across seen, unseen, and accented speakers show stronger continuous emotion control with limited speaker and content degradation. SteerSpeech achieves 1.08x-7.12x baseline target-emotion scores and for a representative emotion subjectively, it receives 78.1%-96.8% intensity preference and 1.43x-1.46x speaker-identity preservation at high steering strengths.

Abstract

Explore similar work

Steerspeech: Activation Steering For Emotion Control In Generated Speech

SteerDuplex: Steerable Duplex Speech Dialogue Models

A Geometric Perspective on Composable Emotion Steering in Text-to-Speech Models