Neuron-Level Emotion Control in Speech-Generative Large Audio-Language Models
Authors: Xiutian Zhao, Ismail Rasim Ulgen, Philipp Koehn, Björn Schuller, Berrak Sisman
Organizations: Center for Language and Speech Processing (CLSP), Johns Hopkins University, USA · Group on Language, Audio & Music (GLAM), Imperial College London, UK
Large audio-language models (LALMs) can produce expressive speech, yet reliable emotion control remains elusive: conversions often miss the target affect and may degrade linguistic fidelity through refusals, hallucinations, or paraphrase. We present, to our knowledge, the first neuron-level study of emotion control in speech-generative LALMs and demonstrate that compact emotion-sensitive neurons (ESNs) are causally actionable, enabling training-free emotion steering at inference time. ESNs are identified via success-filtered activation aggregation enforcing both emotion realization and content preservation. Across three LALMs (Qwen2.5-Omni-7B, MiniCPM-o 4.5, Kimi-Audio), ESN interventions yield emotion-specific gains that generalize to unseen speakers and are supported by automatic and human evaluation. Controllability depends on selector design, mask sparsity, filtering, and intervention strength. Our results establish a mechanistic framework for training-free emotion control in speech generation.
Figures & tables
Figure 1: EVC with LALMs is inherently multi-objective: successful conversion requires both target-emotion realization and linguistic content preservation.
Figure 2: The overview of our four-stage pipeline for identifying and manipulating emotion-sensitive neurons in LALMs for EVC.
Speakers
S1–S6
S7–S8
S9–S10
Total
Identification
001–200 (200)
–
–
1200
Development
201–250 (50)
001–250 (250)
–
1100
Test-seen
251–350 (100)
251–350 (100)
–
800
Test-unseen
–
–
001–350 (350)
700
Table 1: Speaker and utterance-ID split of ESD. Utterance IDs are parallel across the five emotions for each speaker.
Qwen2.5-Omni-7B
MiniCPM-o 4.5
Kimi-Audio
Angry
0.13 / 10.93
3.73 / 12.93
10.60 / 16.00
Happy
45.40 / 10.13
2.13 / 5.33
39.60 / 7.87
Sad
52.20 / 21.60
2.33 / 9.80
42.89 / 31.93
Surprise
2.67 / 5.07
0.07 / 3.47
7.13 / 6.67
Avg.
25.10 / 11.93
2.07 / 7.88
15.62 / 7.13
WER
19.00
2.95
2.82
Table 2: Baseline EVC performance of the three evaluated LALMs on ESD, before any neuron-level intervention. We report the three EVC axes used throughout the paper: target-emotion match rate in % (emotion2vec / Qwen3), content preservation (WER in %), and naturalness (UTMOS) on its 1–5 scale (the higher the more natural).
Figure 3: (Left) Intervention effects on EVC, under activation steering ( c=50 , α=1.0 ). For each selector, we report changes in target-emotion match relative to the unintervened baseline in Table 2 , together with the corresponding changes in content preservation (WER) and post-intervention naturalness (UTMOS). The CAS-H row reports the results using human-filtered instances, see explanations in § 5.4 . (Right) Per-emotion emotion match rate changes by ESN selectors for Qwen2.5-Omni-7B , under the same intervention. In each heatmap, rows denote the identified ESN mask emotion ( emask ) and columns denote the instructed target emotion ( etgt ); entries are signed changes (pp) in emotion match rate relative to the unintervened baseline. Diagonal cells represent self-effects (where emask=etgt ), and off-diagonal cells represent cross-effects (averaged over all emask=etgt pairs).
Figure 4: Sensitivity to ESN selection rate r (Qwen2.5-Omni-7B, CAS, c=50 , steering, α=1.0 ).
Figure 5: Sensitivity to success instance size c (Qwen2.5-Omni-7B, CAS, r=0.5% , steering, α=1.0 ).
Figure 6: Comparison of ESN intervention methods (Qwen2.5-Omni-7B, CAS, c=50 , r=0.5% ). As explained in § 3.4 , α is method-specific, so we use different values to keep intervention strength roughly comparable across methods. Deactivation is parameter-free and hence reported without α .
Δ Emotion Match vs. Base (emotion2vec / Qwen3)
α
Self-Effect ↑
Cross-Effect Avg. ↓
Self–Cross Gap ↑
WER ↓ ( Δ )
UTMOS ↑ ( Δ )
0.3
+0.78 / +0.37
+0.48 / -0.09
+0.30 / +0.45
19.33 (+0.33)
4.00 (0.00)
0.5
+2.00 / +1.13
-0.11 / -0.22
+2.11 / +1.36
20.67 (+1.67)
4.00 (0.00)
1.0
+4.27 / +3.60
+0.34 / -0.05
+3.93 / +3.65
29.65 (+10.65)
3.99 (-0.01)
2.0
+8.89 / +20.67
-0.14 / +0.02
+9.02 / +20.64
203.65 (+184.65)
3.91 (-0.09)
Table 4: Emotion match and WER trade-off when increasing the intervention strength (Qwen2.5-Omni-7B, CAS, c=50 , r=0.5% , steering).
Figure 8: Human listening evaluation under the anchor intervention setting. Win/Tie/Loss rates comparing the intervened system (Qwen2.5-Omni-7B, CAS, c=50 , r=0.5% , steering, α=1.0 ) against the unintervened baseline. 20 participants.
Emotion
Jaccard
Direct Match
JSD
Angry
0.217
35.71%
0.009
Happy
0.075
13.95%
0.040
Sad
0.109
19.57%
0.010
Surprise
0.071
13.27%
0.025
Average
0.118
20.63%
0.021
Table 5: Similarity between ESNs identified from human- vs. SER-filtered instances , measured in Jaccard overlap, direct match rate, and Jensen–Shannon divergence (JSD).
Figure 9: Layer-wise distribution of CAS-identified ESNs across three LALMs ( r=0.5% ). Heatmaps show the number of selected ESNs per decoder layer and target emotion. Qwen2.5-Omni-7B and Kimi-Audio use 28-layer decoders, while MiniCPM-o 4.5 uses a 36-layer decoder. Values indicate where selected neurons are concentrated, not intervention effect size.
Δ Emotion Match vs. Base (emotion2vec / Qwen3)
Selector
Self-Effect ↑
Cross-Effect Avg. ↓
Self–Cross Gap ↑
WER ↓ ( Δ )
UTMOS ↑ ( Δ )
Random
+0.52 / +0.02
–
–
19.21 (+0.21)
4.01 (+0.01)
LAP
+0.32 / -0.08
+0.12 / +0.23
+0.20 / -0.32
19.00 (0.00)
3.96 (-0.04)
LAPE
-0.87 / -0.22
-0.13 / -0.27
-0.73 / +0.06
19.00 (0.00)
4.01 (+0.01)
MAD
-0.68 / 0.08
+0.51 / +0.13
-1.19 / -0.04
19.00 (0.00)
4.00 (0.00)
CAS
+0.12 / -0.42
+0.45 / -0.19
-0.33 / -0.22
19.19 (+0.19)
3.99 (-0.01)
Table 6: Activation steering in the speech synthesis module of Qwen2.5-Omni-7B. Results obtained by applying the same settings ( c=50 , r=0.5% , α=1.0 ) as in Table 3 , but identified and intervening on the synthesis module MLP layers.
Figure 10: Intervention effect on seen and unseen speakers. Δ Emotion match size changes (pp) of intervention (Qwen2.5-Omni-7B, CAS, c=50 , r=0.5% , steering, α=1.0 ) in comparison with the unintervened baseline are shown for seen speakers ( Test-seen split) and unseen speakers ( Test-unseen split), with 95% confidence intervals.