Emotion Across Speech and Faces: Shared Affective Mechanisms in Multimodal Foundation Models

Xiutian Zhao; Luqi Sun; Björn Schuller; Berrak Sisman

cs.CLAug 17, 2026

Emotion Across Speech and Faces: Shared Affective Mechanisms in Multimodal Foundation Models

Authors: Xiutian Zhao, Luqi Sun, Björn Schuller, Berrak Sisman

Organizations: Center for Language and Speech Processing (CLSP), Johns Hopkins University, USA · Group on Language, Audio & Music (GLAM), Imperial College London, UK

Abstract

Modern multimodal foundation models (MFMs) have made rapid progress on tasks requiring integrated perception across speech, vision, and language, including emotion recognition. However, it remains unclear whether they recognize speech and facial emotion through shared affective functional units or modality-specific pathways. We explore emotion-sensitive neurons (ESNs), sparse decoder neurons selectively associated with emotion categories, in three MFMs: Gemma-4-12B-it, MiniCPM-o-4.5, and Qwen2.5-Omni-7B. Using speech emotion recognition and facial expression recognition as complementary probes, we identify acoustic and visual ESNs. Visual ESNs are causally meaningful: deactivating them selectively impairs recognition of the associated facial emotion, whereas steering their activations selectively enhances recognition of that emotion relative to other emotion categories. Acoustic and visual ESNs further show emotion-matched overlap and similar layer-wise distributions, indicating partial structural alignment between affective representations across speech and faces. Finally, cross-modal interventions reveal bidirectional causal transfer: ESNs identified from one modality produce emotion-specific effects when applied to the other. Our findings provide one of the first cross-modality activation-level analyses of affective functional units in MFMs, suggesting that speech and facial emotion recognition partially converge onto sparse decoder-level components that can be localized and manipulated without training.

Figures & tables

Evaluation Setting	MFM	Intervention	Unmasked	Random ( $\Delta$ )	Self-Emotion ( $\Delta$ )	Avg. Cross-Emotion ( $\Delta$ )	Self-Cross Gap
	Gemma-4-	Deactivation $\downarrow$	60.47	60.19 (-0.28)	34.67 (-25.80)	63.00 (+2.53)	-28.33
	12B-it	Steering $\uparrow$	60.47	60.13 (-0.34)	76.93 (+16.46)	55.92 (-4.55)	+21.01
FER	MiniCPM-	Deactivation $\downarrow$	64.00	64.13 (+0.13)	53.67 (-10.33)	64.83 (+0.83)	-11.16
with V-ESN masks	o-4.5	Steering $\uparrow$	64.00	63.99 (-0.01)	68.07 (+4.07)	63.27 (-0.73)	+4.80
	Qwen2.5-	Deactivation $\downarrow$	48.60	47.53 (-1.07)	38.20 (-10.40)	48.85 (+0.25)	-10.65
	Omni-7B	Steering $\uparrow$	48.60	47.76 (-0.84)	51.60 (+3.00)	47.72 (-0.88)	+3.88

TABLE I : Mono-modal causal effects of ESN interventions. V-ESN masks are identified and evaluated on FER; A-ESN masks are identified and evaluated on SER. Columns report the unmasked UAR, random-mask controls, matched-emotion performance, average non-matched-emotion performance, and the Self-Cross Gap, as defined in Section IV-D .

\Delta

values indicate the changes relative to the unmasked UAR. Arrows (

\downarrow

\uparrow

) indicate the expected direction of the matched-emotion performance change.

Panel of Figure 2 — Fig. 1 : Emotion-specific FER effects of V-ESN interventions. Heatmaps report changes in FER accuracy after applying FER-derived V-ESN masks. Rows denote the mask emotion and columns denote the FER test-subset emotion; “Random” gives same-size random-mask controls. Diagonal cells correspond to matched mask–test emotion pairs.

Panel of Figure 3 — Fig. 2 : Layer-wise distribution of selected ESNs (ConAct-selected, $r=0.5\%$ ). Panels show FER-derived V-ESNs and SER-derived A-ESNs for each model; rows denote decoder MLP layers and columns denote emotions. Colors indicate neuron counts on a logarithmic scale. ESNs are sparse but span multiple layers, with stronger concentration in middle and later layers.

Panel of Figure 4 — Fig. 3 : Cross-modal overlap between FER-derived V-ESNs and SER-derived A-ESNs. Each heatmap reports Jaccard similarity coefficient across emotion-specific neuron sets; rows denote FER emotions, columns denote SER emotions, and darker cells indicate larger overlap.

Evaluation Setting	MFM	Intervention	Unmasked	Random ( $\Delta$ )	Self-Emotion ( $\Delta$ )	Avg. Cross-Emotion ( $\Delta$ )	Self-Cross Gap
	Gemma-4-	Deactivation $\downarrow$	60.47	60.19 (-0.28)	42.40 (-18.07)	61.33 (+0.86)	-18.93
	12B-it	Steering $\uparrow$	60.47	60.13 (-0.34)	67.40 (+6.93)	59.12 (-1.35)	+8.28
FER	MiniCPM-	Deactivation $\downarrow$	64.00	64.13 (+0.13)	58.67 (-5.33)	63.73 (-0.27)	-5.06
with A-ESN masks	o-4.5	Steering $\uparrow$	64.00	63.99 (-0.01)	65.80 (+1.80)	63.72 (-0.28)	+2.08
	Qwen2.5-	Deactivation $\downarrow$	48.60	47.53 (-1.07)	41.40 (-7.20)	47.87 (-0.73)	-6.47
	Omni-7B	Steering $\uparrow$	48.60	47.76 (-0.84)	51.80 (+3.20)	47.40 (-1.20)	+4.40

TABLE II : Cross-modal causal effects of ESN interventions. SER-derived A-ESN masks are applied during FER, and FER-derived V-ESN masks are applied during SER. Columns follow Table I .

Panel of Figure 6 — Fig. 4 : Emotion-specific effects of cross-modal ESN transfer for MiniCPM-o-4.5. Heatmaps report changes in recognition accuracy when ESN masks identified from one modality are applied to the other modality. Panels (a,b) apply SER-derived A-ESN masks during FER, and panels (c,d) apply FER-derived V-ESN masks during SER.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Task	SER	FER
Dataset	MSP-Podcast	AffectNet
Anger	36260	25382
Fear	1943	6878
Happiness	58684	134915
Neutral	79117	75374
Sadness	24629	25259

Appendix

TABLE III : Dataset statistics showing utterance/images counts per emotion.

Models	Hugging Face Identifier	License
Gemma-4-12B-it	https://huggingface.co/google/Gemma-4-12B-it	Apache License 2.0
MiniCPM-o-4.5	https://huggingface.co/openbmb/MiniCPM-o-4_5	Apache License 2.0
Qwen2.5-Omni-7B	https://huggingface.co/Qwen/Qwen2.5-Omni-7B	Apache License 2.0

Appendix

TABLE IV : Sources and licenses for the three evaluated MFMs.

Explore similar work

Multilingual Emotion Neurons in Large Audio-Language Models
Aug 9, 2026Xiutian Zhao, Philipp Koehn, Björn Schuller +1Cross-Lingual Representation Alignment Audio-Language Models
Center for Language and Speech Processing (CLSP), Johns Hopkins University, USA · Group on Language, Audio & Music (GLAM), Imperial College London, UK
Differential Attention Unlocks Complementary EEG and Speech Fusion for Emotion Recognition
Sep 25, 2026Philip H. Lee, Shreeram Suresh Chandra, John H. L. HansenMultimodal Emotion Recognition EEG Emotion Recognition
Independent Researcher, USA · Center for Language and Speech Processing (CLSP), Johns Hopkins University, Baltimore, MD, USA · Center for Robust Speech Systems (CRSS), University of Texas at Dallas, Richardson, TX, USA
Multimodal Emotion Recognition via Causal-Diffusion Bridge (Affect-Diff)
May 7, 2026Ankit SanjyalClass-Imbalanced Learning Multimodal Emotion Recognition
Department of Computer Science – Data Science Fordham University New York, USA

Aug 9, 2026J/K move · Enter open · S save

Multilingual Emotion Neurons in Large Audio-Language Models

Xiutian Zhao, Philipp Koehn, Björn Schuller, Berrak Sisman

Center for Language and Speech Processing (CLSP), Johns Hopkins University, USA · Group on Language, Audio & Music (GLAM), Imperial College London, UK

Cross-Lingual Representation Alignment Audio-Language Models Mechanistic Interpretability Emotion Representation in Language Models

Open paper AskPDF arXiv

Emotion is central to human communication, and its expression varies across languages. Large audio-language models (LALMs) achieve strong performance on multilingual speech tasks, yet it remains unclear whether they encode emotion through language-specific correlations or language-agnostic representations. We present the first neuron-level interpretability study of this question. We define Multilingual Emotion Neurons (MLENs) as functional units exhibiting stable emotional selectivity and aligned causal effects across languages, and introduce Consistency-Regularized Fusion (CR-Fusion) to identify them. Across four modern LALMs and 12 typologically diverse languages, emotion-sensitive neurons identified independently per language show minimal overlap, and additional monolingual identification data saturates quickly without isolating more transferable units, motivating identification from pooled cross-lingual evidence. Causal interventions demonstrate that MLENs identified by CR-Fusion provide more precise and transferable affective control than monolingual neuron sets in both zero-shot and low-resource settings. Leave-one-out ablations further reveal asymmetric transfer: individual identification languages, including low-resource ones, contribute non-redundant evidence, while several low-resource languages benefit most from the resulting cross-lingual transfer. Together, our findings provide the first causal, neuron-level account of how LALMs encode emotion across languages, and establish multilingual neuron identification as an effective mechanism for understanding cross-lingual affective behavior.

Abstract

Explore similar work

Multilingual Emotion Neurons in Large Audio-Language Models

Differential Attention Unlocks Complementary EEG and Speech Fusion for Emotion Recognition

Multimodal Emotion Recognition via Causal-Diffusion Bridge (Affect-Diff)