Do Emotion Concepts Generalize Across Sources, Modalities, and Architectures in Vision-Language Models?

Bohao Xing; Xin Liu; Kaishen Yuan; Deng Li; Rong Gao; Guoying Zhao; Xiaolan Fu; Heikki Kälviäinen

cs.CVSep 28, 2026

Do Emotion Concepts Generalize Across Sources, Modalities, and Architectures in Vision-Language Models?

Authors: Bohao Xing, Xin Liu, Kaishen Yuan, Deng Li, Rong Gao, Guoying Zhao, Xiaolan Fu, Heikki Kälviäinen

Organizations: Lappeenranta-Lahti University of Technology LUT · Shanghai Jiao Tong University · The Hong Kong University of Science and Technology (Guangzhou) · University of Oulu · ELLIS Institute Finland · Brno University of Technology

Abstract

Recent studies suggest that large language models encode emotion concepts as structured internal representations, but most existing work focuses on text and a single architecture. Therefore, we ask, do emotion concepts generalize across sources, modalities, and architectures in vision--language models (VLMs)? To address this, we construct CMES (Cross-Modal Emotion Stimuli), a multi-source collection of emotion-conditioned stories, real facial expressions, synthetic portraits, and synthetic emotion-evoking scenes. For each stimulus source, we extract a separate set of six Ekman emotion vectors from each of three VLMs. We report four main findings as follows: 1) Image-derived emotion vectors form a low-dimensional geometry similar to that of text-derived vectors. Valence is relatively stable across sources, while arousal varies more. 2) Text- and image-derived emotion vectors have modest cosine similarity but still show held-out cross-modal correspondence. Text-derived vectors can also steer image interpretation. 3) Cross-architecture correspondence remains even when native cosine is near zero. Transformations estimated from generic ImageNet activations recover both correspondence and causal transfer without using the six emotion vectors or their labels. 4) After aligning representations across architectures, we construct a shared emotion subspace that preserves affective geometry and selective steering effects. The corresponding consensus emotion vectors also generalize to a held-out fourth architecture at two model sizes. These results suggest that emotion representations can share relational structure and causal effects across sources, modalities, and architectures, even when individual vector directions differ.

Figures & tables

Panel of Figure 1 — Figure 1: Motivation and study overview. Left: The map–territory analogy and our motivation. Right: Emotion-vector analysis across sources, modalities, and architectures, shared-subspace construction, and held-out Gemma-3 evaluation.

Figure 2: Emotion-vector readouts and human-axis agreement. (a) Selected promoted ( $\uparrow$ ) and suppressed ( $\downarrow$ ) tokens for Qwen3-VL AffectNet vectors, with English translations. (b) Selected Qwen3-VL AffectNet retrievals (top) and patch-readout examples with the top $30\%$ of patch similarities highlighted (bottom). (c) Absolute correlations with Russell–Mehrabian (R&M) ratings ( Russell and Mehrabian, 1977 ) . Small markers show models, diamonds show means, and lines show ranges.

Figure 3: Leading geometry of the six emotion vectors. (a) Human valence–arousal references from R&M and the NRC Valence–Arousal–Dominance (NRC-VAD) lexicon ( Mohammad, 2018 ) . (b–d) Qwen3-VL PCA projections from stories, real faces, and synthetic images. (e) Three-model consensus vectors. Faint circles mark R&M positions. Display alignment is described in Appendix B.7 .

Figure 4: Representational correspondence. (a) CKA and RSA with source or architecture fixed. (b) Text–image cosine before alignment versus LOEO. Filled markers indicate permutation

z>2

; open markers indicate

z\leq 2

. (c) Ministral-3

\rightarrow

Qwen3-VL. Dots show source means before alignment and with LOEO, individual AffectNet emotions after mapping, and source/target replacement means for controls. Diamonds show means; boxes show quartiles and medians.

Figure 5: Causal effects within models. (a) Qwen3-VL steering and controls. (b) Image- versus T3 text-derived vectors under positive ( $+$ ) and negative ( $-$ ) injection, with filled/open bars showing diag/off. (c) Qwen3-VL continuation excerpts before and after anger injection. (d) Emotion and gender prediction flips across six injections (dots) and their means (diamonds). (e) Changes in safe–dangerous log odds, with positive values favoring safe.

Figure 6: Cross-architecture causal transfer on AffectNet. Ministral-3 image and T2 text vectors are injected into Qwen3-VL, with Qwen3-VL’s own image vectors as a reference. Each condition shows positive (

+

) and negative (

-

) injections. Filled/open bars show diag/off effects.

Figure 7: Shared-subspace effects and consensus transfer. (a) Qwen3-VL interventions. (b) Selected promoted ( $\uparrow$ ) and suppressed ( $\downarrow$ ) tokens for Ministral-3 under ImageNet alignment. (c) Qwen3-VL continuations before and after consensus fear injection. (d) Gemma’s own vectors versus neutral-face/ImageNet-mapped consensus vectors at $s=+0.1$ . Filled/open bars in (a, d) show diag/off. Cos. denotes mean cosine similarity to the corresponding Gemma model’s own AffectNet vectors.

Appendix figures & tables48 assets

Supplementary material from the paper’s appendix.

Appendix

Label	Source	Stimulus type	Per class	Total
Text sources
T1	Gemini 3.1 Pro Preview	Generated text	$1{,}200$	$8{,}400$
T2	Qwen3-Max stories	Generated text	$1{,}200$	$8{,}400$
T3	Qwen3-VL stories	Generated text	$1{,}200$	$8{,}400$
Image sources
AffectNet		Real faces	$3{,}000$	$21{,}000$

Appendix

Table 1: Stimulus sources and sample counts. Unfiltered text counts follow the generation protocol. Every set contains six target emotion classes and a neutral class. Per-class counts cover all seven classes; the filtered text block lists counts per target emotion and for neutral separately. QF means Qwen-filtered, retaining stimuli whose labels are correctly predicted by Qwen3-VL.

Panel of Figure 8 — Figure 8: Prompt templates for CMES stimulus generation. Braces mark placeholders. The story-format example is omitted.

Figure 9: Top-4 promoted (

\uparrow

) and suppressed (

\downarrow

) tokens for story- and AffectNet-derived emotion vectors at layers 18 and 24. Scores use each model’s final normalization and unembedding.

Figure 10: Six random images from each Qwen3-VL emotion vector’s retrievals. Green frames mark label matches; red frames mark mismatches and show the ground-truth label.

Emotion	Retrieval P@50 (%)			Classification Recall (%)
	Qwen3-VL	InternVL3.5	Ministral-3	Qwen3-VL	InternVL3.5	Ministral-3
anger	58	60	38	41	52	29
disgust	80	66	68	34	37	30
fear	72	70	72	52	49	45
happiness	92	84	88	93	89	93
sadness	82	88	90	54	47	50

Appendix

Table 2: Retrieval P@50 and six-way classification recall for AffectNet-trained vectors. P@50 evaluates the top-

50

retrievals; recall evaluates all validation images within each of the six target classes. All values are percentages.

Emotion	Pool share (%)	Retrieval P@50 (%)			Classification Recall (%)
		Qwen3-VL	InternVL3.5	Ministral-3	Qwen3-VL	InternVL3.5	Ministral-3
anger	5.3	72	66	64	69	51	54
disgust	5.2	8	16	6	60	48	53
fear	2.4	28	64	48	62	62	64
happiness	38.6	100	94	98	60	63	65
sadness	15.6	86	80	96	66	61	61

Appendix

Table 3: Retrieval P@50 and six-way classification recall for RAF-DB-trained vectors. P@50 evaluates the top-

50

retrievals; recall uses all test images within each target class. Pool share gives each class’s reference rate in the

3{,}068

-image retrieval pool. All values are percentages.

Emotion vector	Model	Disgust	Fear	Anger	Happiness	Neutral	Surprise	Sadness
AffectNet validation pool
disgust	Qwen3-VL	80	2	18	–	–	–	–
	InternVL3.5	66	4	26	–	–	–	4
	Ministral-3	68	6	18	–	–	–	8
fear	Qwen3-VL	2	72	8	–	–	18	–
	InternVL3.5	2	70	6	–	–	22	–

Appendix

Table 4: Ground-truth composition (%) of the top-

50

images retrieved by the disgust and fear vectors on AffectNet and RAF-DB. Bold values mark the vector’s target class and equal its retrieval P@50. A dash denotes

<0.5\%

Figure 11: Patch cosine similarity to Qwen3-VL emotion vectors. Each emotion shows three label-matching AffectNet validation images sampled from its top- $50$ retrievals. Overlays show the top $30\%$ of image tokens ranked by $\cos(\bm{h}_{t},\bm{v}_{e})$ , with colors normalized for each image.

Figure 12: Cross-layer geometry of T2 story and AffectNet emotion vectors. Each cell compares two layers by the cosine similarity between the $15$ unique off-diagonal entries of their emotion–emotion cosine matrices.

Qwen3-VL
Stimulus source	PC1	PC2	PC3	PC4	PC5	PC1 $+$ PC2
T1	30.7	25.2	19.2	13.9	11.1	55.9
T2	38.2	27.6	20.3	8.4	5.5	65.8
T3	40.2	25.9	19.9	8.0	6.1	66.0
AffectNet	34.0	29.8	18.8	11.6	5.8	63.8
AffectNet (filtered)	32.8	23.0	21.4	15.2	7.5	55.9

Appendix

Table 5: PCA variance explained (%) for six centered emotion vectors. PC1

+

PC2 gives the cumulative share of the leading plane. Consensus rows share the same spectrum because orthogonal mapping into each model preserves variance shares (Appendix F ).

Figure 13: Pearson correlations across six-emotion PC1 (a, valence) and PC2 (b, arousal) coordinates, with signs aligned to R&M. Models are ordered Qwen3-VL, InternVL3.5, and Ministral-3 within each source and consensus group. Human references are ordered R&M, then NRC-VAD. Consensus vectors are mapped into each model’s native frame.

Figure 14: Pairwise linear CKA (a), RSA (b), and mean matched-emotion cosine similarity (c). Models are ordered Qwen3-VL, InternVL3.5, and Ministral-3 within each source and consensus group. Human references are ordered R&M, then NRC-VAD, and are omitted from (c) because they use different coordinates. Consensus vectors are mapped into each model’s native frame. Blue outlines mark pairs with the same source and different architectures; orange outlines mark pairs with the same architecture and different sources.

Figure 15: CKA and RSA for the

21

original model–source sets. Blue varies architecture with source fixed (

21

pairs); purple varies source with architecture fixed (

63

pairs). Points show all pairs and boxes show the interquartile range. Solid lines mark medians and dashed lines mark minima.

Valence agreement $\|r(\mathrm{PC1},\mathrm{valence})\|$
Stimulus source	Qwen3-VL	InternVL3.5	Ministral-3	Mean	Spread
RAF-DB	$0.57$	$0.55$	$0.47$	$0.53$	$0.10$
T3	$0.90$	$0.91$	$0.88$	$0.90$	$0.02$
Synth-S	$0.97$	$0.97$	$0.98$	$0.97$	$0.01$
T1	$0.96$	$0.97$	$0.96$	$0.97$	$0.01$
AffectNet	$0.85$	$0.56$	$0.98$	$0.80$	$0.42$

Appendix

Table 6: Absolute Pearson correlations with R&M ratings across six emotions. PC1 is compared with valence and PC2 with arousal. Mean averages the three models; Spread is their maximum minus minimum. Sources are ordered by mean arousal agreement.

		Arousal $\|r\|$ (PC2)		Valence $\|r\|$ (PC1)
Source	Model	Unfiltered	Filtered	Unfiltered	Filtered
AffectNet	Qwen3-VL	$0.23$	$0.41$	$0.85$	$0.97$
	InternVL3.5	$0.03$	$0.63$	$0.56$	$0.98$
	Ministral-3	$0.64$	$0.88$	$0.98$	$0.97$
RAF-DB	Qwen3-VL	$0.85$	$0.65$	$0.57$	$0.67$
	InternVL3.5	$0.94$	$0.79$	$0.55$	$0.65$

Appendix

Table 7: Effect of success filtering on the two leading emotion axes, measured as sign-aligned

|r|

against the R&M norms. The same subset selected by Qwen3-VL is used for all three architectures. Retained sample sizes are reported in Table 1 .

Qwen3-VL
	R&M		NRC-VAD		vs. T2 text
AffectNet subset	Valence	Arousal	Valence	Arousal	CKA	RSA
Full set	0.85	0.23	0.83	0.20	0.93	0.61
Random subset	$0.81\pm 0.22$	$0.37\pm 0.23$	$0.79\pm 0.22$	$0.34\pm 0.23$	$0.92\pm 0.02$	$0.64\pm 0.05$
Success-filtered	0.97	0.41	0.99	0.28	0.95	0.86
InternVL3.5

Appendix

Table 8: AffectNet sample-size control. Random and filtered subsets contain

377

images per class. Random results are mean

\pm

standard deviation over five seeds. Valence and arousal are correlations of PC1 and PC2 with human ratings. CKA and RSA compare image vectors with the same model’s T2 text vectors.

Text sources
		Mean cosine			Permutation
Model	Source pair	Before alignment	Full-fit	LOEO	$z$
Qwen3-VL	T1–T2	$0.71$	$0.98$	$0.72$	$4.09$
	T1–T3	$0.61$	$0.96$	$0.60$	$4.38$
	T2–T3	$0.69$	$0.98$	$0.72$	$4.52$
InternVL3.5	T1–T2	$0.70$	$0.98$	$0.71$	$4.26$

Appendix

Table 9: Within-modality correspondence. Each row changes the stimulus source within one model. Cosines are means over six emotions. Full-fit uses all six pairs. LOEO fits five and evaluates the sixth. The

z

score compares LOEO with the refitted label-permutation null. Source labels follow Table 1 .

Figure 16: Source differences in the neighbors of disgust (a) and fear (b). Cosine similarities are computed between emotion vectors from AffectNet or RAF-DB. Bars show the mean over three models, and marker shapes identify individual models.

Qwen3-VL
	Mean cosine			Permutation
Text $\to$ image source	Before alignment	Full-fit	LOEO	$z$
T1 $\to$ AffectNet	$0.26$	$0.97$	$0.33$	$4.34$
T1 $\to$ RAF-DB	$0.26$	$0.94$	$0.21$	$2.65$
T1 $\to$ Synth-P	$0.30$	$0.94$	$0.38$	$4.61$
T1 $\to$ Synth-S	$0.26$	$0.97$	$0.34$	$4.29$

Appendix

Table 10: Text–image correspondence within each model. Cosines are means over six emotions. Full-fit uses all six pairs. LOEO fits five and evaluates the sixth. The

z

score compares LOEO with the refitted label-permutation null. Null controls replace image vectors with matched-norm directions. Source labels follow Table 1 .

	Mean cosine			Permutation
Source / control	Before alignment	Full-fit	LOEO	$z$
T1	$0.00$	$1.00$	$0.13$	$3.32$
T2	$0.00$	$1.00$	$0.34$	$3.39$
T3	$0.00$	$1.00$	$0.38$	$3.73$
AffectNet	$0.00$	$0.99$	$0.30$	$3.16$
RAF-DB	$-0.01$	$0.99$	$0.23$	$2.60$

Appendix

Table 11: Ministral-3

\rightarrow

Qwen3-VL correspondence. Cosines are means over six emotions. Before alignment, cosine similarity is measured in native coordinates. Full-fit and LOEO use six and five emotion pairs, respectively. RAF-DB controls replace source or target vectors with matched-norm directions. The

z

score compares LOEO with the refitted label-permutation null. Source labels follow Table 1 .

Figure 17: Alignment sample size and Ministral-3

\rightarrow

Qwen3-VL correspondence. Maps use paired neutral-face or ImageNet activations without emotion-vector fitting. Points show mean cosine across six AffectNet emotion pairs. Sample size is on a logarithmic axis. Reference cosine similarities are

0.00

before alignment,

0.30

with LOEO, and

0.99

with a full fit.

Stimulus source	Anger	Disgust	Fear	Happiness	Sadness	Surprise
T1	$0.10$	$0.11$	$0.21$	$0.15$	$0.12$	$0.07$
T2	$0.50$	$0.54$	$0.37$	$0.25$	$0.17$	$0.20$
T3	$0.53$	$0.55$	$0.44$	$0.31$	$0.21$	$0.24$
AffectNet	$0.26$	$0.28$	$0.46$	$0.31$	$0.21$	$0.31$
RAF-DB	$0.36$	$0.26$	$0.41$	$0.35$	$0.01$	$0.00$
Synth-P	$0.64$	$0.66$	$0.56$	$0.24$	$0.34$	$0.23$

Appendix

Table 12: Per-emotion LOEO cosine for Ministral-3

\rightarrow

Qwen3-VL. Each cell evaluates the named emotion after fitting the other five. Source labels follow Table 1 .

Image vectors (AffectNet)
	Mean cosine
Initial condition	Direct	After LOEO	Held-out range	Permutation $z$
No mapping	$0.000$	$0.305$	$0.206$ – $0.456$	$3.16$
Neutral-face map	$0.697$	$0.755$	$0.694$ – $0.827$	$4.37$
ImageNet map	$0.466$	$0.584$	$0.503$ – $0.675$	$4.71$
Random orthogonal map	$0.020$	$0.316$	$0.190$ – $0.463$	$3.25$

Appendix

Table 13: LOEO after activation-based mapping from Ministral-3 to Qwen3-VL. Direct and After LOEO give mean cosine over six emotions. Held-out range gives the six LOEO scores. The

z

score uses refitted label permutations. Each activation-based map uses

3{,}000

paired images. No mapping gives ordinary LOEO. A random map rotates the real source vectors, whereas the final control replaces them with target-orthogonal random directions.

Figure 18: Image- and text-vector dose responses in Qwen3-VL. Columns compare AffectNet and RAF-DB. The upper row shows mean changes in the injected emotion’s log-probability (diag), and the lower row shows the mean over the other five emotions (off). The two rows use different vertical scales. Points show the means in Tables 14 and 16 ; lines connect tested strengths. Image and T3 text vectors are injected at image-token positions. Random and orthogonal controls have matched norms.

RAF-DB
	$s=-0.5$		$s=-0.1$		$s=+0.1$		$s=+0.5$
Injected vector	diag	off	diag	off	diag	off	diag	off
Image vector	$-4.94$	$+1.32$	$-1.60$	$+0.42$	$+1.85$	$-0.36$	$+5.34$	$-1.55$
Random	$+0.21$	$+0.22$	$+0.01$	$+0.00$	$+0.01$	$+0.02$	$+0.25$	$+0.29$
Orthogonal	$+0.27$	$+0.20$	$+0.02$	$+0.00$	$+0.00$	$+0.02$	$+0.20$	$+0.31$
AffectNet

Appendix

Table 14: Within-model steering for Qwen3-VL at image-token positions. diag and off are mean changes in log-probability for the injected emotion and the other five emotions, respectively. Random and orthogonal controls have matched norms.

Qwen3-VL
	RAF-DB				AffectNet
	$s=-0.5$		$s=+0.5$		$s=-0.5$		$s=+0.5$
Injected vector	diag	off	diag	off	diag	off	diag	off
Image vector	$-4.94$	$+1.32$	$+5.34$	$-1.55$	$-4.69$	$+1.21$	$+4.96$	$-1.43$
Random	$+0.21$	$+0.22$	$+0.25$	$+0.29$	$+0.19$	$+0.16$	$+0.19$	$+0.24$
Orthogonal	$+0.27$	$+0.20$	$+0.20$	$+0.31$	$+0.21$	$+0.17$	$+0.18$	$+0.23$

Appendix

Table 15: Within-model steering across the three core models. diag and off are mean changes in log-probability for the injected emotion and the other five emotions. Controls have matched norms.

Figure 19: Within-model steering across three models and two image datasets. Image and T3 text vectors are compared with matched-norm random and orthogonal controls. The same horizontal scale is used throughout. Filled and open markers show diag and off; circles and triangles show

s=+0.5

and

s=-0.5

. Points show the means in Tables 15 and 17 .

	$s=-0.5$		$s=-0.1$		$s=+0.1$		$s=+0.5$
Evaluation set	diag	off	diag	off	diag	off	diag	off
RAF-DB	$-4.94$	$+1.35$	$-1.66$	$+0.34$	$+1.85$	$-0.33$	$+6.07$	$-1.37$
AffectNet	$-4.00$	$+1.04$	$-1.33$	$+0.27$	$+1.44$	$-0.27$	$+4.77$	$-1.09$

Appendix

Table 16: Steering Qwen3-VL with its T3 text-derived vectors at image-token positions, without cross-modal alignment. diag and off are mean changes in log-probability for the injected emotion and the other five emotions.

Qwen3-VL
	RAF-DB				AffectNet
	$s=-0.5$		$s=+0.5$		$s=-0.5$		$s=+0.5$
Injected vector	diag	off	diag	off	diag	off	diag	off
Text vector (T3)	$-4.94$	$+1.35$	$+6.07$	$-1.37$	$-4.00$	$+1.04$	$+4.77$	$-1.09$
Image vector	$-4.94$	$+1.32$	$+5.34$	$-1.55$	$-4.69$	$+1.21$	$+4.96$	$-1.43$
InternVL3.5

Appendix

Table 17: Text- and image-vector steering within each model. T3 text vectors are injected at image-token positions without alignment. Image vectors provide the within-model reference. diag and off are mean changes in log-probability for the injected emotion and the other five emotions.

Figure 20: Cross-architecture steering on AffectNet for all six directed model pairs. Image and T2 text vectors are compared under no mapping, neutral-face mapping, and ImageNet mapping, with random orthogonal maps and target-orthogonal directions as image-vector controls. Filled/open markers show diag/off, and circles/triangles show positive/negative injection at

|s|=0.5

. All comparisons use the same effect scale. The right-hand columns give cosine similarity to the target vectors of the same modality and source. Values are the means reported in Table 18 .

Ministral-3 $\to$ Qwen3-VL
			$s=-0.5$		$s=+0.5$
Vector source	Mapping / control	Cosine	diag	off	diag	off
Image	No mapping	$0.00$	$+0.25$	$+0.09$	$+0.04$	$+0.18$
	Random orthogonal map	$0.02$	$+0.21$	$+0.21$	$+0.02$	$+0.09$
	Target-orthogonal directions	$0.00$	$+0.06$	$+0.26$	$+0.27$	$+0.09$
	Neutral-face map	$0.70$	$-3.73$	$+1.04$	$+4.11$	$-0.77$

Appendix

Table 18: Cross-architecture steering on AffectNet. Source image or T2 text vectors are injected at the target’s image-token positions. Cosine compares mapped vectors with target vectors of the same modality and source. diag and off are mean

\Delta\log p

for the injected emotion and the other five emotions. Neutral-face and ImageNet maps use neither the six emotion vectors nor their labels.

	Emotion readout		Gender readout
Injected emotion	Flip rate	Accuracy	Flip rate	Accuracy
No intervention	—	$0.982$	—	$0.961$
Anger	$79\%$	$0.204$	$2\%$	$0.969$
Disgust	$13\%$	$0.883$	$3\%$	$0.937$
Fear	$54\%$	$0.460$	$0\%$	$0.961$
Happiness	$38\%$	$0.628$	$4\%$	$0.925$

Appendix

Table 19: Emotion and gender readouts under Qwen3-VL steering at

s=+0.5

. Flip rate (%) compares predictions with the unsteered baseline. Accuracy is a fraction, and Mean averages the six injections.

Panel of Figure 21 — Figure 21: Selected Qwen3-VL continuations under its own AffectNet emotion vectors at image-token positions and $s=+0.5$ . Each block pairs a different validation image with one injected emotion and shows unsteered text above steered text. Green and red highlights mark unsteered and steered text, respectively. Bold marks labels and emotion terms, and italics mark selected visual descriptions. Ellipses mark truncation.

	RAF-DB		AffectNet
Injection positions	diag	off	diag	off
Image tokens (default)	$+5.34$	$-1.55$	$+4.96$	$-1.43$
Question tokens	$+2.50$	$+0.70$	$+2.27$	$+0.13$
Answer prefix	$+4.71$	$-2.31$	$+4.73$	$-1.95$
All positions	$+5.23$	$-5.51$	$+5.55$	$-5.57$

Appendix

Table 20: Injection-position comparison for Qwen3-VL at

s=+0.5

, with vectors held fixed. diag and off are mean changes in log-probability for the injected emotion and the other five emotions.

	RAF-DB		AffectNet
Removed direction or span	$f=0.5$	$f=1$	$f=0.5$	$f=1$
No removal ( $f=0$ )	$0.993$		$0.986$
Single emotion vector (rank 1)	$0.969$	$0.946$	$0.950$	$0.902$
Random direction	$0.991$	$0.991$	$0.988$	$0.987$
Centered emotion-vector span (rank 5)	$0.938$	$\mathbf{0.814}$	$0.901$	$\mathbf{0.781}$
Random subspace	$0.990$	$0.988$	$0.988$	$0.990$

Appendix

Table 21: Emotion recognition accuracy after partial (

f=0.5

) or full (

f=1

) removal in Qwen3-VL at image-token positions. Classification includes the six emotions and neutral. Each random control matches the dimension of the emotion direction or span directly above it.

Figure 22: Safe–dangerous readout under Qwen3-VL emotion steering. Points show mean changes from the unsteered model, with paired markers for

s=-0.5

(open triangles) and

s=+0.5

(filled circles). Negative values shift the prediction toward dangerous and positive values toward safe.

	RAF-DB		AffectNet
Injected emotion	$s=-0.5$	$s=+0.5$	$s=-0.5$	$s=+0.5$
Anger	$+0.83$	$-2.60$	$+0.83$	$-1.42$
Disgust	$-1.25$	$+0.18$	$+0.95$	$-0.31$
Fear	$+0.53$	$-1.98$	$+0.25$	$-1.13$
Happiness	$-1.62$	$+0.77$	$-1.36$	$+0.59$
Sadness	$-0.32$	$+0.52$	$+0.19$	$-0.30$

Appendix

Table 22: Change in the safe–dangerous log-probability contrast under Qwen3-VL emotion steering at image-token positions. Positive values favor safe , and negative values favor dangerous .

	RAF-DB				AffectNet
	$s=-0.5$		$s=+0.5$		$s=-0.5$		$s=+0.5$
Emotion attributed to	diag	off	diag	off	diag	off	diag	off
the depicted person	$-3.43$	$+0.30$	$\mathbf{+2.63}$	$-1.10$	$-2.83$	$+0.34$	$\mathbf{+2.37}$	$-0.83$
the assistant	$-1.94$	$+0.37$	$+1.78$	$-0.53$	$-1.73$	$+0.26$	$+1.61$	$-0.44$
a third party	$-2.20$	$+0.31$	$+1.91$	$-0.60$	$-1.68$	$+0.25$	$+1.63$	$-0.38$

Appendix

Table 23: Effect of the queried subject on Qwen3-VL emotion steering at image-token positions. diag and off are mean changes in log-probability for the injected emotion and the other five emotions.

Figure 23: Depth dependence of steering on RAF-DB at

s=+0.5

. Mean

\Delta\log p

is shown for the matched emotion (diag), the other five emotions (off), and their difference. Lines connect measured layers. Rings mark the default layers. All depths use the same residual-norm scaling rule. Table 24 gives exact values.

Layer	Relative depth	diag	off	Selectivity (diag $-$ off)
Qwen3-VL
9	$0.25$	$+5.62$	$-3.95$	$\mathbf{+9.58}$
14	$0.39$	$\mathbf{+5.97}$	$-3.48$	$+9.45$
18	$0.50$	$+5.84$	$-3.39$	$+9.23$
24	$0.67$	$+5.34$	$-1.55$	$+6.89$
31	$0.86$	$+1.19$	$-0.11$	$+1.30$

Appendix

Table 24: Steering across depth on RAF-DB at

s=+0.5

. diag and off are mean

\Delta\log p

for the injected emotion and the other five emotions. Selectivity is diag minus off. Bold marks each model’s largest diag and selectivity. Differences use unrounded values.

Model / alignment	Emotion	Modality	Model	Residual / interactions
Within each model
Qwen3-VL	$0.474$	$0.004$	—	$0.522$
InternVL3.5	$0.447$	$0.004$	—	$0.550$
Ministral-3	$0.423$	$0.006$	—	$0.571$
Across all three aligned models
Neutral-face alignment	$0.326$	$0.003$	$0.004$	$0.668$

Appendix

Table 25: Variance components of emotion vectors across all seven sources, weighted equally. Entries are fractions of total variance before source averaging. Cross-model estimates follow generalized Procrustes alignment. Residual/interactions includes within-modality source variation and interactions among factors. Values may not sum to one after rounding.

Figure 24: Shared-subspace steering in Qwen3-VL on AffectNet at

s=+0.5

. (a) Shared projection and remainder under both norm conditions, compared with the original vector. (b) Equal-norm remainder controls and consensus vectors under two reference frames. The equal-norm remainder is repeated for comparison. Bars show the means in Table 26 ; bootstrap intervals for selected comparisons are reported in the text.

Equal-norm directions
	Mean $\Delta\log p$
Injected vector	diag	off
Original vector $\bm{v}$	$+4.96$	$-1.43$
Shared projection $\bm{v}_{\mathrm{shared}}$	$+5.39$	$-1.62$
Remainder $\bm{v}_{\perp}$	$+1.00$	$-0.14$
Random outside span $\mathcal{S}_{\mathrm{emo}}$	$+0.19$	$+0.22$

Appendix

Table 26: Shared-subspace steering in Qwen3-VL on AffectNet images at

s=+0.5

. The span uses all three core models and seven sources. Equal-norm directions have unit length. The next block retains the original component lengths. Shuffled labels rotate remainder labels. diag and off are mean

\Delta\log p

for the injected emotion and the other five emotions.

Panel of Figure 25 — Figure 25: Selected continuations for two AffectNet validation images under all six consensus-vector injections in Qwen3-VL at $s=+0.5$ . Both images have ground-truth label disgust . The first row for each image is unsteered. Highlight colors follow the injected emotions in Figure 3 , while unsteered highlights remain black. Bold marks labels and emotion terms, and italics mark selected visual descriptions. The leading They feel is omitted, and ellipses mark truncation.

Figure 26: Lexical readouts of the three-model consensus vectors. After ImageNet alignment and averaging over all

21

model–source sets, each vector is mapped back to each model’s coordinates. The figure shows the five highest- and lowest-scoring tokens per emotion under that model’s lexical readout.

Figure 27: Gemma-3 generalization at 12B and 27B. (a,b) Mean six-emotion LOEO cosine for seven sources and both directions with each core model. The last four rows are RAF-DB replacement controls. (c,d) AffectNet steering at $s=\pm 0.1$ . Filled/open markers show diag/off, and circles/triangles show positive/negative injection. Neutral denotes neutral-face mapping. Gemma-3 is excluded from consensus construction. Tables 27 and 28 give exact values.

Gemma-3-12B
	Source model $\to$ Gemma-3			Gemma-3 $\to$ target model
Source / control	Qwen3-VL	Ministral-3	InternVL3.5	Qwen3-VL	Ministral-3	InternVL3.5
T1	$0.17$	$0.16$	$0.16$	$0.16$	$0.16$	$0.16$
T2	$0.39$	$0.36$	$0.39$	$0.38$	$0.36$	$0.38$
T3	$0.40$	$0.40$	$0.40$	$0.39$	$0.40$	$0.39$
AffectNet	$0.35$	$0.37$	$0.37$	$0.35$	$0.37$	$0.36$

Appendix

Table 27: Mean six-emotion LOEO cosine between Gemma-3 and each core model in both directions. Unequal-width fits follow Appendix B.7 . RAF-DB controls replace source or target vectors with matched-norm random or orthogonal directions. Source labels follow Table 1 .

Gemma-3-12B
		$s=-0.1$		$s=+0.1$
Injected vector	Cosine	diag	off	diag	off
Original Gemma-3 vector	$1.00$	$-2.45$	$+1.02$	$+3.66$	$-0.02$
Neutral-face alignment
Consensus vector	$0.63$	$-1.84$	$+0.89$	$+3.59$	$+0.18$
Qwen3-VL vector	$0.75$	$-1.69$	$+0.82$	$+2.90$	$+0.00$

Appendix

Table 28: Gemma-3 steering on AffectNet at image-token positions. Cosine compares mapped vectors with Gemma-3’s own AffectNet vectors. Consensus construction excludes Gemma-3. Mapping uses paired activations without emotion-vector fitting. diag and off are mean

\Delta\log p

for the injected emotion and the other five emotions.

Explore similar work

Jun 25, 2026cs.CL

Where Do Models Find Happiness? Emotion Vectors in Open-Source LLMs

Recent work identified emotion vectors in Claude Sonnet 4.5, which are internal representations that encode emotion concepts, causally influence behavior, and exhibit geometry mirroring human psychological structure. We test the generality of these findings in two open-weight models, Apertus-8B-Instruct-2509 and Gemma-4-E4B-it, extracting emotion contrast vectors across all layers, using two model-generated corpora. We recover valence geometry for both models, with peak PC1--valence correlations of

r = 0.76

and

r = 0.83

, approaching the

r = 0.81

reported for Claude.Beyond replication, we observe notable differences in how valence representations emerge across model depth. In Gemma-4-E4B-it, valence is strongly encoded in early layers but collapses towards later layers, whereas Apertus-8B-Instruct-2509 exhibits the opposite pattern, with valence representations absent in early layers, but emerging at mid depths. Arousal encoding, in contrast, is sensitive to the extraction corpus: both models show stronger PC2--arousal alignment with Gemma-generated stories (

r

up to

0.45

) than Apertus-generated ones (

r \leq 0.21

), suggesting arousal-relevant cues are unevenly distributed across generated corpora. We open-source our experiment code and dataset for reproducible investigation of emotion representations across language model architectures.

Sinie van der Ben, Raphaël Baur, Yannick Metz +1

Department of Computer Science, ETH, Zurich, Switzerland.

Valence-Arousal Estimation Emotion Text Corpora Poetry

Ask PDF

Apr 16, 2026cs.CV

Why Do Vision Language Models Struggle To Recognize Human Emotions?

Understanding emotions is a fundamental ability for intelligent systems to be able to interact with humans. Vision-language models (VLMs) have made tremendous progress in the last few years for many visual tasks, potentially offering a promising solution for understanding emotions. However, it is surprising that even the most sophisticated contemporary VLMs struggle to recognize human emotions or to outperform even specialized vision-only classifiers. In this paper we ask the question "Why do VLMs struggle to recognize human emotions?", and observe that the inherently continuous and dynamic task of facial expression recognition (DFER) exposes two critical VLM vulnerabilities. First, emotion datasets are naturally long-tailed, and the web-scale data used to pre-train VLMs exacerbates this head-class bias, causing them to systematically collapse rare, under-represented emotions into common categories. We propose alternative sampling strategies that prevent favoring common concepts. Second, temporal information is critical for understanding emotions. However, VLMs are unable to represent temporal information over dense frame sequences, as they are limited by context size and the number of tokens that can fit in memory, which poses a clear challenge for emotion recognition. We demonstrate that the sparse temporal sampling strategy used in VLMs is inherently misaligned with the fleeting nature of micro-expressions (0.25-0.5 seconds), which are often the most critical affective signal. As a diagnostic probe, we propose a multi-stage context enrichment strategy that utilizes the information from "in-between" frames by first converting them into natural language summaries. This enriched textual context is provided as input to the VLM alongside sparse keyframes, preventing attentional dilution from excessive visual data while preserving the emotional trajectory.

Madhav Agarwal, Sotirios A. Tsaftaris, Laura Sevilla-Lara +1

The University of Edinburgh, UK

Vision-Language Foundation Models Emotion Facial Expression Recognition

Ask PDF

May 21, 2026cs.CV

Interpreting and Enhancing Emotional Circuits in Large Vision-Language Models via Cross-Modal Information Flow

Large Vision-Language Models (LVLMs) represent a significant leap towards empathetic agents, demonstrating remarkable capabilities in emotion understanding. However, the internal mechanisms governing how LVLMs translate abstract visual stimuli into coherent emotional narratives remain largely unexplored, primarily due to the scarcity of visual counterfactuals and the diffuse nature of emotional expression. In this paper, we bridge this gap by introducing a steering-vector-based causal attribution framework tailored for descriptive emotional reasoning. To this end, we construct a specialized dataset to demystify the emotional circuits underlying the three-stage ``Adapt-Aggregate-Execute'' mechanism. Crucially, we discover a functional decoupling: visual emotional cues are aggregated in middle layers via sentiment-specific attention heads, but are subsequently translated into narrative generation in deep layers through emotion-general pathways. Guided by these insights, we regulate the emotional information routing to strengthen attention flow and amplify the semantic activation to consolidate expression. Extensive experiments on the comprehensive MER-UniBench demonstrate that our methods significantly improve performance via inference-time intervention, effectively mitigating emotional hallucinations and corroborating the causal fidelity of the discovered circuits.

Chengsheng Zhang, Chenghao Sun, Zhining Xie +1

MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China · AIPD, Tencent

Emotional Regulation Cross-Modal Narrative Generation Interpretation

Ask PDF