Decoding Affective Nuances: Enhancing MLLMs via Hierarchical Emotion Reasoning and Contrastive Discriminative Pruning

Cheng Ye; Weidong Chen; Zhaobo Qi; Beier Zhu; Zhendong Mao

cs.CVSep 29, 2026

Decoding Affective Nuances: Enhancing MLLMs via Hierarchical Emotion Reasoning and Contrastive Discriminative Pruning

Authors: Cheng Ye, Weidong Chen, Zhaobo Qi, Beier Zhu, Zhendong Mao

Organizations: University of Science and Technology of China, Hefei · Harbin Institute of Technology, Weihai

Abstract

While multimodal large language models (MLLMs) have demonstrated exceptional capabilities in objective understanding tasks, their performance in affective reasoning still falls significantly short of human standards. We attribute it to a central capability gap: MLLMs are difficult to reliably distinguish semantically proximal emotions based on fine-grained visual evidence, which could be decoupled as two limitations: 1) Insufficient Attribution. The global reasoning paradigm of conventional MLLMs severely dilutes fine-grained emotion cues, where subtle emotional states are usually implicitly encoded, thereby generating emotional misjudgments in complex scenarios. 2) Insufficient Discrimination. Existing methods could only identify regions generally associated with emotions, which fails to distinguish discriminative regions between semantically similar emotions, leading to ambiguous emotion judgements. To overcome these limitations, we present a training-free inference-time optimization framework, named Decoding Affective Nuances (DAN). Specifically, we propose a Hierarchical Emotional Reasoning Chain (HERC) that enhances the insufficient attribution by harmonizing fine-grained scene/object-level cues and performing a soft-gated reasoning. Furthermore, to discriminate between semantically proximal emotions, we design a Contrastive Discriminative Visual Pruning (CDVP), which isolates discriminative visual tokens to reason the final emotion category by computing the absolute discrepancy between the attention distributions of similar emotions. Performances on several benchmarks demonstrate that DAN significantly improves discrimination for affective nuances without consuming additional training resources, especially achieving +10.47% improvements with Qwen3-VL-8B-Instruct on WebEmo25 dataset that contains 25 fine-grained emotion categories.

Figures & tables

Figure 1: Two coupled limitations underlying the difficulty of MLLMs in distinguishing semantically proximal emotions using fine-grained visual evidence. (a) Insufficient Attribution: Global and undifferentiated visual processing dilutes localized affective cues. (b) Insufficient Discrimination: Failure cases exhibit small confidence margins and concentrated confusion among emotions within the same semantic macro-category, indicating difficulty in resolving ambiguous candidates.

Figure 2: The illustration of our proposed DAN framework. The Hierarchical Emotion Reasoning Chain module (HERC) firstly identifies the scene-level and object-level emotion clues, and then performs a soft-gated reasoning for a stable preliminary classification. Then, the Contrastive Discriminative Visual Pruning module (CDVP) is proposed to handle the affectively nuanced samples, which guides MLLM to focus on the most discriminative regions and reselect the final emotion.

Dataset	Emotion6	EmoSet8	WebEmo7	WebEmo25	Abstract8	Average
Qwen2.5-VL-7B-Instruct
Zero-shot	58.33	56.98	47.70	22.45	23.68	41.83
Zero-shot-CoT	59.76	57.19	47.25	22.15	23.25	41.92
SEPM	61.58	57.94	49.10	22.85	26.32	43.56
DAN(Ours)	63.13	59.10	50.90	24.85	29.82	45.56
Qwen3-VL-4B-Instruct

Table 1: Comparison with state-of-the-art on various emotion datasets. The optimal results are denoted by boldface.

HERC
Scene	Object	CDVP	Emotion6	WebEmo25	Abstract8
$\times$	$\times$	$\times$	55.21	21.10	29.38
$\checkmark$	$\times$	$\times$	54.88	21.25	31.14
$\times$	$\checkmark$	$\times$	56.40	22.50	28.51
$\checkmark$	$\checkmark$	$\times$	57.24	23.15	31.58
$\times$	$\times$	$\checkmark$	58.08	23.80	30.26

Table 2: Ablation study for key components.

Setting	Emotion6	WebEmo25	Abstract8
No-gating	54.55	20.90	28.51
Hard-gating	58.42	24.05	32.02
Soft-gating	59.09	24.80	33.77

Table 3: Discussion on soft-gated reasoning mechanism.

Setting	Emotion6	WebEmo25	Abstract8
Global Average	58.59	24.55	32.46
First-3 Average	57.58	24.30	30.26
Last Layer	58.42	24.50	31.58
Last-3 Average	59.09	24.80	33.77

Table 4: Discussion for strategies of attention scores.

Setting	Emotion6	WebEmo25	Abstract8
Only HERC	57.24	23.15	31.58
+ $S$	56.57	22.95	30.70
+ $S^{*}$	57.41	23.25	32.01
+ $S^{*}$ + $Q_{ctr}$	58.25	24.05	32.89
+ $S^{*}$ + $Q_{ctr}$ + CDP	59.09	24.80	33.77

Table 5: Discussion for CDVP module.

Dataset	Emotion6	WebEmo25	Abstract8
Random	55.56	23.70	27.63
Query-related	56.57	23.95	29.82
FoE-related	57.91	24.25	31.58
Ours	59.09	24.80	33.77

Table 6: Results of different pruning strategies.

Figure 3: Parameter sensitivity analysis.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

$k$	Emotion6	EmoSet	WebEmo7	WebEmo25
1	72.39	72.36	76.70	29.39
2	91.41	94.86	92.10	41.22
3	91.41	94.94	93.00	42.11

Appendix

Table A.1: Result for different truncation threshold

k

Figure A.2: Comparison of discriminative abilities.

Dataset	Emotion6	EmoSet8	Abstract8
SEPM	82.83	90.13	69.74
Scene	87.37	95.17	76.75
Object	86.03	92.91	71.93
HERC(Ours)	89.90	96.44	80.26

Appendix

Table A.2: Results of emotion polarity classification.

Setting	Emotion6	WebEmo25	Abstract8
Entire	58.59	24.60	32.01
Module-based	58.75	24.75	33.33
Alone	59.09	24.80	33.77

Appendix

Table A.3: Discussion for prompt injection strategies.

Setting	CUB	Cars	Aircraft	Dogs
MCQA ( 2025 )	23.30	21.94	31.14	22.50
FINER ( 2024 )	20.67	29.97	32.29	36.30
DAN	25.90	28.64	33.97	34.50

Appendix

Table A.4: Performance on FGIC task.

Setting	Emotion6	EmoSet8	WebEmo7	WebEmo25	Abstract8
Qwen2.5-VL-7B-Instruct	41.25	46.01	56.50	67.70	61.40
Qwen3-VL-4B-Instruct	59.26	62.44	69.85	78.20	71.93
Qwen3-VL-8B-Instruct	38.22	43.33	50.30	58.45	53.51
InternVL3.5-8B	40.57	44.68	52.15	60.05	50.88

Appendix

Table A.5: Result for the trigger rate of CDVP.

Explore similar work

EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models
Feb 27, 2026Yiyang Fang, Wenke Huang, Pei Fu +5Multimodal Large Language Models Emotion Recognition
School of Computer Science, Wuhan University. · MiLM Plus, Xiaomi Inc.
DSPO: Diversity-aware Subjective Policy Optimization for Robust Emotional Reasoning
Sep 29, 2026Cheng Ye, Weidong Chen, Bingyan Xu +1Emotion Recognition Frictive Policy Optimization
University of Science and Technology of China, Hefei
NTDH: Complex Reasoning for Comprehensive Affective Analysis
Aug 5, 2026Tianlei Zhu, Zhiwei Liu, Yuyan Wang +2Affective Computing Sentiment
Columbia University, New York, NY, USA · Department of Computer Science, The University of Manchester, Manchester, United Kingdom

Feb 27, 2026, cs.AIJ/K move · Enter open · S save

EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models

Yiyang Fang, Wenke Huang, Pei Fu, Yihao Yang+4

School of Computer Science, Wuhan University. · MiLM Plus, Xiaomi Inc.

Multimodal Large Language Models Emotion Recognition Offline Reinforcement Learning

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have shown remarkable progress in visual reasoning and understanding tasks but still struggle to capture the complexity and subjectivity of human emotions. Existing approaches based on supervised fine-tuning often suffer from limited generalization and poor interpretability, while reinforcement learning methods such as Group Relative Policy Optimization fail to align with the intrinsic characteristics of emotional cognition. To address these challenges, we propose Reflective Reinforcement Learning for Emotional Reasoning (EMO-R3), a framework designed to enhance the emotional reasoning ability of MLLMs. Specifically, we introduce Structured Emotional Thinking to guide the model to perform step-by-step emotional reasoning in a structured and interpretable manner, and design a Reflective Emotional Reward that enables the model to re-evaluate its reasoning based on visual-text consistency and emotional coherence. Extensive experiments demonstrate that EMO-R3 significantly improves both the interpretability and emotional intelligence of MLLMs, achieving superior performance across multiple visual emotional understanding benchmarks.

Abstract

Explore similar work

EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models

DSPO: Diversity-aware Subjective Policy Optimization for Robust Emotional Reasoning

NTDH: Complex Reasoning for Comprehensive Affective Analysis