Multimodal Emotion Reasoning (MER) requires perceiving and integrating emotion cues from multiple modalities. Recent advances increasingly leverage Multimodal Large Language Models (MLLMs) to enhance reasoning capacity. However, due to the inherent heterogeneity in conveying emotions across modalities, these models often suffer from modality imbalance and confront flawed emotion understanding. In this paper, we conduct an in-depth analysis on MER models and uncover two key empirical insights. (i) In MER, modality imbalance manifests two distinct failure modes: unimodal over-reliance and insufficient cross-modal utilization, which call for different intervention strategies. (ii) MLLM layers exhibit implicit functional hierarchies in multimodal emotion processing, playing separable roles in stabilizing unimodal representations and integrating cross-modal information. Motivated by these observations, we propose a Layer-Aware Modality Rebalancing (LAMR) framework by explicitly aligning intervention strategies with layer roles. Specifically, we first introduce two quantized metrics for adaptive layer role profiling. In each layer, Modality Entropy (ME) measures the dispersion of modality contributions, and Cross-modal Intensity (CMI) captures the strength of cross-modal interactions. Then, we respectively enhance modality equilibrium in stabilization layers and cross-modal propagation in integration layers. Extensive experiments on two comprehensive MER benchmarks demonstrate that LAMR effectively mitigates modality imbalance and improves emotion reasoning across diverse MLLM backbones, without additional training.
Figures & tables
Figure 1. Diagnostic analysis of attention imbalance and layer-wise heterogeneity in MLLM-based emotion reasoning. (a) Attention statistics of AffectGPT. (b) An example of erroneous emotion reasoning given by AffectGPT. (c) Layer-wise self-modal and cross-modal attention heatmaps. (d) Layer-wise evolution of modality attention gaps and cross-modal attention. Four panels diagnose modality imbalance in AffectGPT: modality-level attention statistics, an erroneous multimodal emotion reasoning example, layer-wise self- and cross-modal attention heatmaps, and layer-wise trends in cross-modal attention and modality imbalance.
Figure 2. Overview of the proposed LAMR framework. Adaptive Layer Role Profiling automatically partitions Transformer layers into stabilization layers and integration layers. Layer-wise Attention Rebalance performs layer-specific attention interventions during inference, mitigating unimodal over-reliance in stabilization layers and enhancing cross-modal propagation in integration layers. Pipeline of LAMR. Layer-wise modality entropy and cross-modal intensity are combined to profile stabilization and integration layers, after which modality equilibrium and cross-modal propagation are applied according to the identified layer roles.
Methods
Basic
Sentiment
Fine-grained
Mean ↑
MER23 ↑
MER24 ↑
MELD ↑
IEMO ↑
MOSI ↑
MOSEI ↑
SIMS ↑
SIMS v2 ↑
OV-MERD+ ↑
Video-LLaVA ( Lin et al., 2024 )
36.93
30.25
30.73
38.95
56.37
61.64
53.28
57.45
34.00
44.40
Video-ChatGPT ( Maaz et al., 2024 )
44.86
46.80
37.33
56.83
54.42
63.12
64.82
65.80
39.80
52.64
VideoChat2 ( Li et al., 2024 )
33.67
54.50
36.64
48.70
66.84
54.32
69.49
70.66
39.21
52.67
LLaMA-VID ( Li et al., 2025b )
50.72
57.60
42.75
46.02
61.78
63.89
69.35
67.48
45.01
56.07
VideoChat ( Li et al., 2025a )
48.73
57.30
41.11
48.38
65.13
63.61
69.52
72.14
44.52
56.72
Table 1. Comparison on MER-UniBench ( Lian et al., 2025a ) for multimodal emotion recognition. This benchmark comprises nine datasets covering three tasks: basic emotion recognition, sentiment analysis, and fine-grained emotion recognition. “Mean” represents the average performance across all datasets. We compare various mainstream MLLMs and apply our method to three baselines.
ER-Lab
FG-SA
IR
ML-ER
Noise-ER
SA
Mean
Methods
Rec
Rea
CoT
Rec
Rea
CoT
Rec
Rea
CoT
Rec
Rea
CoT
Rec
Rea
CoT
Rec
Rea
CoT
Rec
Rea
CoT
PandaGPT ( Su et al., 2023 )
15.5
42.6
30.6
34.8
47.8
41.3
13.8
48.5
31.2
2.0
37.8
21.4
21.2
40.9
31.3
35.2
51.0
43.1
20.4
44.8
33.2
+Ours
19.7
45.6
31.1
37.0
49.4
43.2
15.8
48.8
32.3
5.0
42.0
22.0
22.0
41.5
31.4
41.2
54.5
47.9
23.5
47.0
34.7
Emotion-LLaMA ( Cheng et al., 2024 )
11.7
32.8
22.3
26.8
54.4
41.0
11.0
45.1
28.0
8.3
46.9
27.6
44.0
60.3
52.4
36.2
61.5
48.9
23.0
50.2
36.7
+Ours
14.8
37.5
26.2
29.0
55.2
41.7
13.8
47.3
30.6
8.7
49.9
29.3
45.6
60.8
52.9
36.5
63.6
50.1
24.7
52.4
38.5
AffectGPT ( Lian et al., 2025a )
16.2
54.2
35.2
15.6
63.5
39.6
3.0
50.4
26.7
0.6
49.9
25.6
28.8
65.3
47.0
25.5
67.6
46.7
15.0
58.5
36.8
Table 2. Comparison on MME-Emotion ( Zhang et al., 2026a ) for emotional understanding and reasoning. Results cover six emotional tasks and three evaluation metrics, all computed via a multi-agent system framework.
Strategy
MELD
IEMOCAP
SIMS
Mean
AffectGPT
52.16
55.50
86.41
64.69
Random Assignment
30% Integration
53.99
51.53
80.44
61.99
50% Integration
49.71
47.51
80.79
59.34
70% Integration
47.15
46.28
80.68
58.04
Static Assignment
Table 3. Ablation study on layer partition strategies. “Shallow”, “Middle”, and “Deep” denote static heuristic partitions that evenly divide the layers into three contiguous stages. We designate one or two of these stages as integration layers.
CMI
ME
MELD
IEMOCAP
SIMS
Mean
✓
✗
52.25
53.62
83.67
63.18
✗
✓
51.60
53.18
85.47
63.42
✓
✓
53.78
56.17
88.45
66.13
Table 4. Ablation study on the quantitative metrics used for adaptive layer role profiling, i.e.,, CMI and ME.
Strategy
MELD
IEMOCAP
SIMS
Mean
AffectGPT
52.16
55.50
86.41
64.69
Uniform Propagation
47.75
51.23
81.53
60.17
Uniform Equilibrium
53.53
51.11
83.92
62.85
Reversed Intervention
53.28
51.44
84.26
62.99
Ours
53.78
56.17
88.45
66.13
Table 5. Ablation study on layer-wise attention rebalance. “Reversed Intervention” denotes exchanging intervention strategies for two types of layers.
Model
Video-aligned
Audio-aligned
Consistent
Acc
Recall
Avg
Acc
Recall
Avg
Acc
Recall
Avg
AffectGPT
44.92
50.80
47.86
52.26
55.54
53.90
51.86
56.90
54.38
+ Ours
45.89
52.66
49.28
52.56
60.00
56.28
52.72
59.84
56.28
Table 6. Performance comparison on the CA-MER benchmark ( Han et al., 2025 ) under realistic emotion conflicts. The three subsets—video-aligned, audio-aligned, and consistent—represent settings in which the video modality, the audio modality, or all modalities reflect the true emotion, respectively.
Methods
MER23
MER24
MELD
IEMO
SIMS
SIMSv2
OV.
AffectGPT
73.60
73.28
52.16
55.50
86.41
81.91
59.44
+ Devils ( Jiang et al., 2025 )
75.27
70.13
52.34
49.40
79.93
81.95
59.80
+ AVCD ( Jung et al., 2025 )
75.35
75.05
53.56
51.83
80.97
82.43
60.69
+ LISA ( Guo et al., 2025 )
73.14
75.79
52.88
55.53
86.43
85.41
60.33
+ Ours
76.35
73.62
53.78
56.17
88.45
85.54
61.04
Table 7. Comparison with other training-free rebalancing methods on MER-UniBench.
Model Setup
MER24
MELD
IEMOCAP
Avg. Δ
AffectGPT-1.5B
75.39
49.34
52.32
–
+ Ours
76.68
49.97
52.84
0.81 ↑
AffectGPT-7B
73.28
52.16
55.50
–
+ Ours
73.62
53.78
56.17
0.88 ↑
AffectGPT-14B
77.36
55.01
60.80
–
+ Ours
80.02
55.57
65.44
2.62 ↑
Table 8. Scalability analysis across different model sizes ( 1.5 B, 7 B, and 14 B).
Figure 3. Effect of LAMR on modality-level and cross-modal attention rebalancing. (a/b) Modality-wise and cross-modal attention allocation with LAMR . Percentages indicate changes relative to the baseline. Radar charts compare modality-wise and cross-modal attention before and after LAMR for AffectGPT, PandaGPT, and EmotionLLaMA, with percentage changes relative to each baseline.
Figure 4. Two challenging examples of modality imbalance. Cyan, green, and red fonts indicate shared correct, correct, and erroneous interpretations, respectively. Two dialogue examples compare AffectGPT with AffectGPT plus LAMR. Colored text marks interpretations that are shared and correct, newly correct, or erroneous, showing how LAMR reduces modality-driven reasoning errors.
Modality imbalance studies how unequal signal quality, learning dynamics, or model attention make multimodal systems over-rely on some inputs and neglect others.
Multimodal Emotion Recognition (MER) is critical for interpreting real-world interactions. While Multimodal Large Language Models (MLLM) have shown promise in MER, their internal decision-making mechanisms under modality conflict and missingness remain largely underexplored. In this paper, to systematically investigate these behaviors, we introduce EmoMM, a comprehensive benchmark featuring modality-aligned, conflict, and missing subsets. Through extensive evaluation, we uncover a Video Contribution Collapse (VCC) phenomenon, where MLLM marginalize video evidence due to high token redundancy and modality preferences. To address this, we propose Conflict-aware Head-level Attention Steering (CHASE), a lightweight mechanism that detects modality conflicts and performs inference-time attention steering, effectively mitigating decision bias without retraining the backbone. Experimental results demonstrate that CHASE consistently improves performance across various settings, significantly enhancing the reliability of MLLM in complex affective scenarios.
Yueru Sun, Yimeng Zhang, Haoyu Gu +5
School of Future Technology, South China University of Technology, Guangzhou, China
We find that current emotion-oriented Omni-MLLMs still lack reliable omni-modal perception: they (i) underutilize multimodal cues in their reasoning trajectories and (ii) exhibit unfaithful behavior, often hallucinating modality-specific statements from other modalities. Building on these insights, we propose OPPO (Omni-Perception Policy Optimization), a reinforcement learning framework that explicitly optimizes multimodal perception. First, an Omni-Perception Reward decomposes ground-truth reasoning into fine-grained visual, acoustic, and emotion cues and rewards trajectories that semantically recover these cues. Second, an Omni-Perception Loss compares the policy under full and unimodally masked inputs, applying a KL penalty only to modality-specific evidence tokens to suppress cross-modal hallucination. We further introduce MEP-Bench, a diagnostic benchmark that quantifies utilization and faithfulness. Experiments show that OPPO achieves state-of-the-art performance on MER-UniBench and MME-Emotion, while substantially improving utilization and faithfulness scores on MEP-Bench, highlighting the importance of sufficient and faithful omni perception for multimodal emotion reasoning.
Zhiyuan Han, Beier Zhu, Wenwen Tong +6
University of Science and Technology of China · SenseTime · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center +1
Multimodal emotion recognition in conversation (MERC) requires understanding complex interactions between verbal and non-verbal cues. However, most existing approaches fundamentally treat this as a direct input-output (multimodal cues-emotion labels) mapping problem, overlooking the causal reasoning that humans use when interpreting emotions. We propose rationale-guided learning (RGL), a novel framework that transforms MERC into a cognitively-inspired reasoning task. Based on dual-process theory, we decompose emotional reasoning into three facets: Intuitive (immediate perception, System 1), Contextual (situational analysis, System 2), and Integrative (synthesis of both). We leverage an MLLM offline to generate structured rationales, which are encoded as memories to guide model training via aligning internal representations with human-like reasoning patterns. Our final model operates without any MLLM overheads at inference time. Experimental results show that RGL achieves state-of-the-art performance on the IEMOCAP and MELD benchmarks. Further, for interpretation, we demonstrate that the model's internal features effectively retrieve semantically correct rationales for unseen test samples, validating its rationale reasoning capabilities.
Sujung Oh, Jung Uk Kim, Sangmin Lee
Pixel Lab, Sungkyunkwan University, South Korea · Visual AI Lab, Kyung Hee University, South Korea · Pixel Lab, Korea University, South Korea