Multimodal Emotion Reasoning (MER) requires perceiving and integrating emotion cues from multiple modalities. Recent advances increasingly leverage Multimodal Large Language Models (MLLMs) to enhance reasoning capacity. However, due to the inherent heterogeneity in conveying emotions across modalities, these models often suffer from modality imbalance and confront flawed emotion understanding. In this paper, we conduct an in-depth analysis on MER models and uncover two key empirical insights. (i) In MER, modality imbalance manifests two distinct failure modes: unimodal over-reliance and insufficient cross-modal utilization, which call for different intervention strategies. (ii) MLLM layers exhibit implicit functional hierarchies in multimodal emotion processing, playing separable roles in stabilizing unimodal representations and integrating cross-modal information. Motivated by these observations, we propose a Layer-Aware Modality Rebalancing (LAMR) framework by explicitly aligning intervention strategies with layer roles. Specifically, we first introduce two quantized metrics for adaptive layer role profiling. In each layer, Modality Entropy (ME) measures the dispersion of modality contributions, and Cross-modal Intensity (CMI) captures the strength of cross-modal interactions. Then, we respectively enhance modality equilibrium in stabilization layers and cross-modal propagation in integration layers. Extensive experiments on two comprehensive MER benchmarks demonstrate that LAMR effectively mitigates modality imbalance and improves emotion reasoning across diverse MLLM backbones, without additional training.
Figures & tables
Figure 1. Diagnostic analysis of attention imbalance and layer-wise heterogeneity in MLLM-based emotion reasoning. (a) Attention statistics of AffectGPT. (b) An example of erroneous emotion reasoning given by AffectGPT. (c) Layer-wise self-modal and cross-modal attention heatmaps. (d) Layer-wise evolution of modality attention gaps and cross-modal attention. Four panels diagnose modality imbalance in AffectGPT: modality-level attention statistics, an erroneous multimodal emotion reasoning example, layer-wise self- and cross-modal attention heatmaps, and layer-wise trends in cross-modal attention and modality imbalance.
Figure 2. Overview of the proposed LAMR framework. Adaptive Layer Role Profiling automatically partitions Transformer layers into stabilization layers and integration layers. Layer-wise Attention Rebalance performs layer-specific attention interventions during inference, mitigating unimodal over-reliance in stabilization layers and enhancing cross-modal propagation in integration layers. Pipeline of LAMR. Layer-wise modality entropy and cross-modal intensity are combined to profile stabilization and integration layers, after which modality equilibrium and cross-modal propagation are applied according to the identified layer roles.
Methods
Basic
Sentiment
Fine-grained
Mean ↑
MER23 ↑
MER24 ↑
MELD ↑
IEMO ↑
MOSI ↑
MOSEI ↑
SIMS ↑
SIMS v2 ↑
OV-MERD+ ↑
Video-LLaVA ( Lin et al., 2024 )
36.93
30.25
30.73
38.95
56.37
61.64
53.28
57.45
34.00
44.40
Video-ChatGPT ( Maaz et al., 2024 )
44.86
46.80
37.33
56.83
54.42
63.12
64.82
65.80
39.80
52.64
VideoChat2 ( Li et al., 2024 )
33.67
54.50
36.64
48.70
66.84
54.32
69.49
70.66
39.21
52.67
LLaMA-VID ( Li et al., 2025b )
50.72
57.60
42.75
46.02
61.78
63.89
69.35
67.48
45.01
56.07
VideoChat ( Li et al., 2025a )
48.73
57.30
41.11
48.38
65.13
63.61
69.52
72.14
44.52
56.72
Table 1. Comparison on MER-UniBench ( Lian et al., 2025a ) for multimodal emotion recognition. This benchmark comprises nine datasets covering three tasks: basic emotion recognition, sentiment analysis, and fine-grained emotion recognition. “Mean” represents the average performance across all datasets. We compare various mainstream MLLMs and apply our method to three baselines.
ER-Lab
FG-SA
IR
ML-ER
Noise-ER
SA
Mean
Methods
Rec
Rea
CoT
Rec
Rea
CoT
Rec
Rea
CoT
Rec
Rea
CoT
Rec
Rea
CoT
Rec
Rea
CoT
Rec
Rea
CoT
PandaGPT ( Su et al., 2023 )
15.5
42.6
30.6
34.8
47.8
41.3
13.8
48.5
31.2
2.0
37.8
21.4
21.2
40.9
31.3
35.2
51.0
43.1
20.4
44.8
33.2
+Ours
19.7
45.6
31.1
37.0
49.4
43.2
15.8
48.8
32.3
5.0
42.0
22.0
22.0
41.5
31.4
41.2
54.5
47.9
23.5
47.0
34.7
Emotion-LLaMA ( Cheng et al., 2024 )
11.7
32.8
22.3
26.8
54.4
41.0
11.0
45.1
28.0
8.3
46.9
27.6
44.0
60.3
52.4
36.2
61.5
48.9
23.0
50.2
36.7
+Ours
14.8
37.5
26.2
29.0
55.2
41.7
13.8
47.3
30.6
8.7
49.9
29.3
45.6
60.8
52.9
36.5
63.6
50.1
24.7
52.4
38.5
AffectGPT ( Lian et al., 2025a )
16.2
54.2
35.2
15.6
63.5
39.6
3.0
50.4
26.7
0.6
49.9
25.6
28.8
65.3
47.0
25.5
67.6
46.7
15.0
58.5
36.8
Table 2. Comparison on MME-Emotion ( Zhang et al., 2026a ) for emotional understanding and reasoning. Results cover six emotional tasks and three evaluation metrics, all computed via a multi-agent system framework.
Strategy
MELD
IEMOCAP
SIMS
Mean
AffectGPT
52.16
55.50
86.41
64.69
Random Assignment
30% Integration
53.99
51.53
80.44
61.99
50% Integration
49.71
47.51
80.79
59.34
70% Integration
47.15
46.28
80.68
58.04
Static Assignment
Table 3. Ablation study on layer partition strategies. “Shallow”, “Middle”, and “Deep” denote static heuristic partitions that evenly divide the layers into three contiguous stages. We designate one or two of these stages as integration layers.
CMI
ME
MELD
IEMOCAP
SIMS
Mean
✓
✗
52.25
53.62
83.67
63.18
✗
✓
51.60
53.18
85.47
63.42
✓
✓
53.78
56.17
88.45
66.13
Table 4. Ablation study on the quantitative metrics used for adaptive layer role profiling, i.e.,, CMI and ME.
Strategy
MELD
IEMOCAP
SIMS
Mean
AffectGPT
52.16
55.50
86.41
64.69
Uniform Propagation
47.75
51.23
81.53
60.17
Uniform Equilibrium
53.53
51.11
83.92
62.85
Reversed Intervention
53.28
51.44
84.26
62.99
Ours
53.78
56.17
88.45
66.13
Table 5. Ablation study on layer-wise attention rebalance. “Reversed Intervention” denotes exchanging intervention strategies for two types of layers.
Model
Video-aligned
Audio-aligned
Consistent
Acc
Recall
Avg
Acc
Recall
Avg
Acc
Recall
Avg
AffectGPT
44.92
50.80
47.86
52.26
55.54
53.90
51.86
56.90
54.38
+ Ours
45.89
52.66
49.28
52.56
60.00
56.28
52.72
59.84
56.28
Table 6. Performance comparison on the CA-MER benchmark ( Han et al., 2025 ) under realistic emotion conflicts. The three subsets—video-aligned, audio-aligned, and consistent—represent settings in which the video modality, the audio modality, or all modalities reflect the true emotion, respectively.
Methods
MER23
MER24
MELD
IEMO
SIMS
SIMSv2
OV.
AffectGPT
73.60
73.28
52.16
55.50
86.41
81.91
59.44
+ Devils ( Jiang et al., 2025 )
75.27
70.13
52.34
49.40
79.93
81.95
59.80
+ AVCD ( Jung et al., 2025 )
75.35
75.05
53.56
51.83
80.97
82.43
60.69
+ LISA ( Guo et al., 2025 )
73.14
75.79
52.88
55.53
86.43
85.41
60.33
+ Ours
76.35
73.62
53.78
56.17
88.45
85.54
61.04
Table 7. Comparison with other training-free rebalancing methods on MER-UniBench.
Model Setup
MER24
MELD
IEMOCAP
Avg. Δ
AffectGPT-1.5B
75.39
49.34
52.32
–
+ Ours
76.68
49.97
52.84
0.81 ↑
AffectGPT-7B
73.28
52.16
55.50
–
+ Ours
73.62
53.78
56.17
0.88 ↑
AffectGPT-14B
77.36
55.01
60.80
–
+ Ours
80.02
55.57
65.44
2.62 ↑
Table 8. Scalability analysis across different model sizes ( 1.5 B, 7 B, and 14 B).
Figure 3. Effect of LAMR on modality-level and cross-modal attention rebalancing. (a/b) Modality-wise and cross-modal attention allocation with LAMR . Percentages indicate changes relative to the baseline. Radar charts compare modality-wise and cross-modal attention before and after LAMR for AffectGPT, PandaGPT, and EmotionLLaMA, with percentage changes relative to each baseline.
Figure 4. Two challenging examples of modality imbalance. Cyan, green, and red fonts indicate shared correct, correct, and erroneous interpretations, respectively. Two dialogue examples compare AffectGPT with AffectGPT plus LAMR. Colored text marks interpretations that are shared and correct, newly correct, or erroneous, showing how LAMR reduces modality-driven reasoning errors.
Modality imbalance studies how unequal signal quality, learning dynamics, or model attention make multimodal systems over-rely on some inputs and neglect others.