Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition

Zhexian Huang; Bo Zhao; Hui Ma; Zhishu Liu; Jie Zhang; Ruixin Zhang; Shouhong Ding; Zitong Yu

cs.CVMar 7, 2026

Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition

Authors: Zhexian Huang, Bo Zhao, Hui Ma, Zhishu Liu, Jie Zhang, Ruixin Zhang, Shouhong Ding, Zitong Yu

Organizations: Great Bay University · The Chinese University of Hong Kong, Shenzhen · City University of Hong Kong (Dongguan) · Tencent Youtu Lab

Abstract

Multimodal emotion recognition fuses cues such as text, video, and audio to understand individual emotional states. Prior methods face two main limitations: mechanically relying on independent unimodal performance, thereby missing genuine complementary contributions, and coarse-grained fusion conflicting with the fine-grained representations required by emotion tasks. As inconsistent information density across heterogeneous modalities hinders inter-modal feature mining, we propose the Complementarity-Supervised Multi-Band Expert Network, named Atsuko, to model fine-grained complementary features via multi-scale band decomposition and expert collaboration. Specifically, we orthogonally decompose each modality's features into high, mid, and low-frequency components. Building upon this band-level routing, we design a modality-level router with a dual-path mechanism for fine-grained cross-band selection and cross-modal fusion. To mitigate shortcut learning from dominant modalities, we propose the Marginal Complementarity Module (MCM) to quantify performance loss when removing each modality via bi-modal comparison. The resulting complementarity distribution provides soft supervision, guiding the router to focus on modalities contributing unique information gains. Extensive experiments show our method achieves superior performance on the CMU-MOSI, CMU-MOSEI, CH-SIMS, CH-SIMSv2, and MIntRec benchmarks.

Figures & tables

Figure 1: Two fundamental limitations of existing MER methods. (a) Coarse-grained fusion: treating each modality’s temporal features as a homogeneous signal discards fine-grained emotional cues in specific frequency bands. (b) Shortcut learning: performance-based weighting over-relies on the dominant (text) modality, whereas visual and acoustic modalities carry the genuinely complementary cues. Complementarity-based weighting correctly amplifies these contributions.

Figure 2: The overall architecture of Atsuko. (1) Spectral-Band Routing: Multi-modal temporal inputs are orthogonally decomposed into distinct frequency bands via graph Laplacian and dynamically weighted to yield band-enhanced features $\tilde{X}_{m}$ . (2) Modality-Level Routing: A dual-path mechanism (pre- and post-attention) assigns dynamic fusion weights $w_{m}$ to the deep semantic features. (3) Marginal Complementarity Module (MCM): Bimodal contrastive branches quantify each modality’s marginal contribution, providing soft supervision to guide the modality router toward complementary information gains, augmented by feature distillation.

Methods	Setting	MOSI				MOSEI
Methods		ACC7 $\uparrow$	ACC2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	ACC7 $\uparrow$	ACC2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$
EF-LSTM [ 6 ]	Aligned	33.7	75.3	75.2	1.386	47.4	78.2	77.9	0.620
LF-DNN [ 6 ]		31.5	78.4	78.3	0.972	51.7	83.5	83.1	0.568
TFN [ 6 ]		31.9	78.8	78.9	0.953	50.9	80.4	80.7	0.574
LMF [ 7 ]		36.9	78.7	78.7	0.931	52.3	84.7	84.5	0.564
MFN [ 9 ]		35.6	78.4	78.4	0.964	50.8	84.0	84.0	0.574

Table 1: Performance comparison on CMU-MOSI and CMU-MOSEI datasets under aligned and unaligned settings. The best results are highlighted in bold .

\uparrow

indicates higher is better;

\downarrow

indicates lower is better.

Methods	SIMS					SIMS_v2
Methods	ACC5 $\uparrow$	ACC3 $\uparrow$	ACC2 $\uparrow$	F1 $\uparrow$	Avg $\uparrow$	ACC5 $\uparrow$	ACC3 $\uparrow$	ACC2 $\uparrow$	F1 $\uparrow$	Avg $\uparrow$
TFN [ 6 ]	39.30	65.12	78.38	78.62	65.36	52.55	72.21	80.14	80.14	71.26
LMF [ 7 ]	40.53	64.68	77.77	77.88	65.19	47.79	64.90	74.18	73.88	65.19
Self-MM [ 56 ]	41.53	65.47	80.04	80.44	66.87	52.77	72.61	79.69	76.76	70.46
BBFN [ 59 ]	40.92	61.05	78.12	77.88	64.49	53.29	74.47	78.53	78.41	71.18
CENet [ 60 ]	33.92	62.58	77.90	77.53	63.66	53.04	73.1	79.56	79.63	71.33

Table 2: Performance comparison on CH-SIMS and CH-SIMS_v2 datasets. The best results are highlighted in bold .

Methods	ACC $\uparrow$	F1 $\uparrow$	P $\uparrow$	R $\uparrow$
MAG-BERT [ 61 ]	70.34	68.19	68.31	69.36
MulT [ 8 ]	72.58	69.36	70.73	69.47
MISA [ 11 ]	72.36	70.57	71.24	70.41
EMOE [ 5 ]	72.58	70.73	72.08	70.86
Atsuko (Ours)	74.16	71.22	72.93	70.86

Table 3: Results on MIntRec dataset.

SBN	MCM	MOSI				MOSEI
SBN	MCM	ACC7 $\uparrow$	ACC2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	ACC7 $\uparrow$	ACC2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$
		46.0	83.5	83.5	0.716	52.3	83.8	83.7	0.558
✓		47.1	85.5	85.5	0.701	53.0	85.1	85.2	0.548
	✓	46.9	85.4	85.3	0.700	53.8	85.2	85.1	0.535
✓	✓	48.7	86.3	86.2	0.696	54.4	85.6	85.7	0.530

Table 4: Ablation study of the key components in Atsuko on MOSI and MOSEI dataset.

\uparrow

indicates higher is better;

\downarrow

indicates lower is better.

Figure 3: Hyperparameter sensitivity analysis on CMU-MOSI and CMU-MOSEI (Acc-7, %). Each sub-figure sweeps one hyperparameter while holding all others at their default values. The star marker ( $\bigstar$ ) denotes the default configuration adopted in all experiments. Blue lines: MOSI (left $y$ -axis); red lines: MOSEI (right $y$ -axis).

Figure 4: t-SNE visualization of feature distributions on MOSI test set. (a) Baseline (w/o SBN & MCM), (b) w/o MCM, (c) w/o SBN, and (d) Atsuko (Full). Different colors represent varying sentiment intensities. The proposed method shows clearer cluster margins and better separability.

Figure 5: Visualization of modality weights predicted by the router on randomly selected CMU-MOSI test samples. Gray bars denote the baseline model without MCM, and colored bars denote the MCM-enhanced model.

Panel of Figure 6 — Figure 6: Spectral band decomposition of a strongly positive sample (Label: +2.00) from the CMU-MOSI test set across all three modalities.

Explore similar work

Sep 4, 2026

Enhancing Multimodal Emotion Recognition via Multi-Feature Encoding and Attention-Based Fusion

Multimodal emotion recognition has attracted growing interest due to its importance in human-computer interaction, remote education, and healthcare. This paper proposes a novel multimodal emotion recognition framework that integrates rich audio and visual feature extraction with an attention-based fusion strategy. For audio, we extract three complementary feature types: semantic embeddings from Wav2Vec2, MFCC features, and statistical acoustic descriptors such as pitch, energy, and rhythm. These are aligned and fused via a BiLSTM to capture temporal dependencies. For video, we propose a ResNet50-BiLSTM architecture that combines deep residual learning and sequential modeling to extract expressive spatiotemporal features from facial sequences. To enhance multimodal synergy, we introduce a feature-level fusion mechanism based on multi-head attention, allowing the model to adaptively weigh contributions across modalities. Experiments conducted on the MELD and IEMOCAP datasets demonstrate that our model significantly outperforms baselines in both accuracy and robustness. Furthermore, ablation studies show that the attention-based fusion strategy significantly improves performance in unbalanced data settings. Our findings suggest that the proposed framework effectively captures diverse emotional cues from speech and visual expressions, and offers a practical and generalizable approach for real-world multimodal emotion recognition tasks.

Xu Lin, Ke Wang, Hui Kang +1

College of Computer Science and Technology, Jilin University, Changchun 130012, China · Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun 130012, China

Multimodal Emotion Recognition Multimodal Fusion

Ask PDF

Mar 7, 2026

AMB-DSGDN: Adaptive Modality-Balanced Dynamic Semantic Graph Differential Network for Multimodal Emotion Recognition

Multimodal dialogue emotion recognition captures emotional cues by fusing text, visual, and audio modalities. However, existing approaches still suffer from notable limitations in modeling emotional dependencies and learning multimodal representations. On the one hand, they are unable to effectively filter out redundant or noisy signals within multimodal features, which hinders the accurate capture of the dynamic evolution of emotional states across and within speakers. On the other hand, during multimodal feature learning, dominant modalities tend to overwhelm the fusion process, thereby suppressing the complementary contributions of non-dominant modalities such as speech and vision, ultimately constraining the overall recognition performance. To address these challenges, we propose an Adaptive Modality-Balanced Dynamic Semantic Graph Differential Network (AMB-DSGDN). Concretely, we first construct modality-specific subgraphs for text, speech, and vision, where each modality contains intra-speaker and inter-speaker graphs to capture both self-continuity and cross-speaker emotional dependencies. On top of these subgraphs, we introduce a differential graph attention mechanism, which computes the discrepancy between two sets of attention maps. By explicitly contrasting these attention distributions, the mechanism cancels out shared noise patterns while retaining modality-specific and context-relevant signals, thereby yielding purer and more discriminative emotional representations. In addition, we design an adaptive modality balancing mechanism, which estimates a dropout probability for each modality according to its relative contribution in emotion modeling.

Yunsheng Wang, Yuntao Shou, Yilong Tan +3

College of Computer and Mathematics, Central South University of Forestry and Technology · Department of Computer Science, State University of New York

Emotion Recognition in Conversations Multimodal Emotion Recognition Modality Imbalance Multimodal Graph Learning

Ask PDF

Sep 12, 2026

ReH-FUSE: Reliability-Aware Hierarchical Fusion of Experts for Multimodal Emotion Recognition in Conversation

Multimodal emotion recognition in conversation (ERC) requires adapting to the instance-dependent reliability of different evidence sources. Lexical content may be decisive, vocal expression may provide complementary cues, or accurate recognition may require cross-modal interaction; fixed fusion does not explicitly account for this variation. We propose ReH-FUSE, a reliability-aware framework with dialogue-aware text, audio, and cross-modal experts. Its decision-level router first models the relative preference between text and audio and then balances the resulting unimodal mixture against the cross-modal expert. This factorization separates unimodal competition from cross-modal selection. Across three independent runs on IEMOCAP, ReH-FUSE achieves 74.34% weighted F1 and 73.11% macro F1; on MELD, it achieves 68.03% weighted F1. Controlled ablations show that learned routing outperforms uniform expert averaging and benefits from cross-modal interaction.

Guan-Hua Wen, Hou-Chiang Tseng, Kuan-Yu Chen

National Taiwan University of Science and Technology, Taipei, Taiwan

Emotion Recognition in Conversations Multimodal Emotion Recognition Uncertainty-Aware Multimodal Fusion Mixture of Experts

Ask PDF