Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition

Zhexian Huang; Bo Zhao; Hui Ma; Zhishu Liu; Jie Zhang; Ruixin Zhang; Shouhong Ding; Zitong Yu

cs.CVMar 7, 2026

Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition

Authors: Zhexian Huang, Bo Zhao, Hui Ma, Zhishu Liu, Jie Zhang, Ruixin Zhang, Shouhong Ding, Zitong Yu

Organizations: Great Bay University · The Chinese University of Hong Kong, Shenzhen · City University of Hong Kong (Dongguan) · Tencent Youtu Lab

Abstract

Multimodal emotion recognition fuses cues such as text, video, and audio to understand individual emotional states. Prior methods face two main limitations: mechanically relying on independent unimodal performance, thereby missing genuine complementary contributions, and coarse-grained fusion conflicting with the fine-grained representations required by emotion tasks. As inconsistent information density across heterogeneous modalities hinders inter-modal feature mining, we propose the Complementarity-Supervised Multi-Band Expert Network, named Atsuko, to model fine-grained complementary features via multi-scale band decomposition and expert collaboration. Specifically, we orthogonally decompose each modality's features into high, mid, and low-frequency components. Building upon this band-level routing, we design a modality-level router with a dual-path mechanism for fine-grained cross-band selection and cross-modal fusion. To mitigate shortcut learning from dominant modalities, we propose the Marginal Complementarity Module (MCM) to quantify performance loss when removing each modality via bi-modal comparison. The resulting complementarity distribution provides soft supervision, guiding the router to focus on modalities contributing unique information gains. Extensive experiments show our method achieves superior performance on the CMU-MOSI, CMU-MOSEI, CH-SIMS, CH-SIMSv2, and MIntRec benchmarks.

Figures & tables

Figure 1: Two fundamental limitations of existing MER methods. (a) Coarse-grained fusion: treating each modality’s temporal features as a homogeneous signal discards fine-grained emotional cues in specific frequency bands. (b) Shortcut learning: performance-based weighting over-relies on the dominant (text) modality, whereas visual and acoustic modalities carry the genuinely complementary cues. Complementarity-based weighting correctly amplifies these contributions.

Figure 2: The overall architecture of Atsuko. (1) Spectral-Band Routing: Multi-modal temporal inputs are orthogonally decomposed into distinct frequency bands via graph Laplacian and dynamically weighted to yield band-enhanced features $\tilde{X}_{m}$ . (2) Modality-Level Routing: A dual-path mechanism (pre- and post-attention) assigns dynamic fusion weights $w_{m}$ to the deep semantic features. (3) Marginal Complementarity Module (MCM): Bimodal contrastive branches quantify each modality’s marginal contribution, providing soft supervision to guide the modality router toward complementary information gains, augmented by feature distillation.

Methods	Setting	MOSI				MOSEI
Methods		ACC7 $\uparrow$	ACC2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	ACC7 $\uparrow$	ACC2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$
EF-LSTM [ 6 ]	Aligned	33.7	75.3	75.2	1.386	47.4	78.2	77.9	0.620
LF-DNN [ 6 ]		31.5	78.4	78.3	0.972	51.7	83.5	83.1	0.568
TFN [ 6 ]		31.9	78.8	78.9	0.953	50.9	80.4	80.7	0.574
LMF [ 7 ]		36.9	78.7	78.7	0.931	52.3	84.7	84.5	0.564
MFN [ 9 ]		35.6	78.4	78.4	0.964	50.8	84.0	84.0	0.574

Table 1: Performance comparison on CMU-MOSI and CMU-MOSEI datasets under aligned and unaligned settings. The best results are highlighted in bold .

\uparrow

indicates higher is better;

\downarrow

indicates lower is better.

Methods	SIMS					SIMS_v2
Methods	ACC5 $\uparrow$	ACC3 $\uparrow$	ACC2 $\uparrow$	F1 $\uparrow$	Avg $\uparrow$	ACC5 $\uparrow$	ACC3 $\uparrow$	ACC2 $\uparrow$	F1 $\uparrow$	Avg $\uparrow$
TFN [ 6 ]	39.30	65.12	78.38	78.62	65.36	52.55	72.21	80.14	80.14	71.26
LMF [ 7 ]	40.53	64.68	77.77	77.88	65.19	47.79	64.90	74.18	73.88	65.19
Self-MM [ 56 ]	41.53	65.47	80.04	80.44	66.87	52.77	72.61	79.69	76.76	70.46
BBFN [ 59 ]	40.92	61.05	78.12	77.88	64.49	53.29	74.47	78.53	78.41	71.18
CENet [ 60 ]	33.92	62.58	77.90	77.53	63.66	53.04	73.1	79.56	79.63	71.33

Table 2: Performance comparison on CH-SIMS and CH-SIMS_v2 datasets. The best results are highlighted in bold .

Methods	ACC $\uparrow$	F1 $\uparrow$	P $\uparrow$	R $\uparrow$
MAG-BERT [ 61 ]	70.34	68.19	68.31	69.36
MulT [ 8 ]	72.58	69.36	70.73	69.47
MISA [ 11 ]	72.36	70.57	71.24	70.41
EMOE [ 5 ]	72.58	70.73	72.08	70.86
Atsuko (Ours)	74.16	71.22	72.93	70.86

Table 3: Results on MIntRec dataset.

SBN	MCM	MOSI				MOSEI
SBN	MCM	ACC7 $\uparrow$	ACC2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	ACC7 $\uparrow$	ACC2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$
		46.0	83.5	83.5	0.716	52.3	83.8	83.7	0.558
✓		47.1	85.5	85.5	0.701	53.0	85.1	85.2	0.548
	✓	46.9	85.4	85.3	0.700	53.8	85.2	85.1	0.535
✓	✓	48.7	86.3	86.2	0.696	54.4	85.6	85.7	0.530

Table 4: Ablation study of the key components in Atsuko on MOSI and MOSEI dataset.

\uparrow

indicates higher is better;

\downarrow

indicates lower is better.

Figure 3: Hyperparameter sensitivity analysis on CMU-MOSI and CMU-MOSEI (Acc-7, %). Each sub-figure sweeps one hyperparameter while holding all others at their default values. The star marker ( $\bigstar$ ) denotes the default configuration adopted in all experiments. Blue lines: MOSI (left $y$ -axis); red lines: MOSEI (right $y$ -axis).

Figure 4: t-SNE visualization of feature distributions on MOSI test set. (a) Baseline (w/o SBN & MCM), (b) w/o MCM, (c) w/o SBN, and (d) Atsuko (Full). Different colors represent varying sentiment intensities. The proposed method shows clearer cluster margins and better separability.

Figure 5: Visualization of modality weights predicted by the router on randomly selected CMU-MOSI test samples. Gray bars denote the baseline model without MCM, and colored bars denote the MCM-enhanced model.

Panel of Figure 6 — Figure 6: Spectral band decomposition of a strongly positive sample (Label: +2.00) from the CMU-MOSI test set across all three modalities.

Explore similar work

Enhancing Multimodal Emotion Recognition via Multi-Feature Encoding and Attention-Based Fusion
Sep 4, 2026Xu Lin, Ke Wang, Hui Kang +1Multimodal Emotion Recognition Multimodal Fusion
College of Computer Science and Technology, Jilin University, Changchun 130012, China · Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun 130012, China
AMB-DSGDN: Adaptive Modality-Balanced Dynamic Semantic Graph Differential Network for Multimodal Emotion Recognition
Mar 7, 2026Yunsheng Wang, Yuntao Shou, Yilong Tan +3Emotion Recognition in Conversations Multimodal Emotion Recognition
College of Computer and Mathematics, Central South University of Forestry and Technology · Department of Computer Science, State University of New York
ReH-FUSE: Reliability-Aware Hierarchical Fusion of Experts for Multimodal Emotion Recognition in Conversation
Sep 12, 2026Guan-Hua Wen, Hou-Chiang Tseng, Kuan-Yu ChenEmotion Recognition in Conversations Multimodal Emotion Recognition
National Taiwan University of Science and Technology, Taipei, Taiwan

Sep 4, 2026J/K move · Enter open · S save

Enhancing Multimodal Emotion Recognition via Multi-Feature Encoding and Attention-Based Fusion

Xu Lin, Ke Wang, Hui Kang, Xinying Wang

College of Computer Science and Technology, Jilin University, Changchun 130012, China · Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun 130012, China

Multimodal Emotion Recognition Multimodal Fusion

Open paper AskPDF arXiv

Multimodal emotion recognition has attracted growing interest due to its importance in human-computer interaction, remote education, and healthcare. This paper proposes a novel multimodal emotion recognition framework that integrates rich audio and visual feature extraction with an attention-based fusion strategy. For audio, we extract three complementary feature types: semantic embeddings from Wav2Vec2, MFCC features, and statistical acoustic descriptors such as pitch, energy, and rhythm. These are aligned and fused via a BiLSTM to capture temporal dependencies. For video, we propose a ResNet50-BiLSTM architecture that combines deep residual learning and sequential modeling to extract expressive spatiotemporal features from facial sequences. To enhance multimodal synergy, we introduce a feature-level fusion mechanism based on multi-head attention, allowing the model to adaptively weigh contributions across modalities. Experiments conducted on the MELD and IEMOCAP datasets demonstrate that our model significantly outperforms baselines in both accuracy and robustness. Furthermore, ablation studies show that the attention-based fusion strategy significantly improves performance in unbalanced data settings. Our findings suggest that the proposed framework effectively captures diverse emotional cues from speech and visual expressions, and offers a practical and generalizable approach for real-world multimodal emotion recognition tasks.

Abstract

Explore similar work

Enhancing Multimodal Emotion Recognition via Multi-Feature Encoding and Attention-Based Fusion

AMB-DSGDN: Adaptive Modality-Balanced Dynamic Semantic Graph Differential Network for Multimodal Emotion Recognition

ReH-FUSE: Reliability-Aware Hierarchical Fusion of Experts for Multimodal Emotion Recognition in Conversation