Prediction-Layer Branch Calibration for Multimodal Sentiment Analysis

Yulin Sun; Kele Xu; Yong Dou

cs.SDSep 29, 2026

Prediction-Layer Branch Calibration for Multimodal Sentiment Analysis

Authors: Yulin Sun, Kele Xu, Yong Dou

Organizations: National University of Defense Technology

Abstract

Multimodal sentiment analysis integrates textual, acoustic and visual cues, yet current language-model-based fusion methods typically leave prediction-layer branch allocation implicit. We introduce Branch-Calibrated Multimodal Language Fusion (BC-MLF), which explicitly models prediction-layer branch allocation through a Branch-Calibrated Task Head (BCHead), complemented by Fusion Token Contrastive Learning (FTCL) for sentiment-aware fusion-token regularization. FTCL organizes mean-pooled fusion-token representations according to continuous sentiment affinity, while BCHead combines fusion, text and audiovisual predictions through a lightweight sample-adaptive constrained mixture. Without modifying the fusion backbone, BC-MLF consistently improves the reproduced DeepMLF baseline and achieves the strongest results among the compared methods on CMU-MOSEI and CH-SIMS across classification and regression metrics. The controlled ablations show that sample-adaptive prediction-layer branch allocation consistently outperforms static branch aggregation. Code is available at https://github.com/sunyulin0421/BC-MLF.

Figures & tables

Figure 1: Branch-Calibrated Multimodal Language Fusion (BC-MLF). The unchanged fusion backbone produces fusion-token, text and audiovisual representations. During training, FTCL regularizes fusion-token geometry according to continuous sentiment affinity, whereas BCHead calibrates prediction-layer branch allocation through a lightweight constrained mixture.

MODEL	CMU-MOSEI						CH-SIMS
MODEL	Acc2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$	Acc5 $\uparrow$	Acc7 $\uparrow$	Acc2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
LF-DNN $\dagger$	82.78	82.38	0.558	0.731	–	–	76.68	76.48	0.446	0.567
TFN $\dagger$	82.23	81.47	0.573	0.718	–	–	77.07	76.94	0.437	0.582
MAG-BERT $\dagger$	84.87	84.85	0.539	0.764	–	–	74.44	71.75	0.492	0.399
MulT $\dagger$	84.07	83.93	0.564	0.731	53.97	52.56	78.56	78.66	0.453	0.564
MISA $\dagger$	84.51	84.47	0.549	0.759	53.57	51.96	76.54	76.59	0.447	0.563

Table 1: Comparison with representative state-of-the-art MSA methods on CMU-MOSEI and CH-SIMS.

\dagger

: results reported in [ 2 ] ;

*

: results reproduced;

\uparrow/\downarrow

: higher/lower is better. Bold: best result in each column. Results of BC-MLF are averaged over two random seeds.

Method	Acc2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
MOSEI
DeepMLF	86.08	86.10	0.505	0.802
+ FTCL	86.75	86.74	0.501	0.805
+ BCHead	87.40	87.39	0.495	0.807
+ FTCL + Uniform	86.18	86.13	0.518	0.798
+ FTCL + Global	86.15	86.14	0.514	0.799

Table 2: Ablation studies of BC-MLF. The results demonstrate the effectiveness of FTCL and BCHead, and show that sample-adaptive prediction-layer allocation provides additional benefits beyond static branch aggregation.

Figure 2: Pairwise similarity shifts in mean-pooled fusion-token representations between DeepMLF and BC-MLF, shown alongside the continuous label-affinity reference used by FTCL on CH-SIMS.

Dataset	Model	Spearman $\uparrow$	$D_{\mathrm{KL}}(Q\\|P)\downarrow$
MOSEI	DeepMLF	0.3701 $\pm$ 0.0022	0.2410 $\pm$ 0.0025
	DeepMLF + FTCL	0.3862 $\pm$ 0.0043	0.2358 $\pm$ 0.0021
CH-SIMS	DeepMLF	0.2610 $\pm$ 0.0312	0.1793 $\pm$ 0.0035
	DeepMLF + FTCL	0.2749 $\pm$ 0.0387	0.1757 $\pm$ 0.0026

Table 3: Fusion-token alignment with continuous sentiment affinity (mean

\pm

standard deviation over two seeds).

Explore similar work

Segregate, Refine, Integrate: Decomposing Multimodal Fusion for Sentiment Analysis
Jul 14, 2026Alexios Filippakopoulos, Elias Kallioras, Nikolaos Xiros +2Multimodal Fusion Cross-Modal
National Technical University of Athens, Greece · Athena Research Center, Greece · University of Bern, Switzerland +2
MRUF: Multi-granularity Routing with Uncertainty-Aware Fusion for Robust Multimodal Sentiment Analysis
Jul 12, 2026Haoran Ma, Yinfeng Yu, Liejun WangMultimodal Sentiment Analysis Multimodal Fusion
School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China. · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing. · Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center. +2
A Conflict-Aware Penalty and Statistical Loss Framework for Balancing Modalities and Enhancing Stability in Multimodal Sentiment Analysis
May 27, 2026Jianheng Dai, Jiazhang Liang, Sijie MaiMultimodal Sentiment Analysis Complementary Modalities
School of Computer Science, South China Normal University, Guangzhou, Guangdong, China

Jul 14, 2026, cs.CLJ/K move · Enter open · S save

Segregate, Refine, Integrate: Decomposing Multimodal Fusion for Sentiment Analysis

Alexios Filippakopoulos, Elias Kallioras, Nikolaos Xiros, Efthymios Georgiou+1

National Technical University of Athens, Greece · Athena Research Center, Greece · University of Bern, Switzerland+2

Multimodal Fusion Cross-Modal Sentiment Separation Refinement

Open paper AskPDF arXiv

Multimodal fusion must simultaneously refine modality-specific signals and model cross-modal interactions; two competing objectives typically entangled within the same operation. We propose \textbf{SeRIn} (\textbf{Se}gregate, \textbf{R}efine, \textbf{In}tegrate), a multimodal LM fusion scheme that enforces this separation as an architectural prior. Modality-specific representations evolve along isolated pathways, each refined against its respective encoder context, while a dedicated cross-modal pathway accumulates their joint evolution without contaminating unimodal streams. Full cross-modal interaction is deferred to a final prediction step - ablations confirm that structured interactions, not added capacity, drive the gains; gate analysis under visual corruption reveals emergent modality reweighting without explicit supervision. SeRIn achieves state-of-the-art results on CH-SIMS and CMU-MOSEI, improving all metrics on both benchmarks.

Abstract

Explore similar work

Segregate, Refine, Integrate: Decomposing Multimodal Fusion for Sentiment Analysis

MRUF: Multi-granularity Routing with Uncertainty-Aware Fusion for Robust Multimodal Sentiment Analysis

A Conflict-Aware Penalty and Statistical Loss Framework for Balancing Modalities and Enhancing Stability in Multimodal Sentiment Analysis