Evidential Multimodal Transformers with Dempster-Shafer Fusion for Emotion Recognition

Muhammad Usama; Yeoreum Jung

cs.LGOct 9, 2026

Evidential Multimodal Transformers with Dempster-Shafer Fusion for Emotion Recognition

Authors: Muhammad Usama, Yeoreum Jung

Organizations: Braindeck Inc., Seoul, Republic of Korea · School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea

Abstract

Softmax-based multimodal emotion recognition methods often assign high confidence even when sensor data are corrupted or contradictory, for example when facial expressions suggest positive valence while vocal prosody indicates negative arousal. We present an Evidential Multimodal Transformer with Dempster-Shafer Fusion (EMT-DS) that addresses this problem through formal uncertainty quantification. EMT-DS models predictions as Subjective Opinions (Dirichlet distributions), separating epistemic uncertainty (lack of evidence) from aleatoric uncertainty (sensor noise). Modality-specific transformer encoders produce belief mass distributions that are fused with Dempster's Rule of Combination, which normalizes the joint belief mass by a conflict coefficient 1/(1-c), so that contradictory evidence is explicitly quantified as conflict and the fused prediction becomes diffuse rather than being forced into an unjustified consensus. Under subject-independent and speaker-independent evaluation on four benchmarks, EMT-DS achieves 84.3% accuracy on IEMOCAP versus 81.7% for the best baseline (p < 0.01), 82.4% on DEAP under leave-one-subject-out evaluation versus 78.1%, and 96.8% on the highly imbalanced KEMDy20, and it improves accuracy on CMU-MOSEI by 2.4 points over the strongest evidential baseline. In controlled ablation, it reduces Expected Calibration Error from 0.153 (softmax transformer) to 0.042. It maintains 71.2% accuracy under severe Gaussian noise on audio features (sigma = 1.0), where attention baselines drop to 45.3%, and degrades gracefully when entire modalities are absent. The fusion step has O(MK^2) complexity, compared with O(M^2 d^2) for cross-modal attention. The learned conflict metric also flags likely errors: reviewing 20% of samples captures 80% of misclassifications, which supports selective prediction in clinical and safety-critical systems.

Figures & tables

Fig. 1: Overall architecture of EMT-DS. Modality-specific encoders (e.g., Wav2Vec2, ViT-B/16, BERT/KoELECTRA, and a physiological 1D-CNN) extract features

h^{m}\in\mathbb{R}^{d}

from heterogeneous inputs such as audio, video, text, and physiological signals. Evidential heads generate Subjective Opinions

\omega^{m}=(\mathbf{b}^{m},u^{m},\mathbf{a}^{m})

by predicting Dirichlet parameters via softplus-activated layers (Eq. 5 ), yielding belief masses and uncertainty (Eq. 9 ). The Dempster-Shafer fusion module iteratively combines opinions (Eqs. 12 , 13 ), computing conflict

c

to quantify cross-modal disagreement. The fused opinion produces final class probabilities via Eq. 2 .

Quantity	Intuitive meaning	Not the same as
Belief mass $b_{k}^{m}$	Evidence-backed support for class $k$	Softmax confidence
Uncertainty $u^{m}$	Amount of missing evidence (vacuity)	Predictive entropy
Base rate $a_{k}^{m}$	Prior used when evidence is absent	A learned gating weight
Conflict $c$	Extent of cross-modal disagreement	A reliability or similarity score
Projected prob. $p_{k}^{m}$	Decision-time probability $b_{k}^{m}+u^{m}a_{k}^{m}$	Raw softmax output

TABLE I: Intuitive interpretation of the Subjective Logic quantities used in EMT-DS, and the familiar quantities they should not be confused with.

Characteristic	IEMOCAP	DEAP	KEMDy20	CMU-MOSEI
Participants	10	32	80	$\sim$ 1000
Language	English	N/A	Korean	English
Utterances	5,531	1,280	$\sim$ 13k	$\sim$ 23k
Emotion Model	Discrete (4)	Dimensional (VA)	Discrete (5)	Discrete (6)
Modalities Used	A+T	EEG+Physio	A+T+Physio+V	A+T+V
Elicitation	Acted/Improv	Passive Viewing	Naturalistic	In-the-wild

TABLE II: Summary of Benchmark Dataset Characteristics (A: Audio, T: Text, V: Video, Physio: Physiological signals). CV: cross-validation; LOSO: leave-one-subject-out.

Dataset	Method	Acc.	F1	ECE
IEMOCAP	TACFN [ 9 ]	0.798 $\pm$ 0.011	0.782 $\pm$ 0.012	0.147 $\pm$ 0.012
	CAG-MoE [ 13 ]	0.805 $\pm$ 0.009	0.791 $\pm$ 0.010	0.138 $\pm$ 0.010
	DCT [ 55 ]	0.817 $\pm$ 0.010	0.803 $\pm$ 0.011	0.129 $\pm$ 0.009
	MulT [ 25 ]	0.784 $\pm$ 0.012	0.769 $\pm$ 0.013	0.155 $\pm$ 0.014
	MISA [ 60 ]	0.808 $\pm$ 0.010	0.793 $\pm$ 0.011	0.144 $\pm$ 0.011
	TMC [ 26 ]	0.812 $\pm$ 0.008	0.798 $\pm$ 0.009	0.082 $\pm$ 0.007

TABLE III: Main benchmark comparison under subject-independent (DEAP, leave-one-subject-out) and speaker-independent (KEMDy20) protocols. Values are unweighted accuracy, macro F1, and Expected Calibration Error (ECE) as mean

\pm

95% confidence interval over five seeds. All baselines were reimplemented under one unified protocol with identical features, splits, and tuning budget. Dashes denote a method not applicable to that dataset’s modality set. Best per dataset in bold.

Method	Acc.	F1	ECE	NLL	Brier
MulT [ 25 ]	0.498 $\pm$ 0.011	0.475 $\pm$ 0.012	0.174	1.428	0.285
TACFN [ 9 ]	0.505 $\pm$ 0.010	0.482 $\pm$ 0.011	0.161	1.395	0.278
CAG-MoE [ 13 ]	0.518 $\pm$ 0.009	0.496 $\pm$ 0.010	0.152	1.324	0.269
MISA [ 60 ]	0.512 $\pm$ 0.010	0.489 $\pm$ 0.011	0.158	1.352	0.272
ConFEDE [ 61 ]	0.527 $\pm$ 0.009	0.508 $\pm$ 0.010	0.146	1.298	0.259
Atsuko [ 33 ]	0.531 $\pm$ 0.008	0.512 $\pm$ 0.009	0.149	1.286	0.256

TABLE IV: Performance on the CMU-MOSEI six-class joint emotion classification task. Accuracy and F1 are mean

\pm

95% CI over five seeds; calibration metrics are means. Best in bold.

Panel of Figure 6 — Fig. 2: Confusion matrices for EMT-DS on the three benchmark datasets. Diagonal elements represent correct classification rates. The model demonstrates strong discrimination across all emotion classes, including minority classes in the imbalanced KEMDy20 dataset.

Method	Acc. (%)	F1 (%)	ECE	Mem (GB)
Transformer + Softmax + Concat	76.4	74.8	0.153	8.2
Transformer + Softmax + Attention	80.3	78.6	0.147	12.3
Transformer + Evidential + Concat	79.2	77.5	0.089	8.4
Transformer + Evidential + Attention	82.1	80.3	0.067	12.5
Transformer + Evidential + D-S (Ours)	84.3	82.7	0.042	8.7

TABLE V: Controlled Ablation Study on IEMOCAP: Isolating the Contribution of Evidential Encoding and D-S Fusion

Fig. 3: (Left) Average uncertainty by modality on KEMDy20. Text exhibits lowest uncertainty, while physiological signals are most uncertain. Fusion substantially reduces uncertainty. (Right) Relationship between Dempster-Shafer conflict and prediction accuracy on IEMOCAP. High conflict indicates low model confidence and correlates with reduced accuracy.

Fusion Rule	Acc.	F1	ECE	Conflict	AURC
Softmax-then-DS	0.788	0.771	0.118	0.254	0.145
Reliability-Gated	0.809	0.792	0.121	N/A	0.132
Uncertainty-Weighted	0.815	0.798	0.084	N/A	0.114
Yager’s Rule	0.821	0.804	0.072	0.182	0.102
Murphy’s Average	0.824	0.808	0.068	0.165	0.095
PCR5	0.836	0.820	0.048	0.141	0.079

TABLE VI: Comparison of fusion rules on IEMOCAP using identical evidential opinions. Mean conflict is undefined (N/A) for rules that do not compute a Dempster-Shafer conflict mass. Best in bold.

Modality Combination	Acc. (%)	F1 (%)
Audio only	65.3	58.2
Text only	71.2	64.8
Physiological only	58.7	51.3
Video only	62.4	55.9
Audio + Text	88.4	85.1
Audio + Physio	76.9	70.2

TABLE VII: Performance with Different Modality Combinations on KEMDy20 (A: Audio, T: Text, P: Physiological signals, V: Video)

Fig. 4: Sensitivity analysis of key hyperparameters on IEMOCAP. (a) KL annealing coefficient

\lambda_{t}

peaks at 0.8. (b) Conflict regularization

\lambda_{c}

optimal at 0.10. (c) Learning rate plateau between

1\times 10^{-4}

and

5\times 10^{-4}

Fig. 5: Robustness to audio modality corruption. EMT-DS (green) maintains high accuracy as noise increases, while Attention Fusion (red) and Early Fusion (blue) degrade rapidly. Shaded regions indicate

\pm 1

std. deviation.

Dataset	Missing	Attn.	CAG-MoE	EMT-DS	EMT-DS $u$
IEMOCAP	None	0.803	0.819	0.843	0.142
	Audio	0.564	0.621	0.712	0.385
	Text	0.482	0.556	0.653	0.468
KEMDy20	None	0.897	0.945	0.968	0.086
	Audio	0.768	0.852	0.884	0.182
	Text	0.684	0.789	0.826	0.215

TABLE VIII: Accuracy under complete modality missingness on IEMOCAP and KEMDy20. EMT-DS fused uncertainty rises as modalities are removed. Best accuracy per row in bold.

Fig. 6: Reliability diagram comparing EMT-DS (green) and standard Softmax (red). EMT-DS is well-calibrated (ECE=0.042), while Softmax is overconfident (ECE=0.153).

Review strategy	Extra params	Err.@20%	AURC
Random	0	0.201 $\pm$ 0.017	0.170
Predictive entropy	0	0.583 $\pm$ 0.021	0.118
D-S conflict (no conflict loss)	0	0.712 $\pm$ 0.019	0.094
Learned error head	+0.3M	0.809 $\pm$ 0.016	0.074
D-S conflict (EMT-DS)	0	0.802 $\pm$ 0.014	0.071

TABLE IX: Selective-review error detection on IEMOCAP, with operating points fit on a disjoint calibration split. Err.@20%: fraction of errors found within a 20% review budget (mean

\pm

95% CI over five seeds); AURC: area under the risk-coverage curve (lower is better). Best in bold.

Fig. 7: Error detection under selective review (horizontal axis: percentage of data reviewed; vertical axis: percentage of total errors detected). Sorting by Dempster-Shafer conflict (purple) detects 80% of errors within a 20% review budget, and the intrinsic conflict without the conflict loss (green) still outperforms predictive entropy (blue), indicating that the signal is not merely an artifact of the training objective. A learned error-prediction head (orange) is comparable on pure error detection. Operating points are fit on a disjoint calibration split.

Method	Train	Inference	Params
	(ms/sample)	(ms/sample)	(M)
Baseline (Concat)	10.4	6.5	87.2
Attention Fusion	14.7	8.9	94.6
CAG-MoE	43.2	12.3	112.8
DCT	18.9	10.2	98.4
EMT-DS (Ours)	11.5	7.0	89.7

TABLE X: Runtime Performance Comparison (IEMOCAP, Batch Size 32). Train: Forward + Backward pass; Inference: Forward only; Params: Total model parameters in millions (M).

Explore similar work

Jul 19, 2026

EmoEUS: Uncertainty Supervision for Multimodal Emotion Recognition in Conversation

Multimodal emotion recognition in conversation (MERC) can leverage multimodal and contextual cues to boost recognition performance. However, existing fusion approaches in MERC often ignore modality-specific uncertainty across utterances caused by conflicting cues, varying noise, and missing modality-specific signals. We propose EmoEUS, an explicit uncertainty supervision framework for MERC. EmoEUS performs uncertainty-aware multimodal fusion by dynamically weighting modalities using learned variance estimates. We also introduce an explicitly supervised loss that aligns each utterance's predicted variance with the distance between the utterance's distributional representation and its emotion- and modality-specific cluster center. Experiments on IEMOCAP and MELD show that EmoEUS consistently outperforms state-of-the-art methods.

Zilong Huang, Kong Aik Lee, Junjie Li +2

Dept. of Electrical and Electronic Engineering, The Hong Kong Polytechnic University · Speech, Language, and Cognition Laboratory, The University of Hong Kong

Emotion Recognition in Conversations Multimodal Emotion Recognition Uncertainty-Aware Multimodal Fusion Multimodal Fusion

Ask PDF

Sep 9, 2026

Multimodal Emotion Recognition in Conversations via Class-Wise Adaptive Modality Fusion and Affective Geometry

Emotion Recognition in Conversations (ERC) requires integrating heterogeneous textual, audio, and visual cues while accounting for conversational context and emotional dynamics. We extend the Self-Distillation Transformer architecture for ERC with appearance+geometry visual representations, class-wise adaptive modality fusion, and a valence-arousal prior for affective transitions. On the MELD and IEMOCAP datasets, geometry-enhanced visual representations improve weighted F1 by 0.27 and 4.36 points over appearance-only features, respectively, while class-wise adaptive fusion provides further gains of 0.17 and 0.25 points over the original softmax gate. The valence-arousal prior yields targeted improvements of 0.30 and 0.74 accuracy points on emotionally shifted utterances while preserving performance on stable turns. These results indicate that structured facial cues, emotion-dependent modality weighting, and affective geometry provide complementary benefits for multimodal ERC.

Oriol Marín, Roger Marí, Gloria Haro +1

Eurecat, Centre Tecnològic de Catalunya, Barcelona, Spain · Universitat Pompeu Fabra, Barcelona, Spain · Serra Húnter Fellow Programme, Universitat Pompeu Fabra, Barcelona, Spain

Emotion Recognition in Conversations Multimodal Emotion Recognition Multimodal Fusion

Ask PDF

Sep 4, 2026

Emotion as a Distribution: Joint Valence-Arousal Probability Learning for Speaker-Independent Multimodal Emotion Recognition

Human emotion is graded and frequently mixed, yet most multimodal recognizers collapse it onto a single hard label. We argue the recognizer should instead expose a distribution over affective space. Our text+speech system, alongside its categorical decision, emits a

9\times9

probability matrix over the Valence-Arousal plane, trained with a two-dimensional Gaussian soft target under a Kullback-Leibler/cross-entropy objective, aimed at counseling support. Evaluation is strict: speaker-independent 5-fold leave-one-session-out IEMOCAP with rotating-session inner validation, headline metrics only on the held-out session. Within one fixed encoder-fusion-head pipeline we compare Transformer and state-space (Mamba-1/2/3) backbones at matched depth and width, at two operating points (

T\approx550

T\approx2750

). The featured dual-head system reaches 73.0%

\pm

0.3 unweighted accuracy over three seeds (separate rerun: 72.1%), exceeding the Transformer fusion baseline by 3.0 UA points (95% session-bootstrap CI [1.0,4.7]; significant under paired t-test and session-level bootstrap), with no latency or memory advantage at these lengths; swapping the ~1M trainable front-end for frozen WavLM-Large features (learnable layer weights) lifts the same architecture to 76.6%

\pm

1.3. Pre-specified controls scope the claims honestly: simpler valence-arousal auxiliaries reproduce the classification lift within noise, and a dedicated regression head tracks the continuous ratings slightly better, so the head's specific value is the normalized affect distribution itself. That distribution recovers the circumplex: its center of mass tracks valence and arousal (CCC 0.66/0.66; predominantly between-class structure, weaker within-class tracking), and its entropy is weakly but consistently linked to categorical rater ambiguity, not dimensional spread.

Tingyi Lin, Wen-Ren Yang, Kuanwei Chen

Department of Electrical Engineering, National Changhua University of Education, Changhua, Taiwan · Department of Computer Science and Information Engineering, National Central University, Taoyuan, Taiwan

Multimodal Emotion Recognition Emotion Recognition in Conversations Valence-Arousal Modeling

Ask PDF