Emotion as a Distribution: Joint Valence-Arousal Probability Learning for Speaker-Independent Multimodal Emotion Recognition

Tingyi Lin; Wen-Ren Yang; Kuanwei Chen

cs.SDSep 4, 2026

Emotion as a Distribution: Joint Valence-Arousal Probability Learning for Speaker-Independent Multimodal Emotion Recognition

Authors: Tingyi Lin, Wen-Ren Yang, Kuanwei Chen

Organizations: Department of Electrical Engineering, National Changhua University of Education, Changhua, Taiwan · Department of Computer Science and Information Engineering, National Central University, Taoyuan, Taiwan

Abstract

Human emotion is graded and frequently mixed, yet most multimodal recognizers collapse it onto a single hard label. We argue the recognizer should instead expose a distribution over affective space. Our text+speech system, alongside its categorical decision, emits a $9\times9$ probability matrix over the Valence-Arousal plane, trained with a two-dimensional Gaussian soft target under a Kullback-Leibler/cross-entropy objective, aimed at counseling support. Evaluation is strict: speaker-independent 5-fold leave-one-session-out IEMOCAP with rotating-session inner validation, headline metrics only on the held-out session. Within one fixed encoder-fusion-head pipeline we compare Transformer and state-space (Mamba-1/2/3) backbones at matched depth and width, at two operating points ( $T\approx550$ , $T\approx2750$ ). The featured dual-head system reaches 73.0% $\pm$ 0.3 unweighted accuracy over three seeds (separate rerun: 72.1%), exceeding the Transformer fusion baseline by 3.0 UA points (95% session-bootstrap CI [1.0,4.7]; significant under paired t-test and session-level bootstrap), with no latency or memory advantage at these lengths; swapping the ~1M trainable front-end for frozen WavLM-Large features (learnable layer weights) lifts the same architecture to 76.6% $\pm$ 1.3. Pre-specified controls scope the claims honestly: simpler valence-arousal auxiliaries reproduce the classification lift within noise, and a dedicated regression head tracks the continuous ratings slightly better, so the head's specific value is the normalized affect distribution itself. That distribution recovers the circumplex: its center of mass tracks valence and arousal (CCC 0.66/0.66; predominantly between-class structure, weaker within-class tracking), and its entropy is weakly but consistently linked to categorical rater ambiguity, not dimensional spread.

Figures & tables

Method	Mod.	Front-end	Protocol	Params (M)	WA (%)	UA (%)
emotion2vec [ 8 ] ⋆	A	emotion2vec (frozen, emotion SSL)	LOSO (5-fold) †1	$93.8{+}0.2$	71.79	—
TF-Mamba [ 9 ] ⋆	A	WavLM-large	5-fold CV †2	$20.8$ ( $+316$ fr.)	75.3	75.7
EmoBridge [ 12 ] ⋆	A+T	HuBERT (frozen) $+$ LLM (LoRA)	5-fold CV (spk.-indep.) †3	$\sim$ $7{,}000$ (LLM)	76.4	76.8
SDMMKD [ 13 ] ⋆	A+T	HuBERT (LoRA-FT) $+$ SKEP teacher	LOSpkO (10-fold) †4	n/r	79.74	80.68
DFSD-EMO [ 14 ] ⋆	A	768-d pre-trained features	5-fold CV †5	n/r	72.36	72.35
Ours ( mamba_dual_head ) ⋆	A+T	frozen BERT $+$ raw-conv ( $\sim$ 1M, FT)	LOSO, rotating val. (Sec. IV-B )	$16.9$	$71.4$	$73.0$

TABLE I: Representative recent IEMOCAP (4-class) results under different evaluation protocols. Protocol tags mark the partition family — session folds (LOSO; emotion2vec and ours) hold entire speaker pairs out, generic “5-fold CV” entries do not always specify speaker independence, and leave-one-speaker-out (LOSpkO; SDMMKD) uses 10 finer folds — and each paper’s protocol wording is reproduced verbatim in the

\dagger

-numbered table footnotes, so partition differences are visible rather than flattened. Params (M) as reported by each source:

a{+}b

= frozen upstream

+

trainable probe; “fr.” = frozen front-end additional to the stated trainable count; n/r = not reported. Separate WA and UA columns (— where the source does not report the metric; EmoBridge additionally reports wF1

73.8

). A = audio, T = text. Scores are taken verbatim from each source paper; the in-table note states the merge, seed, and encoder-regime annotations. Table body generated from the committed statistics.

Panel of Figure 2 — Fig. 1: Echo-Mind Phase-1 architecture. A frozen text encoder and a swappable speech front-end — frozen Wav2Vec2, fine-tuned raw-convolution, or frozen WavLM-Large (Sec. V-E ) — feed a shared-width projection, an interchangeable temporal backbone, bidirectional cross-attention fusion, and one or two output heads. The $9\times 9$ Valence–Arousal head (Sec. III-E ) is the paper’s headline contribution; the categorical head is retained for comparability with prior work.

(a) Frozen Wav2Vec2 front-end ( $T\!\approx\!550$ )
Model	Params	UA $\pm$ seed (fold)	WA	Macro-F1	$\Delta$ UA vs. cross_attn [95% CI]
late_fusion	$0.46$	$0.672{\pm}0.003\,(0.017)$	$0.657{\pm}0.003$	$0.657{\pm}0.002$	$-0.007\;[-0.015,+0.001]$
cross_attention	$13.2$	$0.679{\pm}0.004\,(0.020)$	$0.658{\pm}0.004$	$0.660{\pm}0.005$	—
mamba_fusion	$15.8$	$0.688{\pm}0.003\,(0.030)$	$0.673{\pm}0.002$	$0.676{\pm}0.003$	$+0.009\;[-0.005,+0.023]$
mamba_v2_fusion	$16.4$	$0.684{\pm}0.005\,(0.024)$	$0.668{\pm}0.008$	$0.671{\pm}0.008$	$+0.005\;[-0.003,+0.012]$
mamba_dual_head	$15.9$	$\mathbf{0.691{\pm}0.004\,(0.018)}$	$\mathbf{0.678{\pm}0.004}$	$\mathbf{0.680{\pm}0.004}$	$+0.012\;[+0.003,+0.022]$

TABLE II: Backbone comparison on IEMOCAP, 5-fold speaker-independent LOSO, at the two acoustic operating points: (a) the frozen Wav2Vec2 front-end (

T\!\approx\!550

); (b) the long-context raw-convolution front-end (

T\!\approx\!2750

, encoder fine-tuned). UA cells are mean

\pm

std over three seeds

\{42,1,2\}

with the per-fold LOSO std in parentheses; WA and Macro-F1 carry the seed std. The last column is the seed-averaged paired UA difference against the Transformer fusion row of the same panel with its enumerated session-cluster bootstrap

95\%

CI (Sec. IV-B ). Best UA/WA/Macro-F1 in bold within each panel. Params = trainable parameters in millions: in (a) the Wav2Vec2 (

\sim

95M) and BERT (

\sim

110M) encoders are frozen and shared across all rows, so only the projection, temporal backbone, fusion, and head(s) are trained; in (b) the fine-tuned raw-conv encoder (

\sim

1M) is included and the frozen BERT (

\sim

110M) is excluded. Table body generated from the released statistics files (Data and Code Availability).

Model	basic UA	mamba UA
late_fusion	$0.695{\pm}0.039$	$0.678{\pm}0.029$
cross_attention	$0.688{\pm}0.039$	$0.708{\pm}0.028$
mamba_v3_fusion	$0.720{\pm}0.030$	$0.718{\pm}0.031$
mamba_v2_fusion	$\mathbf{0.728{\pm}0.034}$	$0.717{\pm}0.033$
mamba_fusion	$0.714{\pm}0.033$	$0.714{\pm}0.019$
mamba_v2_dual_head	$0.725{\pm}0.035$	$0.727{\pm}0.024$

TABLE III: Encoder pre-training ablation: UA (mean

\pm

std over the five LOSO test sessions, single seed 42) for the two LibriSpeech pre-training variants of the raw-convolution front-end (

T\!\approx\!2750

, fine-tuned). basic : conv-only SSL; mamba : SSL assisted by a Mamba-2 context module that is stripped before fine-tuning. The two are within fold std on every model.

Fig. 2: Per-class mean of the predicted $9\times 9$ V–A matrix on the held-out LOSO test sessions ( mamba_dual_head , raw-conv encoder; representative seed of the three-seed replication; shared color scale). Each emotion’s mass concentrates in its expected circumplex quadrant [ 15 ] : the head has learned class-level affective geometry ; whether it also tracks V–A variation within a class is answered in Fig. 4 .

Fig. 3: Expected value (center of mass) of the predicted V–A matrix vs. ground-truth valence and arousal (points: representative seed; annotations: three-seed mean $\pm$ std of $r$ , CCC, and the regression slope), obtained without a point-regression loss (the coordinates supervise only the soft target’s location). Vertical striping reflects the discrete support of rater-averaged annotations; the fitted slope of $\approx\!0.5$ (solid line vs. the dashed identity) quantifies the expected shrinkage of a center-of-mass read-out on a bounded grid.

Fig. 4: Within-class tracking of the continuous ratings ( mamba_dual_head , raw-conv encoder). Top: class-mean-centered Pearson $r$ per emotion and over all classes (bars: three-seed mean; error bars: seed std). The pooled correlation (dashed) includes between-class geometry; the within-class bars remove it. Bottom: class-mean-centered arousal residuals for the representative seed — within-class arousal tracking is genuine ( $r=0.52$ ), while valence retains only $r=0.27$ .

Encoder	$\sigma$	UA	$H$ (nats)	ECE
Frozen	$0.50$	$0.726$	$3.49$	$0.20$
Frozen	$0.25$	$0.730$	$3.07$	$0.19$
Fine-tuned	$0.50$	$0.721$	$3.40$	$0.23$
Fine-tuned	$0.25$	$0.727$	$2.97$	$0.21$

TABLE IV: Effect of the soft-target width

\sigma

on the V–A head (IEMOCAP, mamba_dual_head , raw-conv encoder). UA is mean over 3 seeds

\times

5 LOSO folds; entropy

H

(in nats, max

\ln 81\!=\!4.39

) and ECE are pooled over the test sessions of a representative seed. Lower

\sigma

sharpens the distribution (

H\!\downarrow

) at no UA cost; calibration is unaffected.

Fig. 5: Predicted $9\times 9$ V–A matrices for four of the highest-entropy ( $H\!\approx\!4.1$ nats; max $\ln 81\!=\!4.39$ ) test utterances ( mamba_dual_head , raw-conv encoder). The mass spreads across multiple V–A regions and straddles the true/predicted category boundary — diffuse, multi-peaked model outputs that the grid head can represent and a hard classifier cannot.

Fig. 6: Reliability diagram on the pooled LOSO test sessions (three seeds pooled; ECE annotations are three-seed mean $\pm$ std). Left: raw confidence sits below the diagonal (ECE $0.20$ ) — the categorical head is over-confident. Right: a scalar temperature fitted per fold on the rotating validation session only repairs the test ECE to $0.02$ without changing any argmax decision.

Fig. 7: Selective risk vs. coverage when abstaining on the least-confident utterances, ranked either by softmax confidence or by V–A entropy (lines: three-seed mean; bands: seed range). Confidence ranks errors better at every coverage — the plain reading is that the $9\times 9$ entropy adds no abstention value beyond the classifier’s own confidence, even though abstention itself helps under either ranking.

Fig. 8: Predictive entropy of the V–A matrix for correct vs. incorrect test predictions ( mamba_dual_head , raw-conv encoder; dashed lines mark the group centers). The two distributions overlap almost entirely, so entropy is only a weak indicator of correctness — consistent with the over-confidence in Fig. 6 . A high-entropy prediction reflects mass spread over the V–A grid (Fig. 5 ), but is not a reliable indicator of an incorrect prediction — nor of rater-measured ambiguity (see text).

Arm	UA	$\Delta$ UA [95% CI]	CCC V	CCC A	tests
featured mamba_dual_head	$0.730{\pm}0.003$	—	$0.661{\pm}0.013$	$0.661{\pm}0.005$	reference
head 2 $\to$ scalar $(v,a)$ regression	$0.723{\pm}0.010$	$-0.007\;[-0.019,+0.003]$	$0.687{\pm}0.008$	$0.678{\pm}0.013$	any-aux control / CCC anchor
head 2 $\to$ two 9-way marginals	$0.725{\pm}0.003$	$-0.005\;[-0.016,+0.006]$	$0.673{\pm}0.009$	$0.675{\pm}0.009$	jointness control
anisotropic $\sigma_{V}{=}0.31,\sigma_{A}{=}0.44$	$0.728{\pm}0.005$	$-0.002\;[-0.010,+0.006]$	$0.668{\pm}0.012$	$0.665{\pm}0.012$	rater-std target width
front-end $\to$ frozen WavLM-Large	$0.766{\pm}0.013$	$+0.036\;[+0.028,+0.043]$	$0.743{\pm}0.004$	$0.701{\pm}0.004$	SSL front-end arm
audio-only branch	$0.583{\pm}0.007$	$-0.148\;[-0.179,-0.124]$	—	—	modality ablation

TABLE V: Pre-specified controls and extensions (3 seeds

\times

5 LOSO folds, featured recipe unless stated).

\Delta

UA is the seed-averaged paired difference vs. the featured system with its enumerated session-cluster bootstrap

95\%

CI; CCC columns give the continuous read-out (head-appropriate: regression output, marginal expectations, or center of mass). Indented rows modify only the named component. Table body generated from the released statistics files (Data and Code Availability).

$T$	Transformer	Mamba-1	Mamba-2	Mamba-3
$550$	$\mathbf{5.4}$	$12.5$	$28.5$	$22.4$
$2048$	$\mathbf{10.7}$	$16.6$	$29.1$	$26.4$
$2750$	$\mathbf{16.5}$	$41.0$	$29.2$	$36.0$
$4096$	$\mathbf{29.0}$	$33.7$	$30.6$	$53.5$
$8192$	$89.1$	$67.1$	$\mathbf{63.9}$	$104.9$

TABLE VI: Backbone forward+backward latency (ms) vs. sequence length

T

(isolated

6

-layer bidirectional encoder,

d{=}256

, batch

8

, bf16, median of 3, RTX 5090). Lower is faster.

Explore similar work

Evidential Multimodal Transformers with Dempster-Shafer Fusion for Emotion Recognition
Oct 9, 2026Muhammad Usama, Yeoreum JungMultimodal Emotion Recognition Uncertainty-Aware Multimodal Fusion
Braindeck Inc., Seoul, Republic of Korea · School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea
Team RAS in 10th ABAW Competition: Multimodal Valence and Arousal Estimation Approach
Mar 13, 2026Elena Ryumina, Maxim Markitantov, Alexandr Axyonov +4Valence-Arousal Modeling Affective Computing
St. Petersburg Federal Research Center of the Russian Academy of Sciences, St. Petersburg, Russia · St. Petersburg Federal Research Center of the Russian Academy of Sciences St. Petersburg, Russia · ITMO University, St. Petersburg, Russia +1
Data-Driven Decoding of Russell's Circumplex Model of Affect
Jun 15, 2026Amdjed Belaref, Samir Sadok, Zineb Noumir +1Multimodal Embedding Representation Geometry in Language Models
Alten, France · CentraleSup´elec IETR UMR CNRS 6164, France

Oct 9, 2026J/K move · Enter open · S save

Evidential Multimodal Transformers with Dempster-Shafer Fusion for Emotion Recognition

Muhammad Usama, Yeoreum Jung

Braindeck Inc., Seoul, Republic of Korea · School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea

Multimodal Emotion Recognition Uncertainty-Aware Multimodal Fusion Multimodal Robustness Evidential Deep Learning

Open paper AskPDF arXiv

Softmax-based multimodal emotion recognition methods often assign high confidence even when sensor data are corrupted or contradictory, for example when facial expressions suggest positive valence while vocal prosody indicates negative arousal. We present an Evidential Multimodal Transformer with Dempster-Shafer Fusion (EMT-DS) that addresses this problem through formal uncertainty quantification. EMT-DS models predictions as Subjective Opinions (Dirichlet distributions), separating epistemic uncertainty (lack of evidence) from aleatoric uncertainty (sensor noise). Modality-specific transformer encoders produce belief mass distributions that are fused with Dempster's Rule of Combination, which normalizes the joint belief mass by a conflict coefficient 1/(1-c), so that contradictory evidence is explicitly quantified as conflict and the fused prediction becomes diffuse rather than being forced into an unjustified consensus. Under subject-independent and speaker-independent evaluation on four benchmarks, EMT-DS achieves 84.3% accuracy on IEMOCAP versus 81.7% for the best baseline (p < 0.01), 82.4% on DEAP under leave-one-subject-out evaluation versus 78.1%, and 96.8% on the highly imbalanced KEMDy20, and it improves accuracy on CMU-MOSEI by 2.4 points over the strongest evidential baseline. In controlled ablation, it reduces Expected Calibration Error from 0.153 (softmax transformer) to 0.042. It maintains 71.2% accuracy under severe Gaussian noise on audio features (sigma = 1.0), where attention baselines drop to 45.3%, and degrades gracefully when entire modalities are absent. The fusion step has O(MK^2) complexity, compared with O(M^2 d^2) for cross-modal attention. The learned conflict metric also flags likely errors: reviewing 20% of samples captures 80% of misclassifications, which supports selective prediction in clinical and safety-critical systems.

Abstract

Explore similar work

Evidential Multimodal Transformers with Dempster-Shafer Fusion for Emotion Recognition

Team RAS in 10th ABAW Competition: Multimodal Valence and Arousal Estimation Approach

Data-Driven Decoding of Russell's Circumplex Model of Affect