Incongruence-Aware Evaluation of Multimodal Speech Emotion Recognition in Audio Large Language Models

Fengji Jing; Guy Laban

cs.AIOct 29, 2026

Incongruence-Aware Evaluation of Multimodal Speech Emotion Recognition in Audio Large Language Models

Authors: Fengji Jing, Guy Laban

Organizations: Faculty of Science, The University of Hong Kong, Hong Kong, China · Department of Industrial Engineering and Management, Ben-Gurion University of the Negev, Beer Sheva, Israel · School of Brain Sciences and Cognition, Ben-Gurion University of the Negev, Beer Sheva, Israel · Azrieli National Center for Autism and Neurodevelopment Research, Ben-Gurion University of the Negev, Beer Sheva, Israel

Abstract

Voice-interactive large language models process both what a speaker says and how they say it, yet the two channels do not always agree. We ask whether that disagreement degrades speech emotion recognition. Mapping text and audio into a shared valence-arousal-dominance space, we convert the distance between them into a continuous, threshold-free incongruence score with a two-component Gaussian mixture, and validate the score against human judgement. We evaluate three audiocapable models on acted, conversational and spontaneous speech (eNTERFACE'05, MELD and CMU-MOSEI). With the model held constant across modalities, incongruence raises the odds of error whenever audio is present, in every corpus, after controlling for the ground-truth emotion and for distance in the feature space. Supervised classifiers trained on the same features behave in the opposite way, which suggests that the difficulty lies in how these models combine raw evidence rather than in the signal itself. Text-only effects are unreliable: swapping only the model that reads the transcript reverses their sign on one corpus. Rebuilding the score without the emotion lexicon reproduces the central result, and among six simpler alternatives it is the only label-free score that predicts error in all three corpora. Code, data tables and prompts are released.

Figures & tables

Fig. 1: Overview. Text and audio are mapped into a shared VAD space; their Euclidean distance is converted to a continuous incongruence probability $p_{\text{incon}}$ via a two-component GMM fitted on training distances. LLM predictions are evaluated stratified by $p_{\text{incon}}$ , and compared against VAD-feature baselines.

Fig. 2: Distributions of the extracted valence, arousal, and dominance values by modality on eNTERFACE (text from the NRC-VAD lexicon, audio from the fine-tuned wav2vec2 regressor). The two modalities occupy overlapping but distinct regions of the VAD space, and their per-utterance discrepancy is what

p_{\text{incon}}

quantifies.

Corpus	T1 txt	T1 t+a	T2 txt	T2 t+a
eNTERFACE	$0.0$	$1.4$	$0.5$	$1.5$
MELD	$0.0$	$5.5$	$0.0$	$4.9$
MOSEI	$15.8$	$14.5$	$13.6$	$14.7$

TABLE I: Invalid-response rate (%) per condition (GPT-audio): responses that were unparseable or out-of-vocabulary and hence excluded.

Model	Cond.	Acc.	F1	Acc c	Acc i	OR	$\boldsymbol{\beta}$
GLM-4-Voice	T1 txt	.686	.662	.722	.630	1.20	$+.02$
	T1 t+a	.627	.530	.667	.565	0.69	$-.18^{**}$
	T2 txt	.729	.727	.764	.674	1.36	$+.14^{*}$
	T2 t+a	.720	.709	.750	.674	0.81	$-.14^{*}$
Qwen2-Audio	T1 txt	.767	.751	.767	.766	1.35	$+.18^{*}$
	T1 t+a	.700	.657	.726	.660	1.27	$+.15^{*}$

TABLE II: eNTERFACE’05 (test split): overall accuracy and Macro-F1; congruent vs incongruent accuracy; and the control-adjusted logistic effect of

p_{\text{incon}}

on error (all-split, with text-length and audio-duration controls).

{}^{*}p<0.05

{}^{**}p<0.01

{}^{***}p<0.001

. OR is the unadjusted high-versus-low incongruence odds ratio;

\beta

is the coefficient from the adjusted continuous logistic model.

Emotion	eNTERFACE	MELD	MOSEI
surprise	$1.99^{***}$ (harms)	$1.15$ ns	$0.69$ ns
disgust	$0.81^{*}$ (helps)	$1.17$ ns	$1.42$ ns
fear	$0.91$ ns	$1.58$ ns	$0.55^{*}$ (helps)
anger	$1.05$ ns	$0.66^{***}$ (helps)	$0.68^{***}$ (helps)
sadness	$0.58^{***}$ (helps)	$0.76$ ns	$0.50^{***}$ (helps)
happiness / joy	$0.34$ ns	$1.21$ ns	$1.65^{***}$ (harms)

TABLE III: Within-emotion incongruence effect: odds ratio for misclassification of high- versus low-

p_{\text{incon}}

items, holding the ground-truth class fixed (pooled over all splits and all 12 LLM conditions for eNTERFACE; GPT-audio conditions for MELD and MOSEI). OR

{}>1

= incongruence harms; OR

{}<1

= incongruence helps.

{}^{*}p<0.05

{}^{***}p<0.001

; ns

=

not significant.

Fig. 3: Paradigm dissociation on eNTERFACE (test split, T2 text+audio for the LLMs). Audio LLMs lose accuracy on incongruent utterances, whereas discriminative classifiers trained on the same six VAD features gain accuracy, because a larger cross-modal gap makes classes more separable. This is consistent with the harm arising during raw multimodal integration rather than from the VAD features (the baselines are supervised, however, so this is not by itself proof of mechanism).

Cond.	Acc.	F1	Acc c	Acc i	OR	$\boldsymbol{\beta}$	logit $p$
T1 text-only	.597	.479	.605	.582	1.10	$+.05$	.234
T2 text-only	.606	.487	.626	.569	1.27	$+.10$	.022 ∗
T1 text+audio	.580	.483	.597	.549	1.21	$+.10$	.032 ∗
T2 text+audio	.540	.469	.560	.504	1.25	$+.11$	.017 ∗

TABLE IV: MELD test split using GPT-audio in all four conditions. OR is the high- versus low-

p_{\text{incon}}

error odds ratio and

\beta

is the condition-wise continuous logistic coefficient. The added-audio interaction is not significant for either prompt.

{}^{*}p<0.05

Fig. 4: Within-emotion incongruence effect (OR of error) by corpus. Values ${>}1$ (red) mean incongruence harms recognition of that emotion; ${<}1$ (blue) mean it helps. Sadness and anger, the classes with the most characteristic prosody, are helped wherever their effect is significant; the classes that are harmed differ by corpus. (MELD has no “happiness” class; its positive label is “joy”.)

Corpus	Reader	Acc.	Adj. OR (95% CI)
eNTERFACE	gpt-5.2	$.832$	$1.12$ $[0.91,1.37]$
eNTERFACE	gpt-audio-1.5	$.818$	$\mathbf{0.78}$ $[0.63,0.97]$
MELD	gpt-5.2	$.594$	$1.03$ $[0.94,1.13]$
MELD	gpt-audio-1.5	$.601$	$1.06$ $[0.94,1.18]$
MOSEI	gpt-5.2	$.543$	$\mathbf{1.23}$ $[1.14,1.33]$
MOSEI	gpt-audio-1.5	$.536$	$\mathbf{1.24}$ $[1.14,1.35]$

TABLE V: Text-only condition read by two different models, with transcripts, prompts,

p_{\text{incon}}

and labels held fixed. Accuracy is materially the same; sensitivity to incongruence is not. Odds ratios come from a model fitted to the text-only condition alone, so they differ slightly from the text-only cells of Table VII , which come from the pooled model with an audio interaction.

Corpus	Modality	GPT-audio	Qwen2.5-Omni	GLM-4-Voice
MELD	text-only	$1.11$ $[1.00,1.24]$	$1.04$ $[0.90,1.20]$	$0.99$ $[0.87,1.13]$
MELD	text+audio	$\mathbf{1.15}$ $\mathbf{[1.04,1.28]}$	$1.14$ $[0.97,1.34]$	$1.07$ $[0.94,1.22]$
MOSEI	text-only	$\mathbf{1.22}$ $\mathbf{[1.12,1.33]}$	$0.95$ $[0.79,1.14]$	$1.04$ $[0.86,1.24]$
MOSEI	text+audio	$\mathbf{1.20}$ $\mathbf{[1.11,1.30]}$	$0.96$ $[0.79,1.17]$	$0.93$ $[0.78,1.12]$

TABLE VI: Cross-model incongruence effect on the extension corpora: adjusted OR of error for

p_{\text{incon}}

(95% CI) from the cluster-robust model, for all three audio-LLM families. GPT-audio is on the full corpora; Qwen2.5-Omni and GLM-4-Voice on the matched stratified sample. Bold = CI excludes

1

Corpus	Modality	Adj. OR	95% CI	$q$ (FDR)
eNTERFACE	text-only	$0.85$	$[0.73,0.98]$	$.029$
eNTERFACE	text+audio	$\mathbf{1.25}$	$[1.04,1.49]$	$.022$
MOSEI	text-only	$1.22$	$[1.12,1.33]$	$<.001$
MOSEI	text+audio	$\mathbf{1.20}$	$[1.11,1.30]$	$<.001$
MELD	text-only	$1.11$	$[1.00,1.24]$	$.051$ (ns)
MELD	text+audio	$\mathbf{1.15}$	$[1.04,1.28]$	$.017$

TABLE VII: Adjusted effect of

p_{\text{incon}}

on GPT-audio error from a cluster-robust logistic model per corpus (controls: prompt, ground-truth emotion, text/audio prototype distance, text length, audio duration; SEs clustered by speaker or utterance). Odds ratios with 95% CIs;

q

is the Benjamini–Hochberg FDR-corrected

p

-value across the six effects. The text+audio effect (bold) is harmful and significant in all three corpora; the text-only effects are inconsistent in sign and are model-dependent (Section V-H ).

Fig. 5: Adjusted association between

p_{\text{incon}}

and error from the cluster-robust logistic model, with controls for emotion, prototype distance, length and duration (95% CIs). Circles are GPT-audio on all three corpora; triangles are Qwen2.5-Omni and diamonds GLM-4-Voice on the matched MELD and MOSEI samples. Filled markers have intervals that exclude

1

. For GPT-audio the text+audio association is positive in every corpus, while the text-only association changes sign across corpora. Qwen2.5-Omni agrees in direction on MELD (

1.14

) without reaching significance, and GLM-4-Voice shows no significant association, in line with its much weaker accuracy.

Corpus	Register	Text $\to$ label (VAD)	Transcript-only GPT acc	Embed. probe F1	Text-only effect
eNTERFACE’05	read, emotion-prototypical	$0.733$	$0.818$	$1.00$	protective
CMU-MOSEI	spontaneous monologue	$0.388$	$0.536$	$0.24$	present
MELD	conversational dialogue	$0.188$	$0.601$	$0.27$	absent

TABLE VIII: Lexical emotional transparency by three measures: a lexicon-based one (text-VAD nearest-prototype accuracy, which shares signal with

p_{\text{incon}}

) and two VAD- independent ones (transcript-only GPT accuracy, and the cross-validated macro-F1 of a linear probe on frozen all-mpnet-base-v2 sentence embeddings). All three agree that eNTERFACE is by far the most explicit. The text-only effect (last column) does not order with transparency, which is why we do not treat transparency as governing it. The embedding probe reaches a ceiling on eNTERFACE, whose closed scripted-sentence set makes the transcript trivially diagnostic.

Score	Labels	eNTERFACE	MELD	MOSEI
$p_{\text{incon}}$ (GMM posterior)	no	$\mathbf{1.25}^{*}$	$\mathbf{1.15}^{*}$	$\mathbf{1.20}^{*}$
Euclidean distance	no	$1.10$	$1.15^{*}$	$1.23^{*}$
z-scored distance	no	$1.37^{*}$	$1.00$	$1.21^{*}$
Cosine distance	no	$1.52^{*}$	$1.00$	$1.32^{*}$
Valence gap	no	$1.11$	$0.96$	$1.18^{*}$
Valence-sign flip	no	$1.70^{*}$	$0.91$	$1.31^{*}$

TABLE IX:

p_{\text{incon}}

against simpler stratifiers. Each score replaces

p_{\text{incon}}

in the primary model of Table VII ; entries are GPT-audio text+audio odds ratios per standard deviation (binary flags included). ∗ 95% CI excludes

1

. “Labels” marks scores that require gold emotion labels; prototype disagreement is cross-fitted across speakers, so no utterance’s flag uses its own speaker’s labels.

Fig. 6: Convergent validity: the text+audio incongruence effect under the lexicon-based

p_{\text{incon}}

(NRC lexicon + audeering audio-VAD; blue) versus a fully neural, lexicon-free

p_{\text{incon}}

(EmoBank-trained text-VAD regressor + WavLM audio-VAD; orange), under the identical cluster-robust specification. Filled markers exclude OR

{=}1

. On eNTERFACE and MOSEI the neural score reproduces the text+audio association found with the lexicon-based one; on MELD it does not, so the convergence is partial.

Corpus	Modality	Primary	WavLM
eNTERFACE	text-only	$\mathbf{0.715}$ $\mathbf{[0.598,0.853]}$	$\mathbf{0.827}$ $\mathbf{[0.708,0.966]}$
eNTERFACE	text+audio	$1.006$ $[0.829,1.221]$	$1.140$ $[0.970,1.340]$
MELD	text-only	$1.046$ $[0.955,1.146]$	$1.008$ $[0.928,1.094]$
MELD	text+audio	$1.077$ $[0.989,1.173]$	$0.889$ $[0.786,1.005]$
MOSEI	text-only	$\mathbf{1.104}$ $\mathbf{[1.025,1.188]}$	$\mathbf{1.140}$ $\mathbf{[1.059,1.227]}$
MOSEI	text+audio	$\mathbf{1.085}$ $\mathbf{[1.011,1.164]}$	$\mathbf{1.084}$ $\mathbf{[1.010,1.163]}$

TABLE X: Association between

p_{\text{incon}}

and GPT-audio error under the primary audio-VAD extractor and an independent WavLM extractor. Entries are adjusted ORs with 95% CIs from the same reduced cluster-robust specification. Bold entries have CIs excluding

1

. Robustness is strongest on MOSEI and limited on MELD and eNTERFACE.

Corpus	Modality	$N$ scored	$N$ valid	invalid %
eNTERFACE	text-only	$2574$	$2568$	$0.2$
eNTERFACE	text+audio	$2574$	$2537$	$1.4$
MELD	text-only	$4148$	$4147$	$0.02$
MELD	text+audio	$4148$	$3931$	$5.2$
MOSEI	text-only	$8422$	$7185$	$14.7$
MOSEI	text+audio	$8422$	$7195$	$14.6$

TABLE XI: GPT-audio analysis sample sizes (pooled over the two prompts) and invalid-response rates. Invalid (non-label) outputs are negligible on eNTERFACE and MELD but frequent on MOSEI in both modalities, the audio-capable model declining to answer on roughly one seventh of those utterances; the invalid-as-error sensitivity check (Section V-K ) addresses them.

Transcript	Gold	text/audio	$p_{\text{incon}}$	GPT
“You’re getting paid to work, not drink coffee!”	anger	anger/sadness	$0.20$	anger
“Aaaaah a cockroach!!!”	disgust	disgust/sadness	$0.76$	fear
“Everything was so perfect! I just don’t understand!”	sadness	sadness/sadness	$0.81$	sadness
“Oh no, tell me this is not true, please!”	sadness	anger/sadness	$1.00$	fear

TABLE XII: Illustrative eNTERFACE utterances, with the nearest-prototype emotion of each modality,

p_{\text{incon}}

, and the GPT-audio prediction (T2, text+audio). Category and score capture different things: the first row has discordant prototype labels but a low score, the third concordant labels but a high one, because

p_{\text{incon}}

measures distance in VAD space rather than a change of category.

Fengji Jing is with the Faculty of Science, the University of Hong Kong, Hong Kong, China.

Table 19

Guy Laban is with the Department of Industrial Engineering and Management, the School of Brain Sciences and Cognition, and the Azrieli National Center for Autism and Neurodevelopment Research, Ben-Gurion University of the Negev, Beer Sheva, Israel.

Table 20

Multimodal Emotion Recognition

Multimodal emotion recognition combines signals such as speech, text, facial expressions, video, and physiology to infer affect.

See all 152 papers

Mar 9, 2026

Disentangling Reasoning in Large Audio-Language Models for Ambiguous Emotion Prediction

Speech emotion recognition plays an important role in various applications. However, most existing approaches predict a single emotion label, oversimplifying the inherently ambiguous nature of human emotional expression. Recent large audio-language models show promise in generating richer outputs, but their reasoning ability for ambiguous emotional understanding remains limited. In this work, we reformulate ambiguous emotion recognition as a distributional reasoning problem and present the first systematic study of ambiguity-aware reasoning in LALMs. Our framework comprises two complementary components: an ambiguity-aware objective that aligns predictions with human perceptual distributions, and a structured ambiguity-aware chain-of-thought supervision that guides reasoning over emotional cues. Experiments on IEMOCAP and CREMA-D demonstrate consistent improvements across SFT, DPO, and GRPO training strategies.

Xiaofeng Yu, Jiaheng Dong, Jean Honorio +3

University of Auckland, New Zealand · The University of Melbourne, Australia · University of Birmingham, United Kingdom

Speech Emotion Recognition Emotion Recognition Audio-Language Models CoT Reasoning Label Distribution Learning

Ask PDF

Jun 23, 2026

Comparative Reasoning: Making an Audio Language Model Better at Comparing Emotions

Large audio-language models (LALMs) can reason about audio, yet it remains unclear whether they can perform comparative judgments between two speech signals along emotional, environmental, linguistic, prosodic, and interpersonal dimensions. We study this question in the context of speech emotion recognition (SER), where the model determines which utterance exhibits higher arousal, valence, or dominance. We introduce a reasoning-guided ordinal SER framework that conditions an LALM on paired speech inputs. The model is trained using reasoning traces generated from both semantic audio descriptions and acoustic evidence derived from GeMAPS features, enabling interpretable comparative decisions. Beyond direct supervision, we also employ direct preference optimization to encourage stronger separation for emotional differences. Experiments show that the proposed framework improves preference prediction while requiring only 5% of the training data used by conventional ordinal SER systems.

Abinay Reddy Naini, Jaeyeon Kim, Chao-Han Huck Yang +2

Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA, 15213, US · NVIDIA

Audio-Language Model Evaluation Pairwise Comparison Audio Reasoning Audio-Language Models Speech Emotion Recognition

Ask PDF

Mar 9, 2026

VoxEmo: Benchmarking Speech Emotion Recognition with Speech LLMs

Speech Large Language Models (LLMs) show great promise for speech emotion recognition (SER) via generative interfaces. However, shifting from closed-set classification to open text generation introduces zero-shot stochasticity, making evaluation highly sensitive to prompts. Additionally, conventional speech LLMs benchmarks overlook the inherent ambiguity of human emotion. Hence, we present VoxEmo, a comprehensive SER benchmark encompassing 35 emotion corpora across 15 languages for Speech LLMs. VoxEmo provides a standardized toolkit featuring varying prompt complexities, from direct classification to paralinguistic reasoning. To reflect real-world perception/application, we introduce a distribution-aware soft-label protocol and a prompt-ensemble strategy that emulates annotator disagreement. Experiments reveal that while zero-shot speech LLMs trail supervised baselines in hard-label accuracy, they uniquely align with human subjective distributions.

Hezhao Zhang, Huang-Cheng Chou, Shrikanth Narayanan +1

Department of Computer Science, University of Sheffield, United Kingdom · Signal Analysis and Interpretation Laboratory (SAIL), Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, CA 90089, USA

Speech Emotion Recognition Speech Language Models

Ask PDF

Abstract

Multimodal Emotion Recognition

Explore similar work

Disentangling Reasoning in Large Audio-Language Models for Ambiguous Emotion Prediction

Comparative Reasoning: Making an Audio Language Model Better at Comparing Emotions

VoxEmo: Benchmarking Speech Emotion Recognition with Speech LLMs