Speech emotion recognition plays an important role in various applications. However, most existing approaches predict a single emotion label, oversimplifying the inherently ambiguous nature of human emotional expression. Recent large audio-language models show promise in generating richer outputs, but their reasoning ability for ambiguous emotional understanding remains limited. In this work, we reformulate ambiguous emotion recognition as a distributional reasoning problem and present the first systematic study of ambiguity-aware reasoning in LALMs. Our framework comprises two complementary components: an ambiguity-aware objective that aligns predictions with human perceptual distributions, and a structured ambiguity-aware chain-of-thought supervision that guides reasoning over emotional cues. Experiments on IEMOCAP and CREMA-D demonstrate consistent improvements across SFT, DPO, and GRPO training strategies.
Figures & tables
Figure 1: Overview of the proposed ambiguity-aware objectives and Plug-and-play Learning Paradigms.
Dataset
Methods
JS ↓
BC ↑
R2↑
Brier ↓
IEMOCAP
Base model
0.40
0.64
0.51
0.15
Audio-Reasoner
0.36
0.67
0.52
0.15
SFT
0.29
0.73
0.60
0.12
DPO
0.23
0.79
0.65
0.09
GRPO
0.36
0.67
0.55
0.13
GRPO z
0.20
0.82
0.67
0.07
Table 2: Comparison for ambiguous emotion recognition performance. The best results are in bold . GRPO z indicates ground-truth CoT trajectory injection in GRPO.
Figure 2: Comparison between CE-only (w/o KL supervision) and proposed additional KL supervision. (a) SFT on IEMOCAP, (b) SFT on CREMA-D, (c) DPO on IEMOCAP, and (d) DPO on CREMA-D.
Setting
CoT
KL
JS ↓
BC ↑
R2↑
Brier ↓
In-domain
✓
✓
0.21
0.82
0.57
0.04
In-domain
✗
✓
0.22
0.81
0.57
0.04
Cross-domain
✓
✓
0.38
0.65
0.50
0.15
Cross-domain
✗
✓
0.52
0.52
0.31
0.22
Table 3: Ablation of SFT training objectives. Models are trained on CREMA-D and evaluated under in-domain (CREMA-D) and cross-domain (IEMOCAP) settings.
Large audio-language models (LALMs) can reason about audio, yet it remains unclear whether they can perform comparative judgments between two speech signals along emotional, environmental, linguistic, prosodic, and interpersonal dimensions. We study this question in the context of speech emotion recognition (SER), where the model determines which utterance exhibits higher arousal, valence, or dominance. We introduce a reasoning-guided ordinal SER framework that conditions an LALM on paired speech inputs. The model is trained using reasoning traces generated from both semantic audio descriptions and acoustic evidence derived from GeMAPS features, enabling interpretable comparative decisions. Beyond direct supervision, we also employ direct preference optimization to encourage stronger separation for emotional differences. Experiments show that the proposed framework improves preference prediction while requiring only 5% of the training data used by conventional ordinal SER systems.
Abinay Reddy Naini, Jaeyeon Kim, Chao-Han Huck Yang +2
Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA, 15213, US · NVIDIA
Emotion recognition in conversations is increasingly tackled with language models, but these models can be unstable and expensive to fine-tune or to prompt with long dialogue histories. We propose EmoLASP, a framework that combines a language model with declarative reasoning via Answer Set Programming (ASP) to predict VAD scores (Valence-Arousal-Dominance) in conversations. Experiments on a widely used benchmark dataset (IEMOCAP) across six open-source LLMs (3B-120B) and two PLMs (BERT, RoBERTa) show that EmoLASP improves prediction performance compared to using the language model alone, even when the LLMs/PLMs are given no dialogue history in their prompts or input vectors. The gains are largest for prompt-only LLMs, which EmoLASP uses without any fine-tuning. However, for fine-tuned PLMs, the reasoner adds little once dialogue history is available. EmoLASP's LLM pipeline demonstrates the potential advantages of using a reasoning approach to ensure emotion prediction consistency and to reduce both the cost of fine-tuning and the cost of prompting with long dialogue histories.
Thao Le, Michael Thielscher
School of Computer Science and Engineering, University of New South Wales, Australia
Large audio language models (ALMs) extend LLMs with auditory understanding. A common approach freezes the LLM and trains only an adapter on self-generated targets. However, this fails for reasoning LLMs (RLMs) whose built-in chain-of-thought traces expose the textual surrogate input, yielding unnatural responses. We propose self-rephrasing, converting self-generated responses into audio-understanding variants compatible with RLMs while preserving distributional alignment. We further fuse and compress multiple audio encoders for stronger representations. For training, we construct a 6M-instance multi-task corpus (2.5M unique prompts) spanning 19K hours of speech, music, and sound. Our 4B-parameter ALM outperforms similarly sized models and surpasses most larger ALMs on related audio-reasoning benchmarks, while preserving textual capabilities with a low training cost. Notably, we achieve the best open-source result on the MMAU-speech and MMSU benchmarks and rank third among all the models.