Audio System One Model: Typed Probabilistic Perception of Emotion in Speech
Authors: Jaime Bellver-Soler, Iván Martín-Fernández, Anmol Guragain, Luis Fernando D'Haro
Organizations: Speech Technology and Machine Learning Group (THAU), Information Processing and Telecommunications Center, E.T.S.I. de Telecomunicación, Universidad Politécnica de Madrid, Spain
Speech applications ask many questions about the same voice: which emotion a listener perceives, how aroused the speaker sounds, whether they sound frustrated. Classifiers answer fast, over one fixed label set; audio language models (ALMs) take any question in text, but answer by generation, one pass per question, with poorly calibrated probabilities. We present ASOM (Audio System One Model), a System One model for speech: choice, score and yes/no questions written at request time are answered in one pass, without generation, each with a probability distribution. Frozen speech and text encoders feed a 6.2M-parameter reader with no per-class parameters, trained on listener vote distributions. On MSP-Podcast, ASOM matches a classifier on the same features and data, leads the best of 6 zero-shot ALMs by 0.072 and an ALM fine-tuned on the same data by 0.026, and answers 20 questions in 13.1 ms, 17-41 times faster than the ALMs. It also answers reworded, reduced and new label sets, scales and gender, and on 21 unseen corpora in 13 languages it leads the best zero-shot ALM by 0.138 where the words carry no emotion; where they do, the ALMs mostly read.
Figures & tables
System
Params
F1 (95% CI)
LL 8
ms
Fixed-class classifiers
WavLM-L fine-tuned
0.3B
0.333 (0.302, 0.364)
1.528
10.6
HTSAT (CLAP) + MLP
0.2B
0.259 (0.240, 0.277)
1.672
22.6
emotion2vec+ large
0.3B
0.235 (0.211, 0.255)
4.361
10.3
DeepSER
3.3B
0.400 (0.369, 0.429)
1.544
329
Audio LLMs: zero-shot, and fine-tuned with LoRA
Table 1: MSP-Podcast 2.0 Dev-select, the 3,113 clips every system answers: 8-class macro-F1 (95% interval), log loss against the votes (LL 8 ) and latency of 20 questions in ms (Sec. 4 ). Omni: Thinker parameters. + transcript: an earlier reader on the fixed 30 s window. Bold: best per column.
MSP subset
Unseen corpora
System
Trained
Syn.
New q.
Words
All
Voice
Omni-3B
0.261
0.221
0.258
0.707
0.205
0.156
Omni-7B
0.312
0.282
0.304
0.720
0.313
0.257
Qwen2-Audio
0.288
0.208
0.285
0.696
0.303
0.281
AF3
0.161
0.152
0.145
0.542
0.222
0.224
Phi-4-MM
0.242
0.238
0.247
0.640
0.219
0.184
Table 2: Systems that take their labels as text. Left, 1,000 MSP-Podcast clips: macro-F1 with the trained wording, with synonym names and paraphrased descriptions (Syn.) and with a reformulated question (New q.), each system on its best question (ASOM: its decision question, as in Table 1 ); yes/no AUC for eight emotion words unseen in training (Words). Right: mean macro-F1 on the unseen corpora, all 21 (All; chance 0.172) and the 17 whose transcript is at chance (Voice). Bold: best per column.
Audio-Language Models (ALMs) are making strides in understanding speech and non-speech audio. However, domain-specialist Foundation Models (FMs) remain the best for closed-ended speech processing tasks such as Speech Emotion Recognition (SER). Using ALMs for Zero-shot SER is a popular choice, but their potential to work with specialists to achieve state-of-the-art (SOTA) performance remains unexplored. We propose ZS-Fuse, a late-fusion method that combines zero-shot emotion estimates from a dual-encoder ALM with specialist FMs. To handle ambiguity in emotions and sensitivity to prompt choice, 1) we use a simple prompt ensemble and 2) suggest a novel technique called prompt amplification, which repeats audio and text queries to discover stronger zero-shot capabilities. We demonstrate the efficacy of our technique by evaluating ZS-Fuse with three dual-encoder ALMs and two FMs, and report improvements over SOTA baselines, such as WavLM-Large, on three speech emotion recognition datasets.
Emotion recognition in conversations is increasingly tackled with language models, but these models can be unstable and expensive to fine-tune or to prompt with long dialogue histories. We propose EmoLASP, a framework that combines a language model with declarative reasoning via Answer Set Programming (ASP) to predict VAD scores (Valence-Arousal-Dominance) in conversations. Experiments on a widely used benchmark dataset (IEMOCAP) across six open-source LLMs (3B-120B) and two PLMs (BERT, RoBERTa) show that EmoLASP improves prediction performance compared to using the language model alone, even when the LLMs/PLMs are given no dialogue history in their prompts or input vectors. The gains are largest for prompt-only LLMs, which EmoLASP uses without any fine-tuning. However, for fine-tuned PLMs, the reasoner adds little once dialogue history is available. EmoLASP's LLM pipeline demonstrates the potential advantages of using a reasoning approach to ensure emotion prediction consistency and to reduce both the cost of fine-tuning and the cost of prompting with long dialogue histories.
Thao Le, Michael Thielscher
School of Computer Science and Engineering, University of New South Wales, Australia
Large audio language models (ALMs) extend LLMs with auditory understanding. A common approach freezes the LLM and trains only an adapter on self-generated targets. However, this fails for reasoning LLMs (RLMs) whose built-in chain-of-thought traces expose the textual surrogate input, yielding unnatural responses. We propose self-rephrasing, converting self-generated responses into audio-understanding variants compatible with RLMs while preserving distributional alignment. We further fuse and compress multiple audio encoders for stronger representations. For training, we construct a 6M-instance multi-task corpus (2.5M unique prompts) spanning 19K hours of speech, music, and sound. Our 4B-parameter ALM outperforms similarly sized models and surpasses most larger ALMs on related audio-reasoning benchmarks, while preserving textual capabilities with a low training cost. Notably, we achieve the best open-source result on the MMAU-speech and MMSU benchmarks and rank third among all the models.