Prompt Amplification and Zero-Shot Late Fusion in Audio-Language Models for Speech Emotion Recognition

Saurabh Kataria; Xiao Hu

eess.ASMar 24, 2026

Prompt Amplification and Zero-Shot Late Fusion in Audio-Language Models for Speech Emotion Recognition

Authors: Saurabh Kataria, Xiao Hu

Organizations: School of Nursing, Emory University, Atlanta, USA

Abstract

Audio-Language Models (ALMs) are making strides in understanding speech and non-speech audio. However, domain-specialist Foundation Models (FMs) remain the best for closed-ended speech processing tasks such as Speech Emotion Recognition (SER). Using ALMs for Zero-shot SER is a popular choice, but their potential to work with specialists to achieve state-of-the-art (SOTA) performance remains unexplored. We propose ZS-Fuse, a late-fusion method that combines zero-shot emotion estimates from a dual-encoder ALM with specialist FMs. To handle ambiguity in emotions and sensitivity to prompt choice, 1) we use a simple prompt ensemble and 2) suggest a novel technique called prompt amplification, which repeats audio and text queries to discover stronger zero-shot capabilities. We demonstrate the efficacy of our technique by evaluating ZS-Fuse with three dual-encoder ALMs and two FMs, and report improvements over SOTA baselines, such as WavLM-Large, on three speech emotion recognition datasets.

Figures & tables

Fig. 1: Proposed Zero-shot Fusion (ZS-Fuse) method. The speech foundation fine-tuning branch is complemented by zero-shot estimates from an audio-text dual encoder. The similarity score ( $P\times E$ ) of pre-determined prompt templates ( $P$ ) is computed for all emotion classes ( $E$ ). Additionally, prompt amplification trick can be used to increase the audio length by repeating it and prompt strength by replacing “ <emotion_j> ” with “ N instances of <emotion_j> ”, for further potential gains.

Fig. 2: Zero-shot UAR of three ALMs on three datasets for 16 combinations of audio and text repeats ( $[1,2,3,4]\times[1,2,3,4]$ ).

Explore similar work

Acoustic Cue Alignment in Audio Language Models for Speech Emotion Recognition
Jun 5, 2026Iosif Tsangko, Andreas Triantafyllopoulos, Björn W. SchullerAudio-Language Model Evaluation Audio Understanding
CHI – Chair of Health Informatics, TUM University Hospital, Munich, Germany · MCML – Munich Center for Machine Learning, Munich, Germany · MDSI – Munich Data Science Institute, Munich, Germany +1
Comparative Reasoning: Making an Audio Language Model Better at Comparing Emotions
Jun 23, 2026Abinay Reddy Naini, Jaeyeon Kim, Chao-Han Huck Yang +2Audio-Language Model Evaluation Pairwise Comparison
Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA, 15213, US · NVIDIA
Disentangling Reasoning in Large Audio-Language Models for Ambiguous Emotion Prediction
Mar 9, 2026Xiaofeng Yu, Jiaheng Dong, Jean Honorio +3Speech Emotion Recognition Emotion Recognition
University of Auckland, New Zealand · The University of Melbourne, Australia · University of Birmingham, United Kingdom

Jun 5, 2026J/K move · Enter open · S save

Acoustic Cue Alignment in Audio Language Models for Speech Emotion Recognition

Iosif Tsangko, Andreas Triantafyllopoulos, Björn W. Schuller

CHI – Chair of Health Informatics, TUM University Hospital, Munich, Germany · MCML – Munich Center for Machine Learning, Munich, Germany · MDSI – Munich Data Science Institute, Munich, Germany+1

Audio-Language Model Evaluation Audio Understanding Audio-Language Models Speech Emotion Recognition

Open paper AskPDF arXiv

Instruction-following audio language models (ALMs) can be augmented with explicit acoustic cues, yet it remains unclear whether such cues are used in a grounded way when the raw audio is already available. We study this question in speech emotion recognition (SER) by deriving six interpretable acoustic concept tokens from the standardised eGeMAPS paralinguistic feature set. These tokens summarise energy, pitch, dynamics, brightness, formants, and voice quality, and are appended to the textual prompt while the audio input is kept unchanged. Across the widely used FAU-Aibo and IEMOCAP benchmarks, aligned tokens improve unweighted average recall (UAR), whereas shuffled, conflicting, or corrupted tokens reduce performance relative to aligned tokens and shift confusions toward neutral. Importantly, predictions do not collapse under strong token perturbations, suggesting that the models are sensitive to the symbolic cue channel but remain partly anchored to the audio signal. We argue that token-only interventions provide a practical way to probe audio-grounded cue use, robustness, and interpretability in ALM-based affective computing.

Abstract

Explore similar work

Acoustic Cue Alignment in Audio Language Models for Speech Emotion Recognition

Comparative Reasoning: Making an Audio Language Model Better at Comparing Emotions

Disentangling Reasoning in Large Audio-Language Models for Ambiguous Emotion Prediction