Recent advances in Audio LLMs have achieved human-level speech recognition, yet existing systems struggle to capture paralinguistic aspects such as speaker traits, expressive variations, and environmental acoustic conditions. To address this, we design a framework of 22 paralinguistic characteristics and create a dataset of over 1.2M Audio-QA pairs. We develop ParA-LLM, trained with a two-stage curriculum: first on single-attribute questions to build foundational knowledge, then on multi-attribute questions for joint reasoning over speaker and acoustic characteristics. We also release ParA-Bench, a benchmark of 6,000 multiple-choice questions across speaker-speech, acoustic, and mixed categories, where frontier models like GPT-4o-Audio achieve only 36% accuracy. ParA-LLM surpasses state-of-the-art Audio LLMs like GPT-4o-Audio by 7.5% on ParA-Bench, with additional gains of 1.13% on MMAU-Pro Speech and 7.49% on MMAR Speech.
Figures & tables
Figure 1: Left: Overall accuracy of paralinguistic and acoustic understanding across GPT-4o-Audio and ParA-LLM on ParA-Bench. Right: a QA example, where GPT-4o-Audio misidentifies speaker and speech characteristics.
Figure 2: Our end-to-end pipeline for paralinguistic and acoustic understanding. Data Simulation : Clean speech is augmented with noise and reverberation for diverse acoustic conditions. Characteristic Annotation : Metadata is mapped to 22 well-defined characteristics across acoustic, speaker, and speech properties. QA Generation : Template-based and LLM-based ICL generation yield 1.2M audio–QA pairs across two curriculum stages. Model Training : ParA-LLM is trained on atomic then multi-attribute QA pairs. Benchmark Creation : ParA-Bench comprises 6K multiple-choice questions across speaker-speech, acoustic, and mixed categories.
Model
Speaker- Speech
Acoustic
Mixed
Overall
Qwen2-Audio
34.80
23.45
25.90
28.05
Voxtral
51.95
29.20
35.25
38.80
Audio Flamingo 3
34.35
38.75
33.40
35.50
Mellow
3.60
3.60
3.80
3.67
R1-AQA
26.60
19.00
21.20
22.27
Qwen2.5 Omni 3B
20.85
9.45
10.35
13.55
Table 1: Accuracy of models on ParA-Bench across speaker-speech, acoustic, and mixed categories. Bold denotes the best and underline denotes the second-best in each category.
Model
MMAU-Pro (Speech)
MMAR (Speech)
MMAR (Sound-Speech)
MMAR (Overall)
Qwen2-Audio-7B-Instruct
40.96
35.37
40.83
36.00
ParA-LLM Stage 1
41.98
37.76
45.87
39.40
ParA-LLM Stage 2
42.09
42.86
46.33
39.70
Table 2: Performance of ParA-LLM trained with curriculum learning compared to the Qwen2-Audio-7B-Instruct baseline, demonstrating steady improvement across the benchmarks.
Audio large language models (Audio LLMs) demonstrate strong performance on speech understanding tasks, yet their ability to understand paralinguistic information remains limited. To systematically quantify this issue, we introduce VoxParadox, an adversarial benchmark with 2,000 verified examples, spanning 10 paralinguistic tasks, created with controlled speech synthesis to intentionally mismatch transcript claims and speaking style, enabling direct measurement of speech paralinguistic understanding. Evaluation of a diverse set of Audio LLMs reveals consistently low accuracy on acoustic ground truth and a strong tendency to follow language-implied (incorrect) answers. To understand the cause of this gap, we perform layer-wise probing and find that (i) paralinguistic cues can degrade in deeper encoder layers and at the encoder--LLM interface, and (ii) even when such cues are available in audio tokens, the language model frequently ignores them. To address these problems, we propose Prompt-Conditioned Layer Mixer (PCLM), which adaptively combines information from multiple audio layers based on the input prompt, and pair it with Direct Preference Optimization (DPO) to explicitly prefer acoustically supported options over language-implied alternatives. These methods substantially improve Audio LLM paralinguistic understanding, improving Audio Flamingo 3 from 17.40% to 65.20% on VoxParadox, and from 37.74% to 54.78% on MMSU paralinguistic subset. Our project page is available at https://voxparadox.github.io/.
Jiacheng Pang, Ashutosh Chaubey, Mohammad Soleymani
Institute for Creative Technologies, University of Southern California, Los Angeles, USA
Large Audio-Language Models (LALMs) have been widely used as judge models for the automatic evaluation of generated speech. However, prior approaches predominantly focus on holistic naturalness, leaving fine-grained paralinguistic distinctions underexplored. We introduce ParaPairAudioBench, a pairwise benchmark of 5,175 audio pairs across five paralinguistic dimensions: Style, Rate, Emphasis, Age, and Gender. Our experiments show that current LALM judges still lag behind human judgments by 32%p on average and exhibit severe calibration failures, particularly in Tie cases where the correct decision is to abstain. To further analyze lexical versus acoustic reliance, the benchmark includes both same-transcript and cross-transcript conditions. ParaPairAudioBench enables multi-dimensional, calibration-aware assessment of the reliability of LALM-as-a-Judge for paralinguistic speech evaluation.
Jisu Jeon, Seungyeon Jwa, Joosung Lee +6
Hongik University, South Korea · Seoul National University, South Korea · NAVER Cloud, South Korea +1
Speech large language models (LLMs) observe paralinguistic cues such as prosody, emotion, and non-verbal sounds--crucial for intent understanding. However, leveraging these cues faces challenges: limited training data, annotation difficulty, and models exploiting lexical shortcuts over paralinguistic signals. We propose multi-task reinforcement learning (RL) with chain-of-thought prompting that elicits explicit affective reasoning. To address data scarcity, we introduce a paralinguistics-aware speech LLM (PALLM) that jointly optimizes sentiment classification from audio and paralinguistics-aware response generation via a two-stage pipeline. Experiments demonstrate that our approach improves paralinguistics understanding over both supervised baselines and strong proprietary models (Gemini-2.5-Pro, GPT-4o-audio) by 8-12% on Expresso, IEMOCAP, and RAVDESS. The results show that modeling paralinguistic reasoning with multi-task RL is crucial for building emotionally intelligent speech LLMs.