cs.SDSep 19, 2026

ParA-LLM: A Unified Approach to Paralinguistic and Acoustic Speech Understanding

Authors: Nishit Anand, Jiaqi Su, Ke Chen, Yunyun Wang, Dinesh Manocha, Ramani Duraiswami, Rithesh Kumar, Zeyu Jin

Organizations: Adobe Research, USA · University of Maryland, College Park, USA · OpenAI, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Recent advances in Audio LLMs have achieved human-level speech recognition, yet existing systems struggle to capture paralinguistic aspects such as speaker traits, expressive variations, and environmental acoustic conditions. To address this, we design a framework of 22 paralinguistic characteristics and create a dataset of over 1.2M Audio-QA pairs. We develop ParA-LLM, trained with a two-stage curriculum: first on single-attribute questions to build foundational knowledge, then on multi-attribute questions for joint reasoning over speaker and acoustic characteristics. We also release ParA-Bench, a benchmark of 6,000 multiple-choice questions across speaker-speech, acoustic, and mixed categories, where frontier models like GPT-4o-Audio achieve only 36% accuracy. ParA-LLM surpasses state-of-the-art Audio LLMs like GPT-4o-Audio by 7.5% on ParA-Bench, with additional gains of 1.13% on MMAU-Pro Speech and 7.49% on MMAR Speech.

Figures & tables

Figure 1: Left: Overall accuracy of paralinguistic and acoustic understanding across GPT-4o-Audio and ParA-LLM on ParA-Bench. Right: a QA example, where GPT-4o-Audio misidentifies speaker and speech characteristics.
Figure 2: Our end-to-end pipeline for paralinguistic and acoustic understanding. Data Simulation : Clean speech is augmented with noise and reverberation for diverse acoustic conditions. Characteristic Annotation : Metadata is mapped to 22 well-defined characteristics across acoustic, speaker, and speech properties. QA Generation : Template-based and LLM-based ICL generation yield 1.2M audio–QA pairs across two curriculum stages. Model Training : ParA-LLM is trained on atomic then multi-attribute QA pairs. Benchmark Creation : ParA-Bench comprises 6K multiple-choice questions across speaker-speech, acoustic, and mixed categories.
ModelSpeaker- SpeechAcousticMixedOverall
Qwen2-Audio34.8023.4525.9028.05
Voxtral51.9529.2035.2538.80
Audio Flamingo 334.3538.7533.4035.50
Mellow3.603.603.803.67
R1-AQA26.6019.0021.2022.27
Qwen2.5 Omni 3B20.859.4510.3513.55
Table 1: Accuracy of models on ParA-Bench across speaker-speech, acoustic, and mixed categories. Bold denotes the best and underline denotes the second-best in each category.
ModelMMAU-Pro (Speech)MMAR (Speech)MMAR (Sound-Speech)MMAR (Overall)
Qwen2-Audio-7B-Instruct40.9635.3740.8336.00
ParA-LLM Stage 141.9837.7645.8739.40
ParA-LLM Stage 242.0942.8646.3339.70
Table 2: Performance of ParA-LLM trained with curriculum learning compared to the Qwen2-Audio-7B-Instruct baseline, demonstrating steady improvement across the benchmarks.
More in this topic

Audio Understanding

Audio understanding enables machines to recognize speech, sounds, music, speakers, and events, then answer questions grounded in what they hear.

See all 207 papers

Explore similar work

CardsList

  1. Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox

    May 26, 2026Jiacheng Pang, Ashutosh Chaubey, Mohammad SoleymaniAudio-Language Model EvaluationSpeech Processing

    Institute for Creative Technologies, University of Southern California, Los Angeles, USA

  2. ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge

    Jun 23, 2026Jisu Jeon, Seungyeon Jwa, Joosung Lee +6Audio-Language Model EvaluationPairwise Comparison

    Hongik University, South Korea · Seoul National University, South Korea · NAVER Cloud, South Korea +1

  3. Aligning Paralinguistic Understanding and Generation in Speech LLMs via Multi-Task Reinforcement Learning

    Mar 16, 2026Jingxiang Chen, Minseok Kim, Seong-Gyun Leem +13Speech Language ModelsEmotion Representation in Language Models

    Meta Reality Labs