VocalGrad: Evaluating Acoustic Perception in Audio Language Models

Koki Ryu; Hitomi Yanaka

cs.SDOct 16, 2026

VocalGrad: Evaluating Acoustic Perception in Audio Language Models

Authors: Koki Ryu, Hitomi Yanaka

Organizations: The University of Tokyo · Riken · Tohoku University

Abstract

Recent large audio language models (LALMs) have extended their capabilities beyond lexical speech understanding to tasks requiring paralinguistic and non-linguistic audio understanding, such as emotion recognition. However, these models still struggle to reliably utilize acoustic information, particularly when it co-occurs with spoken linguistic content. Existing benchmarks do not clearly distinguish acoustic perception from higher-level reasoning, making it difficult to identify the source of these failures. To address this issue, we introduce VocalGrad, a benchmark designed to evaluate LALMs' ability to perceive low-level acoustic variations. VocalGrad formulates this problem as a simple binary classification of temporal change direction, enabling evaluation largely independent of higher-level reasoning or semantic understanding. Our benchmark results reveal a substantial gap in acoustic perception between human annotators and state-of-the-art LALMs: while human annotators achieve high accuracy, LALMs perform at near-chance levels. We further conduct linear probing to identify structural bottlenecks in these models. Our analysis reveals two distinct failure modes: for some categories, task-relevant information is preserved in internal representations but not utilized by the language model head; for others, the information becomes progressively less linearly decodable across model layers. Overall, our results reveal substantial limitations of LALMs in modeling acoustic variation under controlled settings and expose representation differences not observable from text outputs.

Figures & tables

Figure 1: Overview of VocalGrad. Each source speech clip is transformed with a temporal change applied to a single acoustic factor. The task is formulated as a binary classification of change direction, enabling evaluation of acoustic perception in isolation from semantic understanding required for downstream tasks.

Statistics	Value	Statistics	Value
Total Questions	21,600	Intensity Tiers	3
Speaker-Intrinsic Categories	3	Transformation Trajectories	4
Environment-Induced Categories	6	Avg. Audio Length	3.98 s

Table 1: Key statistics of VocalGrad.

Model	Avg.	speed	pitch	volume	distortion	roughness	noise	echo	clarity	vibration
Human Avg.	94.4	95.3	84.7	100.0	86.0	99.3	100.0	94.7	90.7	99.3
Gemini 3 Flash	50.2	50.0	51.5	49.9	48.1	44.8	55.5	50.3	54.0	47.6
Kimi-Audio	51.8	53.1	50.5	50.5	51.6	48.0	56.5	54.5	53.6	47.6
MiMo-Audio	50.3	50.7	50.5	52.9	49.8	43.2	53.2	50.0	52.2	50.3
Step-Audio 2 mini	50.0	49.9	50.2	49.9	50.2	49.9	49.9	50.0	49.8	49.8
Audio Flamingo 3	49.6	50.0	50.0	50.0	50.0	46.7	50.0	50.0	49.9	50.0

Table 2: Human annotator average and model accuracies (%) on VocalGrad by category.

Model	pitch	volume	noise	vibration
Gemini 3 Flash	48.3	50.0	47.5	49.2
Kimi-Audio	50.4	57.5	50.0	50.0
MiMo-Audio	50.0	55.8	64.2	50.0
Step-Audio 2 mini	50.0	50.0	50.0	50.0
Audio Flamingo 3	50.0	50.0	50.0	50.0

Table 3: Model accuracies (%) on VocalGrad ablation-beep subsets by category.

Figure 2: Linear probing accuracies with different audio representations.

Figure 3: Linear probing accuracies with text prediction representations across language decoder layers.

Method	speed	pitch	volume	distortion	roughness	noise	echo	clarity	vibration
LM-Head	73.2	57.0	88.2	52.2	97.7	98.6	61.5	57.4	53.3
All-Linear	99.0	98.7	100.0	88.5	100.0	100.0	96.8	87.0	99.2

Table 4: Kimi-Audio fine-tuning accuracy (%) by category.

Panel of Figure 4 — Figure 4: Analysis of LoRA fine-tuning effects on Kimi-Audio. (Left) Cross-category performance. (Right) Layer-wise probing accuracy for models fine-tuned on Volume and Voice Pitch.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Screenshot of the web-based tool used for human annotation of VocalGrad.

Model	jump	linear	quad_first_flat	quad_last_flat
Human Avg.	98.5	93.7	92.0	93.8
Gemini 3 Flash	50.5	49.9	49.4	50.9
Kimi-Audio	52.8	51.3	50.1	53.0
MiMo-Audio	51.6	49.9	49.3	50.4
Step-Audio 2 mini	50.0	50.0	49.8	50.0
Audio Flamingo 3	49.7	49.6	49.5	49.7

Appendix

Table 5: Human annotator average and model accuracies (%) on VocalGrad by temporal trajectory.

Figure 6: Tier-wise accuracy for Background Noise and Voice Pitch.

Model	Exact match	Rule-based match
Gemini 3 Flash	83.9	99.1
Kimi-Audio	99.9	99.9
MiMo-Audio	99.9	100.0
Step-Audio 2 mini	100.0	100.0
Audio Flamingo 3	100.0	100.0

Appendix

Table 6: Rates (%) of raw model outputs that exactly match the target direction words or match the rule-based parser on VocalGrad.

Model	Avg.	speed	pitch	volume	distortion	roughness	noise	echo	clarity	vibration
Gemini 3 Flash	42.8	45.8	50.2	47.6	27.1	15.3	52.0	49.0	52.8	45.4
Kimi-Audio	51.8	53.1	50.5	50.5	51.6	48.0	56.5	54.5	53.6	47.6
MiMo-Audio	50.3	50.7	50.5	52.7	49.8	43.2	53.2	50.0	52.2	50.3
Step-Audio 2 mini	50.0	49.9	50.2	49.9	50.2	49.9	49.9	50.0	49.8	49.8
Audio Flamingo 3	49.6	50.0	50.0	50.0	50.0	46.7	50.0	50.0	49.9	50.0

Appendix

Table 7: Model accuracies (%) on VocalGrad when only exact raw outputs are counted as correct.

Model	Avg.	speed	pitch	volume	distortion	roughness	noise	echo	clarity	vibration
Gemini 3 Flash	16.1	9.1	3.3	5.0	43.5	67.1	7.0	2.4	2.7	4.5
Kimi-Audio	0.1	0.0	0.0	0.0	0.0	0.1	0.0	0.4	0.0	0.0
MiMo-Audio	0.1	0.0	0.0	0.6	0.0	0.0	0.0	0.1	0.0	0.0
Step-Audio 2 mini	0.0	0.1	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
Audio Flamingo 3	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0

Appendix

Table 8: Category-wise non-exact-match rates (%) for raw model outputs on VocalGrad.

Figure 7: Accuracy (%) of humans and LALMs on the VocalGrad benchmark, with error bars representing binomial simulation-based 95% confidence intervals. Error bars are omitted for 100% accuracy.

Clips	Model	Avg.	Max category	Max value	Min category	Min value
VocalGrad	Gemini 3 Flash	45.8	distortion	85.7	echo	2.1
	Kimi-Audio	24.5	noise	69.5	volume	0.8
	MiMo-Audio	56.6	echo	90.9	noise	3.9
	Step-Audio 2 mini	87.0	roughness	99.9	echo	0.0
	Audio Flamingo 3	0.7	roughness	5.9	noise	0.0
VCTK source	Gemini 3 Flash	47.8	distortion	90.7	echo	2.0

Appendix

Table 9: Increase response bias (%) on original VocalGrad data and source clips.

Model	Avg.	speed	pitch	volume	distortion	roughness	noise	echo	clarity	vibration
Kimi-Audio	51.4	50.7	50.2	50.1	49.7	62.0	47.4	50.0	52.8	50.1
MiMo-Audio	49.0	50.5	49.4	46.3	50.0	47.3	49.0	48.8	50.5	49.2
Step-Audio 2 mini	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0
Audio Flamingo 3	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0

Appendix

Table 10: VocalGrad benchmarking accuracy (%) with the swap_order prompt.

Model	Avg.	speed	pitch	volume	distortion	roughness	noise	echo	clarity	vibration
Kimi-Audio	51.1	51.7	50.5	50.0	49.7	50.9	53.9	51.8	52.5	49.0
MiMo-Audio	49.8	50.0	50.2	49.2	51.4	44.5	53.0	50.2	49.9	50.2
Step-Audio 2 mini	50.2	50.2	50.0	50.1	50.1	50.2	50.1	51.0	50.2	50.1
Audio Flamingo 3	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0

Appendix

Table 11: VocalGrad benchmarking accuracy (%) with the audio_ref prompt.

Panel of Figure 8 — Figure 8: Cross-attribute VocalGrad accuracies (%) for Kimi-Audio and MiMo-Audio.

Category	Pattern	Query
Volume	rise_fall	Does the loudness rise or fall as the audio progresses?
Volume	intensity_inc_dec	Does the sound intensity increase or decrease throughout the recording?
Volume	intensity_ing	Is the audio intensity increasing or decreasing over time?
Voice Pitch	tone_rise_fall	Does the tone rise or fall over time?
Voice Pitch	freq_up_down	Does the sound frequency change upward or downward over time?
Voice Pitch	freq_rise_decline	Over time, does the frequency of the voice rise or decline?

Appendix

Table 12: Alternative category-specific query patterns used to evaluate prompt sensitivity on VocalGrad.

Category	Pattern	Accuracy	Increase ratio
Volume	rise_fall	50.1	0.6
	intensity_inc_dec	50.5	0.8
	intensity_ing	50.0	0.0
Voice Pitch	tone_rise_fall	50.0	0.1
	freq_up_down	50.4	5.7
	freq_rise_decline	51.5	40.1

Appendix

Table 13: Kimi-Audio performance with category-specific alternative query patterns on VocalGrad. Accuracy and the proportion of outputs mapped to the increasing direction are reported in percent.

Model	Avg.	speed	pitch	volume	distortion	roughness	noise	echo	clarity	vibration
Kimi-Audio	50.2	51.6	49.3	51.1	50.0	49.3	49.2	50.7	51.0	49.5
MiMo-Audio	49.9	47.6	49.3	50.0	50.3	52.0	51.0	49.8	50.1	48.8
Step-Audio 2 mini	51.0	53.2	49.0	52.8	50.0	49.6	51.3	51.1	50.3	51.8

Appendix

Table 14: Few-shot prompting accuracies (%) on VocalGrad by category.

Dataset	Avg.	speed	pitch	volume	distortion	roughness	noise	echo	clarity	vibration
VCTK	51.8	53.1	50.5	50.5	51.6	48.0	56.5	54.5	53.6	47.6
Loquacious Set	50.6	52.5	50.4	50.8	50.0	49.1	49.7	51.3	51.3	50.6
Common Voice	52.9	59.0	54.0	51.2	50.5	56.1	50.3	53.7	53.0	48.4

Appendix

Table 15: Kimi-Audio accuracies (%) on VocalGrad across different source datasets.

Problem Format	speed	pitch	volume	distortion	roughness	noise	echo	clarity	vibration
Yes/No	59.4	58.3	47.9	53.3	44.9	58.0	45.7	51.0	50.4
Ternary	26.5	24.5	24.2	24.4	24.1	23.6	21.6	27.6	21.8

Appendix

Table 16: Kimi-Audio performance (% macro-F1) on alternative VocalGrad problem formats.

Format	Prediction	Increasing	Decreasing	Source
Yes/No	Yes	87.6	85.6	72.8
Yes/No	No	12.4	14.4	27.2
Ternary	Increase	2.2	1.6	0.2
	Decrease	92.0	91.1	89.2
	Constant	5.6	7.1	10.3
	Unparsed	0.2	0.2	0.2

Appendix

Table 17: Prediction distributions (%) of Kimi-Audio under the binary change-detection and ternary variation tasks. Columns indicate the input clip type.

Layer	$a_{1}$	$a_{2}$	$a_{3}$	$a_{4}$	$o$
1	85.5	78.3	78.5	92.4	93.1
10	82.8	69.6	69.5	89.8	88.8
25	75.1	65.7	64.7	80.5	85.5

Appendix

Table 18: Position-wise linear probing accuracy (%) of Kimi-Audio under the Query

\rightarrow

Audio context ordering.

a_{i}

denotes the final audio token of the

i

-th temporal segment, and

o

denotes the output-prediction position.

Layer	$a_{1}$	$a_{2}$	$a_{3}$	$a_{4}$	$q_{\mathrm{attr}}$	$q_{p}$	$q_{l}$	$o$
1	86.5	80.3	80.6	92.9	91.6	87.4	89.7	90.6
10	81.1	66.6	66.7	84.8	80.6	79.3	84.6	84.7
25	74.1	59.9	59.9	70.6	71.1	78.2	78.7	76.8

Appendix

Table 19: Position-wise linear probing accuracy (%) of Kimi-Audio under the Audio

\rightarrow

Query context ordering.

a_{i}

denotes the final audio token of the

i

-th temporal segment,

q_{\mathrm{attr}}

the final token of the attribute expression,

q_{p}

the penultimate query token,

q_{l}

the final query token, and

o

the output-prediction position.

Method	Model	speed	pitch	volume	distortion	roughness	noise	echo	clarity	vibration
LM-Head	Kimi-Audio	73.2	57.0	88.2	52.2	97.7	98.6	61.5	57.4	53.3
	MiMo-Audio	53.8	50.4	85.2	68.0	83.8	72.3	52.9	78.1	77.5
	Step-Audio 2 mini	56.7	50.2	52.9	51.7	66.7	76.4	50.0	50.5	50.8
	Audio Flamingo 3	58.9	55.6	69.3	51.3	84.2	78.9	51.6	55.2	58.8
All-Linear	Kimi-Audio	99.0	98.7	100.0	88.5	100.0	100.0	96.8	87.0	99.2
	MiMo-Audio	98.4	97.2	100.0	99.1	100.0	100.0	94.6	100.0	99.7

Appendix

Table 20: Performance (%) of all fine-tuned models on VocalGrad.

Layer index	Base	Volume-tuned	Voice-Pitch-tuned
5	0.005 / 0.148	0.004 / 0.296	0.019 / 0.146
10	0.020 / 0.153	-0.059 / 0.490	0.028 / 0.187
15	0.023 / 0.132	-0.082 / 0.586	0.038 / 0.231
20	0.079 / 0.190	-0.089 / 0.599	0.139 / 0.709
25	0.094 / 0.234	-0.084 / 0.657	0.134 / 0.737
28	0.054 / 0.205	-0.095 / 0.729	0.116 / 0.682

Appendix

Table 21: Mean signed cosine similarity and mean absolute cosine similarity between category-specific increase–decrease directions in Kimi-Audio hidden representations. Each cell reports signed / absolute cosine similarity, averaged over the 36 unordered pairs of the nine augmentation categories.

Figure 9: Signed cosine similarity between category-specific increase–decrease directions in Kimi-Audio hidden representations at layer 28. Results are shown for the base, Volume-tuned, and Voice-Pitch-tuned models.

Recall
Model	Macro	Anger	Disgust	Fear	Joy	Neutral	Sadness	Surprise
Base	41.7	46.7	17.6	16.5	46.6	67.2	49.3	48.3
Volume-tuned	40.7	46.6	17.4	15.2	44.3	62.5	51.8	47.3
Voice-Pitch-tuned	35.3	41.5	14.9	11.7	31.9	67.6	42.0	37.5
Model	Macro	Anger	Disgust	Fear	Joy	Neutral	Sadness	Surprise
Base	42.7	47.2	30.9	30.0	52.7	65.7	32.7	39.5

Appendix

Table 22: Performance (%) of the base and VocalGrad-fine-tuned Kimi-Audio models on the MELD emotion recognition dataset. The best result among the three models for each metric and category is shown in bold.

Audio-Language Model Evaluation

Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.

See all 206 papers

Aug 23, 2026

MRMAD: A Multi-Round Multi-Audio Benchmark for Evaluating Acoustic Degradation Perception in Large Audio-Language Models

Large audio-language models (LALMs) have shown promising progress in understanding speech, music, and general sound events, yet their ability to reason about how audio signals are degraded remains underexplored. Existing benchmarks primarily evaluate semantic understanding, event recognition, or high-level audio reasoning, leaving a basic question unanswered: Do LALMs understand the differences in audio quality? We introduce MRMAD, a Multi-Round Multi-Audio Degradation benchmark for evaluating audio degradation perception and understanding in LALMs. MRMAD spans speech, music, and sound, and frames evaluation as multi-turn dialogues across multiple audio inputs, requiring models to identify types of degradation, compare severity, and perceive corruption changes across turns. Unlike current single-turn audio-language benchmarks, MRMAD evaluates whether LALMs can maintain consistent degradation hypotheses with new evidence and comprehend low-level acoustic phenomena over multi-turn dialogues. Through a systematic evaluation of 18 representative LALMs from non-thinking to reasoning and Omni models, we find that current models often recognize coarse content while failing to diagnose, compare, or reason about degradations reliably. Human evaluations further reveal a significant perception gap between LALMs and human listeners. MRMAD thus exposes a critical yet overlooked aspect of audio-language understanding and provides a diagnostic foundation for building future LALMs that are robust to real-world acoustic conditions.

Yize Li, Ningyuan Yang, Sile Yin +6

Northeastern University · Stony Brook University · Bose Corporation

Audio-Language Model Evaluation Audio Reasoning

Ask PDF

May 26, 2026

Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox

Audio large language models (Audio LLMs) demonstrate strong performance on speech understanding tasks, yet their ability to understand paralinguistic information remains limited. To systematically quantify this issue, we introduce VoxParadox, an adversarial benchmark with 2,000 verified examples, spanning 10 paralinguistic tasks, created with controlled speech synthesis to intentionally mismatch transcript claims and speaking style, enabling direct measurement of speech paralinguistic understanding. Evaluation of a diverse set of Audio LLMs reveals consistently low accuracy on acoustic ground truth and a strong tendency to follow language-implied (incorrect) answers. To understand the cause of this gap, we perform layer-wise probing and find that (i) paralinguistic cues can degrade in deeper encoder layers and at the encoder--LLM interface, and (ii) even when such cues are available in audio tokens, the language model frequently ignores them. To address these problems, we propose Prompt-Conditioned Layer Mixer (PCLM), which adaptively combines information from multiple audio layers based on the input prompt, and pair it with Direct Preference Optimization (DPO) to explicitly prefer acoustically supported options over language-implied alternatives. These methods substantially improve Audio LLM paralinguistic understanding, improving Audio Flamingo 3 from 17.40% to 65.20% on VoxParadox, and from 37.74% to 54.78% on MMSU paralinguistic subset. Our project page is available at https://voxparadox.github.io/.

Jiacheng Pang, Ashutosh Chaubey, Mohammad Soleymani

Institute for Creative Technologies, University of Southern California, Los Angeles, USA

Audio-Language Model Evaluation Speech Processing Audio Understanding Audio-Language Models

Ask PDF

Mar 17, 2026

DEAF: A Benchmark for Diagnostic Evaluation of Acoustic Faithfulness in Audio Language Models

Recent Audio Multimodal Large Language Models (Audio MLLMs) demonstrate impressive performance on speech benchmarks, yet it remains unclear whether these models genuinely process acoustic signals or rely on text-based semantic inference. To systematically study this question, we introduce DEAF (Diagnostic Evaluation of Acoustic Faithfulness), a benchmark of over 2,700 conflict stimuli spanning three acoustic dimensions: emotional prosody, background sounds, and speaker identity. Then, we design a controlled multi-level evaluation framework that progressively increases textual influence, ranging from semantic conflicts in the content to misleading prompts and their combination, allowing us to disentangle content-driven bias from prompt-induced sycophancy. We further introduce diagnostic metrics to quantify model reliance on textual cues over acoustic signals. Our evaluation of seven Audio MLLMs reveals a consistent pattern of text dominance: models are sensitive to acoustic variations, yet predictions are predominantly driven by textual inputs, revealing a gap between high performance on standard speech benchmarks and genuine acoustic understanding.

Jiaqi Xiong, Yunjia Qi, Qi Cao +6

University of Oxford · XJTLU · HNU +3

Audio-Language Model Evaluation Audio-Language Models Audio Understanding

Ask PDF

Abstract

Audio-Language Model Evaluation

Explore similar work

MRMAD: A Multi-Round Multi-Audio Benchmark for Evaluating Acoustic Degradation Perception in Large Audio-Language Models

Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox

DEAF: A Benchmark for Diagnostic Evaluation of Acoustic Faithfulness in Audio Language Models