Recent Audio Multimodal Large Language Models (Audio MLLMs) demonstrate impressive performance on speech benchmarks, yet it remains unclear whether these models genuinely process acoustic signals or rely on text-based semantic inference. To systematically study this question, we introduce DEAF (Diagnostic Evaluation of Acoustic Faithfulness), a benchmark of over 2,700 conflict stimuli spanning three acoustic dimensions: emotional prosody, background sounds, and speaker identity. Then, we design a controlled multi-level evaluation framework that progressively increases textual influence, ranging from semantic conflicts in the content to misleading prompts and their combination, allowing us to disentangle content-driven bias from prompt-induced sycophancy. We further introduce diagnostic metrics to quantify model reliance on textual cues over acoustic signals. Our evaluation of seven Audio MLLMs reveals a consistent pattern of text dominance: models are sensitive to acoustic variations, yet predictions are predominantly driven by textual inputs, revealing a gap between high performance on standard speech benchmarks and genuine acoustic understanding.
Figures & tables
Figure 1: Illustration of the three acoustic–semantic conflict types in DEAF . ESC : happy text vs. depressed tone. BSC : quiet-setting text vs. noisy traffic audio. SIC : elderly-female text vs. young-boy voice. Correct prediction requires following the audio , not the text.
Figure 2: Word clouds of the happy class under Explicit and Implicit semantic conditions in ESC.
Figure 3: Overview of the DEAF dataset construction pipeline.
Type
Scripts
Match.
Mismatch.
Total
ESC
104
312
936
1,248
BSC
84
360
720
1,260 †
SIC
82
104
104
248 ‡
All
270
776
1,760
2,756
Table 1: Dataset statistics. † Across 5 SNR levels, including 180 neutral clips (36 per SNR level); single-SNR total is 252. ‡ Including 40 neutral clips shared across sub-dimensions.
Figure 4: Overview of the DEAF framework. Each conflict type is evaluated at three levels of increasing textual interference. Correct answers always require following acoustic evidence; trap answers follow textual cues.
Type
Question
ESC
What is the emotional tone of the speaker’s voice?
BSC
What environment do the background sounds suggest?
SIC G
What is the perceived gender of the speaker?
SIC A
How old does the speaker sound?
SIC C
What is the perceived gender and approximate age of the speaker?
Table 2: Question Templates.
ESC
BSC
SIC
Model
L1
L2
L3
Avg
L1
L2
L3
Avg
L1
L2
L3
Avg
Gemini 2.5 Flash
11.4
26.6
0.2
12.7
8.3
11.6
2.6
7.5
47.1
49.7
43.5
46.8
Gemini 3 Flash
5.2
2.5
0.0
2.6
7.8
16.2
2.0
8.7
43.1
42.2
34.0
39.8
GPT-4o-Audio
0.9
2.5
0.0
1.1
1.3
4.8
0.0
2.0
1.7
0.0
6.5
2.7
Audio Flamingo 3
24.8
30.8
6.6
20.7
14.5
16.5
4.6
11.9
44.6
15.0
43.4
34.3
Qwen2-Audio
14.7
34.0
5.8
18.2
5.5
20.5
3.5
9.8
40.0
30.2
32.6
34.3
Table 3: Acoustic Robustness Score (ARS, %) across conflict types and levels. Higher values indicate stronger acoustic grounding under textual interference.
Figure 5: Radar comparison of acoustic perception performance across conflict types.
ESC
BSC
SIC
Model
L1
L2
L3
Avg
L1
L2
L3
Avg
L1
L2
L3
Avg
Gemini 2.5 Flash
6.1
15.9
0.1
7.4
4.3
6.2
1.3
3.9
54.7
52.2
34.1
47.0
Gemini 3 Flash
2.7
1.3
0.0
1.3
4.3
10.1
1.0
5.1
37.9
30.6
23.7
30.7
GPT-4o-Audio
0.4
1.3
0.0
0.6
0.7
2.4
0.0
1.0
0.9
0.0
3.4
1.4
Audio Flamingo 3
14.6
19.2
3.4
12.4
7.9
9.1
2.3
6.4
66.4
8.2
31.9
35.5
Qwen2-Audio
8.1
22.2
3.0
11.1
2.9
12.9
1.8
5.9
32.8
19.4
22.0
24.7
Table 4: Accuracy (Acc, %) on mismatched samples across conflict types and levels.
Model
L1
L2
L3
Gemini 2.5 Flash
6.7
0.7
2.2
Gemini 3 Flash
3.5
-6.3
1.7
GPT-4o-Audio
1.1
-2.4
0.0
Audio Flamingo 3
2.5
1.3
3.9
Qwen2-Audio
0.2
3.0
-1.2
Qwen3-Omni
10.0
-0.9
12.2
Table 5: The Environment Discrimination Index (EDI) for BSC.
Model
ESC
BSC
SIC
Closed-source
Gemini 2.5 Flash
−1.8
+0.7
−3.5 †
Gemini 3 Flash
+2.6 †
+0.4
−1.5
GPT-4o-Audio
+0.8
+0.0
+1.1
Open-source
Audio Flamingo 3
−8.9 ‡
−5.0 †
−2.5
Table 6: Δ ARS (%, Explicit − Implicit) across conflict types. Significance is assessed with bootstrap tests (10,000 resamples); † denotes p<0.01 and ‡ denotes p<0.001 .
Condition
Acc
TDS
OR
Correct prompt
93.6
–
6.4
L1
1.0
64.7
34.3
L2
2.9
94.8
2.4
L3
0.1
99.7
0.2
Table 7: GPT-4o-Audio response breakdown on ESC (%), as assigned by the LLM judge. Correct prompt: the L1 clips with a prompt stating the correct emotion (e.g., “The speaker sounds happy.”). Since no textual cue points to a wrong answer, all incorrect responses are counted as OR. All rows come from a single evaluation run, so Acc may differ slightly from Table 4 .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Word clouds of the three datasets: (a) ESC, (b) BSC, and (c) SIC.
Category
Code
Description
Domestic
DWASHING
Laundry room
DKITCHEN
Kitchen, food prep
DLIVING
Living room
Nature
NFIELD
Sports field
NRIVER
River, flowing water
NPARK
City park
Appendix
Table 8: BSC environment categories from DEMAND.
ESC
BSC
SIC
Model
L1
L2
L3
Avg
L1
L2
L3
Avg
L1
L2
L3
Avg
Gemini 2.5 Flash
89.0
81.6
96.2
88.9
91.1
89.7
94.0
91.6
41.4
47.4
60.3
49.7
Gemini 3 Flash
95.4
96.9
98.3
96.9
39.6
41.0
41.4
40.7
50.0
68.1
59.9
59.3
GPT-4o-Audio
94.0
96.4
98.8
96.4
90.2
89.4
90.8
90.1
35.8
43.1
64.7
47.9
Audio Flamingo 3
81.1
77.9
92.7
83.9
91.4
87.3
97.2
92.0
33.6
91.8
68.1
64.5
Qwen2-Audio
79.3
72.1
87.5
79.6
62.6
50.0
77.8
63.5
51.3
68.5
62.9
60.9
Appendix
Table 9: Acoustic Sensitivity Score (ASS, %) across conflict types and levels. Higher values indicate greater sensitivity to acoustic changes. ASS is computed from changes in judge labels (C/T/O) and is an upper bound on class-level response change (Section 2.5 ).
Voice Profile
Voice ID
Young Male
1wzJ0Fr9SDexsF2IsKU4
Young Female
aFueGIISJUmscc05ZNfD
Elderly Male
Av4Fi2idMFuA8kTbVZgv
Elderly Female
0rEo3eAjssGDUCXHYENf
Appendix
Table 10: ElevenLabs voice profiles used for speech synthesis.
Subtask
n
Human ACC
Human ACC
Krippendorff
items
(full vote)
(high-conf, ≥ 3-agree)
α
ESC
100
51.7%
69.4%
0.35
BSC
100
75.0%
88.4%
0.68
SIC
100
100.0%
100.0%
1.000
Overall
300
75.6%
86.0%
0.72
Appendix
Table 11: Human evaluation results across different subtasks.
Large Audio-Language Models show consistent performance gains across speech and audio benchmarks, yet high scores may not reflect true auditory perception. If a model can answer questions without processing the acoustic signal, the benchmark fails as a measure of auditory understanding. We present a diagnostic framework using two axes: text prior, which measures answerability from text and general knowledge alone, and audio reliance, which assesses actual dependency on the acoustic signal. Evaluating eight LALMs across three benchmarks, we find that models retain 60-72% of their full audio scores even without any audio input. Moreover, among items that require audio, only 3.0-4.2% need the complete audio clip; the majority can be resolved using localized fragments. These findings challenge the assumption that benchmark performance equals robust audio understanding, and we conclude with practical guidelines for improving evaluation reliability and benchmark design.
Leonardo Haw-Yang Foo, Chih-Kai Yang, Chen-An Li +2
National Taiwan University · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)
We introduce HEAR (Human-recorded Evaluation of Audio-LLM bias by Real speakers), a large-scale, ecologically valid benchmark comprising 87k real human audio samples from 843 demographically diverse participants. HEAR enables comprehensive evaluation through Multiple Choice Question Answering (MCQA) and open-ended long-form tasks. To our knowledge, this is the first large-scale voice benchmark grounded entirely in authentic human speech. We evaluate model behavior across both real-time speech-to-speech and speech-to-text architectures. Our results reveal that voice-conditioned bias is a model-specific property. Furthermore, we demonstrate that personalization instructions consistently exacerbate demographic disparities. Our findings establish that voice bias is a controllable model characteristic, providing a foundational framework for future bias mitigation and evaluation in Audio-LLM development.
Audio language models (ALMs) are increasingly used for speech-based understanding, yet their ability to perform semantic reasoning beyond transcription, Text-to-Audio Retrieval, Captioning, and Question-Answering accuracy remains insufficiently benchmarked. In particular, the effects of accent variation, domain shift, and semantic over-inference on audio reasoning are poorly understood. We evaluate audio language models across five semantic and paralinguistic reasoning tasks: entailment, consistency, plausibility, accent drift, and accent restraint. Collectively, these tasks assess a model's ability to reason over spoken audio as the primary evidence source, including whether a textual hypothesis can be inferred, contradicted, or left undetermined by the audio, whether statements align or conflict with spoken content, whether claims are plausible given the discourse, and whether model predictions remain stable or appropriately constrained across accent variation. These findings highlight critical limitations in current audio reasoning evaluations and hope to provide guidance for more robust and equitable ALM design and assessment