Large language models (LLMs) have been widely used as knowledge backbones of Large Audio Language Models (LALMs), yet how much auditory knowledge they encode through text-only pre-training and how this affects downstream performance remains unclear. We study this gap by comparing different LLMs under two text-only and one audio-grounded setting: (1) direct probing on AKB-2000, a curated benchmark testing the breadth and depth of auditory knowledge; (2) cascade evaluation, where LLMs reason over text descriptions from an audio captioner; and (3) audio-grounded evaluation, where each LLM is fine-tuned into a Large Audio Language Model (LALM) with an audio encoder. Our findings reveal that auditory knowledge varies substantially across families, and text-only results are strongly correlated with audio performance. Our work provides empirical grounding for a comprehensive understanding of LLMs in audio research.
Figures & tables
Figure 1: Overview of the three evaluations introduced in this work. (Top) AKB-2000 construction pipeline: a two-level taxonomy guides LLM-assisted question generation, followed by human verification. (Middle) Cascade evaluation: a captioner converts audio to text descriptions fed to a text-only LLM. (Bottom) Audio-grounded evaluation: each LLM is fine-tuned into a LALM using the DeSTA self-distillation framework and evaluated with audio input.
Model
Avg.
Sound
Paralinguistic
Phonetic
Music
Quality
Technical
Proprietary LLMs
Gemini-2.5-Pro
96.05
96.37
96.46
94.93
95.76
97.45
95.35
GPT-5
94.35
95.16
94.44
93.55
95.51
94.90
92.44
Claude-Sonnet-4.5
95.70
95.16
96.97
91.71
96.01
95.41
96.22
Gemini-2.0-Flash
91.85
93.15
92.42
88.94
92.77
94.39
89.24
GPT-4o
92.90
95.97
93.94
90.32
95.01
93.88
87.50
Table 1: Text-only Auditory Knowledge Benchmark (AKB-2000) evaluation accuracy (%) across six categories. Shading indicates relative ranking among open-weight models (darker = higher); values in parentheses show the gap from Gemini-2.5-Pro.
Figure 2: Pearson correlation heatmap across all five evaluation metrics. The white line separates text-only metrics (top-left) from audio-grounded metrics (bottom-right).
Subcategory
Avg.
Phonetics & Phonology
60.2
Stress & Emphasis
58.3
Music Theory
52.9
Syllable & Stress
49.1
Rhyme
48.7
Table 2: The five most challenging subcategories in AKB-2000, averaged over 12 open-weight LLMs (%).
Model
MMAU (Text)
MMAR (Text)
Avg.
Sound
Music
Speech
Avg.
Sound
Music
Speech
Captioner Comparison (LLM: Gemini-2.5-Pro)
Official Baselines †
57.3
57.35
49.70
64.86
50.7
46.1
40.3
60.9
Whisper-large-v3
61.7
52.55
53.59
78.98
61.6
41.21
44.66
77.55
Omni-captioner
68.9
69.97
61.08
75.68
65.5
51.52
46.12
77.21
Gemini-caption
70.9
68.77
66.47
77.48
71.8
64.85
49.03
81.97
Table 3: Text-only cascade evaluation on MMAU and MMAR benchmark (%). † denotes official cascade baselines from each benchmark [ 51 , 52 ] . Shading indicates relative ranking among open-weight models; values in parentheses show the gap from Gemini-2.5-Pro.
Model
MMAU
MMAR
(Audio)
(Audio)
Published Systems
Gemini-2.5-Pro (Audio)
71.60
74.70
Audio Flamingo 3 [ 20 ]
73.30
58.60
Qwen2.5-Omni [ 11 ]
71.50
56.70
DeSTA2.5-Audio [ 14 ]
66.00
50.80
Table 4: Audio-grounded MMAU and MMAR performance (%). The upper block lists published LALM systems; the lower block reports our fine-tuned LALMs with DeSTA framework.
Figure 3: Category-level scatter plots comparing cascade and audio-grounded accuracy (%) for 8 fine-tuned LALMs, broken down by Sound, Music, and Speech domains.
Imperial College London, UK · Technische Universität München München, Germany · Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, AE +2