Audio Large Language Models (Audio LLMs) have advanced in audio understanding, yet they can still predict the answer by reasoning from textual cues or linguistic priors rather than the provided audio. A common remedy is to train models on data whose answers cannot be inferred from text alone. This approach can improve performance, but what changes within the model remains unclear. In this paper, we ask what must happen inside the model for the audio to actually determine the answer. Our findings are threefold. (1) Replacing the audio with silence or unrelated audio causes substantially larger performance degradation in the trained model than in the pretrained model. (2) Acoustic information most strongly shapes the model's representations of the answer choices in early-to-middle layers, while training mainly increases the influence of audio information on the final prediction in middle-to-late layers. (3) The weights learned during training have their largest impact in specific layer bands. Together, these results provide a mechanistic account of how training strengthens the use of acoustic evidence in Audio LLMs.
Figures & tables
Figure 1: How audio information is used across layers in Audio LLMs. Connections from audio to the answer choices are most important in early-to-middle layers both before and after training, while training increases the contribution of the answer choices to the final prediction in middle-to-late layers.
Dataset
ZS
FT
ΔA
ΔSaudio
Qwen2-Audio
ADQA
36.9
51.1
+14.3±2.0
+9.3±2.0
MMAR
41.2
52.5
+11.3±3.9
+6.9±3.6
MMAU
59.7
58.6
−1.1±3.6
−9.1±3.5
MMSU
47.3
58.8
+11.5±1.6
+6.8±1.5
Qwen2.5-Omni
Table 1: Accuracy and acoustic sensitivity. Accuracies are in percentages. ΔA and ΔSaudio denote changes in accuracy and behavioral sensitivity to valid acoustic evidence, respectively, in percentage points. ± values indicate 95% paired-bootstrap CI half-widths from 10,000 resamples.
Figure 2: Layer-wise attention knockout. Audio ↛ All Options is most influential in early-to-middle layers for both ZS and FT , whereas All Options ↛ Answer becomes prominent in middle-to-late layers after training.
Figure 3: Correct- and wrong-option contributions. Qwen2-Audio shows opposite effects for correct and wrong options in both ZS and FT . Blocking the correct option lowers the correct-answer probability, while blocking wrong options raises it. Qwen2.5-Omni shows a strong increase in correct-option contribution after training, with little change from wrong options.
Figure 4: Functional effect of LoRA layer bands. Accuracy change after disabling the learned LoRA updates in each layer band, relative to the full trained model. Negative values indicate performance degradation after removal, whereas positive values indicate improved accuracy.
Qwen2-Audio
Data
All
0–7
8–15
16–23
24–31
All r=4
ADQA
51.13
49.73
51.13
38.80
41.60
39.70
MMAR
52.51
49.50
52.81
42.11
44.92
43.82
MMAU
58.60
56.60
55.60
55.90
57.50
57.90
MMSU
58.78
56.68
59.72
51.34
52.56
51.58
Mean
55.26
53.13
54.82
47.04
49.15
48.25
Table 2: Layer-restricted LoRA training. Accuracy (%) when LoRA updates are applied to all layers or restricted to a single layer band during training. Unless otherwise specified, LoRA rank is r=16 . Bold and underline indicate the best and second-best results in each row, respectively.
Large language models (LLMs) have been widely used as knowledge backbones of Large Audio Language Models (LALMs), yet how much auditory knowledge they encode through text-only pre-training and how this affects downstream performance remains unclear. We study this gap by comparing different LLMs under two text-only and one audio-grounded setting: (1) direct probing on AKB-2000, a curated benchmark testing the breadth and depth of auditory knowledge; (2) cascade evaluation, where LLMs reason over text descriptions from an audio captioner; and (3) audio-grounded evaluation, where each LLM is fine-tuned into a Large Audio Language Model (LALM) with an audio encoder. Our findings reveal that auditory knowledge varies substantially across families, and text-only results are strongly correlated with audio performance. Our work provides empirical grounding for a comprehensive understanding of LLMs in audio research.
Ke-Han Lu, Szu-Wei Fu, Chao-Han Huck Yang +13
National Taiwan University, Taiwan · NVIDIA · Academia Sinica, Taiwan
Large Audio Language Models (LALMs) have demonstrated strong performance across a wide range of audio understanding tasks. However, they are typically evaluated on single, coherent audio segments, leaving their behavior under less familiar input configurations underexplored. We study this issue through a controlled setting in which two audio segments are concatenated into a single input. Across multiple LALMs, we observe a striking task-dependent robustness gap: automatic speech recognition (ASR) remains comparatively stable, whereas audio question answering (AQA) degrades substantially. To investigate the mechanisms underlying this disparity, we analyze how audio information is routed through LALM decoders using layer-wise attention knockout. The results reveal distinct task-dependent pathways. ASR relies primarily on direct retrieval from audio tokens by answer tokens, whereas AQA depends more strongly on a mediated route in which audio information is first integrated into prompt tokens and subsequently accessed during generation. We further probe prompt-token representations under audio concatenation and find that task-relevant audio attributes remain readily decodable, particularly in middle and later decoder layers, even when AQA performance deteriorates sharply. This dissociation indicates that the failure cannot be explained by complete loss of audio information from the decoder states and is instead consistent with a downstream bottleneck in retrieving or utilizing prompt-mediated information during answer generation. Together, our findings reveal task-dependent audio information routing in LALMs and highlight information utilization as a potential limitation on their generalization.
Yizhou Zhang, Wangjin Zhou, Xin Gu +6
Graduate School of Informatics, Kyoto University, Japan · WXG, Tencent, China
Large audio-language models (LALMs) fuse an audio encoder into a large language model (LLM) through multi-stage training. This coupling means that a new domain or a stronger LLM requires retraining, and their answers cannot be traced to what the model heard: a chain-of-thought is a post-hoc account. We propose Listen-to-Reason (L2R), an interpretable-by-design pipeline that passes audio to the LLM through an explicit, human-readable tree: small heads on frozen expert encoders map each chunk of a clip to semantically meaningful nodes on the tree (for speech, music and environmental sound), and a frozen text-only LLM answers from these nodes and an ASR transcript without hearing the clip. Every answer can therefore be traced to the nodes and transcript it read, and the nodes are causal: replacing the deciding node with a distractor overturns 78% of correct answers on SAKURA. With a 7B reader, L2R outperforms all LALMs we compare against on SAKURA and trails them by 6-12 points on MMAU and MMAR, despite training about 1,400x fewer parameters on orders of magnitude less audio data. However, because any LLM can serve as the reader, we show that a stronger reader narrows this gap without retraining any audio component. A new domain is added with one small head: with five labelled clips per species, it outperforms QLoRA fine-tuning of an LALM on the same clips by 13-26 points.
Pooneh Mousavi, Mirco Ravanelli, Cem Subakan
Concordia University · Mila – Quebec AI Institute · Laval University