Large Audio-Language Models (LALMs) have shown strong performance in speech understanding, making speech a natural interface for accessing factual information. Yet they are trained on static corpora and may encode incorrect facts. Existing model editing methods localize and update facts in text-only LLMs, but do not account for continuous speech representations, or where knowledge is stored across acoustic or language modules, or their cross-modal module. We construct the first audio benchmark for knowledge localization and editing in LALMs and propose a speech-driven locate-then-edit framework. First, we use speech-aware causal tracing to localize layers and modules that support factual retrieval and then apply editing at identified sites. Experiments show that factual knowledge is jointly encoded in audio and text modules, and that audio editing yields more effective updates than text editing or fine-tuning, enabling fine-grained knowledge control in speech AI systems.
Figures & tables
Figure 1: Causal tracing consists of three runs: i) a clean run for the correct object; ii) a corrupted run, where the subject span are corrupted to obtain a low-probability object prediction; and iii) a corrupted-with-restoration run, which restores the corrupted hidden representations at selected layers to their clean values, evaluating the contribution of those layers via changes in object probability.
Figure 2: Layer-wise Average Indirect Effect (AIE) of hidden states (left), MLP modules (middle), and attention modules (right) for the CounterFact (top) and Known-1000 (bottom) datasets. The dashed vertical line separates the audio encoder layers (left) from the LLM backbone layers (right).
Mod.
Method
Score ↑
ES ↑
PS ↑
NS ↑
Baseline
29.09
20.4
24.4
76.18
Audio
FT
61.50
97.2
45.9
59.86
Single-layer
71.36
78.2
66.5
70.34
Multi-layer
63.52
80.7
67.9
49.72
Text
FT
68.21
97.6
70.0
51.42
Single-layer
64.10
79.7
56.3
60.64
Table 1: Editing performance across modalities and strategies.
Mod.
Word
Top-1
Top-2
Top-3
Audio
subj. first (29,28,27)
71.70
73.34
73.34
subj. last (29,28,27)
71.05
70.76
68.49
Text
subj. first (2,1,7)
51.77
32.33
51.54
subj. last (12,10,8)
56.47
57.63
63.88
Table 2: Editing scores S for Top-3 layers by modality and word position. Layer numbers are ranked in parentheses.
Large language models (LLMs) have been widely used as knowledge backbones of Large Audio Language Models (LALMs), yet how much auditory knowledge they encode through text-only pre-training and how this affects downstream performance remains unclear. We study this gap by comparing different LLMs under two text-only and one audio-grounded setting: (1) direct probing on AKB-2000, a curated benchmark testing the breadth and depth of auditory knowledge; (2) cascade evaluation, where LLMs reason over text descriptions from an audio captioner; and (3) audio-grounded evaluation, where each LLM is fine-tuned into a Large Audio Language Model (LALM) with an audio encoder. Our findings reveal that auditory knowledge varies substantially across families, and text-only results are strongly correlated with audio performance. Our work provides empirical grounding for a comprehensive understanding of LLMs in audio research.
Ke-Han Lu, Szu-Wei Fu, Chao-Han Huck Yang +13
National Taiwan University, Taiwan · NVIDIA · Academia Sinica, Taiwan
Large audio-language models (LALMs) fuse an audio encoder into a large language model (LLM) through multi-stage training. This coupling means that a new domain or a stronger LLM requires retraining, and their answers cannot be traced to what the model heard: a chain-of-thought is a post-hoc account. We propose Listen-to-Reason (L2R), an interpretable-by-design pipeline that passes audio to the LLM through an explicit, human-readable tree: small heads on frozen expert encoders map each chunk of a clip to semantically meaningful nodes on the tree (for speech, music and environmental sound), and a frozen text-only LLM answers from these nodes and an ASR transcript without hearing the clip. Every answer can therefore be traced to the nodes and transcript it read, and the nodes are causal: replacing the deciding node with a distractor overturns 78% of correct answers on SAKURA. With a 7B reader, L2R outperforms all LALMs we compare against on SAKURA and trails them by 6-12 points on MMAU and MMAR, despite training about 1,400x fewer parameters on orders of magnitude less audio data. However, because any LLM can serve as the reader, we show that a stronger reader narrows this gap without retraining any audio component. A new domain is added with one small head: with five labelled clips per species, it outperforms QLoRA fine-tuning of an LALM on the same clips by 13-26 points.
Pooneh Mousavi, Mirco Ravanelli, Cem Subakan
Concordia University · Mila – Quebec AI Institute · Laval University
Multimodal large language models can exhibit text dominance, over-relying on linguistic priors instead of grounding predictions in non-text inputs. One example is large audio-language models (LALMs) where decisive audio evidence can be under-utilized even when it contains important information. To address this issue we use mechanistic interpretability to identify a small set of audio-specialist attention heads whose audio attention yields a ``listening'' signal. We show that this signal increases when audio evidence affects the model's output, providing an indicator of audio engagement under standard prompting. Leveraging this localization, we construct an audio--silence steering direction and apply an inference-time activation intervention to the final representation, amplifying the model's audio effect. To demonstrate the utility of this intervention, we show on MMAU that this improves accuracy by up to +8.0 percentage points on two Qwen-based LALMs, without any parameter updates.
Neta Glazer, Lenny Aharon, Ethan Fetaya
Bar-Ilan University, Ramat Gan, Israel · Columbia University, New York, NY, USA