An audio language model is a black box in a specific way: we see what it says, never what it works out on the way there, and chain-of-thought monitoring helps only if the model writes its reasoning down. Reading a base Qwen3-Omni with a logit lens at the audio-token positions, we find that the answer to a spoken question becomes legible - in words - in the model's middle layers, before it emits any token. Five findings follow. (1) The readout carries concepts in neither the question, the options, nor the model's own transcription: on a clip whose verbatim transcription is empty garbling, it reconstructs Watergate and scandal, passes through the role president, and resolves to Nixon - a hidden multi-hop chain, read with no chain-of-thought. (2) The content is language-agnostic: one audio-inferred concept surfaces in several scripts at once, and 38% of top-1 readouts are Chinese on English inputs. (3) It is paralinguistic: given the same clip as audio and as the model's own emotion-free caption, the audio mind forms the sound source, speaker role, or affect that the caption discards, and answers correctly more often. (4) The audio-driven signal is absent at the input, turns on about a tenth of the way into the network, separates most cleanly from the text prior in the middle band (35-80% of depth), and activation patching shows it is causally used and committed before the last fifth of the layers. (5) Deleting single layers maps the pipeline: reading the sound in is localized to the entry layers and answer delivery to the output layer, while retrieval is distributed across the interior. Throughout, a waveform-swap control - identical text, only the sound changed - isolates the audio-driven signal from a prior over the printed options. This is a qualitative account of what an audio model works out before it speaks: the quantities are controls, not benchmark scores.
Figures & tables
Figure 1: Reading one clip’s mind, concept by concept (Nixon clip). One row per concept word; each is that word’s logit-lens rank over the full input ( x : audio positions, then question + options) against layer depth ( y ). Darker = better rank; the shaded band is the workspace. The event concepts ( water / gate , scandal ) are rank 1 from the early layers and the actor concepts ( president , Nixon ) strengthen deeper—an event-then-actor progression rather than a strict token-by-token order; the answer is reached over the audio, not only at the printed option.
Figure 2: Reading an audio LLM’s workspace before it speaks (Kennedy clip; Q: “which U.S. city?”; answer: Dallas; larger marker = better rank). (a) At an audio position the event concept assassination stabilizes at rank 1 across the workspace band alongside assassin and perpetrated . It appears in neither the printed question nor the answer, but the model’s own ASR partly transcribes it, so we treat this clip as illustrative rather than pure inference; the beyond-transcript readout here is the late victim name Kennedy ( Section 4.1 ). (b) At the generation position the answer Dallas commits only in the motor layers: concept in the middle band, answer at the output.
Answer
Top readout
Layer
Depth
Rank
# ws. hits
Dallas
assassination
L30
62%
1
56
Kidney
kidney
L17
35%
1
44
Lion
lions
L19
40%
1
37
Bird
niǎolèi (birds)
L33
69%
1
12
Music
música
L18
38%
1
11
Train
rail
L19
40%
1
27
Table 1: Seven curated thinking-path cases. For each we report the strongest (lowest-rank) workspace-band readout of a task-relevant concept over the audio span, the layer (of 48) where it is most stably rank 1, its depth, its rank among the 152,064-token vocabulary, and the number of workspace cells with the concept in the top-8. For Dallas we list assassination as the workspace-band scene concept, but it is partly transcribed by the model’s own ASR ( Section 4.1 ). Kidney is speech-content recall; the rest are sound-source identifications.
Figure 3: Watching the model free-associate. (top) Spontaneous neighbour words that light up at the audio positions—spoken nowhere in the clip, in no question or option; corpus-rare. (bottom) Timetable: for each family, the first readable layer of the evoking concept (blue) vs. each associate (orange); the associate always follows its evoker.
Figure 4: The workspace is multilingual. The same concept surfaces across languages at the audio positions (most at rank 1). The model holds a language-agnostic concept rendered into whichever token sits nearest, not a memorized English string.
Figure 5: Two minds, one clip. The roaring clip read two ways, with the concept lion pinned (darker = better rank). Top (audio): lion lights up across the workspace over the audio span; the model answers Lion . Bottom (the model’s own emotion-free caption): lion appears almost nowhere but the printed option, and the model answers Wolf .
Band
Real
Sil.
Mism.
R − S
R − M
Sensory
41.0
38.0
31.8
+3.0
+9.2
Workspace
40.0
21.8
32.2
+18.2
+7.8
Motor
48.9
34.7
26.3
+14.2
+22.6
Table 2: Balanced four-way readout accuracy (%) by functional band. R − S (real − silence) tests separation from the text prior—null in the sensory band ( p=0.78 ), largest and highly significant in the workspace band ( p=4.7×10−5 ) where silence alone collapses to chance, intermediate in motor ( p=1.5×10−3 ). R − M (real − mismatch, position-matched) is significant in the workspace ( p=0.015 ) but grows toward motor ( p<10−3 ), so raw magnitude keeps sharpening toward the output while separability from the prior peaks mid-stack. p : paired two-sided McNemar.
Figure 6: The audio-driven gap over the text prior, layer by layer. Balanced four-way readout accuracy with real audio minus the same quantity with silence, with a 95% bootstrap band; filled markers are depths where the paired test is significant. The gap is null at the input, becomes significant at ∼ 12% depth, and stays open, densest inside the workspace band.
Figure 7: Deleting one layer at a time. (a) Accuracy drop from bypassing each layer, over 40 clips. Only the entry layer L0 is critical ( −65 points); every interior layer costs at most 10. (b) Which ability is lost. Perception fails only for L0–L1, delivery only for L47, and retrieval for no single layer—listening and delivery are localized to the two ends, retrieval is distributed across the interior.
Figure 8: Where a text-mode hallucination is born. Depth at which the wrong concept first takes over the caption-fed run, on 73 clips the model answers correctly from the audio. Most wrong answers are early commitments, formed inside the sensory band and carried forward unchanged.
Audio Large Language Models (Audio LLMs) have advanced in audio understanding, yet they can still predict the answer by reasoning from textual cues or linguistic priors rather than the provided audio. A common remedy is to train models on data whose answers cannot be inferred from text alone. This approach can improve performance, but what changes within the model remains unclear. In this paper, we ask what must happen inside the model for the audio to actually determine the answer. Our findings are threefold. (1) Replacing the audio with silence or unrelated audio causes substantially larger performance degradation in the trained model than in the pretrained model. (2) Acoustic information most strongly shapes the model's representations of the answer choices in early-to-middle layers, while training mainly increases the influence of audio information on the final prediction in middle-to-late layers. (3) The weights learned during training have their largest impact in specific layer bands. Together, these results provide a mechanistic account of how training strengthens the use of acoustic evidence in Audio LLMs.
Hyebin Cho, Suho Yoo, Jihoo Jung +1
Korea Advanced Institute of Science and Technology, South Korea
Large audio-language models (LALMs) fuse an audio encoder into a large language model (LLM) through multi-stage training. This coupling means that a new domain or a stronger LLM requires retraining, and their answers cannot be traced to what the model heard: a chain-of-thought is a post-hoc account. We propose Listen-to-Reason (L2R), an interpretable-by-design pipeline that passes audio to the LLM through an explicit, human-readable tree: small heads on frozen expert encoders map each chunk of a clip to semantically meaningful nodes on the tree (for speech, music and environmental sound), and a frozen text-only LLM answers from these nodes and an ASR transcript without hearing the clip. Every answer can therefore be traced to the nodes and transcript it read, and the nodes are causal: replacing the deciding node with a distractor overturns 78% of correct answers on SAKURA. With a 7B reader, L2R outperforms all LALMs we compare against on SAKURA and trails them by 6-12 points on MMAU and MMAR, despite training about 1,400x fewer parameters on orders of magnitude less audio data. However, because any LLM can serve as the reader, we show that a stronger reader narrows this gap without retraining any audio component. A new domain is added with one small head: with five labelled clips per species, it outperforms QLoRA fine-tuning of an LALM on the same clips by 13-26 points.
Pooneh Mousavi, Mirco Ravanelli, Cem Subakan
Concordia University · Mila – Quebec AI Institute · Laval University
Multimodal large language models can exhibit text dominance, over-relying on linguistic priors instead of grounding predictions in non-text inputs. One example is large audio-language models (LALMs) where decisive audio evidence can be under-utilized even when it contains important information. To address this issue we use mechanistic interpretability to identify a small set of audio-specialist attention heads whose audio attention yields a ``listening'' signal. We show that this signal increases when audio evidence affects the model's output, providing an indicator of audio engagement under standard prompting. Leveraging this localization, we construct an audio--silence steering direction and apply an inference-time activation intervention to the final representation, amplifying the model's audio effect. To demonstrate the utility of this intervention, we show on MMAU that this improves accuracy by up to +8.0 percentage points on two Qwen-based LALMs, without any parameter updates.
Neta Glazer, Lenny Aharon, Ethan Fetaya
Bar-Ilan University, Ramat Gan, Israel · Columbia University, New York, NY, USA