cs.SDAug 25, 2026

Can We Read the Mind of an Audio LLM? A Verbalizable, Multilingual Middle-Layer Workspace

Authors: Jiajun Fan, Jingyuan Li, Prashanth Gurunath Shivakumar, Qi Luo, Jia-Hong Huang, M. Maruf, Roger Ren, Yile Gu, 

Organizations: Amazon AGI Foundations · University of Illinois Urbana-Champaign

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

An audio language model is a black box in a specific way: we see what it says, never what it works out on the way there, and chain-of-thought monitoring helps only if the model writes its reasoning down. Reading a base Qwen3-Omni with a logit lens at the audio-token positions, we find that the answer to a spoken question becomes legible - in words - in the model's middle layers, before it emits any token. Five findings follow. (1) The readout carries concepts in neither the question, the options, nor the model's own transcription: on a clip whose verbatim transcription is empty garbling, it reconstructs Watergate and scandal, passes through the role president, and resolves to Nixon - a hidden multi-hop chain, read with no chain-of-thought. (2) The content is language-agnostic: one audio-inferred concept surfaces in several scripts at once, and 38% of top-1 readouts are Chinese on English inputs. (3) It is paralinguistic: given the same clip as audio and as the model's own emotion-free caption, the audio mind forms the sound source, speaker role, or affect that the caption discards, and answers correctly more often. (4) The audio-driven signal is absent at the input, turns on about a tenth of the way into the network, separates most cleanly from the text prior in the middle band (35-80% of depth), and activation patching shows it is causally used and committed before the last fifth of the layers. (5) Deleting single layers maps the pipeline: reading the sound in is localized to the entry layers and answer delivery to the output layer, while retrieval is distributed across the interior. Throughout, a waveform-swap control - identical text, only the sound changed - isolates the audio-driven signal from a prior over the printed options. This is a qualitative account of what an audio model works out before it speaks: the quantities are controls, not benchmark scores.

Figures & tables

Figure 1: Reading one clip’s mind, concept by concept (Nixon clip). One row per concept word; each is that word’s logit-lens rank over the full input ( xx : audio positions, then question +{+} options) against layer depth ( yy ). Darker == better rank; the shaded band is the workspace. The event concepts ( water / gate , scandal ) are rank 1 from the early layers and the actor concepts ( president , Nixon ) strengthen deeper—an event-then-actor progression rather than a strict token-by-token order; the answer is reached over the audio, not only at the printed option.
Figure 2: Reading an audio LLM’s workspace before it speaks (Kennedy clip; Q: “which U.S. city?”; answer: Dallas; larger marker == better rank). (a) At an audio position the event concept assassination stabilizes at rank 1 across the workspace band alongside assassin and perpetrated . It appears in neither the printed question nor the answer, but the model’s own ASR partly transcribes it, so we treat this clip as illustrative rather than pure inference; the beyond-transcript readout here is the late victim name Kennedy ( Section 4.1 ). (b) At the generation position the answer Dallas commits only in the motor layers: concept in the middle band, answer at the output.
AnswerTop readoutLayerDepthRank# ws. hits
DallasassassinationL3062%156
KidneykidneyL1735%144
LionlionsL1940%137
Birdniǎolèi (birds)L3369%112
MusicmúsicaL1838%111
TrainrailL1940%127
Table 1: Seven curated thinking-path cases. For each we report the strongest (lowest-rank) workspace-band readout of a task-relevant concept over the audio span, the layer (of 48) where it is most stably rank 1, its depth, its rank among the 152,064-token vocabulary, and the number of workspace cells with the concept in the top-8. For Dallas we list assassination as the workspace-band scene concept, but it is partly transcribed by the model’s own ASR ( Section 4.1 ). Kidney is speech-content recall; the rest are sound-source identifications.
Figure 3: Watching the model free-associate. (top) Spontaneous neighbour words that light up at the audio positions—spoken nowhere in the clip, in no question or option; corpus-rare. (bottom) Timetable: for each family, the first readable layer of the evoking concept (blue) vs. each associate (orange); the associate always follows its evoker.
Figure 4: The workspace is multilingual. The same concept surfaces across languages at the audio positions (most at rank 1). The model holds a language-agnostic concept rendered into whichever token sits nearest, not a memorized English string.
Figure 5: Two minds, one clip. The roaring clip read two ways, with the concept lion pinned (darker == better rank). Top (audio): lion lights up across the workspace over the audio span; the model answers Lion . Bottom (the model’s own emotion-free caption): lion appears almost nowhere but the printed option, and the model answers Wolf .
BandRealSil.Mism.R −- SR −- M
Sensory41.038.031.8+3.0+3.0+9.2+9.2
Workspace40.021.832.2+18.2+18.2+7.8+7.8
Motor48.934.726.3+14.2+14.2+22.6+22.6
Table 2: Balanced four-way readout accuracy (%) by functional band. R −- S (real −- silence) tests separation from the text prior—null in the sensory band ( p=0.78p{=}0.78 ), largest and highly significant in the workspace band ( p=4.7×10−5p{=}4.7\times 10^{-5} ) where silence alone collapses to chance, intermediate in motor ( p=1.5×10−3p{=}1.5\times 10^{-3} ). R −- M (real −- mismatch, position-matched) is significant in the workspace ( p=0.015p{=}0.015 ) but grows toward motor ( p<10−3p{<}10^{-3} ), so raw magnitude keeps sharpening toward the output while separability from the prior peaks mid-stack. pp : paired two-sided McNemar.
Figure 6: The audio-driven gap over the text prior, layer by layer. Balanced four-way readout accuracy with real audio minus the same quantity with silence, with a 95% bootstrap band; filled markers are depths where the paired test is significant. The gap is null at the input, becomes significant at ∼\sim 12% depth, and stays open, densest inside the workspace band.
Figure 7: Deleting one layer at a time. (a) Accuracy drop from bypassing each layer, over 40 clips. Only the entry layer L0 is critical ( −65-65 points); every interior layer costs at most 10. (b) Which ability is lost. Perception fails only for L0–L1, delivery only for L47, and retrieval for no single layer—listening and delivery are localized to the two ends, retrieval is distributed across the interior.
Figure 8: Where a text-mode hallucination is born. Depth at which the wrong concept first takes over the caption-fed run, on 73 clips the model answers correctly from the audio. Most wrong answers are early commitments, formed inside the sensory band and carried forward unchanged.
More in this topic

Audio-Language Models

Audio-language models connect audio and natural language for understanding, retrieval, dialogue, speech generation, and music or sound analysis.

See all 303 papers

Explore similar work

CardsList

  1. Tracing Audio Grounding and Answer Selection in Audio LLMs

    Sep 4, 2026Hyebin Cho, Suho Yoo, Jihoo Jung +1Audio-Language ModelsAudio QA

    Korea Advanced Institute of Science and Technology, South Korea

  2. Listen-to-Reason: Listen with Experts, Retrieve over a Graph, Reason with LLMs

    Oct 7, 2026Pooneh Mousavi, Mirco Ravanelli, Cem SubakanAudio-Language ModelsAudio Reasoning

    Concordia University · Mila – Quebec AI Institute · Laval University

  3. Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering

    Mar 6, 2026Neta Glazer, Lenny Aharon, Ethan FetayaAudio-Language ModelsAudio-Language Model Evaluation

    Bar-Ilan University, Ramat Gan, Israel · Columbia University, New York, NY, USA