cs.LGSep 12, 2026

Inherited Heads: Audio language models track speakers with their text backbone's attention, and an attention-mass ranking retrieves a different set

Authors: Bojro Das

Organizations: Cornell University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Asked to describe what one of six speakers in a recording talks about, audio language models describe the right one on 6 to 16% of trials, below the 16.7% a guess would give. Adding a fixed bias to the attention logits of a hundred heads, under a tenth of the model's and with no training, redirects the description to whichever speaker we choose, on 90.7% to 99.0% of trials. Those heads are largely not specific to audio. Rank the text-only language model an audio model was built from, or a released model of the same family, on a written version of the task, take its top hundred heads, and carry them over unchanged: they redirect the audio model on 80.8% to 95.0% of trials, with nothing about audio entering the selection. The audio and text head sets share 66 to 74 of 100 where chance would give about 20, and the shared part alone reproduces almost all of the steering. What that does not show is that sharing is what makes the heads work: an equal-sized draw from the same discovered hundred does nearly as well, and none of our three models separates the two explanations. A second finding concerns how such heads are found. Ranking heads by how much attention they place on the segment asked about, as an established score does, or by how much of their attention moves with the question, as a per-head normalised variant does, gives top hundreds that share 69, 37 and 4 heads across our three models. In Ultravox, where they share 4, the established score's heads leave output the judge cannot place on any segment on 69.7% of trials, against 40.0% with no intervention and 1.0% for the normalised variant. That is one arm of six; on the other five the established score steers above a random draw.

Figures & tables

Figure 1: The task and the edit. Six speakers read in turn in one 33.5-second strip; the model is asked to describe what one of them talks about. The intervention adds a constant +B+B to the pre-softmax attention logits of the selected heads at every key inside the target segment’s token span and −B-B inside the other five, in prefill and during decoding. It is positional and content-independent: no learned direction, no second prompt.
Figure 2: Head-set overlap between each multimodal model and its text-only backbone (for Qwen2-Audio, a released relative with the same layout), against a null that resamples 100 heads with the same layer histogram. The null, not the naive k2/Nk^{2}/N baseline, is the right comparison because these heads concentrate in the middle layers. Split-half ceilings are drawn where a reliability run exists; where one does not, no bound is drawn rather than a stand-in.
Figure 3: Each subset steering alone. The third bar is the one that constrains the reading: a random draw of the shared subset’s size, taken from the same discovered hundred, reaches within 0.039 of the shared subset on two models and exceeds it on SALMONN. Any two-thirds of this head set carries most of the behaviour, so sufficiency is a claim about the subset rather than about sharedness. Diamonds are the five draws of a layer-matched random control for the shared subset, taken from outside the discovered hundred; the shared subset exceeds every one of them on every arm.
  shared with the backbonenot shared
modelall 100these headsrandom, same count, from the 100these headsrandom, same count, from the 100
Qwen2-Audio0.9830.975 (66)0.9360.275 (34)0.644
Ultravox0.9900.950 (71)0.9170.383 (29)0.444
SALMONN0.9070.775 (74)0.8640.592 (26)0.511
Table 1: Each part of the head set steering on its own, against a random draw of the same size from the same discovered hundred. Head counts in parentheses; same held-out strips throughout; chance is 1/61/6 ; 20 strips ×\times 6 targets == 120 trials per cell. Each random figure is the mean of three random draws of the same size taken from inside the discovered hundred, the exception to the usual pool noted in Section 3 . Per-draw values are in Appendix A .
Figure 4: Changing cross-modal token geometry. Patches are flattened row by row, so tiling the panels side by side scatters each one across the sequence in 24 runs of 7 tokens, while stacking them makes each panel a single contiguous block. Heads discovered on one layout steer the other. The two discovered sets overlap on 80 of 100 heads, which is why we read this as suggestive rather than decisive.
Figure 5: Steering accuracy against bias strength, and where the target’s gained attention comes from. The effect saturates by B=5B{=}5 , so the large value used for comparability with published work buys no accuracy, and at B=5B{=}5 most of the gained attention already comes from the prompt and the attention sink rather than from the other segments.
Figure 6: The two head sets read two ways. (a) Steering accuracy across six arms, beside one band-matched random draw per arm. Each draw spans the selectivity-ranked hundred’s layers; on Ultravox only 32 of the mass-ranked hundred’s heads lie in that span, so there the draw is not a control for the mass ranking. (b) Ultravox read for unattributability on the same items: the judge cannot place the output on any segment on 69.7% of trials under the mass-ranked heads, against 40.0% with no intervention and 1.0% under the selectivity-ranked heads. The two head sets sit at different depths, so the difference is not yet attributable to the score alone.
  audiomass ranking’s
  attentionscore
modelcutmedianbelowmedianbelow
Qwen2-Audio90.287.75294.431
SALMONN93.891.66491.863
Ultravox90.262.39350.596
Table 2: Where a top-100 cut falls in each model’s own distribution, as a percentile, against the median of the selectivity-ranked hundred and the number of those hundred heads below the cut. The cut falls at a different percentile in the 1024-head models than in 1600-head SALMONN, so it is given per model.
 selectivity rankingmass ranking
modelnamescannotnamescannot
 targetplacetargetplace
Qwen2-Audio0.9830.0130.6470.143
SALMONN0.9070.0800.8070.153
Ultravox0.9900.0100.0170.697
Table 3: Proportion of trials on which the model names the target segment, and on which the judge can place the description on no segment at all. The two need not sum to 1: the remainder names a segment, and the wrong one.
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

modelheadsdraw 1draw 2draw 3subset% of each draw shared
size of the shared part: narrow contrast
Qwen2-Audio660.9580.9330.9170.97568, 73, 65
Ultravox710.8420.9330.9750.95069, 70, 72
SALMONN740.8420.7920.9580.77572, 70, 72
size of the unshared part: wide contrast
Qwen2-Audio340.3580.6830.8920.27562, 65, 68
Appendix
Table 4: The three draws behind each random figure in Table 1 , beside the fixed subset they control: the shared part in the upper block and the unshared part in the lower. The means quoted in Table 1 are 0.936, 0.917 and 0.864 above and 0.644, 0.444 and 0.511 below.
modelheadsdraw 1draw 2draw 3draw 4draw 5subset
controls for the shared part
Qwen2-Audio660.0750.1830.1000.2330.1670.975
Ultravox710.1500.2170.4080.2000.2080.950
SALMONN740.1920.2420.0670.0670.4500.775
controls for the unshared part
Qwen2-Audio340.1420.1580.1330.1580.1500.275
Appendix
Table 5: Five draws per control, beside the fixed subset each one controls. The shared part exceeds all five of its draws on every model. The unshared part does so on Qwen2-Audio and SALMONN and not on Ultravox, where it is above three draws, level with a fourth and below the fifth. That Ultravox row, spanning 0.642, is the widest of these controls and the first example in Section 6 ’s discussion of control spread.
 accuracyunattributablemass ranking
modeldraw 1draw 2draw 1draw 2accuracyunattributable
Qwen2-Audio0.0600.1530.1270.1070.6470.143
SALMONN0.1700.1870.4970.4130.8070.153
Ultravox0.4530.5270.3930.2770.0170.697
Appendix
Table 6: Both band-matched draws at K=100K{=}100 on the three audio models, beside the mass ranking on the same items.
head setaccuracyunattributable
draw 10.3930.223
draw 20.2830.240
draw 30.6630.120
draw 40.2000.143
draw 50.1470.257
draw 60.1770.450
Appendix
Table 7: Ten random hundreds on Ultravox at the selectivity-ranked hundred’s layers, with that hundred on the same items for reference.
More in this topic

Audio-Language Models

Audio-language models connect audio and natural language for understanding, retrieval, dialogue, speech generation, and music or sound analysis.

See all 303 papers

Explore similar work

CardsList
Mar 6, 2026

Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering

Multimodal large language models can exhibit text dominance, over-relying on linguistic priors instead of grounding predictions in non-text inputs. One example is large audio-language models (LALMs) where decisive audio evidence can be under-utilized even when it contains important information. To address this issue we use mechanistic interpretability to identify a small set of audio-specialist attention heads whose audio attention yields a ``listening'' signal. We show that this signal increases when audio evidence affects the model's output, providing an indicator of audio engagement under standard prompting. Leveraging this localization, we construct an audio--silence steering direction and apply an inference-time activation intervention to the final representation, amplifying the model's audio effect. To demonstrate the utility of this intervention, we show on MMAU that this improves accuracy by up to +8.0 percentage points on two Qwen-based LALMs, without any parameter updates.

Neta Glazer, Lenny Aharon, Ethan Fetaya

Bar-Ilan University, Ramat Gan, Israel · Columbia University, New York, NY, USA

Audio-Language ModelsAudio-Language Model EvaluationCausal Interventions in Language Models
AskPDF
May 11, 2026

Afrispeech Semantics: Evaluating Audio Semantic Reasoning in Spoken Language Models Across Domains and Accents

Audio language models (ALMs) are increasingly used for speech-based understanding, yet their ability to perform semantic reasoning beyond transcription, Text-to-Audio Retrieval, Captioning, and Question-Answering accuracy remains insufficiently benchmarked. In particular, the effects of accent variation, domain shift, and semantic over-inference on audio reasoning are poorly understood. We evaluate audio language models across five semantic and paralinguistic reasoning tasks: entailment, consistency, plausibility, accent drift, and accent restraint. Collectively, these tasks assess a model's ability to reason over spoken audio as the primary evidence source, including whether a textual hypothesis can be inferred, contradicted, or left undetermined by the audio, whether statements align or conflict with spoken content, whether claims are plausible given the discourse, and whether model predictions remain stable or appropriately constrained across accent variation. These findings highlight critical limitations in current audio reasoning evaluations and hope to provide guidance for more robust and equitable ALM design and assessment

Chibuzor Okocha, Christan Grant

University of Florida

Audio-Language Model EvaluationSpoken Language UnderstandingAudio ReasoningAudio UnderstandingAudio-Language ModelsLanguage Model Robustness
AskPDF
Oct 20, 2026

Uncovering Audio-Text Alignment in Large Audio-Language Models through Cross-Modal Concept Neurons

While the language model backbone is pretrained on text alone, large audio-language models (LALMs) can, to a great extent, recognize acoustic attributes, such as emotion, speaker gender, and spoken language. However, it is unclear how the audio representation is integrated, if at all, with the other text representation before the model produces text. We hypothesize that there is a shared subspace where the audio and text representations are aligned. To test this hypothesis, we identify cross-modal neurons that are responsible for representing both audio and text. We find alignment between audio and text neurons to various degrees across three LALMs and four acoustic attribute types. Causal interventions, steering and deactivating the neurons, confirm that the cross-modal neurons are not just necessary for prediction but can also be manipulated.

Tzu-Quan Lin, Chih-Kai Yang, Yi-Cheng Lin +2

Centre for Speech Technology Research, University of Edinburgh, United Kingdom · Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE), Taiwan

Audio-Text AlignmentAudio-Language ModelsCross-Modal Representation LearningMechanistic Interpretability
AskPDF