Multimodal large language models can exhibit text dominance, over-relying on linguistic priors instead of grounding predictions in non-text inputs. One example is large audio-language models (LALMs) where decisive audio evidence can be under-utilized even when it contains important information. To address this issue we use mechanistic interpretability to identify a small set of audio-specialist attention heads whose audio attention yields a ``listening'' signal. We show that this signal increases when audio evidence affects the model's output, providing an indicator of audio engagement under standard prompting. Leveraging this localization, we construct an audio--silence steering direction and apply an inference-time activation intervention to the final representation, amplifying the model's audio effect. To demonstrate the utility of this intervention, we show on MMAU that this improves accuracy by up to +8.0 percentage points on two Qwen-based LALMs, without any parameter updates.
Figures & tables
Figure 1: Specialist-Guided Steering (SGS). (a) We identify audio-specialist attention heads by computing each head’s audio-attention share and selecting the Top- K heads whose audio attention is most predictive of correctness on a calibration set. (b) We run audio and matched-duration silence forward passes and form a layer-localized steering direction by aggregating residual differences (hℓaud−hℓsil) over the specialist layer set L (layers containing the discovered heads). We scale this direction by β and add it to the final representation to obtain h∗ for prediction.
Figure 2: Effect of steering strength β and specialist count K on performance for R1-AQA (top) and Qwen2-Audio-7B (bottom). Lines show improvement in percentage points (pp) for different Top- K specialist head sets; each K induces a specialist layer set L and we apply layer-localized steering within L .
Table 2: Accuracy (%) by domain on MMAU test-mini; Overall reports gain in percentage points (pp).
Qwen2-Audio
R1-AQA
K
Random
Specialists
Random
Specialists
10
+3.1%
+7.2%
+1.0%
+4.7%
20
+2.2%
+8.0%
+2.0%
+4.9%
30
+3.1%
+7.4%
+1.3%
+4.6%
Table 3: Layer-guided steering with specialist-selected heads vs. matched random-head sets: improvement over baseline (pp). Random heads induce a layer set Lrand using the same procedure as specialists.
Qwen2-Audio
R1-AQA
K
Added layers
∣L∣
Added layers
∣L∣
10
19, 21, 22, 23
4
16, 17, 18, 19, 22
5
20
14, 18, 20, 24, 26
9
20, 24, 27, 28, 30
10
30
25, 27
11
26
11
Table 4: Induced specialist layer set L as a function of K (see Section 4.2 ). “Added layers” lists layers newly included in L when increasing K ; ∣L∣ is the resulting set size.