Organizations: Centre for Speech Technology Research, University of Edinburgh, United Kingdom · Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE), Taiwan
While the language model backbone is pretrained on text alone, large audio-language models (LALMs) can, to a great extent, recognize acoustic attributes, such as emotion, speaker gender, and spoken language. However, it is unclear how the audio representation is integrated, if at all, with the other text representation before the model produces text. We hypothesize that there is a shared subspace where the audio and text representations are aligned. To test this hypothesis, we identify cross-modal neurons that are responsible for representing both audio and text. We find alignment between audio and text neurons to various degrees across three LALMs and four acoustic attribute types. Causal interventions, steering and deactivating the neurons, confirm that the cross-modal neurons are not just necessary for prediction but can also be manipulated.
Figures & tables
Figure 1: Why we compare neurons rather than entire vectors in the ambient space (schematic). Audio and text are fed to the LALM in separate forward passes. Cosine values follow Qwen2.5-Omni on emotion (§ 4.1 ); neuron counts are illustrative (§ 2 , § 4.2 ).
Model
Task
Cosine
Neuron
Qwen2.5-Omni
animal
8.74%
70.08%
emotion
0.88%
25.04%
gender
0.16%
76.04%
language
0.23%
55.08%
DeSTA2.5-Audio
animal
0.07%
9.22%
emotion
1.58%
52.03%
Table 1: Matched-label variance share (§ 4.1 ): how much of the variation in each audio-by-text label matrix comes from whether the labels match, rather than from labels similar to everything or from noise. Cosine: cosine similarity of pooled activations; Neuron: Jaccard overlap of cross-modal concept neuron sets (§ 4.2 ).
Model
Anim.
Emo.
Gen.
Lang.
Qwen2.5-Omni
0.245 ∗∗
0.174 ∗
0.144 †
0.177 ∗∗
DeSTA2.5-Audio
0.094 ∗∗
0.121 ∗∗
0.127 †
0.125 ∗∗
Audio Flamingo 3
0.264 ∗∗
0.202 ∗∗
0.098 †
0.216 ∗∗
Table 2: Cross-modal concept neurons (§ 4.2 ): mean per-label Jaccard overlap between the audio and text top- k sets on the held-out test split. Stars: distinctiveness, exact label-permutation test of the matched-vs-mismatched gap (Appendix A.3 ); ∗p<0.05 , ∗∗p<0.01 ; † not testable (gender has only two labels, so its smallest attainable p is 0.5 ). Exact values: Table 6 .
Figure 2: Overview of the causal tests (schematic; activation values are illustrative). Steering (§ 4.3 ): the direction is computed once from text-only forward passes as the difference between the target and carrier labels’ activations, and is added to the target label’s cross-modal concept neurons while the model processes a carrier input (Eq. 5 ); the carrier can also be silence or noise. Deactivation (§ 4.4 ): deactivating a task’s cross-modal concept neurons (Eq. 6 ) impairs that task before the other tasks, whereas deactivating a random set of the same size does far less damage.
Figure 3: Contrastive neuron steering toward the emotion label “happy” in Qwen2.5-Omni, one panel per carrier label. x -axis: steering coefficient λ ; y -axis: fraction of samples predicted as “happy”. Blue: cross-modal concept neurons; grey: median of five matched-size random neuron sets, with their min–max range shaded (the null).
Model
Anim.
Emo.
Gen.
Lang.
Total
Qwen2.5-Omni
4/9
5/5
2/2
7/8
18/24
DeSTA2.5-Audio
2/9
4/5
0/2
1/8
7/24
Audio Flamingo 3
6/9
4/5
1/2
6/8
17/24
Total
12/27
13/15
3/6
14/24
42/72
Table 3: Number of labels for which steering cross-modal concept neurons yields a significantly higher target rate than five matched-size random neuron sets (one-sided prediction-interval t -test, q<0.05 after Benjamini–Hochberg correction across all 72 labels; Appendix A.4 ).
Figure 4: Task-level deactivation on the emotion task. Solid lines: target task accuracy (blue) and mean control-task accuracy (orange) versus the number of deactivated neurons N (log scale). Dashed lines: matched-size random-neuron control (target task: median of the five random trials, with their min–max range shaded; control tasks: mean). Purple stars: audio-only neurons, a reference control (§ 4.4 ). Vertical dashed lines mark the target’s and controls’ decline onsets, shaded when the target’s comes first.
Task
Model
Target N
Control N
Sel.
animal
Qwen2.5-Omni
1,745
36,014
yes
DeSTA2.5-Audio
5,973
5,973
no
Audio Flamingo 3
984
123,275
yes
emotion
Qwen2.5-Omni
1,618
71,812
yes
DeSTA2.5-Audio
10,336
26,763
yes
Audio Flamingo 3
5,862
180,203
yes
Table 4: Task-level deactivation (§ 4.4 ). The number of deactivated neurons N at which target and control accuracy each begin to decline (more than 10% relative to baseline, sustained for the rest of the sweep), and whether the target’s onset precedes the controls’ (selective necessity window).
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Qwen2.5-Omni
DeSTA2.5-Audio
Audio Flamingo 3
Task
Matched
Mismatched
Ratio
Matched
Mismatched
Ratio
Matched
Mismatched
Ratio
animal
0.245
0.093
2.62
0.094
0.075
1.26
0.264
0.100
2.63
emotion
0.174
0.113
1.54
0.121
0.074
1.65
0.202
0.115
1.75
gender
0.144
0.070
2.07
0.127
0.046
2.77
0.098
0.072
1.35
language
0.177
0.089
1.99
0.125
0.087
1.43
0.216
0.090
2.40
Appendix
Table 5: Mean Jaccard overlap of matched (diagonal) and mismatched (off-diagonal) audio-text label pairs in the neuron-overlap matrix of § 4.2 , and their ratio.
Model
Task
k
Jaccard
Random baseline
Max. p (exist.)
Perm. p (distinct.)
Qwen2.5-Omni
animal
200
0.245
0.000185
1.4×10−139
2.8×10−6†
emotion
100
0.174
0.000096
2.2×10−55
0.042
gender
200
0.144
0.000185
4.0×10−100
–
language
100
0.177
0.000096
2.2×10−55
2.5×10−5†
DeSTA2.5-Audio
animal
100
0.094
0.000111
6.0×10−30
0.0025
emotion
100
0.121
0.000111
8.8×10−45
0.0083†
Appendix
Table 6: Existence of cross-modal concept neurons (§ 2 ): tuned neuron-set size k , mean per-label Jaccard overlap on the held-out test split, the corresponding random-overlap baseline, and the largest per-label hypergeometric p -value among the task’s labels (existence; Appendix A.2 ), and the exact label-permutation p -value of the diagonal-vs-off-diagonal Jaccard gap (distinctiveness; Appendix A.3 ). † : the smallest attainable p -value, 1/∣C∣! for a task with ∣C∣ labels. All testable combinations are significant at p<0.05 . Gender (two labels) cannot be tested by label permutation, since its smallest attainable p -value is 0.5 .
Task
Model
Target onset
95% interval
Control onset
95% interval
P(selective)
animal
Qwen2.5-Omni
1,745
932–3,406
36,014
36,014–36,014
1.00
DeSTA2.5-Audio
5,973
2,693–40,280
5,973
2,693–15,110
0.25
Audio Flamingo 3
984
984–1,804
123,275
63,872–123,275
1.00
emotion
Qwen2.5-Omni
1,618
133–5,996
71,812
71,812–189,409
1.00
DeSTA2.5-Audio
10,336
4,395–26,763
26,763
26,763–26,763
0.90
Audio Flamingo 3
5,862
444–69,437
180,203
69,437–180,203
1.00
Appendix
Table 7: Bootstrap stability of the necessity windows of Table 4 (2,000 resamples of the test samples). Onsets are numbers of deactivated neurons N ; P(selective) is the fraction of resamples in which the target task’s decline onset precedes the control tasks’.
Figure 5: Neuron-Jaccard overlap matrices for Qwen2.5-Omni, by task.
Figure 6: Neuron-Jaccard overlap matrices for DeSTA2.5-Audio, by task.
Figure 7: Neuron-Jaccard overlap matrices for Audio Flamingo 3, by task.
Figure 8: Contrastive neuron steering toward the emotion label “happy” in DeSTA2.5-Audio, one panel per carrier label. x -axis: steering coefficient; y -axis: fraction of samples whose predicted answer is “happy”. Blue: cross-modal concept neurons; grey: median of five matched-size random neuron sets, with their min–max range shaded (the null).
Figure 9: Contrastive neuron steering toward the emotion label “happy” in Audio Flamingo 3, one panel per carrier label. x -axis: steering coefficient; y -axis: fraction of samples whose predicted answer is “happy”. Blue: cross-modal concept neurons; grey: median of five matched-size random neuron sets, with their min–max range shaded (the null).
Figure 10: Task-level deactivation on the animal task, for Qwen2.5-Omni, DeSTA2.5-Audio, and Audio Flamingo 3 (left to right). Solid lines: target task accuracy (blue) and mean control-task accuracy (orange) as a function of the number of deactivated neurons N (log scale). Dashed lines: matched-size random-neuron control (target task: median of the five random trials, with their min–max range shaded; control tasks: mean). Purple stars: audio-only neurons, a reference control (§ 4.4 ). Vertical dashed lines and the shaded band mark the target’s and controls’ decline onsets; a band is shown only when the target’s onset precedes the controls’. Unlike emotion, DeSTA2.5-Audio shows no selective window here.
Figure 11: Task-level deactivation on the gender task, for Qwen2.5-Omni, DeSTA2.5-Audio, and Audio Flamingo 3 (left to right). Solid lines: target task accuracy (blue) and mean control-task accuracy (orange) as a function of the number of deactivated neurons N (log scale). Dashed lines: matched-size random-neuron control (target task: median of the five random trials, with their min–max range shaded; control tasks: mean). Purple stars: audio-only neurons, a reference control (§ 4.4 ). Vertical dashed lines and the shaded band mark the target’s and controls’ decline onsets; a band is shown only when the target’s onset precedes the controls’.
Figure 12: Task-level deactivation on the language task, for Qwen2.5-Omni, DeSTA2.5-Audio, and Audio Flamingo 3 (left to right). Solid lines: target task accuracy (blue) and mean control-task accuracy (orange) as a function of the number of deactivated neurons N (log scale). Dashed lines: matched-size random-neuron control (target task: median of the five random trials, with their min–max range shaded; control tasks: mean). Purple stars: audio-only neurons, a reference control (§ 4.4 ). Vertical dashed lines and the shaded band mark the target’s and controls’ decline onsets; a band is shown only when the target’s onset precedes the controls’. Like animal, DeSTA2.5-Audio shows no selective window here.
Emotion is central to human communication, and its expression varies across languages. Large audio-language models (LALMs) achieve strong performance on multilingual speech tasks, yet it remains unclear whether they encode emotion through language-specific correlations or language-agnostic representations. We present the first neuron-level interpretability study of this question. We define Multilingual Emotion Neurons (MLENs) as functional units exhibiting stable emotional selectivity and aligned causal effects across languages, and introduce Consistency-Regularized Fusion (CR-Fusion) to identify them. Across four modern LALMs and 12 typologically diverse languages, emotion-sensitive neurons identified independently per language show minimal overlap, and additional monolingual identification data saturates quickly without isolating more transferable units, motivating identification from pooled cross-lingual evidence. Causal interventions demonstrate that MLENs identified by CR-Fusion provide more precise and transferable affective control than monolingual neuron sets in both zero-shot and low-resource settings. Leave-one-out ablations further reveal asymmetric transfer: individual identification languages, including low-resource ones, contribute non-redundant evidence, while several low-resource languages benefit most from the resulting cross-lingual transfer. Together, our findings provide the first causal, neuron-level account of how LALMs encode emotion across languages, and establish multilingual neuron identification as an effective mechanism for understanding cross-lingual affective behavior.
Xiutian Zhao, Philipp Koehn, Björn Schuller +1
Center for Language and Speech Processing (CLSP), Johns Hopkins University, USA · Group on Language, Audio & Music (GLAM), Imperial College London, UK
Despite the strong performance of large audio language models (LALMs) in various tasks, exactly how and where they integrate acoustic features with textual context remains unclear. We adapt causal tracing to investigate the internal information flow of LALMs during audio comprehension. By conducting layer-wise and token-wise analyses across DeSTA, Qwen, and Voxtral, we evaluate the causal effects of individual hidden states. Layer-wise analysis identifies different fusion strategies, from progressive integration in DeSTA to abrupt late-stage fusion in Qwen. Token-wise analysis shows that the final sequence token acts as an informational bottleneck where the network decisively retrieves relevant information from the audio. We also observe an attention-like query mechanism at intermediate token positions that triggers the model to pull task-relevant audio context. These findings provide a clear characterization of when and where multi-modal integration occurs within LALMs.
Wei-Chih Chen, Chien-yu Huang, Hung-yi Lee
National Taiwan University, Taiwan · Carnegie Mellon University, United States
Multimodal large language models can exhibit text dominance, over-relying on linguistic priors instead of grounding predictions in non-text inputs. One example is large audio-language models (LALMs) where decisive audio evidence can be under-utilized even when it contains important information. To address this issue we use mechanistic interpretability to identify a small set of audio-specialist attention heads whose audio attention yields a ``listening'' signal. We show that this signal increases when audio evidence affects the model's output, providing an indicator of audio engagement under standard prompting. Leveraging this localization, we construct an audio--silence steering direction and apply an inference-time activation intervention to the final representation, amplifying the model's audio effect. To demonstrate the utility of this intervention, we show on MMAU that this improves accuracy by up to +8.0 percentage points on two Qwen-based LALMs, without any parameter updates.
Neta Glazer, Lenny Aharon, Ethan Fetaya
Bar-Ilan University, Ramat Gan, Israel · Columbia University, New York, NY, USA