Uncovering Audio-Text Alignment in Large Audio-Language Models through Cross-Modal Concept Neurons

Tzu-Quan Lin; Chih-Kai Yang; Yi-Cheng Lin; Hao Tang; Hung-yi Lee

cs.SDOct 20, 2026

Uncovering Audio-Text Alignment in Large Audio-Language Models through Cross-Modal Concept Neurons

Authors: Tzu-Quan Lin, Chih-Kai Yang, Yi-Cheng Lin, Hao Tang, Hung-yi Lee

Organizations: Centre for Speech Technology Research, University of Edinburgh, United Kingdom · Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE), Taiwan

Abstract

While the language model backbone is pretrained on text alone, large audio-language models (LALMs) can, to a great extent, recognize acoustic attributes, such as emotion, speaker gender, and spoken language. However, it is unclear how the audio representation is integrated, if at all, with the other text representation before the model produces text. We hypothesize that there is a shared subspace where the audio and text representations are aligned. To test this hypothesis, we identify cross-modal neurons that are responsible for representing both audio and text. We find alignment between audio and text neurons to various degrees across three LALMs and four acoustic attribute types. Causal interventions, steering and deactivating the neurons, confirm that the cross-modal neurons are not just necessary for prediction but can also be manipulated.

Figures & tables

Figure 1: Why we compare neurons rather than entire vectors in the ambient space (schematic). Audio and text are fed to the LALM in separate forward passes. Cosine values follow Qwen2.5-Omni on emotion (§ 4.1 ); neuron counts are illustrative (§ 2 , § 4.2 ).

Model	Task	Cosine	Neuron
Qwen2.5-Omni	animal	8.74%	70.08%
	emotion	0.88%	25.04%
	gender	0.16%	76.04%
	language	0.23%	55.08%
DeSTA2.5-Audio	animal	0.07%	9.22%
	emotion	1.58%	52.03%

Table 1: Matched-label variance share (§ 4.1 ): how much of the variation in each audio-by-text label matrix comes from whether the labels match, rather than from labels similar to everything or from noise. Cosine: cosine similarity of pooled activations; Neuron: Jaccard overlap of cross-modal concept neuron sets (§ 4.2 ).

Model	Anim.	Emo.	Gen.	Lang.
Qwen2.5-Omni	0.245 ∗∗	0.174 ∗	0.144 †	0.177 ∗∗
DeSTA2.5-Audio	0.094 ∗∗	0.121 ∗∗	0.127 †	0.125 ∗∗
Audio Flamingo 3	0.264 ∗∗	0.202 ∗∗	0.098 †	0.216 ∗∗

Table 2: Cross-modal concept neurons (§ 4.2 ): mean per-label Jaccard overlap between the audio and text top-

k

sets on the held-out test split. Stars: distinctiveness, exact label-permutation test of the matched-vs-mismatched gap (Appendix A.3 );

{}^{*}p<0.05

{}^{**}p<0.01

; † not testable (gender has only two labels, so its smallest attainable

p

0.5

). Exact values: Table 6 .

Figure 2: Overview of the causal tests (schematic; activation values are illustrative). Steering (§ 4.3 ): the direction is computed once from text-only forward passes as the difference between the target and carrier labels’ activations, and is added to the target label’s cross-modal concept neurons while the model processes a carrier input (Eq. 5 ); the carrier can also be silence or noise. Deactivation (§ 4.4 ): deactivating a task’s cross-modal concept neurons (Eq. 6 ) impairs that task before the other tasks, whereas deactivating a random set of the same size does far less damage.

Figure 3: Contrastive neuron steering toward the emotion label “happy” in Qwen2.5-Omni, one panel per carrier label. $x$ -axis: steering coefficient $\lambda$ ; $y$ -axis: fraction of samples predicted as “happy”. Blue: cross-modal concept neurons; grey: median of five matched-size random neuron sets, with their min–max range shaded (the null).

Model	Anim.	Emo.	Gen.	Lang.	Total
Qwen2.5-Omni	4/9	5/5	2/2	7/8	18/24
DeSTA2.5-Audio	2/9	4/5	0/2	1/8	7/24
Audio Flamingo 3	6/9	4/5	1/2	6/8	17/24
Total	12/27	13/15	3/6	14/24	42/72

Table 3: Number of labels for which steering cross-modal concept neurons yields a significantly higher target rate than five matched-size random neuron sets (one-sided prediction-interval

t

-test,

q<0.05

after Benjamini–Hochberg correction across all 72 labels; Appendix A.4 ).

Figure 4: Task-level deactivation on the emotion task. Solid lines: target task accuracy (blue) and mean control-task accuracy (orange) versus the number of deactivated neurons $N$ (log scale). Dashed lines: matched-size random-neuron control (target task: median of the five random trials, with their min–max range shaded; control tasks: mean). Purple stars: audio-only neurons, a reference control (§ 4.4 ). Vertical dashed lines mark the target’s and controls’ decline onsets, shaded when the target’s comes first.

Task	Model	Target $N$	Control $N$	Sel.
animal	Qwen2.5-Omni	1,745	36,014	yes
	DeSTA2.5-Audio	5,973	5,973	no
	Audio Flamingo 3	984	123,275	yes
emotion	Qwen2.5-Omni	1,618	71,812	yes
	DeSTA2.5-Audio	10,336	26,763	yes
	Audio Flamingo 3	5,862	180,203	yes

Table 4: Task-level deactivation (§ 4.4 ). The number of deactivated neurons

N

at which target and control accuracy each begin to decline (more than 10% relative to baseline, sustained for the rest of the sweep), and whether the target’s onset precedes the controls’ (selective necessity window).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

	Qwen2.5-Omni			DeSTA2.5-Audio			Audio Flamingo 3
Task	Matched	Mismatched	Ratio	Matched	Mismatched	Ratio	Matched	Mismatched	Ratio
animal	0.245	0.093	2.62	0.094	0.075	1.26	0.264	0.100	2.63
emotion	0.174	0.113	1.54	0.121	0.074	1.65	0.202	0.115	1.75
gender	0.144	0.070	2.07	0.127	0.046	2.77	0.098	0.072	1.35
language	0.177	0.089	1.99	0.125	0.087	1.43	0.216	0.090	2.40

Appendix

Table 5: Mean Jaccard overlap of matched (diagonal) and mismatched (off-diagonal) audio-text label pairs in the neuron-overlap matrix of § 4.2 , and their ratio.

Model	Task	$k$	Jaccard	Random baseline	Max. $p$ (exist.)	Perm. $p$ (distinct.)
Qwen2.5-Omni	animal	200	0.245	0.000185	$1.4\times 10^{-139}$	$2.8\times 10^{-6}\,\dagger$
	emotion	100	0.174	0.000096	$2.2\times 10^{-55}$	0.042
	gender	200	0.144	0.000185	$4.0\times 10^{-100}$	–
	language	100	0.177	0.000096	$2.2\times 10^{-55}$	$2.5\times 10^{-5}\,\dagger$
DeSTA2.5-Audio	animal	100	0.094	0.000111	$6.0\times 10^{-30}$	0.0025
	emotion	100	0.121	0.000111	$8.8\times 10^{-45}$	$0.0083\,\dagger$

Appendix

Table 6: Existence of cross-modal concept neurons (§ 2 ): tuned neuron-set size

k

, mean per-label Jaccard overlap on the held-out test split, the corresponding random-overlap baseline, and the largest per-label hypergeometric

p

-value among the task’s labels (existence; Appendix A.2 ), and the exact label-permutation

p

-value of the diagonal-vs-off-diagonal Jaccard gap (distinctiveness; Appendix A.3 ).

\dagger

: the smallest attainable

p

-value,

1/|\mathcal{C}|!

for a task with

|\mathcal{C}|

labels. All testable combinations are significant at

p<0.05

. Gender (two labels) cannot be tested by label permutation, since its smallest attainable

p

-value is

0.5

Task	Model	Target onset	95% interval	Control onset	95% interval	P(selective)
animal	Qwen2.5-Omni	1,745	932–3,406	36,014	36,014–36,014	1.00
	DeSTA2.5-Audio	5,973	2,693–40,280	5,973	2,693–15,110	0.25
	Audio Flamingo 3	984	984–1,804	123,275	63,872–123,275	1.00
emotion	Qwen2.5-Omni	1,618	133–5,996	71,812	71,812–189,409	1.00
	DeSTA2.5-Audio	10,336	4,395–26,763	26,763	26,763–26,763	0.90
	Audio Flamingo 3	5,862	444–69,437	180,203	69,437–180,203	1.00

Appendix

Table 7: Bootstrap stability of the necessity windows of Table 4 (2,000 resamples of the test samples). Onsets are numbers of deactivated neurons

N

; P(selective) is the fraction of resamples in which the target task’s decline onset precedes the control tasks’.

Panel of Figure 5 — Figure 5: Neuron-Jaccard overlap matrices for Qwen2.5-Omni, by task.

Panel of Figure 6 — Figure 6: Neuron-Jaccard overlap matrices for DeSTA2.5-Audio, by task.

Panel of Figure 7 — Figure 7: Neuron-Jaccard overlap matrices for Audio Flamingo 3, by task.

Figure 8: Contrastive neuron steering toward the emotion label “happy” in DeSTA2.5-Audio, one panel per carrier label. $x$ -axis: steering coefficient; $y$ -axis: fraction of samples whose predicted answer is “happy”. Blue: cross-modal concept neurons; grey: median of five matched-size random neuron sets, with their min–max range shaded (the null).

Figure 9: Contrastive neuron steering toward the emotion label “happy” in Audio Flamingo 3, one panel per carrier label. $x$ -axis: steering coefficient; $y$ -axis: fraction of samples whose predicted answer is “happy”. Blue: cross-modal concept neurons; grey: median of five matched-size random neuron sets, with their min–max range shaded (the null).

Figure 10: Task-level deactivation on the animal task, for Qwen2.5-Omni, DeSTA2.5-Audio, and Audio Flamingo 3 (left to right). Solid lines: target task accuracy (blue) and mean control-task accuracy (orange) as a function of the number of deactivated neurons $N$ (log scale). Dashed lines: matched-size random-neuron control (target task: median of the five random trials, with their min–max range shaded; control tasks: mean). Purple stars: audio-only neurons, a reference control (§ 4.4 ). Vertical dashed lines and the shaded band mark the target’s and controls’ decline onsets; a band is shown only when the target’s onset precedes the controls’. Unlike emotion, DeSTA2.5-Audio shows no selective window here.

Figure 11: Task-level deactivation on the gender task, for Qwen2.5-Omni, DeSTA2.5-Audio, and Audio Flamingo 3 (left to right). Solid lines: target task accuracy (blue) and mean control-task accuracy (orange) as a function of the number of deactivated neurons $N$ (log scale). Dashed lines: matched-size random-neuron control (target task: median of the five random trials, with their min–max range shaded; control tasks: mean). Purple stars: audio-only neurons, a reference control (§ 4.4 ). Vertical dashed lines and the shaded band mark the target’s and controls’ decline onsets; a band is shown only when the target’s onset precedes the controls’.

Figure 12: Task-level deactivation on the language task, for Qwen2.5-Omni, DeSTA2.5-Audio, and Audio Flamingo 3 (left to right). Solid lines: target task accuracy (blue) and mean control-task accuracy (orange) as a function of the number of deactivated neurons $N$ (log scale). Dashed lines: matched-size random-neuron control (target task: median of the five random trials, with their min–max range shaded; control tasks: mean). Purple stars: audio-only neurons, a reference control (§ 4.4 ). Vertical dashed lines and the shaded band mark the target’s and controls’ decline onsets; a band is shown only when the target’s onset precedes the controls’. Like animal, DeSTA2.5-Audio shows no selective window here.

Audio-Text Alignment

See all 21 papers

Multilingual Emotion Neurons in Large Audio-Language Models
Aug 9, 2026Xiutian Zhao, Philipp Koehn, Björn Schuller +1Cross-Lingual Representation Alignment Neuron-Level Interpretability
Center for Language and Speech Processing (CLSP), Johns Hopkins University, USA · Group on Language, Audio & Music (GLAM), Imperial College London, UK
Causal Tracing of Audio-Text Fusion in Large Audio Language Models
Mar 14, 2026Wei-Chih Chen, Chien-yu Huang, Hung-yi LeeLLM Interpretability Audio Understanding
National Taiwan University, Taiwan · Carnegie Mellon University, United States
Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering
Mar 6, 2026Neta Glazer, Lenny Aharon, Ethan FetayaAudio-Language Models Audio-Language Model Evaluation
Bar-Ilan University, Ramat Gan, Israel · Columbia University, New York, NY, USA

Aug 9, 2026J/K move · Enter open · S save

Multilingual Emotion Neurons in Large Audio-Language Models

Xiutian Zhao, Philipp Koehn, Björn Schuller, Berrak Sisman

Center for Language and Speech Processing (CLSP), Johns Hopkins University, USA · Group on Language, Audio & Music (GLAM), Imperial College London, UK

Cross-Lingual Representation Alignment Neuron-Level Interpretability Audio-Language Models Mechanistic Interpretability Emotion Representation in Language Models

Open paper AskPDF arXiv

Emotion is central to human communication, and its expression varies across languages. Large audio-language models (LALMs) achieve strong performance on multilingual speech tasks, yet it remains unclear whether they encode emotion through language-specific correlations or language-agnostic representations. We present the first neuron-level interpretability study of this question. We define Multilingual Emotion Neurons (MLENs) as functional units exhibiting stable emotional selectivity and aligned causal effects across languages, and introduce Consistency-Regularized Fusion (CR-Fusion) to identify them. Across four modern LALMs and 12 typologically diverse languages, emotion-sensitive neurons identified independently per language show minimal overlap, and additional monolingual identification data saturates quickly without isolating more transferable units, motivating identification from pooled cross-lingual evidence. Causal interventions demonstrate that MLENs identified by CR-Fusion provide more precise and transferable affective control than monolingual neuron sets in both zero-shot and low-resource settings. Leave-one-out ablations further reveal asymmetric transfer: individual identification languages, including low-resource ones, contribute non-redundant evidence, while several low-resource languages benefit most from the resulting cross-lingual transfer. Together, our findings provide the first causal, neuron-level account of how LALMs encode emotion across languages, and establish multilingual neuron identification as an effective mechanism for understanding cross-lingual affective behavior.

Abstract

Audio-Text Alignment

Explore similar work

Multilingual Emotion Neurons in Large Audio-Language Models

Causal Tracing of Audio-Text Fusion in Large Audio Language Models

Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering