Concentrated Causal Influence, Distributed Diagnostic Information: Cognitive Impairment Decisions in Large Audio-Language Models

Xiutian Zhao; Luqi Sun; Björn Schuller; Berrak Sisman

eess.ASOct 23, 2026

Concentrated Causal Influence, Distributed Diagnostic Information: Cognitive Impairment Decisions in Large Audio-Language Models

Authors: Xiutian Zhao, Luqi Sun, Björn Schuller, Berrak Sisman

Organizations: Center for Language and Speech Processing (CLSP), Johns Hopkins University, USA · Group on Language, Audio & Music (GLAM), Imperial College London, UK

Abstract

Large audio-language models (LALMs) can assess cognitive impairment (CI) from speech, yet their answers reveal neither which internal components control the decision nor how much CI information is accessible in their representations. We investigate both questions through CI-contrast neurons (CICNs), selected by class-contrastive activation frequency in decoder MLPs. Across eight open LALMs and two corpora (Pitt and PROCESS-2), we causally test CICNs against size- and layer-matched random masks and activity-matched controls. In three of the five models that meet our eligibility criteria, ablation of small CICN sets causes large, class-directed prediction shifts; speech-identified masks also control transcript-only decisions, and three of four primary Pitt masks retain specificity on PROCESS-2. Position-restricted interventions locate the influence before the final answer position, most strongly after the speech or transcript evidence under letter-labeled options, largely as a decision-direction shift. Beyond causal localization, supervised linear readouts of frozen CICN activations achieve 0.92--0.97 AUC within Pitt, versus 0.67--0.84 for the models' direct answers. Matched random and all-neuron readouts reach comparable performance; top-ranked CICNs show only an exploratory advantage at very small unit budgets. These results distinguish concentrated causal control over output decisions from broadly distributed diagnostic information for CI detection in LALMs.

Figures & tables

Figure 1: Ablating CICN masks on Pitt in their identification condition (24 masks). Each row shows the NC mask (blue circle), CI mask (red square), and matched random masks (gray). Left:

\Delta P(\text{CI})

; right:

\Delta\ell

. Both axes are signed so that positive values are shifts away from the mask’s class (toward CI for NC masks, toward NC for CI masks), the direction expected under ablation. Filled markers pass the predefined significance test (Section 3.8 ); hollow markers do not. Right margin:

\text{flip}_{\text{own}}

of the real mask / random-mask mean (descriptive).

Model	Cond.	Mask	real	random	$q$	real	random	share
			$\Delta$ AUC			Spearman $\rho$		offset
Kimi-Audio	AT-chat	NC	$-$ 0.059	+0.002 $\pm$ 0.019	0.009	0.925	0.968	0.91
MiniCPM-o-4.5	A	CI	$-$ 0.040	$-$ 0.003 $\pm$ 0.010	0.003	0.953	0.989	0.89
MiniCPM-o-4.5	AT	CI	+0.001	+0.003 $\pm$ 0.009	0.95	0.964	0.986	0.89
Qwen3-Omni-30B	AT	NC	0.000	0.000 $\pm$ 0.009	0.95	0.935	0.981	0.61
Kimi-Audio	A	NC	$-$ 0.023	+0.004 $\pm$ 0.017	0.16	0.896	0.967	0.69

Table 1: Discrimination versus intercept shift under CICN ablation on the Pitt evaluation split. The four primary masks are above the divider.

\Delta

AUC is the change in diagnostic AUC after ablation;

\rho

is the Spearman correlation between unmodified and ablated log-odds; offset share is the fraction of mean squared per-item log-odds change explained by one additive offset.

Model	Mask	identified	T	T random	A	AT	AT-chat
			evaluated under
Kimi-Audio	NC	A	+0.164 ∗	$-$ 0.003 $\pm$ 0.055	+0.139	+0.131	+0.144
		AT	+0.258 ∗	+0.008 $\pm$ 0.042	+0.230	+0.228	+0.146
		AT-chat	+0.246 ∗	+0.036 $\pm$ 0.044	+0.266	+0.240	+0.267
MiniCPM-o-4.5	CI	A	$-$ 0.178 ∗	+0.012 $\pm$ 0.042	$-$ 0.212	$-$ 0.202	$-$ 0.118
		AT	$-$ 0.237 ∗	+0.017 $\pm$ 0.044	$-$ 0.221	$-$ 0.253	$-$ 0.109

Table 2: Cross-condition transfer under ablation on Pitt:

\Delta P(\text{CI})

for masks identified under the row condition and evaluated under the column condition; diagonal in italics. “ T random” gives the matched-random mean

\pm

sd under transcript-only evaluation. ∗ marks cells that pass the predefined transfer test. Diagonal effects are tested in Figure 1 ; other cells without displayed random controls are descriptive.

Figure 2: Position-restricted ablation on Pitt: change in

P(\text{CI})

away from the mask’s class when ablation is restricted to one set of prefill positions. Each bar is a separate intervention, colored as in the prompt-order strip; the bars are not an additive decomposition. Black tick: all positions but the last; right column: last position only. Values and tests are in Appendix G.1 .

Figure 3: Diagnostic readout AUC on Pitt against neuron budget. Bold lines average seven cells; thin lines show individual cells. Blue: top-ranked CICNs; gray: layer-matched random neurons, with mean

\pm

2 sd over 20 runs. Dashed lines show the mean all-neuron readout and zero-shot answer. Whole-mask results are shown separately because mask size varies by model. Small-budget differences are descriptive.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Hugging Face checkpoint
Qwen2.5-Omni-7B	Qwen/Qwen2.5-Omni-7B
Qwen3-Omni-30B	Qwen/Qwen3-Omni-30B-A3B-Instruct
Kimi-Audio	moonshotai/Kimi-Audio-7B-Instruct
MiniCPM-o-4.5	openbmb/MiniCPM-o-4_5
Gemma-4-12B	google/gemma-4-12B-it
Voxtral-Small-24B	mistralai/Voxtral-Small-24B-2507

Appendix

Table 3: Model names used in this paper and the Hugging Face checkpoints they refer to.

Model	Layers	MLP width	Neurons	Mask ( $r{=}0.5\%$ )
Qwen2.5-Omni-7B	28	18,944	530,432	2,652
Qwen3-Omni-30B	48	$128\times 768$	4,718,592	23,592
Kimi-Audio	28	18,944	530,432	2,652
MiniCPM-o-4.5	36	12,288	442,368	2,211
Gemma-4-12B	48	15,360	737,280	3,686

Appendix

Table 4: Decoder MLP sizes of the models that reach identification on Pitt (Audio-Flamingo-Next, Voxtral-Small-24B and MOSS-Audio-8B do not meet the analysis-eligibility criteria, § 4 ). For Qwen3-Omni-30B a neuron is one unit of one expert.

		eval.	identification
Model	Cond.	UAR	UAR [95% CI]
Gemma-4-12B	AT	0.67	0.663 [0.608, 0.715]
	AT-chat	0.68	0.639 [0.582, 0.695]
Kimi-Audio	A	0.67	0.634 [0.573, 0.690]
	AT	0.64	0.615 [0.554, 0.673]
	AT-chat	0.64	0.589 [0.528, 0.652]

Appendix

Table 5: Analysis eligibility recomputed on the Pitt identification split for the 12 cells that reach identification. “eval.”: UAR on the evaluation split (Table 9 ). The eligible identification recordings of each cell are in Table 13 .

	HC	MCI	Dementia
Labeled style ( $n$ )	99	124	38
Plain style ( $n$ )	101	26	12
Plain style (%)	50.5	17.3	24.0
Fillers / 100 words, labeled	3.9	3.7	2.8
Fillers / 100 words, plain	1.4	2.8	3.7
Median words, labeled	163	127	120

Appendix

Table 6: PROCESS-2 Cookie Theft transcripts by transcription style (all 400 participants, before exclusions).

CHAT	Rendered as
(.) , (..) , (...)	(short pause), (pause), (long pause)
timed pause (2.5)	(pause 2.5s)
filler &-uh	uh
fragment &+wa	wa-
[/] , [//] , [///]	[repeats], [self-correction], [reformulation]
[/-] , [/?]	[false start], [unclear retracing]

Appendix

Table 7: Annotated rendering (convention v1 ).

Analysis	Null	$K$	Family
Ablation, own condition (§ 5.1 )	matched random masks	20	24 masks
Ablation, own condition (§ 5.1 )	sets matched on activation statistics	20	8 masks
Ablation, transfer to T (§ 5.3 )	matched random masks under T	20	24 masks
Class-mean shift (§ 5.1 )	norm-matched random masks	20	21 mask– $\alpha$ cells
CICN masks on gender (§ 5.1 )	matched random masks	20	cross-task, $\|\Delta\ell\|$ only
Span masks (App. F.1 )	matched random masks	20	36 masks

Appendix

Table 8: Overview of the analyses: the null each is compared with, the number

K

of random masks per real mask, and the family over which the Benjamini–Hochberg correction runs. Unless stated, each mask enters with the conjunction of its two co-primary metrics. Details and caveats are in the sections given.

Model	Cond.	UAR	95% CI	F1	$P(\text{CI})$	pred. CI
Qwen2.5-Omni-7B	T	0.664	[0.580, 0.751]	0.634	0.464	0.430
	A	0.587	[0.527, 0.644]	0.471	0.342	0.159
	AT	0.736	[0.646, 0.817]	0.715	0.492	0.495
	AT-chat	0.582	[0.542, 0.629]	0.435	0.377	0.103
Qwen3-Omni-30B	T	0.736	[0.668, 0.806]	0.682	0.343	0.346
	A	0.726	[0.636, 0.808]	0.699	0.432	0.458

Appendix

Table 9: Unmodified Pitt results. “pred. CI” is the fraction of items predicted CI.

Model	Cond.	UAR	95% CI	F1	$P(\text{CI})$	pred. CI
Qwen2.5-Omni-7B	T	0.549	[0.468, 0.652]	0.553	0.330	0.121
	A	0.568	[0.480, 0.665]	0.576	0.310	0.091
	AT	0.606	[0.506, 0.719]	0.618	0.369	0.131
Qwen3-Omni-30B	T	0.487	[0.467, 0.500]	0.438	0.077	0.020
	A	0.525	[0.500, 0.583]	0.494	0.086	0.010
	AT	0.562	[0.494, 0.661]	0.565	0.166	0.051

Appendix

Table 10: Unmodified PROCESS-2 results, pooled over transcription styles.

	T		A		AT
Model	lab.	plain	lab.	plain	lab.	plain
Qwen2.5-Omni-7B	0.59	0.51	0.56	0.58	0.67	0.44
Qwen3-Omni-30B	0.47	0.50	0.50	0.60	0.54	0.60
Kimi-Audio	0.53	0.55	0.66	0.59	0.70	0.53
MiniCPM-o-4.5	0.62	0.56	0.67	0.44	0.61	0.48
Gemma-4-12B	0.71	0.73	0.57	0.43	0.47	0.61

Appendix

Table 11: Unmodified PROCESS-2 UAR within transcription style (lab.: labeled; plain: plain prose; Table 6 ). Labeled style: 54 evaluation recordings (39 NC / 15 CI); plain style: 45 (40 NC / 5 CI). With 5 CI recordings, each plain-style value moves by 0.10 per CI recording.

	Pitt			PROCESS-2
Model	AT $-$ T	AT $-$ A	chat $-$ AT	AT $-$ T	AT $-$ A
Qwen2.5-Omni-7B	+0.07	+0.15 ∗	$-$ 0.15 ∗	+0.06	+0.04
Qwen3-Omni-30B	+0.01	+0.02	+0.00	+0.08	+0.04
Kimi-Audio	+0.01	$-$ 0.04	+0.00	+0.13	+0.01
MiniCPM-o-4.5	+0.09	$-$ 0.03	+0.03	$-$ 0.03	$-$ 0.04
Gemma-4-12B	+0.10	+0.07	+0.02	$-$ 0.20 ∗	$-$ 0.02

Appendix

Table 12: Paired UAR differences between input conditions of the unmodified model ( chat = AT-chat ). ∗ marks a significant paired-bootstrap contrast (Appendix B ).

Pitt
		eligible / total		tokens pooled ( $N=15$ )
Model	Cond.	NC	CI	NC	CI
Qwen2.5-Omni-7B	AT	24 / 152	16 / 145	42,988	61,434
Qwen3-Omni-30B	A	64 / 152	20 / 145	21,960	23,438
	AT	47 / 152	46 / 145	23,286	28,064
	AT-chat	73 / 152	19 / 145	25,008	36,058

Appendix

Table 13: Identification recordings that the unmodified model classifies correctly under both option orders with margin

\geq 0.1

(“eligible”), and the prompt tokens pooled from the 15 used per class (both orders, span all ).

Model	Cond.	Mask	$\Delta\ell$ real	$\Delta\ell$ sets	$p$	$q$	paraphrase	numeric labels
				Activity-matched sets			Prompt variants, $q$
Qwen3-Omni-30B	AT	NC	+1.77	$-$ 0.16 $\pm$ 0.76	0.030	0.040	$<$ 0.001	$<$ 0.001
MiniCPM-o-4.5	A	CI	$-$ 2.51	$-$ 0.27 $\pm$ 0.95	0.033	0.040	$<$ 0.001	0.006
MiniCPM-o-4.5	AT	CI	$-$ 2.63	$-$ 0.64 $\pm$ 0.86	0.035	0.040	$<$ 0.001	0.002
Kimi-Audio	AT-chat	NC	+1.60	+0.05 $\pm$ 0.60	0.033	0.040	0.006	0.005
Kimi-Audio	AT	NC	+1.46	$-$ 0.05 $\pm$ 0.56	0.020	0.040	–	–

Appendix

Table 14: Robustness of the eight strongest Pitt CICN masks under activity-matched and prompt-variant controls. The real-mask

\Delta\ell

is retained for effect-size context; its uniform-random comparison is shown in Figure 1 .

\Delta P(\text{CI})

also enters each significance test.

p

is the uncorrected conjunction value; the equal

q

values follow from the Benjamini–Hochberg step-up. The first four rows are the primary masks. –: not evaluated under that prompt variant. † indicates an activity-matched comparison with a larger effect in the opposite direction, which is not counted as passing.

Model	Cond.	Mask	$\Delta P$	flip	$\Delta P$	flip	$\Delta P$	flip	$\Delta P$	flip	$\Delta P$	flip
			main		$N=5$		$N=10$		$r=0.1\%$		$r=1\%$
Kimi-Audio	A	NC	+0.139	0.50 §	+0.125	0.63	+0.105	0.42	+0.011	0.17	+0.220	0.79 §
Kimi-Audio	AT	NC	+0.228	0.95 §	+0.164	0.57	+0.132	0.67 §	+0.046	0.33 §	+0.262	1.00 §
Kimi-Audio	AT-chat	NC	+0.267	0.88 §	+0.250	0.88 §	+0.207	0.77 §	+0.152	0.58 §	+0.064	0.35
MiniCPM-o-4.5	A	CI	$-$ 0.212	0.38 §	$-$ 0.105	0.12 §	$-$ 0.189	0.36 §	$-$ 0.105	0.14 §	$-$ 0.203	0.41 §
MiniCPM-o-4.5	AT	CI	$-$ 0.253	0.55 §	$-$ 0.355	0.68 §	$-$ 0.133	0.23	$-$ 0.087	0.16	$-$ 0.268	0.68 §

Appendix

Table 15: Sensitivity of Pitt ablation effects to identification-pool size and CICN mask size for the eight strongest masks. The main setting is

N=15

r=0.5\%

; each variant uses its own matched random controls.

\Delta P=\Delta P(\text{CI})

; flip denotes

\text{flip}_{\text{own}}

			pool (30)			all identification (297)
Model	Cond.	Mask	raw	adjusted	kept	raw	adjusted	kept
Kimi-Audio	A	NC	0.127	0.086	68%	0.051	0.033	65%
Kimi-Audio	AT	NC	0.092	0.080	86%	0.046	0.042	91%
Kimi-Audio	AT-chat	NC	0.094	0.069	73%	0.039	0.032	82%
MiniCPM-o-4.5	A	CI	0.139	0.080	58%	0.056	0.035	61%
MiniCPM-o-4.5	AT	CI	0.119	0.078	66%	0.054	0.035	64%

Appendix

Table 16: Class gap in CICN activation probability (mask’s class minus the other class, averaged over mask neurons) before and after adjustment for duration, silence proportion, and speech rate, shown for the selection pool and all identification recordings. “kept” is the adjusted/raw gap ratio.

	CICN mask $\rightarrow$ gender		gender mask $\rightarrow$ CI
CICN cell (own $\|\Delta\ell\|$ on CI)	real	random	male	female
Qwen3 AT NC (1.87)	0.62	0.49 $\pm$ 0.08	0.87 (0.65 $\pm$ 0.09)	0.97 (0.70 $\pm$ 0.15)
Qwen3 AT CI (1.10)	1.61 ∗	0.48 $\pm$ 0.05	(same cell as above)
Kimi AT NC (1.46)	0.21	0.22 $\pm$ 0.08	0.89 ∗ (0.40 $\pm$ 0.12)	0.35 (0.41 $\pm$ 0.17)
Kimi AT-chat NC (1.60)	0.23	0.21 $\pm$ 0.05	0.89 (0.42 $\pm$ 0.21)	0.30 (0.57 $\pm$ 0.28)
MiniCPM A CI (2.51)	1.35 ∗	0.57 $\pm$ 0.27	0.42 (0.52 $\pm$ 0.25)	0.47 (0.44 $\pm$ 0.20)

Appendix

Table 17: Cross-task ablation control measured by

|\Delta\ell|

. Left: CICN masks evaluated on gender; right: gender masks identified in the same model and condition and evaluated on CI. The own-task value in parentheses is the mean absolute per-item change, which is at least as large as the magnitude of the signed mean

\Delta\ell

on the same recordings (Table 14 ). ∗ marks cells that pass the predefined cross-task test; others are n.s.

		$\Delta P(\text{CI})$			$\Delta\ell$				$\text{flip}_{\text{own}}$
Model / cond. / mask	$\alpha$	real	norm-m.	$p$	real	norm-m.	$p$	$q$	real	norm-m.
Kimi AT NC	$-4$	+0.197	$-$ 0.010	0.007	+1.27	$-$ 0.06	0.006	0.072	0.90 §	0.13
Kimi AT NC	$+4$	$-$ 0.148	+0.020	0.024	$-$ 0.91	+0.12	0.027	0.094	0.20 §	0.03
Kimi AT-chat NC	$-4$	+0.225	+0.002	0.010	+1.25	+0.01	0.007	0.072	0.77 §	0.14
Kimi AT-chat NC	$+1$	$-$ 0.059	+0.002	0.013	$-$ 0.31	0.00	0.013	0.072	0.14 §	0.02
Kimi AT-chat NC	$+4$	$-$ 0.220	+0.001	0.017	$-$ 1.23	$-$ 0.01	0.015	0.072	0.57 §	0.09

Appendix

Table 18: Steering against norm-matched random directions; “norm-m.” gives the matched-control mean.

Model	Mask	A – AT	A – AT-chat	AT – AT-chat
Qwen3-Omni-30B	NC	0.28	0.32	0.42
	CI	0.19	0.57	0.29
Kimi-Audio	NC	0.19	0.20	0.30
	CI	0.20	0.28	0.30
MiniCPM-o-4.5	NC	0.33	0.24	0.21
	CI	0.40	0.24	0.22

Appendix

Table 19: Jaccard overlap between ConAct masks (span all unless stated). The Jaccard expected for independent masks with the same per-layer counts is 0.002–0.008 in every cell.

Figure 4: Layer distribution of the ConAct masks (span all ): share of a mask’s neurons in each decoder layer, one line per identification condition. Rows are models and corpora; the

y

axis differs between rows.

		evaluated under
Model (NC mask)	span	T	A	AT
Kimi-Audio	audio	+0.27 ∗ (4.5)	+0.29 ∗ (3.3)	+0.27 ∗ (2.9)
	audio (tok.-matched)	+0.28 ∗ (4.7)	+0.34 ∗ (5.4)	+0.28 ∗ (3.8)
	text	$-$ 0.04 ( $-$ 1.4)	$-$ 0.07 ( $-$ 1.3)	$-$ 0.02 ( $-$ 0.3)
Qwen3-Omni-30B	audio	+0.31 ∗ (9.1)	+0.16 ∗ (5.5)	+0.17 ∗ (6.1)
	audio (tok.-matched)	+0.19 ∗ (4.1)	+0.08 (1.7)	+0.09 ∗ (2.6)

Appendix

Table 20: Span dissociation within AT :

\Delta P(\text{CI})

when ablating NC masks identified from the audio-token span, a token-matched subset of that span, or the transcript-token span;

z

relative to matched random masks is shown in parentheses.

			$\Delta P(\text{CI})$			$\Delta\ell$
Model	Cond.	Mask	real	random	$p$	real	random	$p$	$q$	$\text{flip}_{\text{own}}$ ( $n$ )
Kimi-Audio	AT	NC	+0.298	+0.002	$<$ 0.001	+1.72	$-$ 0.01	0.001	0.006	0.65 (65)
MiniCPM-o-4.5	A	CI	$-$ 0.143	+0.018	0.001	$-$ 2.26	+0.14	$<$ 0.001	0.006	– (7)
Qwen3-Omni-30B	A	NC	+0.112	+0.019	$<$ 0.001	+1.85	+0.27	$<$ 0.001	0.006	0.06 (79)
Qwen3-Omni-30B	AT	NC	+0.110	+0.015	0.001	+1.91	+0.21	$<$ 0.001	0.006	0.14 (77)
Kimi-Audio	A	NC	+0.202	$-$ 0.008	0.002	+1.15	$-$ 0.10	0.010	0.033	0.29 (75)

Appendix

Table 21: Pitt CICN masks ablated on PROCESS-2 with Pitt-matched random controls. Upper part: A and AT masks; lower part: Pitt AT-chat masks evaluated under PROCESS-2 AT .

n

is the denominator of

\text{flip}_{\text{own}}

; flip rates are omitted for

n<10

. The two blocks are separate correction families.

Pitt masks
			$\Delta P(\text{CI})$		$\text{flip}_{\text{own}}$
Model	Cond.	Mask	lab.	plain	lab.	plain
Kimi-Audio	A	NC	+0.202 ( $-$ 0.013)	+0.203 ( $-$ 0.002)	0.28 (0.03)	0.31 (0.04)
Kimi-Audio	AT	NC	+0.282 ( $-$ 0.001)	+0.317 (+0.006)	0.65 (0.06)	0.65 (0.06)
Kimi-Audio	AT-chat $\rightarrow$ AT	NC	+0.291 (+0.037)	+0.309 (+0.043)	0.71 (0.09)	0.59 (0.11)
MiniCPM-o-4.5	A	CI	$-$ 0.145 (+0.015)	$-$ 0.142 (+0.022)	–	–

Appendix

Table 22: Within-style PROCESS-2 ablation: real-mask effect with the matched-random mean in parentheses (recording counts in Table 11 ). “–” indicates fewer than 10 baseline-correct items of the mask’s class, so no flip rate is reported. These values are descriptive and are not tested separately.

Pitt held-out (16 MCI, 4 vascular, 3 other)
			$\Delta P(\text{CI})$			$\Delta\ell$
Model	Cond.	Mask	real	random	$p$	real	random	$p$	$q$
Kimi-Audio	A	NC	+0.200	$-$ 0.027 $\pm$ 0.085	0.018	+1.09	$-$ 0.17	0.026	0.034
Kimi-Audio	AT	NC	+0.301	$-$ 0.002 $\pm$ 0.068	$<$ 0.001	+1.84	$-$ 0.03	$<$ 0.001	0.001
Kimi-Audio	AT-chat	NC	+0.290	+0.021 $\pm$ 0.056	$<$ 0.001	+1.75	+0.11	$<$ 0.001	0.001
MiniCPM-o-4.5	A	CI	$-$ 0.203	+0.016 $\pm$ 0.054	0.001	$-$ 2.20	+0.17	0.002	0.003

Appendix

Table 23: CICN ablation on held-out diagnoses with matched random controls. The eight strongest Pitt masks are used; on PROCESS-2, AT-chat masks are evaluated under AT . Flip rates are undefined because these recordings have no NC/CI evaluation label. Each block is a separate correction family.

	Kimi-Audio AT NC	Kimi-Audio AT-chat NC			MiniCPM-o-4.5 A CI			MiniCPM-o-4.5 AT CI
Ablated positions	orig.	orig.	para.	num.	orig.	para.	num.	orig.	para.	num.
all (Figure 1 )	+0.228 ∗	+0.267 ∗	–	–	$-$ 0.212 ∗	–	–	$-$ 0.253 ∗	–	–
all but the last	+0.222 ∗	+0.252 ∗	+0.179 ∗	+0.223 ∗	$-$ 0.217 ∗	$-$ 0.324 ∗	$-$ 0.135 ∗	$-$ 0.261 ∗	$-$ 0.330 ∗	$-$ 0.218 ∗
audio span	+0.069 ∗	+0.038	+0.030	+0.041	$-$ 0.100 ∗	$-$ 0.109 ∗	$-$ 0.085 ∗	$-$ 0.084 ∗	$-$ 0.083 ∗	$-$ 0.083 ∗
transcript span	+0.022 ∗	+0.016	–	–	–	–	–	$-$ 0.034 ∗	–	–
template	+0.152 ∗	+0.227 ∗	–	–	$-$ 0.123 ∗	–	–	$-$ 0.161 ∗	–	–

Appendix

Table 24: Position-restricted ablation on Pitt under the original prompt (orig.) and the two prompt variants (para.: paraphrase; num.: numeric option labels; Appendix D.5 ):

\Delta P(\text{CI})

when the mask is ablated only at the listed prefill positions. “Template” is every position outside the audio and transcript spans except the last; “before the content” cannot attend to the recording. ∗ marks cells that pass the predefined test for that position set and prompt. –: not run (the variants cover three masks and four position sets; MiniCPM-o-4.5 A has no transcript span); the random-mask row is given for the original prompt. The shares quoted in Appendix G.2 are computed on

\Delta\ell

Figure 5: Logit lens on Pitt. Top: AUC of the option-logit readout from each layer’s residual stream at the final prompt position, one line per input condition. Bottom: depth distribution of the CICN masks used in the position-restricted ablations (blue: NC masks; red: CI masks).

Figure 6: Token-level correlates of CICN activity on Pitt. Left: CI-direction score (positive: CI-like) around fillers, rendered pauses, and Cookie Theft information-unit words, relative to each recording’s transcript mean; thin gray lines are matched random masks and “rec.” counts recordings containing the event. Right: within-recording token-category contrasts, with matched-random mean

\pm

2 sd; filled markers pass the predefined token-level test. Values, including AT , are in Table 25 .

		Kimi-Audio		MiniCPM-o-4.5
Contrast	Mask	AT	AT-chat	AT	AT-chat
info. unit $-$ content	CI	$-$ 0.055 ∗ ( $-$ 0.008)	$-$ 0.047 ∗ ( $-$ 0.008)	$-$ 0.128 ∗ ( $-$ 0.039)	$-$ 0.128 ∗ ( $-$ 0.036)
	NC	$-$ 0.039 ∗ ( $-$ 0.010)	$-$ 0.056 ∗ ( $-$ 0.015)	$-$ 0.076 ∗ ( $-$ 0.036)	$-$ 0.076 ∗ ( $-$ 0.032)
filler $-$ content	CI	–	+0.185 ∗ (+0.058)	–	+0.332 ∗ (+0.088)
fragment $-$ content	CI	–	+0.210 ∗ (+0.088)	–	+0.350 ∗ (+0.103)
pause $-$ content	CI	–	+0.070 ∗ (+0.001)	–	+0.244 ∗ (+0.083)

Appendix

Table 25: Token-level within-recording differences in CI-direction score on Pitt (positive: more CI-like); matched-random means are in parentheses. Pause, filler, fragment, and retracing markers occur only in AT-chat . ∗ marks contrasts that pass the predefined token-level test; ‡ marks the same criterion with an effect opposite to the expected sign.

Figure 7: Predictive utility of CI-contrast neurons: whole-mask readouts. Blue, gray and green are supervised linear readouts trained on Pitt (CICNs, matched-random neurons with mean

\pm

2 sd over 20 masks, and all neurons); yellow is the model’s zero-shot answer, joined to the CICN readout by a line. (a) Within-corpus readout: CICN readouts reach 0.92–0.97 AUC on Pitt, against 0.67–0.84 for the model’s own answer. (b) Cross-corpus generalization: the same readouts evaluated on PROCESS-2, with rows in the same order as (a).

Audio-Language Models

Audio-language models connect audio and natural language for understanding, retrieval, dialogue, speech generation, and music or sound analysis.

See all 303 papers

Oct 9, 2026

Does Location Matter? A Black-Box Causal Audit Reveals Model-Dependent Audio Grounding in LALM Hallucination

Large audio-language models (LALMs) frequently exhibit object hallucination, affirming absent sounds. The dominant mitigation, Audio-Aware Decoding (AAD), contrasts logits under real audio against an all-zero "blank" waveform, flagged by prior review as out-of-distribution. We introduce a black-box causal test that isolates the claimed evidence itself: localize a claim's evidence region via CLAP similarity, remove it by spectral gating, and compare against a random-window control across a 2x2x2 grid on three LALMs. On Qwen2-Audio-7B, localized ablation's raw AUROC trails blank's (0.613 vs. 0.721); this is a duration confound, not a localization failure, since blank removes the entire clip versus the grid's 2-4 s. Holding duration fixed, localization beats random removal at every cell on both models. Raw detection value remains model-dependent, especially on Audio Flamingo 3. The same evidence corrects decoding: contrastive decoding combining blank and language-prior contrasts, gated and ensembled, raises Qwen2-Audio accuracy from 57.1% to 71.9%, beating AAD alone. A unified causal-grounding profile does not transfer between models, indicating different grounding regimes. The source code and a detailed report of our findings, including identified limitations, are available at https://github.com/khalooei/causa.

Mohammad Khalooei, Mohammad Sabokrou

Sharif University of Technology, Iran · Amirkabir University of Technology, Iran · New Uzbekistan University, Uzbekistan +1

Audio-Language Models Audio-Language Model Hallucination Detection Causal Interventions in Language Models Temporal Audio Grounding

Ask PDF

Oct 8, 2026

Selective Listening: Mechanism-Guided Control of Audio Influence in Large Audio-Language Models

Large audio-language models (LALMs) exploit multimodal evidence, yet task-irrelevant audio can alter text-reasoning decisions when listening is unnecessary. Aggregate Accuracy can hide this paired drift because audio-induced repairs and damages may cancel. Paired drift analysis and targeted interventions identify architecture-specific, intervention-sensitive late audio pathways as actionable control points. We introduce ICAP-Gate, which applies mechanism-guided, task-conditioned control to each model's pathway. Across four LALMs, two reasoning benchmarks, and environmental-sound and natural-speech interference, ICAP-Gate has lower point estimates for Influence Rate and Answer Flip than ungated inference in all 16 full-split model--condition evaluations. Fixed suppression degrades automatic speech recognition (ASR) across all four models, whereas ICAP-Gate matches ungated ASR performance by preserving the pathway for explicit audio-demand instructions. ICAP-Gate has lower paired-drift point estimates than mitigation prompting in all four evaluated settings and provides competitive stabilization relative to eight-sample Self-Consistency while using one generation per query; in controlled ARC measurements, Self-Consistency incurs

7.0

9.2\times

ungated latency. These results establish selective modality influence control as a design principle for robust multimodal reasoning.

Yulin Sun, Kele Xu, Yong Dou

College of Computer Science and Technology, National University of Defense Technology · State Key Laboratory of Complex & Critical Software Environment

Audio-Language Models Multimodal Reasoning

Ask PDF

Sep 6, 2026

Causal Localization of the Refusal Direction in Audio Language Models

A large audio language model (LALM) attaches a speech front end to a text language model (LM) that is already safety-aligned. When such a model refuses a harmful spoken request, is the refusal carried by the front end, or inherited from the text LM? We test this with causal interventions. At each model's audio-to-LM interface and at tested LM residual layers, we fit a direction separating harmful from benign prompts, ablate its component, and measure the resulting change in the model's first-token refusal margin. Four of the five models are evaluated under held-out category shift. Across five LALMs spanning three backbone families, with three models passing a baseline safety gate, the largest tested effects occur in a mid-to-late LM band, while ablations of the tested interface directions have little effect. On Qwen2.5-Omni, ablating the L16 direction changes the margin by -7.10, versus -0.013 at the projector. The audio pathway is still in use: zeroing the encoder output changes the margin by -4.7. In the same model, the contrast is linearly decodable at an early layer where single-layer ablation has little effect, and a direction fitted on the LM backbone alone transfers to the full audio model. Because harmful and benign prompts also differ in form, we interpret the direction as refusal-linked rather than harmfulness-specific. These interventions localize dependence of the refusal margin, not where refusal is computed. Moving the margin also does not always change what the model writes. Safety audits of these models should use interventions rather than rely on probes alone, and should examine the inherited text LM alongside the audio interface.

Leonardo Haw-Yang Foo, Hung-yi Lee

National Taiwan University, Taiwan · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE), Taiwan

Refusal Behavior in Language Models Audio-Language Models LLM Auditing

Ask PDF

Abstract

Audio-Language Models

Explore similar work

Does Location Matter? A Black-Box Causal Audit Reveals Model-Dependent Audio Grounding in LALM Hallucination

Selective Listening: Mechanism-Guided Control of Audio Influence in Large Audio-Language Models

Causal Localization of the Refusal Direction in Audio Language Models