Concentrated Causal Influence, Distributed Diagnostic Information: Cognitive Impairment Decisions in Large Audio-Language Models
Authors: Xiutian Zhao, Luqi Sun, Björn Schuller, Berrak Sisman
Organizations: Center for Language and Speech Processing (CLSP), Johns Hopkins University, USA · Group on Language, Audio & Music (GLAM), Imperial College London, UK
Large audio-language models (LALMs) can assess cognitive impairment (CI) from speech, yet their answers reveal neither which internal components control the decision nor how much CI information is accessible in their representations. We investigate both questions through CI-contrast neurons (CICNs), selected by class-contrastive activation frequency in decoder MLPs. Across eight open LALMs and two corpora (Pitt and PROCESS-2), we causally test CICNs against size- and layer-matched random masks and activity-matched controls. In three of the five models that meet our eligibility criteria, ablation of small CICN sets causes large, class-directed prediction shifts; speech-identified masks also control transcript-only decisions, and three of four primary Pitt masks retain specificity on PROCESS-2. Position-restricted interventions locate the influence before the final answer position, most strongly after the speech or transcript evidence under letter-labeled options, largely as a decision-direction shift. Beyond causal localization, supervised linear readouts of frozen CICN activations achieve 0.92--0.97 AUC within Pitt, versus 0.67--0.84 for the models' direct answers. Matched random and all-neuron readouts reach comparable performance; top-ranked CICNs show only an exploratory advantage at very small unit budgets. These results distinguish concentrated causal control over output decisions from broadly distributed diagnostic information for CI detection in LALMs.
Figures & tables
Figure 1: Ablating CICN masks on Pitt in their identification condition (24 masks). Each row shows the NC mask (blue circle), CI mask (red square), and matched random masks (gray). Left: ΔP(CI) ; right: Δℓ . Both axes are signed so that positive values are shifts away from the mask’s class (toward CI for NC masks, toward NC for CI masks), the direction expected under ablation. Filled markers pass the predefined significance test (Section 3.8 ); hollow markers do not. Right margin: flipown of the real mask / random-mask mean (descriptive).
Δ AUC
Spearman ρ
offset
Model
Cond.
Mask
real
random
q
real
random
share
Kimi-Audio
AT-chat
NC
− 0.059
+0.002 ± 0.019
0.009
0.925
0.968
0.91
MiniCPM-o-4.5
A
CI
− 0.040
− 0.003 ± 0.010
0.003
0.953
0.989
0.89
MiniCPM-o-4.5
AT
CI
+0.001
+0.003 ± 0.009
0.95
0.964
0.986
0.89
Qwen3-Omni-30B
AT
NC
0.000
0.000 ± 0.009
0.95
0.935
0.981
0.61
Kimi-Audio
A
NC
− 0.023
+0.004 ± 0.017
0.16
0.896
0.967
0.69
Table 1: Discrimination versus intercept shift under CICN ablation on the Pitt evaluation split. The four primary masks are above the divider. Δ AUC is the change in diagnostic AUC after ablation; ρ is the Spearman correlation between unmodified and ablated log-odds; offset share is the fraction of mean squared per-item log-odds change explained by one additive offset.
evaluated under
Model
Mask
identified
T
T random
A
AT
AT-chat
Kimi-Audio
NC
A
+0.164 ∗
− 0.003 ± 0.055
+0.139
+0.131
+0.144
AT
+0.258 ∗
+0.008 ± 0.042
+0.230
+0.228
+0.146
AT-chat
+0.246 ∗
+0.036 ± 0.044
+0.266
+0.240
+0.267
MiniCPM-o-4.5
CI
A
− 0.178 ∗
+0.012 ± 0.042
− 0.212
− 0.202
− 0.118
AT
− 0.237 ∗
+0.017 ± 0.044
− 0.221
− 0.253
− 0.109
Table 2: Cross-condition transfer under ablation on Pitt: ΔP(CI) for masks identified under the row condition and evaluated under the column condition; diagonal in italics. “ T random” gives the matched-random mean ± sd under transcript-only evaluation. ∗ marks cells that pass the predefined transfer test. Diagonal effects are tested in Figure 1 ; other cells without displayed random controls are descriptive.
Figure 2: Position-restricted ablation on Pitt: change in P(CI) away from the mask’s class when ablation is restricted to one set of prefill positions. Each bar is a separate intervention, colored as in the prompt-order strip; the bars are not an additive decomposition. Black tick: all positions but the last; right column: last position only. Values and tests are in Appendix G.1 .
Figure 3: Diagnostic readout AUC on Pitt against neuron budget. Bold lines average seven cells; thin lines show individual cells. Blue: top-ranked CICNs; gray: layer-matched random neurons, with mean ± 2 sd over 20 runs. Dashed lines show the mean all-neuron readout and zero-shot answer. Whole-mask results are shown separately because mask size varies by model. Small-budget differences are descriptive.
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Hugging Face checkpoint
Qwen2.5-Omni-7B
Qwen/Qwen2.5-Omni-7B
Qwen3-Omni-30B
Qwen/Qwen3-Omni-30B-A3B-Instruct
Kimi-Audio
moonshotai/Kimi-Audio-7B-Instruct
MiniCPM-o-4.5
openbmb/MiniCPM-o-4_5
Gemma-4-12B
google/gemma-4-12B-it
Voxtral-Small-24B
mistralai/Voxtral-Small-24B-2507
Appendix
Table 3: Model names used in this paper and the Hugging Face checkpoints they refer to.
Model
Layers
MLP width
Neurons
Mask ( r=0.5% )
Qwen2.5-Omni-7B
28
18,944
530,432
2,652
Qwen3-Omni-30B
48
128×768
4,718,592
23,592
Kimi-Audio
28
18,944
530,432
2,652
MiniCPM-o-4.5
36
12,288
442,368
2,211
Gemma-4-12B
48
15,360
737,280
3,686
Appendix
Table 4: Decoder MLP sizes of the models that reach identification on Pitt (Audio-Flamingo-Next, Voxtral-Small-24B and MOSS-Audio-8B do not meet the analysis-eligibility criteria, § 4 ). For Qwen3-Omni-30B a neuron is one unit of one expert.
eval.
identification
Model
Cond.
UAR
UAR [95% CI]
Gemma-4-12B
AT
0.67
0.663 [0.608, 0.715]
AT-chat
0.68
0.639 [0.582, 0.695]
Kimi-Audio
A
0.67
0.634 [0.573, 0.690]
AT
0.64
0.615 [0.554, 0.673]
AT-chat
0.64
0.589 [0.528, 0.652]
Appendix
Table 5: Analysis eligibility recomputed on the Pitt identification split for the 12 cells that reach identification. “eval.”: UAR on the evaluation split (Table 9 ). The eligible identification recordings of each cell are in Table 13 .
HC
MCI
Dementia
Labeled style ( n )
99
124
38
Plain style ( n )
101
26
12
Plain style (%)
50.5
17.3
24.0
Fillers / 100 words, labeled
3.9
3.7
2.8
Fillers / 100 words, plain
1.4
2.8
3.7
Median words, labeled
163
127
120
Appendix
Table 6: PROCESS-2 Cookie Theft transcripts by transcription style (all 400 participants, before exclusions).
CHAT
Rendered as
(.) , (..) , (...)
(short pause), (pause), (long pause)
timed pause (2.5)
(pause 2.5s)
filler &-uh
uh
fragment &+wa
wa-
[/] , [//] , [///]
[repeats], [self-correction], [reformulation]
[/-] , [/?]
[false start], [unclear retracing]
Appendix
Table 7: Annotated rendering (convention v1 ).
Analysis
Null
K
Family
Ablation, own condition (§ 5.1 )
matched random masks
20
24 masks
Ablation, own condition (§ 5.1 )
sets matched on activation statistics
20
8 masks
Ablation, transfer to T (§ 5.3 )
matched random masks under T
20
24 masks
Class-mean shift (§ 5.1 )
norm-matched random masks
20
21 mask– α cells
CICN masks on gender (§ 5.1 )
matched random masks
20
cross-task, ∣Δℓ∣ only
Span masks (App. F.1 )
matched random masks
20
36 masks
Appendix
Table 8: Overview of the analyses: the null each is compared with, the number K of random masks per real mask, and the family over which the Benjamini–Hochberg correction runs. Unless stated, each mask enters with the conjunction of its two co-primary metrics. Details and caveats are in the sections given.
Model
Cond.
UAR
95% CI
F1
P(CI)
pred. CI
Qwen2.5-Omni-7B
T
0.664
[0.580, 0.751]
0.634
0.464
0.430
A
0.587
[0.527, 0.644]
0.471
0.342
0.159
AT
0.736
[0.646, 0.817]
0.715
0.492
0.495
AT-chat
0.582
[0.542, 0.629]
0.435
0.377
0.103
Qwen3-Omni-30B
T
0.736
[0.668, 0.806]
0.682
0.343
0.346
A
0.726
[0.636, 0.808]
0.699
0.432
0.458
Appendix
Table 9: Unmodified Pitt results. “pred. CI” is the fraction of items predicted CI.
Model
Cond.
UAR
95% CI
F1
P(CI)
pred. CI
Qwen2.5-Omni-7B
T
0.549
[0.468, 0.652]
0.553
0.330
0.121
A
0.568
[0.480, 0.665]
0.576
0.310
0.091
AT
0.606
[0.506, 0.719]
0.618
0.369
0.131
Qwen3-Omni-30B
T
0.487
[0.467, 0.500]
0.438
0.077
0.020
A
0.525
[0.500, 0.583]
0.494
0.086
0.010
AT
0.562
[0.494, 0.661]
0.565
0.166
0.051
Appendix
Table 10: Unmodified PROCESS-2 results, pooled over transcription styles.
T
A
AT
Model
lab.
plain
lab.
plain
lab.
plain
Qwen2.5-Omni-7B
0.59
0.51
0.56
0.58
0.67
0.44
Qwen3-Omni-30B
0.47
0.50
0.50
0.60
0.54
0.60
Kimi-Audio
0.53
0.55
0.66
0.59
0.70
0.53
MiniCPM-o-4.5
0.62
0.56
0.67
0.44
0.61
0.48
Gemma-4-12B
0.71
0.73
0.57
0.43
0.47
0.61
Appendix
Table 11: Unmodified PROCESS-2 UAR within transcription style (lab.: labeled; plain: plain prose; Table 6 ). Labeled style: 54 evaluation recordings (39 NC / 15 CI); plain style: 45 (40 NC / 5 CI). With 5 CI recordings, each plain-style value moves by 0.10 per CI recording.
Pitt
PROCESS-2
Model
AT − T
AT − A
chat − AT
AT − T
AT − A
Qwen2.5-Omni-7B
+0.07
+0.15 ∗
− 0.15 ∗
+0.06
+0.04
Qwen3-Omni-30B
+0.01
+0.02
+0.00
+0.08
+0.04
Kimi-Audio
+0.01
− 0.04
+0.00
+0.13
+0.01
MiniCPM-o-4.5
+0.09
− 0.03
+0.03
− 0.03
− 0.04
Gemma-4-12B
+0.10
+0.07
+0.02
− 0.20 ∗
− 0.02
Appendix
Table 12: Paired UAR differences between input conditions of the unmodified model ( chat = AT-chat ). ∗ marks a significant paired-bootstrap contrast (Appendix B ).
eligible / total
tokens pooled ( N=15 )
Model
Cond.
NC
CI
NC
CI
Pitt
Qwen2.5-Omni-7B
AT
24 / 152
16 / 145
42,988
61,434
Qwen3-Omni-30B
A
64 / 152
20 / 145
21,960
23,438
AT
47 / 152
46 / 145
23,286
28,064
AT-chat
73 / 152
19 / 145
25,008
36,058
Appendix
Table 13: Identification recordings that the unmodified model classifies correctly under both option orders with margin ≥0.1 (“eligible”), and the prompt tokens pooled from the 15 used per class (both orders, span all ).
Activity-matched sets
Prompt variants, q
Model
Cond.
Mask
Δℓ real
Δℓ sets
p
q
paraphrase
numeric labels
Qwen3-Omni-30B
AT
NC
+1.77
− 0.16 ± 0.76
0.030
0.040
< 0.001
< 0.001
MiniCPM-o-4.5
A
CI
− 2.51
− 0.27 ± 0.95
0.033
0.040
< 0.001
0.006
MiniCPM-o-4.5
AT
CI
− 2.63
− 0.64 ± 0.86
0.035
0.040
< 0.001
0.002
Kimi-Audio
AT-chat
NC
+1.60
+0.05 ± 0.60
0.033
0.040
0.006
0.005
Kimi-Audio
AT
NC
+1.46
− 0.05 ± 0.56
0.020
0.040
–
–
Appendix
Table 14: Robustness of the eight strongest Pitt CICN masks under activity-matched and prompt-variant controls. The real-mask Δℓ is retained for effect-size context; its uniform-random comparison is shown in Figure 1 . ΔP(CI) also enters each significance test. p is the uncorrected conjunction value; the equal q values follow from the Benjamini–Hochberg step-up. The first four rows are the primary masks. –: not evaluated under that prompt variant. † indicates an activity-matched comparison with a larger effect in the opposite direction, which is not counted as passing.
main
N=5
N=10
r=0.1%
r=1%
Model
Cond.
Mask
ΔP
flip
ΔP
flip
ΔP
flip
ΔP
flip
ΔP
flip
Kimi-Audio
A
NC
+0.139
0.50 §
+0.125
0.63
+0.105
0.42
+0.011
0.17
+0.220
0.79 §
Kimi-Audio
AT
NC
+0.228
0.95 §
+0.164
0.57
+0.132
0.67 §
+0.046
0.33 §
+0.262
1.00 §
Kimi-Audio
AT-chat
NC
+0.267
0.88 §
+0.250
0.88 §
+0.207
0.77 §
+0.152
0.58 §
+0.064
0.35
MiniCPM-o-4.5
A
CI
− 0.212
0.38 §
− 0.105
0.12 §
− 0.189
0.36 §
− 0.105
0.14 §
− 0.203
0.41 §
MiniCPM-o-4.5
AT
CI
− 0.253
0.55 §
− 0.355
0.68 §
− 0.133
0.23
− 0.087
0.16
− 0.268
0.68 §
Appendix
Table 15: Sensitivity of Pitt ablation effects to identification-pool size and CICN mask size for the eight strongest masks. The main setting is N=15 , r=0.5% ; each variant uses its own matched random controls. ΔP=ΔP(CI) ; flip denotes flipown .
pool (30)
all identification (297)
Model
Cond.
Mask
raw
adjusted
kept
raw
adjusted
kept
Kimi-Audio
A
NC
0.127
0.086
68%
0.051
0.033
65%
Kimi-Audio
AT
NC
0.092
0.080
86%
0.046
0.042
91%
Kimi-Audio
AT-chat
NC
0.094
0.069
73%
0.039
0.032
82%
MiniCPM-o-4.5
A
CI
0.139
0.080
58%
0.056
0.035
61%
MiniCPM-o-4.5
AT
CI
0.119
0.078
66%
0.054
0.035
64%
Appendix
Table 16: Class gap in CICN activation probability (mask’s class minus the other class, averaged over mask neurons) before and after adjustment for duration, silence proportion, and speech rate, shown for the selection pool and all identification recordings. “kept” is the adjusted/raw gap ratio.
CICN mask → gender
gender mask → CI
CICN cell (own ∣Δℓ∣ on CI)
real
random
male
female
Qwen3 AT NC (1.87)
0.62
0.49 ± 0.08
0.87 (0.65 ± 0.09)
0.97 (0.70 ± 0.15)
Qwen3 AT CI (1.10)
1.61 ∗
0.48 ± 0.05
(same cell as above)
Kimi AT NC (1.46)
0.21
0.22 ± 0.08
0.89 ∗ (0.40 ± 0.12)
0.35 (0.41 ± 0.17)
Kimi AT-chat NC (1.60)
0.23
0.21 ± 0.05
0.89 (0.42 ± 0.21)
0.30 (0.57 ± 0.28)
MiniCPM A CI (2.51)
1.35 ∗
0.57 ± 0.27
0.42 (0.52 ± 0.25)
0.47 (0.44 ± 0.20)
Appendix
Table 17: Cross-task ablation control measured by ∣Δℓ∣ . Left: CICN masks evaluated on gender; right: gender masks identified in the same model and condition and evaluated on CI. The own-task value in parentheses is the mean absolute per-item change, which is at least as large as the magnitude of the signed mean Δℓ on the same recordings (Table 14 ). ∗ marks cells that pass the predefined cross-task test; others are n.s.
ΔP(CI)
Δℓ
flipown
Model / cond. / mask
α
real
norm-m.
p
real
norm-m.
p
q
real
norm-m.
Kimi AT NC
−4
+0.197
− 0.010
0.007
+1.27
− 0.06
0.006
0.072
0.90 §
0.13
Kimi AT NC
+4
− 0.148
+0.020
0.024
− 0.91
+0.12
0.027
0.094
0.20 §
0.03
Kimi AT-chat NC
−4
+0.225
+0.002
0.010
+1.25
+0.01
0.007
0.072
0.77 §
0.14
Kimi AT-chat NC
+1
− 0.059
+0.002
0.013
− 0.31
0.00
0.013
0.072
0.14 §
0.02
Kimi AT-chat NC
+4
− 0.220
+0.001
0.017
− 1.23
− 0.01
0.015
0.072
0.57 §
0.09
Appendix
Table 18: Steering against norm-matched random directions; “norm-m.” gives the matched-control mean.
Model
Mask
A – AT
A – AT-chat
AT – AT-chat
Qwen3-Omni-30B
NC
0.28
0.32
0.42
CI
0.19
0.57
0.29
Kimi-Audio
NC
0.19
0.20
0.30
CI
0.20
0.28
0.30
MiniCPM-o-4.5
NC
0.33
0.24
0.21
CI
0.40
0.24
0.22
Appendix
Table 19: Jaccard overlap between ConAct masks (span all unless stated). The Jaccard expected for independent masks with the same per-layer counts is 0.002–0.008 in every cell.
Figure 4: Layer distribution of the ConAct masks (span all ): share of a mask’s neurons in each decoder layer, one line per identification condition. Rows are models and corpora; the y axis differs between rows.
evaluated under
Model (NC mask)
span
T
A
AT
Kimi-Audio
audio
+0.27 ∗ (4.5)
+0.29 ∗ (3.3)
+0.27 ∗ (2.9)
audio (tok.-matched)
+0.28 ∗ (4.7)
+0.34 ∗ (5.4)
+0.28 ∗ (3.8)
text
− 0.04 ( − 1.4)
− 0.07 ( − 1.3)
− 0.02 ( − 0.3)
Qwen3-Omni-30B
audio
+0.31 ∗ (9.1)
+0.16 ∗ (5.5)
+0.17 ∗ (6.1)
audio (tok.-matched)
+0.19 ∗ (4.1)
+0.08 (1.7)
+0.09 ∗ (2.6)
Appendix
Table 20: Span dissociation within AT : ΔP(CI) when ablating NC masks identified from the audio-token span, a token-matched subset of that span, or the transcript-token span; z relative to matched random masks is shown in parentheses.
ΔP(CI)
Δℓ
Model
Cond.
Mask
real
random
p
real
random
p
q
flipown ( n )
Kimi-Audio
AT
NC
+0.298
+0.002
< 0.001
+1.72
− 0.01
0.001
0.006
0.65 (65)
MiniCPM-o-4.5
A
CI
− 0.143
+0.018
0.001
− 2.26
+0.14
< 0.001
0.006
– (7)
Qwen3-Omni-30B
A
NC
+0.112
+0.019
< 0.001
+1.85
+0.27
< 0.001
0.006
0.06 (79)
Qwen3-Omni-30B
AT
NC
+0.110
+0.015
0.001
+1.91
+0.21
< 0.001
0.006
0.14 (77)
Kimi-Audio
A
NC
+0.202
− 0.008
0.002
+1.15
− 0.10
0.010
0.033
0.29 (75)
Appendix
Table 21: Pitt CICN masks ablated on PROCESS-2 with Pitt-matched random controls. Upper part: A and AT masks; lower part: Pitt AT-chat masks evaluated under PROCESS-2 AT . n is the denominator of flipown ; flip rates are omitted for n<10 . The two blocks are separate correction families.
ΔP(CI)
flipown
Model
Cond.
Mask
lab.
plain
lab.
plain
Pitt masks
Kimi-Audio
A
NC
+0.202 ( − 0.013)
+0.203 ( − 0.002)
0.28 (0.03)
0.31 (0.04)
Kimi-Audio
AT
NC
+0.282 ( − 0.001)
+0.317 (+0.006)
0.65 (0.06)
0.65 (0.06)
Kimi-Audio
AT-chat → AT
NC
+0.291 (+0.037)
+0.309 (+0.043)
0.71 (0.09)
0.59 (0.11)
MiniCPM-o-4.5
A
CI
− 0.145 (+0.015)
− 0.142 (+0.022)
–
–
Appendix
Table 22: Within-style PROCESS-2 ablation: real-mask effect with the matched-random mean in parentheses (recording counts in Table 11 ). “–” indicates fewer than 10 baseline-correct items of the mask’s class, so no flip rate is reported. These values are descriptive and are not tested separately.
ΔP(CI)
Δℓ
Model
Cond.
Mask
real
random
p
real
random
p
q
Pitt held-out (16 MCI, 4 vascular, 3 other)
Kimi-Audio
A
NC
+0.200
− 0.027 ± 0.085
0.018
+1.09
− 0.17
0.026
0.034
Kimi-Audio
AT
NC
+0.301
− 0.002 ± 0.068
< 0.001
+1.84
− 0.03
< 0.001
0.001
Kimi-Audio
AT-chat
NC
+0.290
+0.021 ± 0.056
< 0.001
+1.75
+0.11
< 0.001
0.001
MiniCPM-o-4.5
A
CI
− 0.203
+0.016 ± 0.054
0.001
− 2.20
+0.17
0.002
0.003
Appendix
Table 23: CICN ablation on held-out diagnoses with matched random controls. The eight strongest Pitt masks are used; on PROCESS-2, AT-chat masks are evaluated under AT . Flip rates are undefined because these recordings have no NC/CI evaluation label. Each block is a separate correction family.
Kimi-Audio AT NC
Kimi-Audio AT-chat NC
MiniCPM-o-4.5 A CI
MiniCPM-o-4.5 AT CI
Ablated positions
orig.
orig.
para.
num.
orig.
para.
num.
orig.
para.
num.
all (Figure 1 )
+0.228 ∗
+0.267 ∗
–
–
− 0.212 ∗
–
–
− 0.253 ∗
–
–
all but the last
+0.222 ∗
+0.252 ∗
+0.179 ∗
+0.223 ∗
− 0.217 ∗
− 0.324 ∗
− 0.135 ∗
− 0.261 ∗
− 0.330 ∗
− 0.218 ∗
audio span
+0.069 ∗
+0.038
+0.030
+0.041
− 0.100 ∗
− 0.109 ∗
− 0.085 ∗
− 0.084 ∗
− 0.083 ∗
− 0.083 ∗
transcript span
+0.022 ∗
+0.016
–
–
–
–
–
− 0.034 ∗
–
–
template
+0.152 ∗
+0.227 ∗
–
–
− 0.123 ∗
–
–
− 0.161 ∗
–
–
Appendix
Table 24: Position-restricted ablation on Pitt under the original prompt (orig.) and the two prompt variants (para.: paraphrase; num.: numeric option labels; Appendix D.5 ): ΔP(CI) when the mask is ablated only at the listed prefill positions. “Template” is every position outside the audio and transcript spans except the last; “before the content” cannot attend to the recording. ∗ marks cells that pass the predefined test for that position set and prompt. –: not run (the variants cover three masks and four position sets; MiniCPM-o-4.5 A has no transcript span); the random-mask row is given for the original prompt. The shares quoted in Appendix G.2 are computed on Δℓ .
Figure 5: Logit lens on Pitt. Top: AUC of the option-logit readout from each layer’s residual stream at the final prompt position, one line per input condition. Bottom: depth distribution of the CICN masks used in the position-restricted ablations (blue: NC masks; red: CI masks).
Figure 6: Token-level correlates of CICN activity on Pitt. Left: CI-direction score (positive: CI-like) around fillers, rendered pauses, and Cookie Theft information-unit words, relative to each recording’s transcript mean; thin gray lines are matched random masks and “rec.” counts recordings containing the event. Right: within-recording token-category contrasts, with matched-random mean ± 2 sd; filled markers pass the predefined token-level test. Values, including AT , are in Table 25 .
Kimi-Audio
MiniCPM-o-4.5
Contrast
Mask
AT
AT-chat
AT
AT-chat
info. unit − content
CI
− 0.055 ∗ ( − 0.008)
− 0.047 ∗ ( − 0.008)
− 0.128 ∗ ( − 0.039)
− 0.128 ∗ ( − 0.036)
NC
− 0.039 ∗ ( − 0.010)
− 0.056 ∗ ( − 0.015)
− 0.076 ∗ ( − 0.036)
− 0.076 ∗ ( − 0.032)
filler − content
CI
–
+0.185 ∗ (+0.058)
–
+0.332 ∗ (+0.088)
fragment − content
CI
–
+0.210 ∗ (+0.088)
–
+0.350 ∗ (+0.103)
pause − content
CI
–
+0.070 ∗ (+0.001)
–
+0.244 ∗ (+0.083)
Appendix
Table 25: Token-level within-recording differences in CI-direction score on Pitt (positive: more CI-like); matched-random means are in parentheses. Pause, filler, fragment, and retracing markers occur only in AT-chat . ∗ marks contrasts that pass the predefined token-level test; ‡ marks the same criterion with an effect opposite to the expected sign.
Figure 7: Predictive utility of CI-contrast neurons: whole-mask readouts. Blue, gray and green are supervised linear readouts trained on Pitt (CICNs, matched-random neurons with mean ± 2 sd over 20 masks, and all neurons); yellow is the model’s zero-shot answer, joined to the CICN readout by a line. (a) Within-corpus readout: CICN readouts reach 0.92–0.97 AUC on Pitt, against 0.67–0.84 for the model’s own answer. (b) Cross-corpus generalization: the same readouts evaluated on PROCESS-2, with rows in the same order as (a).
Large audio-language models (LALMs) frequently exhibit object hallucination, affirming absent sounds. The dominant mitigation, Audio-Aware Decoding (AAD), contrasts logits under real audio against an all-zero "blank" waveform, flagged by prior review as out-of-distribution. We introduce a black-box causal test that isolates the claimed evidence itself: localize a claim's evidence region via CLAP similarity, remove it by spectral gating, and compare against a random-window control across a 2x2x2 grid on three LALMs. On Qwen2-Audio-7B, localized ablation's raw AUROC trails blank's (0.613 vs. 0.721); this is a duration confound, not a localization failure, since blank removes the entire clip versus the grid's 2-4 s. Holding duration fixed, localization beats random removal at every cell on both models. Raw detection value remains model-dependent, especially on Audio Flamingo 3. The same evidence corrects decoding: contrastive decoding combining blank and language-prior contrasts, gated and ensembled, raises Qwen2-Audio accuracy from 57.1% to 71.9%, beating AAD alone. A unified causal-grounding profile does not transfer between models, indicating different grounding regimes. The source code and a detailed report of our findings, including identified limitations, are available at https://github.com/khalooei/causa.
Mohammad Khalooei, Mohammad Sabokrou
Sharif University of Technology, Iran · Amirkabir University of Technology, Iran · New Uzbekistan University, Uzbekistan +1
Large audio-language models (LALMs) exploit multimodal evidence, yet task-irrelevant audio can alter text-reasoning decisions when listening is unnecessary. Aggregate Accuracy can hide this paired drift because audio-induced repairs and damages may cancel. Paired drift analysis and targeted interventions identify architecture-specific, intervention-sensitive late audio pathways as actionable control points. We introduce ICAP-Gate, which applies mechanism-guided, task-conditioned control to each model's pathway. Across four LALMs, two reasoning benchmarks, and environmental-sound and natural-speech interference, ICAP-Gate has lower point estimates for Influence Rate and Answer Flip than ungated inference in all 16 full-split model--condition evaluations. Fixed suppression degrades automatic speech recognition (ASR) across all four models, whereas ICAP-Gate matches ungated ASR performance by preserving the pathway for explicit audio-demand instructions. ICAP-Gate has lower paired-drift point estimates than mitigation prompting in all four evaluated settings and provides competitive stabilization relative to eight-sample Self-Consistency while using one generation per query; in controlled ARC measurements, Self-Consistency incurs 7.0--9.2× ungated latency. These results establish selective modality influence control as a design principle for robust multimodal reasoning.
Yulin Sun, Kele Xu, Yong Dou
College of Computer Science and Technology, National University of Defense Technology · State Key Laboratory of Complex & Critical Software Environment
A large audio language model (LALM) attaches a speech front end to a text language model (LM) that is already safety-aligned. When such a model refuses a harmful spoken request, is the refusal carried by the front end, or inherited from the text LM? We test this with causal interventions. At each model's audio-to-LM interface and at tested LM residual layers, we fit a direction separating harmful from benign prompts, ablate its component, and measure the resulting change in the model's first-token refusal margin. Four of the five models are evaluated under held-out category shift. Across five LALMs spanning three backbone families, with three models passing a baseline safety gate, the largest tested effects occur in a mid-to-late LM band, while ablations of the tested interface directions have little effect. On Qwen2.5-Omni, ablating the L16 direction changes the margin by -7.10, versus -0.013 at the projector. The audio pathway is still in use: zeroing the encoder output changes the margin by -4.7. In the same model, the contrast is linearly decodable at an early layer where single-layer ablation has little effect, and a direction fitted on the LM backbone alone transfers to the full audio model. Because harmful and benign prompts also differ in form, we interpret the direction as refusal-linked rather than harmfulness-specific. These interventions localize dependence of the refusal margin, not where refusal is computed. Moving the margin also does not always change what the model writes. Safety audits of these models should use interventions rather than rely on probes alone, and should examine the inherited text LM alongside the audio interface.
Leonardo Haw-Yang Foo, Hung-yi Lee
National Taiwan University, Taiwan · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE), Taiwan