Causal Localization of the Refusal Direction in Audio Language Models

Leonardo Haw-Yang Foo; Hung-yi Lee

cs.MMSep 6, 2026

Causal Localization of the Refusal Direction in Audio Language Models

Authors: Leonardo Haw-Yang Foo, Hung-yi Lee

Organizations: National Taiwan University, Taiwan · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE), Taiwan

Abstract

A large audio language model (LALM) attaches a speech front end to a text language model (LM) that is already safety-aligned. When such a model refuses a harmful spoken request, is the refusal carried by the front end, or inherited from the text LM? We test this with causal interventions. At each model's audio-to-LM interface and at tested LM residual layers, we fit a direction separating harmful from benign prompts, ablate its component, and measure the resulting change in the model's first-token refusal margin. Four of the five models are evaluated under held-out category shift. Across five LALMs spanning three backbone families, with three models passing a baseline safety gate, the largest tested effects occur in a mid-to-late LM band, while ablations of the tested interface directions have little effect. On Qwen2.5-Omni, ablating the L16 direction changes the margin by -7.10, versus -0.013 at the projector. The audio pathway is still in use: zeroing the encoder output changes the margin by -4.7. In the same model, the contrast is linearly decodable at an early layer where single-layer ablation has little effect, and a direction fitted on the LM backbone alone transfers to the full audio model. Because harmful and benign prompts also differ in form, we interpret the direction as refusal-linked rather than harmfulness-specific. These interventions localize dependence of the refusal margin, not where refusal is computed. Moving the margin also does not always change what the model writes. Safety audits of these models should use interventions rather than rely on probes alone, and should examine the inherited text LM alongside the audio interface.

Figures & tables

Intervention	Group-blocked	LOGO
Projector, rank-1	$-0.013$ [ $-0.033$ , $0.005$ ]	$-0.005$ [ $-0.018$ , $0.008$ ]
Projector, rank-16 subspace	$-0.14$ [ $-0.21$ , $-0.08$ ]	—
LM L08	$-0.10$ [ $-0.12$ , $-0.09$ ]	$-0.12$ [ $-0.14$ , $-0.11$ ]
LM L14	$-2.51$ [ $-2.64$ , $-2.39$ ]	$-3.29$ [ $-3.40$ , $-3.19$ ]
LM L16	$-7.10$ [ $-7.28$ , $-6.93$ ]	$-7.53$ [ $-7.67$ , $-7.38$ ]
LM L17 †	$-9.19$ [ $-9.38$ , $-8.99$ ]	—

Table 1: Qwen2.5-Omni held-out ablation effects.

\Delta G

on harmful clips with paired-bootstrap 95% CIs, under group-blocked (

N{=}238

) and leave-one-group-out (

N{=}584

) splits; random-direction nulls satisfy

|\Delta G|\leq 0.02

at every site.

\dagger

: dense group-blocked follow-up.

Figure 1: Ablation localizes the margin effect to an LM band. Held-out

|\Delta G|

at each intervention site, under group-blocked (

N{=}238

) and LOGO (

N{=}584

) category shift; shading marks L14–L26.

Test at L16	Value	Random control
Scalar rescue (recovery)	$0.950$ [ $0.947$ , $0.952$ ]	$0.000$ [ $-0.001$ , $0.001$ ]
Add $\mathbf{r}^{(16)}$ to benign ( $\Delta G$ )	$+7.24$	$-0.24$
Text-fit $\to$ audio pass	$0.99$	—
Audio-fit $\to$ text pass	$0.73$	—
Paraphrased audio ( $N{=}47$ )	$0.75$ [ $0.65$ , $0.84$ ]	—

Table 2: Readout-level rescue and transfer at L16 (Qwen2.5-Omni). Group-blocked split unless noted. Transfer rows ablate a direction fitted in one modality during the other modality’s forward pass, and give the result as a fraction of that pass’s own effect; the paraphrase row is a fraction of the verbatim effect on the same

47

items.

Model	H-refuse	Baseline $G$	LM site ( $\Delta G$ )	Harm flip
Qwen2.5-Omni	$1.00$	$6.85$	L16 ( $-7.10$ )	$0.75$
AF3 †	$0.68$	$2.97$	L20 ( $-4.22$ )	$0.24$
Voxtral †	$0.60$	$4.08$	L18 ( $-5.56$ )	$0.49$
Ultravox	$0.83$	$13.12$	L20 ( $-7.69$ )	$0.10$
DeSTA	$0.87$	$8.18$	L20 ( $-6.03$ )	$0.00$

Table 3: Cross-model summary. H-refuse is the baseline harmful-refusal rate; the gate requires it

\geq 0.70

with answer-benign

\geq 0.80

, and

\dagger

marks models that fail it. Harm flip is the fraction of baseline refusals ablation turns into harmful answers.

\Delta G

is held-out group-blocked ablation, except AF3, whose in-sample fit makes its magnitude incomparable; Baseline

G

and Harm flip come from a separate behavioral run.

Figure 2: Layer-wise localization across models. For each model,

|\Delta G|

is normalized by its peak over the tested LM sites and plotted against fractional LM depth; absolute magnitudes are therefore not comparable across models. Open markers left of the divider are the tested audio interfaces; their horizontal positions carry no depth information.

Refusal Behavior in Language Models

Refusal behavior in language models concerns when systems decline requests, how they detect risk, and how to balance safety with helpfulness.

See all 148 papers

Sep 24, 2026

AEGIS: Audio Endogenous Guarding via Internal Signals Against Large Audio-Language Model Jailbreaks

Large audio-language models (LALMs) expand language models to process and interpret audio, but also expose them to heterogeneous audio jailbreaks. We ask whether successful jailbreaks reflect failures to recognize harmful intent or failures occurring after such recognition. Layer-wise probing reveals the latter: risk-related information remains decodable from intermediate representations, yet the internal risk signal fails to translate into refusal in later-layer processing. We identify this discrepancy as the risk-to-refusal gap. Building on this finding, we propose AEGIS, a detect-then-intervene defense whose mid-layer risk gate selectively activates downstream safety adapters. Across six LALMs and three heterogeneous audio jailbreak benchmarks, AEGIS reduces the average unsafe rate from 17.9% to 0.4%, while causing only a marginal increase in over-refusal on benign inputs. These results establish selective internal intervention as an effective path toward more robust refusal in LALMs. The code is available at https://github.com/azzzzliao/aegis-audio-defense.

Yu-Ling Liao, Tzu-Chin Chiu, Zong-You Chen +2

National Taiwan University

Audio-Language Models Jailbreak Defense Language Model Robustness Refusal Behavior in Language Models

Ask PDF

Mar 6, 2026

Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering

Multimodal large language models can exhibit text dominance, over-relying on linguistic priors instead of grounding predictions in non-text inputs. One example is large audio-language models (LALMs) where decisive audio evidence can be under-utilized even when it contains important information. To address this issue we use mechanistic interpretability to identify a small set of audio-specialist attention heads whose audio attention yields a ``listening'' signal. We show that this signal increases when audio evidence affects the model's output, providing an indicator of audio engagement under standard prompting. Leveraging this localization, we construct an audio--silence steering direction and apply an inference-time activation intervention to the final representation, amplifying the model's audio effect. To demonstrate the utility of this intervention, we show on MMAU that this improves accuracy by up to +8.0 percentage points on two Qwen-based LALMs, without any parameter updates.

Neta Glazer, Lenny Aharon, Ethan Fetaya

Bar-Ilan University, Ramat Gan, Israel · Columbia University, New York, NY, USA

Audio-Language Models Audio-Language Model Evaluation Causal Interventions in Language Models

Ask PDF

Oct 9, 2026

Does Location Matter? A Black-Box Causal Audit Reveals Model-Dependent Audio Grounding in LALM Hallucination

Large audio-language models (LALMs) frequently exhibit object hallucination, affirming absent sounds. The dominant mitigation, Audio-Aware Decoding (AAD), contrasts logits under real audio against an all-zero "blank" waveform, flagged by prior review as out-of-distribution. We introduce a black-box causal test that isolates the claimed evidence itself: localize a claim's evidence region via CLAP similarity, remove it by spectral gating, and compare against a random-window control across a 2x2x2 grid on three LALMs. On Qwen2-Audio-7B, localized ablation's raw AUROC trails blank's (0.613 vs. 0.721); this is a duration confound, not a localization failure, since blank removes the entire clip versus the grid's 2-4 s. Holding duration fixed, localization beats random removal at every cell on both models. Raw detection value remains model-dependent, especially on Audio Flamingo 3. The same evidence corrects decoding: contrastive decoding combining blank and language-prior contrasts, gated and ensembled, raises Qwen2-Audio accuracy from 57.1% to 71.9%, beating AAD alone. A unified causal-grounding profile does not transfer between models, indicating different grounding regimes. The source code and a detailed report of our findings, including identified limitations, are available at https://github.com/khalooei/causa.

Mohammad Khalooei, Mohammad Sabokrou

Sharif University of Technology, Iran · Amirkabir University of Technology, Iran · New Uzbekistan University, Uzbekistan +1

Audio-Language Models Audio-Language Model Hallucination Detection Causal Interventions in Language Models Temporal Audio Grounding

Ask PDF

Abstract

Refusal Behavior in Language Models

Explore similar work

AEGIS: Audio Endogenous Guarding via Internal Signals Against Large Audio-Language Model Jailbreaks

Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering

Does Location Matter? A Black-Box Causal Audit Reveals Model-Dependent Audio Grounding in LALM Hallucination