Attention Sees, MLP Refuses: Tracing the Attention-to-MLP Safety Pathway in LLMs

Zixuan Weng; Haoqi He; Weifei Jin; Yue Huang; Yang Bai

cs.AIOct 9, 2026

Attention Sees, MLP Refuses: Tracing the Attention-to-MLP Safety Pathway in LLMs

Authors: Zixuan Weng, Haoqi He, Weifei Jin, Yue Huang, Yang Bai

Organizations: Georgia Institute of Technology · Independent Researcher · Duke University · University of Notre Dame

Abstract

Safety-aligned large language models (LLMs) remain vulnerable to jailbreaks and can over-refuse benign requests. Understanding these failures requires distinguishing harmfulness recognition from the computation that writes refusal. We reframe this as a pathway question and make it measurable on residual-stream updates in decoder-only Transformers, separating the harmfulness information a module update exposes from the change it produces in readable refusal evidence. Across 18 Base-Instruct pairs spanning Qwen, Llama, and Gemma, harmfulness is decoded more readily from attention updates, particularly in early layers, whereas MLP updates contribute more strongly to harm-selective refusal writing in aligned models. Post-training strengthens MLP refusal writing in all 18 pairs without systematically improving harmfulness decodability, while positive MLP writing becomes more concentrated in deeper layers after post-training. Across six reasoning-enabled models, harmfulness remains decodable during reasoning while strong refusal writing concentrates at answer onset, a pattern that persists with reasoning disabled. Causal interventions further support this pathway: resetting downstream MLP updates to their clean values substantially reduces the refusal shift caused by upstream attention interventions. Building on these findings, we introduce Sentinel, which combines an attention-derived perception score P with an MLP-derived decision score D. Their joint profiles characterize jailbreaks and over-refusals, and conditional routing with both signals achieves a lower aggregate error rate than single-signal routing and evaluated fixed reminders across eight models.

Figures & tables

Figure 1: Early attention exposes harmfulness information and increases residual-stream separability. (a) Layerwise probe AUC on attention and MLP updates. (b) Change in residual-stream probe AUC after adding each update. Curves show cross-model medians over relative depth; shaded regions denote interquartile ranges.

Figure 2: Harmfulness is already decodable in Base checkpoints, while Instruct checkpoints exhibit stronger and more specific MLP refusal writing. Comparison across 18 Base–Instruct pairs. (a) layer-averaged probe

\mathrm{AUC}^{U}

; (b) signed cumulative writing,

\sum_{\ell}\Delta\mathrm{RWS}^{U}_{\ell}

; (c) specificity relative to matched control vocabularies across 18 Base–Instruct pairs. Specificity is the ratio of positive refusal-writing mass to the median positive mass over 16 matched control vocabularies.

Figure 3: Positive MLP refusal writing becomes more concentrated in late layers after post-training. (a) Cumulative positive

\Delta\mathrm{RWS}^{M}

mass, normalized to one for each model. (b–c) Per-model-normalized layerwise profiles for Instruct and Thinking checkpoints, respectively. Normalization preserves the shape and location of writing but does not support comparisons of absolute magnitude.

Panel of Figure 4 — Figure 4: Harmfulness remains decodable during reasoning, while strong MLP refusal writing appears at answer start. Results across six Qwen3 and Qwen3.5 models. (a) Harmfulness probe Median probe $\mathrm{AUC}^{U}$ at six generation positions with reasoning enabled. Lines show cross-model medians, shaded regions denote interquartile ranges, and shuffled-label controls remain near chance. (b) Cumulative harm-selective refusal writing, $\sum_{\ell}\Delta\mathrm{RWS}^{U}_{\ell}(t)$ . Purple curves show cross-model medians with thinking enabled, and the shaded band denotes the MLP interquartile range. Blue markers show the corresponding thinking-off values at answer start. The background shading marks the reasoning span.

Figure 5: Prompt categories in the Sentinel

P

–

D

space. Each panel shows one of three non-reasoning models. Points represent group-mean perception scores

P

and decision scores

D

, standardized using each model’s calibration data and the original

D

layer-selection rule. Circles denote jailbreak attack groups, with area encoding ASR. Filled and open squares denote over-refused (OR) and normally answered (OK) subsets of XSTest-safe and EVOREFUSE-TEST, respectively. Diamonds mark the AlpacaEval and AdvBench reference groups. Axis ranges vary across panels; zero is not a behavioral decision threshold.

Method	Jailbreak ASR $\downarrow$	Benign False refusal $\downarrow$	Over-refusal False refusal $\downarrow$	AVG $\downarrow$
No reminder ( $N$ )	95.97	0.00	88.73	61.57
Helpfulness reminder ( $H_{\mathrm{end}}$ )	68.22	0.00	59.95	42.72
Self-Reminder ( Xie et al., 2023 )	41.03	0.13	83.64	41.60
In-Context Defense (ICD) ( Wei et al., 2026 )	50.03	0.80	72.59	41.14
Balanced reminder ( $B_{\mathrm{end}}$ )	45.73	0.13	76.70	40.85
Llama Guard 3 ( Dubey et al., 2024 )	26.58	1.08	92.53	40.06

Table 1: Conditional reminders versus fixed-reminder baselines. All values are percentages, averaged equally across eight non-reasoning models. AVG is the arithmetic mean of jailbreak ASR, benign false refusal rate, and over-refusal false refusal rate. Lower is better for every column. Rows are ordered by decreasing AVG; the best value among the displayed methods in each column is bold.

Appendix figures & tables42 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Module-update counterfactuals in non-reasoning configurations. The left column patches Attention updates, and the right column patches MLP updates. The upper panels report changes in final refusal evidence,

\Delta\mathrm{RES}_{\mathrm{final}}

; the lower panels report changes in same-layer MLP writing,

\Delta\mathrm{RWS}_{\ell}^{M}

. Red denotes harmful-donor-to-benign-target patching, and blue denotes benign-donor-to-harmful-target patching. Both quantities are divided by the same within-configuration clean harmful–benign final refusal-evidence gap

G_{c}

. Lines show medians across 21 non-reasoning configurations at matched relative depths; shaded bands show interquartile ranges, not confidence intervals. Attention-update patching shifts the final refusal readout with a comparatively small aggregate same-layer MLP response, whereas MLP-update patching produces directionally aligned local and final responses, particularly in later layers.

Figure 7: Module-update counterfactuals at the answer boundary after reasoning. The left column patches Attention updates, and the right column patches MLP updates. The upper panels report changes in final refusal evidence,

\Delta\mathrm{RES}_{\mathrm{final}}

; the lower panels report changes in same-layer MLP writing,

\Delta\mathrm{RWS}_{\ell}^{M}

G_{c}

. Lines show 10 reasoning configuration medians at matched relative depths; shaded bands show interquartile ranges, not confidence intervals. The aggregate results show a similar local–final distinction to the non-reasoning analysis, extending the intervention evidence to answer formation after reasoning.

Figure 8: Joint path-cutting intervention design. We first patch donor Attention updates at the frozen

P

layers, then either allow downstream computation to proceed normally or restore the recipient’s clean MLP updates at the frozen

D

layers.

Figure 9: Downstream MLP restoration blocks upstream Attention effects. Each point shows the mean over 100 matched pairs for one configuration; error bars are 95% confidence intervals obtained by prompt-pair bootstrap. Filled circles show the total

P

-patch effect

T

, open circles show the residual effect

R

after restoring clean

D

, and the connecting segment shows the blocked component

B

. All effects are divided by the same clean harmful–benign gap

G_{c}

used in Figure 6 and Figure 7 . For the benign-to-harmful direction, we flip the sign so that positive values in both panels indicate movement toward donor-consistent behavior. Right-side percentages report

B/T

; “n.s.” indicates that the 95% bootstrap confidence interval for

T

includes zero.

Figure 10: Generation-level behavioral outcomes. The main number in each cell is the number of donor-consistent labels out of 100 (refusal for harmful-to-benign; compliance for benign-to-harmful), as assigned by the GPT-based judge.

Model	MLP $B_{M}/T$	Attention $B_{A}/T$	$B_{M}-B_{A}$ [95% CI]
Llama-3.1-8B	26.5%	$-15.4\%$	2.3851 [2.0207, 2.7460]
Qwen3-8B	79.4%	$-34.6\%$	8.3055 [7.4179, 9.1967]
Gemma4-12B	82.8%	19.2%	6.2235 [5.5384, 6.9308]

Appendix

Table 2: Held-out matched module-restoration control using the strict post-

P

portion of each onset-selected

D

component.

B_{M}/T

and

B_{A}/T

are ratios of direction-signed means pooled over 50 held-out prompt pairs and both patching directions. The final column reports the mean within-observation MLP-minus-Attention difference in each model’s final refusal-evidence units, pooling the 100 prompt–direction observations; its 95% bootstrap confidence interval resamples those observations (10,000 replicates; seed 20260924).

Model	Direction	Selected $D$	Random $C$	$\Delta_{\mathrm{DC}}$ [95% CI]
Llama-3.1-8B	harmful $\rightarrow$ benign	64.4%	49.3%	0.0370 [0.0267, 0.0473]
Qwen3-8B	harmful $\rightarrow$ benign	65.4%	30.4%	0.2148 [0.1819, 0.2496]
Qwen3-8B	benign $\rightarrow$ harmful	69.3%	30.0%	0.1316 [0.1039, 0.1611]
Gemma4-12B	harmful $\rightarrow$ benign	82.3%	73.5%	0.0531 [0.0396, 0.0665]
Gemma4-12B	benign $\rightarrow$ harmful	86.6%	84.4%	0.0091 [0.0032, 0.0152]

Appendix

Table 3: Selected-

D

and fixed random post-

P

restoration controls. Blocking fractions are ratios of means,

B/T

. The final column reports the donor-aligned paired difference

\Delta_{\mathrm{DC}}=s(B_{D}-B_{C})/G_{c}

with a pointwise 95% bootstrap confidence interval. The Llama-3.1-8B benign-to-harmful direction is omitted because its total-effect interval includes zero.

Figure 11: Reasoning timelines across refusal readouts. Cumulative MLP writing

S^{M}(t)=\sum_{\ell}\Delta\mathrm{RWS}_{\ell}^{M}(t)

at six sampled positions for (a) Qwen3-8B and (b) Qwen3-14B under thinking-on. Colors denote the four readout constructions; the shaded strip marks the true answer boundary. In all eight model–readout configurations, writing at answer_start exceeds the maximum at the three sampled reasoning-mid positions, with gaps of 10.00–26.47. Positions are categorical; connecting lines do not represent measurements at intermediate tokens.

Figure 12: True-answer-boundary control under thinking-on and thinking-off. Each segment connects cumulative MLP writing at assistant_start (open circle) to writing at the true answer_start (filled circle). Rows correspond to readouts, while panels separate models and thinking conditions. Blue denotes thinking-off and purple denotes thinking-on. All eight model–readout configurations show positive gains in each condition: +2.58 to +19.97 under thinking-off and +7.80 to +24.15 under thinking-on. Segments show paired positions, not uncertainty intervals.

Figure 13: Layerwise accumulation of MLP writing across refusal readouts. Cumulative

\Delta\mathrm{RWS}^{M}

through each layer for (a) Llama3.1-8B, (b) Qwen3-8B, and (c) Qwen3-14B. Colors identify readouts; solid and dashed curves denote Instruct and Base, respectively. The shaded region marks the final quarter of network depth. Instruct is measured at the visible-answer boundary, using thinking-off for Qwen3; Base uses the final input-token position. The final Instruct–Base MLP layer-sum difference is positive in all 12 configurations, ranging from

+8.96

+21.49

Figure 14: Attention and MLP net writing across refusal readouts. Signed layer sums

S^{U}=\sum_{\ell}\Delta\mathrm{RWS}_{\ell}^{U}

for Attention (blue) and MLP (purple), measured in Instruct models at the visible-answer boundary, with Qwen3 in thinking-off mode. Gray segments connect the two module values within each configuration. MLP net writing exceeds Attention net writing in 11 of 12 configurations. Qwen3-8B with data-driven

k=8

has

S^{M}=7.16

and

S^{A}=7.73

. Segments represent module differences, not uncertainty intervals.

Model	$k$	Refusal tokens	Compliance tokens
Llama3.1-8B	3	’t , I , ␣can	␣of , The , Here
Llama3.1-8B	8	’t , I , ␣can , ␣request , ␣assist , ␣fulfill , ␣Is , ␣help	␣of , The , Here , ␣is , ␣are , ␣the , ’s , ␣sentence
Qwen3-8B	3	I , ␣cannot , ␣instructions	Here , ␣are , ␣the
Qwen3-8B	8	I , ␣cannot , ␣instructions , ␣request , ’m , ␣provide , ␣unable , ␣illegal	Here , ␣are , ␣the , ␣sentence , ␣five , ’s , ␣three , ’s
Qwen3-14B	3	I , ␣or , ’m	␣of , Here , ␣are
Qwen3-14B	8	I , ␣or , ’m , ␣unable , ␣any , ␣cannot , ␣assist , ␣instructions	␣of , Here , ␣are , ␣the , ␣in , The , To , ’s

Appendix

Table 4: Data-driven refusal and compliance token sets independently estimated from the prompt-disjoint calibration generations. The symbol ␣ denotes a leading whitespace in the token string.

Model	Readout	Maximum reasoning-mid $S^{M}$	answer_start $S^{M}$	Boundary gap
Qwen3-8B	Main token set	1.73	18.72	+16.99
	Arditi-style	-3.18	21.82	+25.00
	Data-driven ( $k=3$ )	0.59	21.91	+21.32
	Data-driven ( $k=8$ )	2.32	12.32	+10.00
Qwen3-14B	Main token set	1.95	17.92	+15.97
	Arditi-style	-0.43	26.05	+26.47

Appendix

Table 5: Cumulative MLP writing at the strongest sampled reasoning-mid position and at the true answer boundary under thinking-on.

Model	Readout	Thinking-off boundary gain	Thinking-on boundary gain
Qwen3-8B	Main token set	+11.50	+13.56
	Arditi-style	+8.19	+14.63
	Data-driven ( $k=3$ )	+9.26	+15.86
	Data-driven ( $k=8$ )	+2.58	+7.80
Qwen3-14B	Main token set	+18.18	+17.84
	Arditi-style	+17.54	+24.15

Appendix

Table 6: True-answer-boundary gains under thinking-off and thinking-on across alternative refusal readouts.

Model	Readout	$S^{A}$	$S^{M}$	$S^{M}-S^{A}$
Llama3.1-8B	Main / Arditi-style / $k=3$ / $k=8$	-0.05 / 3.84 / 0.47 / 0.51	12.32 / 11.10 / 14.65 / 14.61	12.37 / 7.25 / 14.18 / 14.10
Qwen3-8B	Main / Arditi-style / $k=3$ / $k=8$	-1.94 / 0.47 / -0.52 / 7.73	16.73 / 15.39 / 15.40 / 7.16	18.67 / 14.92 / 15.92 / -0.57
Qwen3-14B	Main / Arditi-style / $k=3$ / $k=8$	-1.46 / -2.19 / -0.83 / 3.08	18.28 / 19.57 / 19.50 / 13.21	19.74 / 21.77 / 20.34 / 10.13

Appendix

Table 7: Attention and MLP signed layer sums under the four refusal-readout constructions.

Method	Jailbreak ASR $\downarrow$	Benign false refusal $\downarrow$	Over-refusal false refusal $\downarrow$	AVG $\downarrow$
No reminder ( $\mathrm{N}$ )	96.28%	0.00%	93.33%	63.20%
Helpfulness reminder ( $\mathrm{H}_{\mathrm{end}}$ )	67.59%	0.00%	68.36%	45.32%
Balanced reminder ( $\mathrm{B}_{\mathrm{end}}$ )	53.47%	0.34%	85.77%	46.53%
Safety reminder ( $\mathrm{S}_{\mathrm{end}}$ )	30.89%	6.95%	92.71%	43.52%
Sentinel P+D — Main token set	40.00%	0.35%	59.76%	33.37%
Sentinel P+D — Arditi-style	38.01%	0.35%	67.98%	35.45%

Appendix

Table 8: Conditional-reminder routing under alternative refusal readouts. All values are macro averages over Llama3.1-8B-Instruct, Qwen3-8B-Instruct, and Qwen3-14B-Instruct. AVG is the equally weighted mean of Jailbreak ASR, Benign false refusal, and Over-refusal false refusal. These three-model averages are not directly comparable to the eight-model averages in Table 1 of the main text.

Figure 15: Matched-input, matched-position Base–Instruct control for six checkpoint pairs. Each panel shows the layerwise MLP

\Delta\mathrm{RWS}

profile for Base and Instruct checkpoints under the same plain-text response cue. The token sequence and measurement position are identical within each pair. Instruct checkpoints have larger signed layer sums in all six pairs, while their precise depth profiles remain sensitive to the shared interface context.

	MLP signed sum		Late-quarter positive-mass share		Peak depth
Checkpoint pair	Base	Instruct	Base	Instruct	Base	Instruct
Llama-3.1-8B	1.105	6.169	25.41%	22.60%	0.742	0.581
Qwen2.5-7B	4.893	14.652	72.47%	79.36%	0.852	0.852
Qwen3.5-9B	0.655	1.060	32.71%	21.52%	0.871	0.613
Gemma-2-9B	3.665	4.678	49.52%	43.16%	0.439	0.756
Gemma-3-12B	0.990	5.583	21.29%	60.81%	0.723	0.894

Appendix

Table 9: MLP refusal-writing statistics for six matched-input, matched-position checkpoint pairs. We report the signed MLP layer sum

S^{M}

, the late-quarter positive-mass share

C_{\mathrm{late}}^{M}

, and the normalized peak depth

d_{\mathrm{peak}}

Cohort	Source and behavioral condition	Maximum per model
Benign	AlpacaEval requests answered normally without a reminder	100
Raw harmful	AdvBench reference requests with independent behavioral judgments	100
Historical over-refusal	Allowed XSTest-safe or EvoRefuse requests refused without a reminder	100
Historical jailbreak	ReNeLLM, PAIR, AutoDAN, or GCG requests answered compliantly without a reminder	100
Matched XSTest	XSTest-safe requests answered normally without a reminder	50
Matched EvoRefuse	EvoRefuse requests answered normally without a reminder	50

Appendix

Table 10: Construction of the routing dataset.

Model	Jailbreak	Benign	Over-refusal
Llama-3.1-8B-Instruct	99	98	93
Qwen2.5-7B-Instruct	98	99	95
Qwen3-8B-Instruct	100	94	88
Qwen3-14B-Instruct	98	97	91
Gemma2-9B-Instruct	96	96	96
Gemma4-12B-Instruct	69	89	79

Appendix

Table 11: Requests with outputs for all four actions in the three main evaluation cohorts.

Router input	Dim.	Jailbreak ASR $\downarrow$	Benign False refusal $\downarrow$	Over-refusal False refusal $\downarrow$	AVG $\downarrow$
$P$ mean	1	57.34	0.13	71.40	42.96
$D$ mean	1	55.02	0.78	68.27	41.36
$P$ mean $+$ $D$ mean	2	50.01	0.13	67.89	39.34

Appendix

Table 12: Low-dimensional P/D control. Each router reads at most two scalars. All values are percentages, averaged equally across the same eight non-reasoning models as Table 1 . AVG is the arithmetic mean of the three error rates, and lower is better in every column.

Model	Layers	$\mathcal{L}_{P}$	$\mathcal{L}_{D}$
Llama-3.1-8B-Instruct	32	11–13	14–18
Qwen2.5-7B-Instruct	28	18–19	14–25
Qwen3-8B-Instruct	36	18–21	21–23
Qwen3-14B-Instruct	40	38–39	39
Qwen3.8-27B	64	42–48	47–53
Gemma-2-9B-IT	42	18–23	28–31

Appendix

Table 13: Automatically selected layers for the eight non-reasoning models used in Section 4 . Layer indices are zero-based;

a\text{--}b

denotes the contiguous interval from layer

a

through layer

b

Model	Mode	Attention	Position	$\mathcal{L}_{P}$	$\mathcal{L}_{D}$
Llama-3.1-8B-Instruct	non-reasoning	self-attention	assistant_start	11–13	14–18
Qwen3-8B-Instruct	non-reasoning	self-attention	assistant_start	18–21	21–23
Qwen3-8B-Instruct	reasoning	self-attention	answer_start	22–23	27–30
Qwen3.5-9B-Instruct	non-reasoning	DeltaNet	assistant_start	16	29–30
Qwen3.5-9B-Instruct	reasoning	self-attention	answer_start	19	21–22
Gemma-4-12B-IT	non-reasoning	sliding attention	assistant_start	25–28	32–35

Appendix

Table 14: Frozen layer configurations for the joint

P

-patch and

D

-restoration experiments. Non-reasoning and reasoning modes are treated as separate configurations and measured at assistant_start or answer_start , respectively.

Family	Scales	Pairs
Qwen2.5	3B, 7B, 14B	3
Qwen3	4B, 8B, 14B	3
Qwen3.5	2B, 4B, 9B	3
Llama	Llama3.2-1B, Llama3.2-3B, Llama3.1-8B	3
Gemma2	2B, 9B, 27B	3
Gemma3	4B, 12B, 27B	3

Appendix

Table 15: Base–Instruct pairs used in the representation and refusal-writing analyses. Each family–scale combination includes both the Base and corresponding Instruct checkpoint.

Mode

Models

NR (21)

Gemma2-9B-Instruct, Gemma2-27B-Instruct; Gemma3-4B-Instruct, Gemma3-12B-Instruct, Gemma3-27B-Instruct; Gemma4-12B-Instruct, Gemma4-31B-Instruct; Llama3.1-8B-Instruct, Llama3.2-1B-Instruct, Llama3.2-3B-Instruct; Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, Qwen2.5-14B-Instruct; Qwen3-4B-Instruct, Qwen3-8B-Instruct, Qwen3-14B-Instruct; Qwen3.5-4B-Instruct, Qwen3.5-9B-Instruct; Qwen3.5-27B, Qwen3.6-27B, and Qwen3.8-27B.

R (10)

Gemma4-12B-Instruct, Gemma4-31B-Instruct; MUSE-Glimmer-30B; Qwen3-4B-Instruct, Qwen3-8B-Instruct, Qwen3-14B-Instruct; Qwen3.5-4B-Instruct, Qwen3.5-9B-Instruct, Qwen3.5-27B, and Qwen3.6-27B.

Appendix

Table 16: Complete model inventory for the single-module counterfactual analysis. NR and R denote non-reasoning and reasoning configurations.

Figure 16: Individual module-update counterfactuals in non-reasoning configurations. Results are shown for all 21 non-reasoning configurations included in the Appendix B . Within each configuration, the left column patches attention updates and the right column patches MLP updates. Upper panels report changes in final refusal evidence,

\Delta\mathrm{RES}_{\mathrm{final}}

; lower panels report changes in same-layer MLP writing,

\Delta\mathrm{RWS}_{\ell}^{M}

. Red denotes harmful-donor-to-benign-target patching, and blue denotes benign-donor-to-harmful-target patching.

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Figure 39

Figure 40

Figure 41

Figure 42

Figure 17: Individual module-update counterfactuals at the answer boundary after reasoning. Results are shown for all 10 reasoning configurations included in the Appendix B . Interventions replace a single module update at answer_start following the cached reasoning context. Within each configuration, the left column patches attention updates and the right column patches MLP updates. Upper panels report changes in final refusal evidence,

\Delta\mathrm{RES}_{\mathrm{final}}

; lower panels report changes in same-layer MLP writing,

\Delta\mathrm{RWS}_{\ell}^{M}

. Red denotes harmful-donor-to-benign-target patching, and blue denotes benign-donor-to-harmful-target patching.

Figure 44

Figure 45

Figure 46

Figure 47

Metric	What it measures	Corresponding process
Harmfulness probe	Whether a module update contains linearly decodable harmfulness information.	Harmfulness recognition
Refusal Evidence Score (RES)	How strongly the current residual state supports refusal-oriented continuations over compliance-oriented continuations.	Refusal-evidence readout
Refusal Writing Score (RWS)	Whether adding a module update actually increases refusal evidence in the current residual state.	Refusal writing
$\Delta\mathrm{RWS}$	Whether harmful prompts receive stronger refusal writing than harmless prompts.	Harm-selective refusal writing

Appendix

Table 17: Measurement map for the recognition-to-writing analysis.

Refusal Behavior in Language Models

Refusal behavior in language models concerns when systems decline requests, how they detect risk, and how to balance safety with helpfulness.

See all 141 papers

LLMs Encode Harmfulness and Refusal Separately
Jul 16, 2025Jiachen Zhao, Jing Huang, Zhengxuan Wu +2LLM Interpretability LLM Refusal Behavior
Northeastern University · Stanford University
Why Do Aligned LLMs Remain Jailbreakable: Refusal-Escape Directions, Operator-Level Sources, and Safety-Utility Trade-off
May 9, 2026Yu Chen, Yuanhao Liu, Qi CaoLLM Alignment Language Model Steering
Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
Addressing Over-Refusal in LLMs with Competing Rewards
Jun 30, 2026Taeyoun Kim, Aviral KumarRL for Language Model Reasoning LLM Safety
Carnegie Mellon University · SynthLabs

Jul 16, 2025J/K move · Enter open · S save

LLMs Encode Harmfulness and Refusal Separately

Jiachen Zhao, Jing Huang, Zhengxuan Wu, David Bau+1

Northeastern University · Stanford University

LLM Interpretability LLM Refusal Behavior LLM Safety LLM Jailbreak Attacks Causal Interventions in Language Models Refusal Behavior in Language Models

Open paper AskPDF arXiv

LLMs are trained to refuse harmful instructions, but do they truly understand harmfulness beyond just refusing? Prior work has shown that LLMs' refusal behaviors can be mediated by a one-dimensional subspace, i.e., a refusal direction. In this work, we identify a new dimension to analyze safety mechanisms in LLMs, i.e., harmfulness, which is encoded internally as a separate concept from refusal. There exists a harmfulness direction that is distinct from the refusal direction. As causal evidence, steering along the harmfulness direction can lead LLMs to interpret harmless instructions as harmful, but steering along the refusal direction tends to elicit refusal responses directly without reversing the model's judgment on harmfulness. Furthermore, using our identified harmfulness concept, we find that certain jailbreak methods work by reducing the refusal signals without reversing the model's internal belief of harmfulness. We also find that adversarially finetuning models to accept harmful instructions has minimal impact on the model's internal belief of harmfulness. These insights lead to a practical safety application: The model's latent harmfulness representation can serve as an intrinsic safeguard (Latent Guard) for detecting unsafe inputs and reducing over-refusals that is robust to finetuning attacks. For instance, our Latent Guard achieves performance comparable to or better than Llama Guard 3 8B, a dedicated finetuned safeguard model, across different jailbreak methods. Our findings suggest that LLMs' internal understanding of harmfulness is more robust than their refusal decision to diverse input instructions, offering a new perspective to study AI safety.

Abstract

Refusal Behavior in Language Models

Explore similar work

LLMs Encode Harmfulness and Refusal Separately

Why Do Aligned LLMs Remain Jailbreakable: Refusal-Escape Directions, Operator-Level Sources, and Safety-Utility Trade-off

Addressing Over-Refusal in LLMs with Competing Rewards