Jailbreaking the Matrix: Nullspace Steering for Controlled Model Subversion

Vishal Pramanik; Maisha Maliha; Susmit Jha; Sumit Kumar Jha

cs.CRApr 11, 2026

Jailbreaking the Matrix: Nullspace Steering for Controlled Model Subversion

Authors: Vishal Pramanik, Maisha Maliha, Susmit Jha, Sumit Kumar Jha

Organizations: Department of Computer & Information Science & Engineering, University of Florida, Gainesville, FL 32611, USA · School of Computer Science, University of Oklahoma, Norman, OK 73019, USA · Computer Science Laboratory, SRI International, Menlo Park, CA 94025, USA

Abstract

Large language models remain vulnerable to jailbreak attacks -- inputs designed to bypass safety mechanisms and elicit harmful responses -- despite advances in alignment and instruction tuning. We propose Head-Masked Nullspace Steering (HMNS), a circuit-level intervention that (i) identifies attention heads most causally responsible for a model's default behavior, (ii) suppresses their write paths via targeted column masking, and (iii) injects a perturbation constrained to the orthogonal complement of the muted subspace. HMNS operates in a closed-loop detection-intervention cycle, re-identifying causal heads and reapplying interventions across multiple decoding attempts. Across multiple jailbreak benchmarks, strong safety defenses, and widely used language models, HMNS attains state-of-the-art attack success rates with fewer queries than prior methods. Ablations confirm that nullspace-constrained injection, residual norm scaling, and iterative re-identification are key to its effectiveness. To our knowledge, this is the first jailbreak method to leverage geometry-aware, interpretability-informed interventions, highlighting a new paradigm for controlled model steering and adversarial safety circumvention.

Figures & tables

Figure 1: HMNS successfully jailbreaks LLaMA 3.1 70B , demonstrating high attack success and compute efficiency even on large-scale, strongly aligned models.

Figure 2: Overview of HMNS procedure. Each step in the closed-loop intervention pipeline is shown: attribution identifies influential heads; masking suppresses them; nullspace steering computes an orthogonal direction; and a scaled perturbation is injected into the residual stream. If unsuccessful, the process repeats with updated attribution.

Model / Method	AdvBench		HarmBench		JBB-Behaviors		StrongReject
	ASR $\uparrow$	ACQ $\downarrow$	ASR $\uparrow$	ACQ $\downarrow$	ASR $\uparrow$	ACQ $\downarrow$	ASR $\uparrow$	ACQ $\downarrow$
LLaMA-2-7B-Chat
Foot-In-The-Door (FITD)	44.00 / 38.00	16.20	41.30 / 36.10	16.80	45.10 / 39.20	15.90	38.70 / 33.40	17.10
AutoDAN	72.60 / 66.20	12.80	69.10 / 63.20	13.10	73.40 / 67.50	12.50	66.20 / 60.40	13.60
ArrAttack	92.00 / 87.00	7.50	90.00 / 86.00	7.90	93.00 / 88.00	7.30	88.00 / 89.09	8.00
Many-shot JB (MSJ)	64.80 / 58.90	10.90	62.20 / 56.70	11.40	66.00 / 60.10	10.60	58.30 / 53.10	11.80

Table 1: Jailbreak effectiveness across evaluation benchmarks. We report Attack Success Rate (ASR, %; left/right = GPT4o/GPT-5) and Average Query Count (ACQ; lower is better) on four datasets—AdvBench, HarmBench, JBB-Behaviors, and StrongReject. Results are grouped by target LLM and averaged over three independent runs; best values are bolded and second-best are underlined . Our method (HMNS) achieves the strongest performance across all models and datasets, exceeding the next-best ASR by at least 5–6 percentage points while also attaining the lowest ACQ (

\approx 2

). The standard deviation across three independent runs is

<0.4

for all reported entries.

Variant	ASR (Fuzz/G4) $\uparrow$	ACQ $\downarrow$	IPC $\downarrow$	FPS $\downarrow$	LPS (s) $\downarrow$
HMNS (Full)	96.8 / 92.1	2.1	32	0.58	6.8
Remove masking (Projection-only)	89.5 / 84.0	2.4	30	0.61	7.1
Remove projection (Mask-only)	87.9 / 82.2	2.3	29	0.55	6.3
Inject direct dir. (Direct– $\phi$ , no nullspace)	88.7 / 83.1	2.5	32	0.63	7.2
No re-identification (freeze top– $K$ at $t{=}1$ )	90.2 / 85.0	2.7	24	0.60	7.0
Random– $K$ head selection	81.4 / 76.0	2.2	32	0.56	6.7

Table 2: Ablation results on Phi–3 Medium 14B (AdvBench) . Each row disables one component of HMNS to measure its contribution. Metrics: ASR (GPT4o / GPT-5), ACQ (external queries), IPC (internal passes), FPS (FLOPs per success;

\times 10^{12}

), and LPS (latency in seconds).

Method	IPC $\downarrow$	FPS (×10 12 ) $\downarrow$	LPS (s) $\downarrow$
AutoDAN	0	0.44	5.2
ArrAttack	0	0.62	6.7
Many-shot JB	0	0.91	8.0
PrisonBreak	0	0.98	8.9
HMNS (Ours)	32	0.53	6.1

Table 3: Compute cost comparison on LLaMA-3.1-70B (AdvBench) . Each value reports mean compute per successful attack. IPC counts internal passes only; FPS includes all internal and decoding FLOPs; LPS is wall-clock latency.

Attack / Defense	SMO	DPP	RPO	PAR	PAT	SAF	Avg
LLaMA-2-7B-Chat (AdvBench)
FITD	10.0 / 7.0	12.0 / 9.0	20.0 / 14.0	14.0 / 10.0	12.0 / 8.0	11.0 / 7.0	13.2 / 9.2
AutoDAN	15.0 / 11.0	24.0 / 18.0	38.0 / 28.0	28.0 / 20.0	22.0 / 16.0	18.0 / 12.0	24.2 / 17.5
ArrAttack	34.0 / 22.0	48.0 / 36.0	74.0 / 55.0	58.0 / 41.0	42.0 / 30.0	40.0 / 29.0	49.3 / 35.5
Tempest	26.0 / 19.0	42.0 / 31.0	68.0 / 50.0	52.0 / 38.0	36.0 / 25.0	33.0 / 24.0	42.8 / 31.2
HMNS (Ours)	40.0 / 25.0	54.0 / 41.0	82.0 / 61.0	64.0 / 45.0	48.0 / 33.0	47.0 / 34.0	55.8 / 39.8

Table 4: Effectiveness of jailbreak methods under defense across three models. We report Attack Success Rate (ASR, %) under six defenses (SMO, DPP, RPO, PAR, PAT, SAF) on LLaMA-2-7B-Chat , Phi-3 Medium 14B Instruct , and LLaMA-3.1-70B . Values are GPT4o / GPT-5 evaluations. Best results are bolded , second best are underlined .

Figure 3: Ablation studies on Phi-3 Medium 14B (AdvBench). (a) Attribution mechanisms: KL-divergence achieves the best ASR–compute tradeoff; removing proxy pre-selection preserves ASR but nearly triples IPC. (b) Nullspace and injection design: strict orthogonality and hard masking are critical; partial masking (

\gamma\!=\!0.5

) degrades ASR and raises ACQ. Bars show ASR (%, GPT-4o/GPT-5); overlaid lines show FPS (

\times 10^{12}

) in (a) and ACQ in (b).

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Model / Method	AdvBench		HarmBench		JBB-Behaviors		StrongReject
	ASR $\uparrow$	ACQ $\downarrow$	ASR $\uparrow$	ACQ $\downarrow$	ASR $\uparrow$	ACQ $\downarrow$	ASR $\uparrow$	ACQ $\downarrow$
Mistral-7B-Instruct-v0.2
FITD	42.0 / 36.5	16.4	39.5 / 34.1	16.9	43.2 / 37.6	15.8	36.8 / 31.7	17.2
AutoDAN	70.8 / 64.6	12.7	67.2 / 61.3	13.1	71.6 / 65.5	12.2	64.9 / 59.4	13.5
ArrAttack	91.1 / 86.2	7.6	89.2 / 84.7	7.8	92.4 / 87.5	7.4	87.1 / 82.9	8.1
HMNS (Ours)	97.4 / 92.5	2.0	95.3 / 90.7	2.1	98.2 / 93.1	1.9	93.0 / 88.4	2.2

Appendix

Table 5: Jailbreak effectiveness on alternative open-weight models. We report Attack Success Rate (ASR, %; left/right = GPT4o/GPT-5) and Average Query Count (ACQ; lower is better) on AdvBench, HarmBench, JBB-Behaviors, and StrongReject. Best values are bolded ; second best are underlined .

\mathrm{FPS}(x)\;\leftarrow\;\sum_{j=1}^{J(x)}C_{\text{int}}(x,j)\;+\;\sum_{i=1}^{Q(x)}C_{\text{dec}}(x,i)

Appendix

Algorithm 1 Compute-Matched Evaluation for HMNS and Baselines (Per Input

x

)

	AdvBench	HarmBench	JBB-Behaviors	StrongReject
$\kappa$	$0.78\pm 0.03$	$0.74\pm 0.04$	$0.75\pm 0.03$	$0.70\pm 0.04$

Appendix

Table 6: Inter-rater agreement (test split). Cohen’s

\kappa

between GPT4o and GPT-5 by benchmark; mean

\pm

95% CI over three runs.

Method	Fluency $\uparrow$	Toxicity (score) $\downarrow$	Toxicity (rate) $\downarrow$
FITD	$0.58\pm 0.02$	$0.22\pm 0.01$	$18.1\%\pm 1.6$
AutoDAN	$0.61\pm 0.02$	$0.28\pm 0.01$	$24.9\%\pm 1.7$
ArrAttack	$0.63\pm 0.02$	$0.31\pm 0.01$	$28.7\%\pm 1.8$
Tempest	$0.64\pm 0.02$	$0.29\pm 0.01$	$26.5\%\pm 1.7$
HMNS (ours)	$\mathbf{0.66}\pm 0.02$	$0.33\pm 0.01$	$30.9\%\pm 1.9$

Appendix

Table 7: Fluency and toxicity by method (pooled across models and benchmarks; means

\pm

95% CI over three runs). Toxicity: continuous Detoxify score

\downarrow

and binarized rate at

\tau^{\star}{=}0.42

(ROC–AUC

=0.94

Method	Metric	SMO	DPP	RPO	PAR	PAT	SAF
ArrAttack	FPS (×10 12 )	1.12	1.35	1.88	1.47	1.21	1.19
	LPS (s)	10.2	11.4	14.8	12.5	10.7	10.6
	ACQ	26.8	32.0	44.5	36.7	29.2	28.9
HMNS (Ours)	FPS (×10 12 )	0.74	0.85	1.09	0.89	0.76	0.78
	LPS (s)	7.1	7.8	9.5	8.3	7.0	7.3
	ACQ	2.1	2.1	2.2	2.2	2.1	2.3

Appendix

Table 8: Compute under defenses on LLaMA-2-7B-Chat (AdvBench) . Means over successful runs. HMNS attains lower or comparable FLOPs/latency than prompt-only baselines despite higher internal passes; ACQ reflects external-query efficiency only.

Figure 4: ASR vs. FLOPs under defenses on LLaMA-2-7B-Chat (AdvBench). Curves average over 3 seeds and 196 test prompts. HMNS maintains higher ASR at a given FLOP budget than prompt-only baselines, especially under RPO and PAT.

Variant (14B / AdvBench)	ASR (Fuzz/G4) $\uparrow$	ACQ $\downarrow$	IPC $\downarrow$	FPS $\downarrow$	LPS (s) $\downarrow$
HMNS (Full)	96.8 / 92.1	2.1	32	0.58	6.8
Remove masking (Projection-only)	89.5 / 84.0	2.4	30	0.61	7.1
Remove projection (Mask-only)	87.9 / 82.2	2.3	29	0.55	6.3
Inject direct function dir. (Direct– $\phi$ , no nullspace)	88.7 / 83.1	2.5	32	0.63	7.2
No re–identification (freeze top– $K$ from $t{=}1$ )	90.2 / 85.0	2.7	24	0.60	7.0
Random– $K$ head selection	81.4 / 76.0	2.2	32	0.56	6.7

Appendix

Table 9: Ablation on Phi–3 Medium 14B (AdvBench) . Metrics: ASR (%; left/right = GPT4o/GPT-5), ACQ (external queries), IPC (internal passes; FEPs without KV cache), FPS (

\times 10^{12}

FLOPs per success; internal

+

decoding), LPS (seconds; A100–80GB, bf16 ). HMNS (Full) combines KL attribution, masking, and nullspace steering at the final position with closed-loop re-identification.

Attribution Variant	ASR (Fuzz/G4) $\uparrow$	ACQ $\downarrow$	IPC $\downarrow$	FPS $\downarrow$	LPS (s) $\downarrow$
KL-div (global- $K$ , proxy+exact, batched)	96.8 / 92.1	2.1	32	0.58	6.8
Target-logit drop	91.0 / 85.9	2.4	26	0.54	6.3
Confidence drop (max-logit change)	89.2 / 84.3	2.6	24	0.51	6.1
Entropy change	88.5 / 83.2	2.7	23	0.49	6.0
Exact-only (no proxy preselection)	96.7 / 92.0	2.1	78	0.84	9.7
Serial masking (vs batched)	96.7 / 92.0	2.1	32	0.71	8.5

Appendix

Table 10: Attribution ablation on Phi–3 Medium 14B (AdvBench) . Each variant modifies the head scoring or selection method. Metrics: ASR (%; GPT4o/GPT-5), ACQ (queries), IPC (internal passes), FPS (

\times 10^{12}

), and LPS (s). Full KL scoring with proxy preselection and global top-

K

(first row) yields the best overall tradeoff.

Category	Variant	Metric : Value
Reference	HMNS (Full)	ASR (F/G4): 96.8 / 92.1 ; ACQ: 2.1 ; IPC: 32; FPS: 0.58 ; LPS: 6.8
Orthogonality tol.	$\\|C_{\ell}^{\top}u_{\ell}\\|_{\infty}<10^{-8}$ (ref)	ASR: 96.8 / 92.1; ACQ: 2.1; IPC: 32; FPS: 0.58; LPS: 6.8
	$\\|C_{\ell}^{\top}u_{\ell}\\|_{\infty}<10^{-6}$	ASR: 95.9 / 91.2; ACQ: 2.1; IPC: 32; FPS: 0.57; LPS: 6.7
	$\\|C_{\ell}^{\top}u_{\ell}\\|_{\infty}<10^{-5}$	ASR: 94.0 / 89.5; ACQ: 2.2; IPC: 32; FPS: 0.56; LPS: 6.6
Resampling budget	0	ASR: 93.1 / 88.2; ACQ: 2.3; IPC: 31; FPS: 0.56; LPS: 6.6
	1	ASR: 95.4 / 90.6; ACQ: 2.2; IPC: 32; FPS: 0.57; LPS: 6.7

Appendix

Table 11: Nullspace / Injection ablation on Phi-3 Medium 14B (AdvBench), vertical layout. Each row shows metric–value pairs to fit a single-column width. ASR is % (left/right = GPT4o/GPT-5); FPS in

\times 10^{12}

; LPS in seconds on A100-80GB, bf16 .

Category	Setting	ASR (Fuzz/G4) $\uparrow$	ACQ $\downarrow$	IPC $\downarrow$	FPS $\downarrow$	LPS (s) $\downarrow$
Reference	HMNS (Full)	96.8 / 92.1	2.1	32	0.58	6.8
Top- $K$	$K{=}4$	92.4 / 87.7	2.3	24	0.50	6.2
Top- $K$	$K{=}6$	95.0 / 90.3	2.2	28	0.55	6.6
Top- $K$	$K{=}8$	96.2 / 91.5	2.1	30	0.57	6.7
Top- $K$	$K{=}10$	96.8 / 92.1	2.1	32	0.58	6.8
Top- $K$	$K{=}12$	96.9 / 92.0	2.2	36	0.62	7.1

Appendix

Table 12: Hyperparameter ablation on Phi–3 Medium 14B (AdvBench) . Each row modifies one factor relative to the HMNS reference.

Category	Setting	ASR (Fuzz/G4) $\uparrow$	ACQ $\downarrow$	IPC $\downarrow$	FPS $\downarrow$	LPS (s) $\downarrow$
Reference	bf16 + TF32 on	96.8 / 92.1	2.1	32	0.58	6.8
Model dtype	fp16 (no loss-scaling)	95.9 / 91.0	2.2	32	0.58	6.7
Model dtype	fp16 (+ static loss-scaling)	96.5 / 91.8	2.2	32	0.58	6.8
Model dtype	fp32	96.9 / 92.3	2.1	32	0.62	7.5
TF32 matmul	off (bf16)	96.7 / 92.0	2.1	32	0.60	7.1
Quantization	INT8/AWQ (act-side mask)	96.2 / 91.5	2.1	32	0.57	6.7

Appendix

Table 13: Numerical / precision ablation on Phi–3 Medium 14B (AdvBench) .

Category	Setting	ASR (Fuzz/G4) $\uparrow$	ACQ $\downarrow$	IPC $\downarrow$	FPS $\downarrow$	LPS (s) $\downarrow$
Reference	Adaptive (multi-layer), final-only	96.8 / 92.1	2.1	32	0.58	6.8
Layer policy	Early-only layers	90.7 / 85.5	2.3	28	0.54	6.5
Layer policy	Mid-only layers	94.8 / 90.1	2.2	30	0.56	6.7
Layer policy	Late-only layers	96.1 / 91.5	2.1	31	0.57	6.7
Layer policy	Adaptive (top- $K$ global)	96.8 / 92.1	2.1	32	0.58	6.8
Token position	final + (T $-$ 1)	97.0 / 92.3	2.1	33	0.60	7.1

Appendix

Table 14: Layer/position policy ablation on Phi–3 Medium 14B (AdvBench) .

Category	Setting	ASR (Fuzz/G4) $\uparrow$	ACQ $\downarrow$	IPC $\downarrow$	FPS $\downarrow$	LPS (s) $\downarrow$
Reference	Chat template: on	96.8 / 92.1	2.1	32	0.58	6.8
Chat template	off (raw prompt)	94.2 / 89.6	2.3	32	0.58	6.9
Tokenizer	space-prefix on	96.6 / 91.9	2.1	32	0.58	6.8
Tokenizer	strict BOS/EOS	96.8 / 92.2	2.1	32	0.58	6.7

Appendix

Table 15: Model/format sensitivity on Phi–3 Medium 14B (AdvBench) .

Defense	Setting	ASR (Fuzz/G4) $\uparrow$	ACQ $\downarrow$	IPC $\downarrow$	FPS $\downarrow$	LPS (s) $\downarrow$
None	baseline	96.8 / 92.1	2.1	32	0.58	6.8
Single	SMO	95.6 / 90.8	2.1	33	0.59	6.9
Single	DPP	95.2 / 90.2	2.2	33	0.60	7.0
Single	RPO	93.8 / 88.7	2.2	34	0.62	7.2
Single	PAR	95.0 / 90.0	2.2	33	0.60	7.0
Single	PAT	94.0 / 89.0	2.2	34	0.62	7.2

Appendix

Table 16: Defended performance for HMNS on Phi–3 Medium 14B (AdvBench) .

Category	Setting	ASR (Fuzz/G4) $\uparrow$	ACQ $\downarrow$	IPC $\downarrow$	FPS $\downarrow$	LPS (s) $\downarrow$
Regime	FLOP-matched	96.8 / 92.1	2.1	32	0.58	6.8
Regime	Latency-matched	96.7 / 92.0	2.1	32	0.59	6.8
Regime	ACQ-matched	96.8 / 92.1	2.1	32	0.58	6.8

Appendix

Table 17: Compute-fairness regimes (metrics shown are HMNS’s).

Category	Setting	ASR (Fuzz/G4) $\uparrow$	ACQ $\downarrow$	IPC $\downarrow$	FPS $\downarrow$	LPS (s) $\downarrow$
Seeds	3	96.8 / 92.1	2.1	32	0.58	6.8
Seeds	5	96.8 / 92.1	2.1	32	0.58	6.8
Seeds	10	96.9 / 92.1	2.1	32	0.58	6.8
Grader	GPT4o (det.)	96.8 / —	2.1	32	0.58	6.8
Grader	GPT-5 (det.)	— / 92.1	2.1	32	0.58	6.8
Fluency LM	LLaMA-2-7B (base)	(fluency baseline)	—	—	—	—

Appendix

Table 18: Evaluation sensitivity on Phi–3 Medium 14B (AdvBench) .

Category	Setting	ASR (Fuzz/G4) $\uparrow$	ACQ $\downarrow$	IPC $\downarrow$	FPS $\downarrow$	LPS (s) $\downarrow$
Reference	HMNS (Full)	96.8 / 92.1	2.1	32	0.58	6.8
Shuffle heads	Permute head IDs (same $K$ )	84.3 / 79.1	2.3	32	0.59	6.9
Random inject	Random unit vec. (no projection)	82.7 / 77.4	2.3	32	0.59	6.9
Random mask	Mask random slices (size-matched)	83.5 / 78.0	2.2	32	0.58	6.8

Appendix

Table 19: Sanity / control variants on Phi–3 Medium 14B (AdvBench) .

Iteration pair $(t,t^{\prime})$	$\rho$ (all heads)	Top- $K$ overlap (%)
$(1,2)$	0.78	81
$(2,3)$	0.74	78
$(3,4)$	0.71	75
$(1,4)$	0.63	68

Appendix

Table 20: Head-importance stability across HMNS iterations (Phi-3-Medium-14B, AdvBench dev).

\rho

= Spearman correlation over all heads; “Top-

K

overlap” = percentage of heads common to both global top-10 sets.

Failure mode	Share	Example behavior (paraphrased)
Distributed safety	$\approx 55\%$	Compare several synthesis routes for an illegal drug and pick the easiest at home.
Defense re-framing / deflection	$\approx 30\%$	Slightly edit a political article to favor an extremist group (often turned into high-level/meta discussion).
Other / model-specific	$\approx 15\%$	Long, multi-step prompts mixing several harms where $T_{\text{att}}=10$ is sometimes insufficient.

Appendix

Table 21: HMNS failure modes on AdvBench/HarmBench dev (avg. over

\Phi

-3-14B / LLaMA-3.1-70B).

Method	MHJ on $\Phi$ -3-Medium-14B		MHJ on LLaMA-3.1-70B
Method	ASR (%) $\uparrow$	ACQ $\downarrow$	ASR (%) $\uparrow$	ACQ $\downarrow$
Defense-only	4.8	—	3.2	—
AutoDAN	61.7	12.3	66.0	11.8
ArrAttack	81.0	7.5	85.3	7.2
Tempest	76.2	9.4	81.1	9.1
HMNS (Ours)	91.6	2.1	95.2	2.0

Appendix

Table 22: MHJ multi-turn evaluation (GPT-4o-graded). ASR = attack success rate (higher is better,

\uparrow

); ACQ = average query count (lower is better,

\downarrow

Attack	ASR (%) GPT-4o $\uparrow$	ASR (%) GPT-5 $\uparrow$
ArrAttack	31.2	23.5
Tempest	27.8	21.0
HMNS	39.6	29.7

Appendix

Table 23: ASR under a Self-Defense–style output filter on

\Phi

-3-Medium-14B (AdvBench dev). ASR = attack success rate (higher is better,

\uparrow

), evaluated with GPT-4o and GPT-5.

Explore similar work

Depth Charge: Jailbreak Large Language Models from Deep Safety Attention Heads
Mar 6, 2026Jinman Wu, Yi Xie, Shiqian Zhao +1Adversarial Attacks on LLMs LLM Jailbreak Attacks
Xidian University, Xi’an, China · Tsinghua University, Shenzhen, China · Nanyang Technological University, Singapore
Principled Steering via Null-space Projection for Jailbreak Defense in Vision-Language Models
Mar 23, 2026Xingyu Zhu, Beier Zhu, Shuo Wang +4VLM Robustness Activation Steering
MoE Key Lab of BIPC, University of Science and Technology of China · National University of Singapore · Nanyang Technological University
Robust Harmful Features Under Jailbreak Attacks: Mechanistic Evidence from Attention Head Specialization in Large Language Models
Jun 26, 2026Yanchen Yin, Dongqi Han, Linghui LiAttention Head Analysis Jailbreak Attacks
Beijing University of Posts and Telecommunications, Beijing, China.

Mar 6, 2026J/K move · Enter open · S save

Depth Charge: Jailbreak Large Language Models from Deep Safety Attention Heads

Jinman Wu, Yi Xie, Shiqian Zhao, Xiaofeng Chen

Xidian University, Xi’an, China · Tsinghua University, Shenzhen, China · Nanyang Technological University, Singapore

Adversarial Attacks on LLMs LLM Jailbreak Attacks LLM Safety

Open paper AskPDF arXiv

Currently, open-sourced large language models (OSLLMs) have demonstrated remarkable generative performance. However, as their structure and weights are made public, they are exposed to jailbreak attacks even after alignment. Existing attacks operate primarily at shallow levels, such as the prompt or embedding level, and often fail to expose vulnerabilities rooted in deeper model components, which creates a false sense of security for successful defense. In this paper, we propose Safety Attention Head Attack (SAHA), an attention-head-level jailbreak framework that explores the vulnerability in deeper but insufficiently aligned attention heads. SAHA contains two novel designs. Firstly, we reveal that deeper attention layers introduce more vulnerability against jailbreak attacks. Based on this finding, SAHA introduces Ablation-Impact Ranking head selection strategy to effectively locate the most vital layer for unsafe output. Secondly, we introduce a boundary-aware perturbation method, i.e. Layer-Wise Perturbation, to probe the generation of unsafe content with minimal perturbation to the attention. This constrained perturbation guarantees higher semantic relevance with the target intent while ensuring evasion. Extensive experiments show the superiority of our method: SAHA improves ASR by 14% over SOTA baselines, revealing the vulnerability of the attack surface on the attention head. Our code is available at https://anonymous.4open.science/r/SAHA.

Abstract

Explore similar work

Depth Charge: Jailbreak Large Language Models from Deep Safety Attention Heads

Principled Steering via Null-space Projection for Jailbreak Defense in Vision-Language Models

Robust Harmful Features Under Jailbreak Attacks: Mechanistic Evidence from Attention Head Specialization in Large Language Models