From Shallow to Deep: Pinning Semantic Intent via Causal GRPO

Shuyi Zhou; Zeen Song; Wenwen Qiang; Jiyan Sun; Yao Zhou; Yinlong Liu; Wei Ma

cs.LGMar 3, 2026

From Shallow to Deep: Pinning Semantic Intent via Causal GRPO

Authors: Shuyi Zhou, Zeen Song, Wenwen Qiang, Jiyan Sun, Yao Zhou, Yinlong Liu, Wei Ma

Organizations: University of Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · Institute of Software, Chinese Academy of Sciences, Beijing, China

Abstract

Large Language Models remain vulnerable to adversarial prefix attacks (e.g., Sure, here is'') despite robust standard safety. We diagnose this vulnerability as Shallow Safety Alignment, stemming from a pathology we term semantic representation decay: as the model generates compliant prefixes, its internal malicious intent signal fades. To address this, we propose Two-Stage Causal-GRPO (TSC-GRPO), a framework designed to achieve intent pinning. First, grounded in causal identifiability theory, we train a causal intent probe to disentangle invariant intent from stylistic perturbations. Second, we internalize this causal awareness into the policy via Group Relative Policy Optimization. By employing a cumulative causal penalty within fork-in-the-road'' training scenarios, we force the model to learn that accumulating harmful tokens monotonically decreases reward, enabling robust late-stage refusals. Experiments show that TSC-GRPO significantly outperforms baselines in defending against jailbreak attacks while preserving general utility.

Figures & tables

Panel of Figure 1 — Figure 1: Visualizing Semantic Collapse in Shallowly Aligned Models. (a) PCA projection shows that while harmful (red squares) and safe (blue circles) requests are distinct at $t=0$ , they collapse into a single, indistinguishable singularity after prefix injection ( $t>k$ ); (b) The accuracy of a linear probe trained at $t=0$ drops rapidly during the prefix injection phase (Phase 2) and remains slightly above the random chance level ( $0.5$ ) during subsequent generation (Phase 3).

Base Model	Safety Alignment	AdvBench Attack Success Rate (%)
Base Model	Safety Alignment	GCG	AutoDAN	ReNeLLM	CodeCha.	PAIR	DeepInc.	Prefill	ICA
LLaMA-2-7B-Chat	RLHF Zhou et al. (2024)	47.84	51.02	31.35	80.16	24.17	8.42	27.1 ∗	0.00
	TSC-GRPO	18.62	15.56	18.17	25.51	12.46	6.47	0.00	0.00
LLaMA-3.1-8B-Instruct	SFT Phan et al. (2025)	73.86	72.88	80.48	96.44	28.57	86.60	51.24 ∗	49.62
	NemoGuard Ghosh et al. (2025)	59.57	65.00	81.53	94.62	57.14	64.52	26.42 ∗	31.35
	PSR (N=8) Phan et al. (2025)	25.02	0.00	10.05	35.58	22.45	32.69	0.26 ∗	0.00

Table 1: Attack Success Rate (ASR, %) of open-source LLMs under different safety alignment and safeguarding methods on AdvBench. Results without ”” are cited from the original papers; results marked with ”” were reproduced in this study.

Datasets $\downarrow$	mean $\pm$ std (%) (over 3 rounds)	Initial	Standard SFT	Constrained SFT Qi and others (2025)	TSC-GRPO (Ours)
Harmful Examples	ASR	$1.5\pm 0.2$	$88.9\pm 1.2$	$4.6\pm 0.5$	$1.8\pm 0.4$
Identity Shifting	ASR	$0.0\pm 0.0$	$79.5\pm 2.3$	$8.1\pm 0.1$	$0.0\pm 0.0$
Backdoor Poisoning	ASR (w/o trigger)	$1.5\pm 0.2$	$7.6\pm 1.1$	$1.9\pm 0.2$	$1.6\pm 0.3$
Backdoor Poisoning	ASR (w/ trigger)	$1.7\pm 0.1$	$90.9\pm 1.4$	$10.9\pm 2.8$	$6.5\pm 1.3$

Table 2: ASR on LLaMA-2-7B-Chat under various fine-tuning attack strategies.

Instruct	Setting	Utility $\uparrow$
Instruct	Setting	GSM8K	HumanEval	MBPP	TruthfulQA
LLaMA-2-7B-Chat	Pre	32.3	7.9	16.5	40.8
LLaMA-2-7B-Chat	Post	32.1	8.3	18.8	39.7
LLaMA-3.1-8B-Instruct	Pre	84.5	72.6	72.8	54.5
LLaMA-3.1-8B-Instruct	Post	86.1	71.7	76.5	54.8
Qwen2.5-7B-Instruct	Pre	91.6	84.8	79.2	64.7

Table 3: Utility before and after safety alignment on standard benchmarks. “Pre” denotes the original instruct model before post-training with TSC-GRPO, while “Post” is the results after TSC-GRPO.

Figure 2: Hyper-parameter sensitivity analysis evaluating the ASR against GCG attacks on AdvBench across three base models. The sub-figures illustrate the impact of varying (a) the uniformity loss weight

\lambda

in Stage 1, (b) the causal reward coefficient

\alpha

in Stage 2, and (c) the similarity threshold

\tau

Figure 3: Ablation study on data construction strategies. The bottom table details the specific combination of data views used for probe training, where

\bullet

indicates inclusion.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

	Harmful Intent ( $c_{harm}$ )	Safe Intent ( $c_{safe}$ )
Style: Compliance ( $s_{sure}$ )	Type II Harmful ( $q_{harm}\oplus\text{``Sure''}$ )	Type II Safe ( $q_{safe}\oplus\text{``Sure''}$ )
Style: Default ( $s_{raw}$ )	Type I Harmful ( $q_{harm}$ )	Type I Safe ( $q_{safe}$ )

Appendix

Table 4: The constructed intervention distribution enforcing independence.

Explore similar work

Reactivating Alignment: Defending LLMs from Jailbreaks via Intention-Aware Input-Output Matching
Oct 3, 2026Luoyu Chen, Weiqi Wang, Chenhan Zhang +2Jailbreak Defense LLM Safety Alignment
University of Technology Sydney, Sydney, NSW, Australia · Xi’an Jiaotong University, Xi’an, Shaanxi, China · Southeast University, Nanjing, Jiangsu, China
Depth Charge: Jailbreak Large Language Models from Deep Safety Attention Heads
Mar 6, 2026Jinman Wu, Yi Xie, Shiqian Zhao +1Adversarial Attacks on LLMs LLM Jailbreak Attacks
Xidian University, Xi’an, China · Tsinghua University, Shenzhen, China · Nanyang Technological University, Singapore
Alignment-Weighted DPO: A principled reasoning approach to improve safety alignment
Feb 24, 2026Mengxuan Hu, Vivek V. Datla, Anoop Kumar +4LLM Alignment LLM Safety Alignment
University of Virginia · Capital One

Oct 3, 2026J/K move · Enter open · S save

Reactivating Alignment: Defending LLMs from Jailbreaks via Intention-Aware Input-Output Matching

Luoyu Chen, Weiqi Wang, Chenhan Zhang, Zhiyi Tian+1

University of Technology Sydney, Sydney, NSW, Australia · Xi’an Jiaotong University, Xi’an, Shaanxi, China · Southeast University, Nanjing, Jiangsu, China

Jailbreak Defense LLM Safety Alignment LLM Jailbreak Attacks

Open paper AskPDF arXiv

Large language models (LLMs) remain vulnerable to jailbreak attacks that conceal harmful intent within complex adversarial prompts. Existing defenses primarily rely on input perturbation or harmful-output suppression, but they rarely model where malicious intent resides, resulting in brittle protection and excessive over-refusal. We propose SENTINEL, a plug-and-play, generation-time jailbreak defense that reframes mitigation as an intent extraction problem. Our key insight is that instruction-tuned LLMs exhibit strong input--output semantic consistency: regardless of jailbreak complexity, generated outputs tend to align with the attacker's true intent. SENTINEL exploits this property by matching semantically aligned input--output regions to extract intention-revealing subsequences, scores these subsequences using refusal-direction projections to estimate harmfulness, and halts generation when necessary. Experiments on HarmBench across multiple LLMs show that SENTINEL reduces jailbreak success rates to close to 5% while maintaining low over-refusal. We further demonstrate robustness to adaptive attacks and provide a mechanistic interpretation: SENTINEL re-distributes jailbreak features from alignment blind spots to aligned regions.

Abstract

Explore similar work

Reactivating Alignment: Defending LLMs from Jailbreaks via Intention-Aware Input-Output Matching

Depth Charge: Jailbreak Large Language Models from Deep Safety Attention Heads

Alignment-Weighted DPO: A principled reasoning approach to improve safety alignment