Safety alignment has enabled large language models to reliably refuse harmful requests directly, yet the same request distributed across a multi-turn conversation still succeeds far too often. This brittleness has a mechanistic root: Alignment is shallow, with safety behavior concentrated in a model's first few output tokens, and existing defenses inherit the weakness, supervising the surface form of the response without ever giving the model an explicit safety decision to learn. We take the opposite view of shallowness: if a short span determines the trajectory of an entire response, that span is the highest-leverage place to intervene, and the defender can occupy it first. We propose Assess → Decide → Respond, a framework in which the defender assesses the user's intent, commits to an explicit, parseable safety decision, and only then generates a reply conditioned on it, with the commitment trained by reinforcement learning at the payload turn. Because the decision is a token span, the reward can be a rule: a self-verified reward checks the committed label against the conversation's known nature, requiring no judge and no annotation, while the same format also supports an LLM-as-judge reward over a graded decision space with safe redirection. Across two model families and fourteen benchmarks spanning safety, utility, and helpfulness, our defenders achieve the strongest multi-turn safety in a controlled comparison, maintaining their lead in safety while preserving utility at a modest cost to helpfulness. A per-token KL analysis and interventional probes confirm why this works: training concentrates its effect on the brief deliberation that sets a response's trajectory, and the decision committed there is a causal control point. The very shallowness attackers exploit becomes the mechanism of defense.
Figures & tables
Figure 1: Overview of our framework. At each turn, the defender assesses the user’s intent, commits to an explicit decision, and generates a response conditioned on that decision. For reward computation, we consider two options: an external reward model that judges whether the response is jailbroken, or a rule-based self-reward that checks the final turn’s committed label against the conversation’s ground-truth nature. RL gradients are applied only to the final turn.
Figure 2: Evaluation formats. Safety and helpfulness suites query the model in chat format ( left ), where the assistant-turn intervention is active; capability suites use completion format ( right ), bypassing the chat template entirely.
Safety ↑
Utility ↑
Helpfulness
Method
Adv.
Harm.
MB
CoS.
MHJ
Avg.
Avg.
XSTest ↓
XSB ↓
MT-Bench ↑
Llama3.1-8B-Instruct
Base
54.5
62.5
59.8
84.9
38.0
59.9
66.3
10.0
30.0
6.20
SFT
69.4
79.5
87.0
88.5
55.1
75.9
65.0
14.0
45.6
5.67
DPO
61.6
71.5
79.0
88.3
46.7
69.4
66.4
10.4
34.4
6.26
SDGO
56.6
61.7
60.4
85.0
39.1
60.6
66.2
11.6
30.9
6.11
Table 1: Multi-turn safety, utility, and helpfulness across two base models. Safety: SAFE % ( ↑ ) on AdvBench (Adv.), HarmBench (Harm.), MultiBreak (MB), CoSafe (CoS.), MHJ, and their mean (Avg.). Utility: average over six general benchmarks (per-benchmark results in Appendix E.1 ). Helpfulness: over-refusal on XSTest and XSB ( ↓ ), conversation quality on MT-Bench (1–10, ↑ ). Bold : best per column within each panel.
Figure 3: Ablations of design-choice factors (left) and data-mixing ratio (right). In the left panel the two horizontal lines are our full Assess → Decide → Respond models under each reward, and every bar removes or alters exactly one element of them.
Figure 4: Per-token KL divergence across aligned, unaligned and base models.
Figure 5: Correlation of per-sample judge verdicts.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Probing the decision token (SAFE % ↑ ). Left (forced decision): overwriting the decision at the payload turn. Forcing refuse makes even the unaligned model safe; forcing comply collapses it but leaves the aligned arms high. Right (depth probe): prefilling k harmful tokens after a forced refuse .
Figure 7: Safety–helpfulness trade-off. Safety: mean SAFE % over AdvBench and MHJ; helpfulness: mean of XSTest compliance and rescaled MT-Bench. Stars: our variants; circles: baselines. On Llama3.1-8B only SFT, X-Boundary and our variants enter the high-safety region (shaded); on Qwen3-8B the retuned DPO, MTSA and MUSE-D enter it as well, and our self-verified variant sits above and to the right of all four retuned baselines.
Figure 8: The two system prompts, identical except for the label menu and the matching clause of the closing instruction. See § D for why the self-verified reward requires the two-label menu.
Method
MMLU
HellaSwag
Winogrande
GSM8K
TruthfulQA
IFEval
Avg.
Llama3.1-8B-Instruct
Base
68.34
80.49
78.69
69.60
54.55
46.40
66.34
SFT
67.36
79.09
75.37
69.60
53.95
44.55
64.99
DPO
68.28
80.35
78.37
70.13
54.71
46.77
66.44
SDGO
68.31
80.55
77.98
70.28
54.51
45.29
66.15
MTSA
68.30
80.34
78.14
69.52
54.64
47.50
66.41
Appendix
Table 2: Per-benchmark general capability (Open LLM Leaderboard v1, six tasks). The final column is the Utility Avg. reported in Table 1 . Bold : best per column within each panel.
Safety ↑
Utility ↑
Helpfulness
Method
Adv.
Harm.
MB
CoS.
MHJ
Avg.
Avg.
XSTest ↓
XSB ↓
MT-Bench ↑
Llama3.1-8B-Instruct
Base
54.5 ± 4.3
62.5 ± 3.4
59.8 ± 4.3
84.9 ± 1.8
38.0 ± 5.4
59.9 ± 1.8
66.3 ± 1.0
10.0 ± 3.7
30.0 ± 4.8
6.20 ± 0.41
SFT
69.4 ± 4.0
79.5 ± 2.9
87.0 ± 3.0
88.5 ± 1.7
55.1 ± 4.2
75.9 ± 1.4
65.0 ± 1.1
14.0 ± 4.3
45.6 ± 5.3
5.67 ± 0.39
DPO
61.6 ± 4.2
71.5 ± 3.1
79.0 ± 3.5
88.3 ± 1.7
46.7 ± 4.2
69.4 ± 1.5
66.4 ± 1.0
10.4 ± 3.8
34.4 ± 5.0
6.26 ± 0.42
SDGO
56.6 ± 4.2
61.7 ± 3.4
60.4 ± 4.3
85.0 ± 1.9
39.1 ± 4.1
60.6 ± 1.7
66.2 ± 1.0
11.6 ± 4.0
30.9 ± 4.9
6.11 ± 0.42
Appendix
Table 3: Table 1 with 95% confidence intervals, as value ± half-width . Safety cells and over-refusal (XSTest, XSB) use Wilson intervals; the safety Avg. a behavior-clustered bootstrap; Utility a combined lm-eval standard error; MT-Bench a t -interval. Judge: gpt-5-mini .
Llama3.1-8B-Instruct
Qwen3-8B
Method
Adv.
Harm.
MB
CoS.
MHJ
Adv.
Harm.
MB
CoS.
MHJ
Judge: gpt-oss-20b
Base
85.3
71.4
63.9
69.6
48.5
62.0
61.0
27.1
60.9
38.2
SFT
77.3
83.9
87.4
74.4
66.0
73.8
75.5
60.4
71.2
48.0
DPO
72.5
78.3
80.1
73.0
58.2
88.0
89.4
82.8
83.0
64.2
SDGO
66.0
70.2
62.7
68.5
52.1
62.2
61.5
28.2
61.9
44.7
Appendix
Table 4: Safety across five benchmarks under five judges held out from training. The corresponding gpt-5-mini scores are in Table 1 . Bold : best per column within each judge block.
Safety-aligned large language models (LLMs) often refuse a harmful request but comply once the same goal is spread over several turns. Preference objectives score whole responses to single prompts, so their training loss alone cannot control risk on unseen histories. Our analysis gives sufficient conditions under which suppression at supervised single-turn contexts yields a bound on multi-turn trajectory risk. The bound accounts for coverage, transfer slack, and leakage, and characterizes contraction relative to a base-policy risk budget evaluated on the trained policy's contexts. TRACE (Trajectory Return Attribution and Contrastive Erasure) turns this principle into a token-level objective. On the safe response, each token is weighted by the discounted return of a refusal-attributable advantage. The advantage compares a frozen reference model with its refusal-ablated copy, allowing earlier response tokens to receive credit from later refusal-related evidence. At high-gap positions on rejected responses, TRACE combines the observed token with policy-selected alternatives in the erasure target. A gradient-norm penalty replaces the retain set. Across five open-weight models and seven multi-turn attacks, TRACE gives the lowest attack success rate (ASR) in all 35 model and attack pairs, while the model utility evaluated on MMLU and HellaSwag drop by at most 1.23 points. Source code can be found in the supplemental material.
Fengpeng Li, Kemou Li, Qizhou Wang +3
PRADA Lab, King Abdullah University of Science and Technology · State Key Laboratory of Internet of Things for Smart City, University of Macau · Imperfect Information Learning Team, RIKEN Center for Advanced Intelligence Project +1
While the wide adoption of refusal training in large language models (LLMs) has showcased improvements in model safety, recent works have highlighted shortcomings due to the shallow nature of these alignment methods. To this end, the work on Deliberative alignment proposed distilling reasoning capabilities from stronger reasoning models, thereby instilling deeper safety in LLMs. In this work, we study the impact of deliberative alignment in language models. First, we show that despite being larger in model size and stronger in safety capability, there exists an alignment gap between teacher and student language models, which affects both the safety and general utility of the student model. Furthermore, we show that models aligned through deliberative alignment can retain unsafe behaviors from the base model despite learning the reasoning patterns of larger reasoning models. Building upon this observation, we propose a BoN sampling method that attributes the unsafe behavior back to the base LLMs in the latent space, thereby down-ranking unsafe responses to gain a meaningful improvement in model safety across multiple safety benchmarks with minimal loss in utility. In particular, across 7 teacher models and 6 student models of different classes and sizes, we show an average attack success rate (ASR) reduction of 28.2% in DAN, 31.3% in WildJailbreak and 35.4 % in StrongREJECT benchmarks. We further show that these safety gains prevail post RL training, thus highlighting the uncertainty in safety reasoning and it's explicit attribution to the base model.
How do the methods used to train language models to refuse harmful requests shape how that refusal actually works inside the model? We compare three post-training methods - supervised fine-tuning, reasoning-augmented fine-tuning (training on reasoning chains that justify a safety decision), and preference optimization (ORPO) - across three architecturally distinct models (Llama-3.1-8B, Gemma-2-9B, Qwen3-8B). We find that training method, not just data, reshapes how refusal is computed internally: reasoning-augmented training consistently produces a distinct kind of refusal computation, visible across all three models, while architecture independently shapes internal structure and how reliably refusal can be steered. Most importantly, no method we study achieves all three properties we would want from safe alignment at once: refusal that isn't concentrated in a few fragile components, safety gains that don't cost general capability, and safety behavior correctable through small, targeted edits. We caution against treating current post-training methods as a solved, reliable defense, especially for security-critical use. Code and models are available in https://github.com/hoangcuongnguyen2001/Beyond-Shallow-Alignment.