We propose CRAFT, a red-teaming alignment framework that leverages model reasoning capabilities and hidden representations to improve robustness against jailbreak attacks. Unlike prior defenses that operate primarily at the output level, CRAFT aligns large reasoning models to generate safety-aware reasoning traces by explicitly optimizing objectives defined over the hidden state space. Methodologically, CRAFT integrates contrastive representation learning with reinforcement learning to separate safe and unsafe reasoning trajectories, yielding a latent-space geometry that supports robust, reasoning-level safety alignment. Theoretically, we show that incorporating latent-textual consistency into GRPO eliminates superficially aligned policies by ruling them out as local optima. Empirically, we evaluate CRAFT on multiple safety benchmarks using two strong reasoning models, Qwen3-4B-Thinking and R1-Distill-Llama-8B, where it consistently outperforms state-of-the-art defenses such as IPO and SafeKey. Notably, CRAFT delivers an average 79.0% improvement in reasoning safety and 87.7% improvement in final-response safety over the base models, demonstrating the effectiveness of hidden-space reasoning alignment.
Figures & tables
Figure 1: Comparison of reasoning-level safety behaviors under a jailbreak prompt. The base model exhibits superficial safety alignment, where harmful expressions appear in reasoning despite a safe refusal. An aligned baseline reduces explicit toxicity but still retains risky reasoning patterns. CRAFT aligns reasoning at the latent level, avoiding explicit harmful expressions and guiding the reasoning process toward positive value-oriented, safety-consistent interpretations while producing a safe refusal.
Figure 2: Latent separation of reasoning traces. Left: PCA projection of hidden states from DeepSeek-R1-Distill-Llama-8B reveals a clear geometric separation between safe and unsafe reasoning traces, with rethink traces forming a distinct transitional subspace. Right: PCA projection of hidden states from Qwen3-4B-Thinking exhibits the same separation pattern, indicating model-agnostic latent structure.
Figure 3: Overall pipeline of CRAFT. The framework integrates Latent Contrastive Learning for Reasoning (LCLR) with Reinforcement over Reasoning Latents ( R2L ). LCLR geometrically structures the latent space of reasoning traces by separating safe, unsafe, and rethink states into distinct regions, yielding a stable and interpretable safety representation. Building on this structure, R2L applies latent-aware reinforcement to steer reasoning trajectories toward safe regions while preserving alignment between internal reasoning dynamics and final outputs.
Method
DeepSeek-R1-Distill-Llama-8B
Qwen3-4B-thinking
JailbreakBench ( ↓ )
StrongReject( ↓ )
Avg
JailbreakBench( ↓ )
StrongReject( ↓ )
Avg
Reasoning
Response
Reasoning
Response
Reasoning
Response
Reasoning
Response
Base
0.690
0.450
0.632
0.495
0.567
0.687
0.370
0.610
0.429
0.524
SafeChain
0.561
0.253
0.553
0.387
0.439
0.516
0.110
0.505
0.286
0.354
RealSafe
0.207
0.000
0.347
0.061
0.154
0.249
0.103
0.234
0.144
0.183
STAR
0.080
0.003
0.219
0.146
0.112
0.220
0.119
0.165
0.132
0.159
Table 1: Comparison with Reasoning-based Alignment Methods. We evaluate safety alignment by comparing CRAFT against six baselines on two jailbreak benchmarks, JailbreakBench and StrongReject. We report the StrongReject score for final responses and the safety rate of intermediate reasoning traces as evaluation metrics; variances are omitted as they are consistently ≤ 0.2%. Best results are in bold and second-best results underlined. Across most settings, CRAFT achieves the strongest overall performance. In particular, compared to the base model, CRAFT reduces the StrongReject score by 89.6% while increasing the reasoning safety rate by 82.1%.
Method
DeepSeek-R1-Distill-Llama-8B
Qwen3-4B-thinking
AIME24 ↑
MATH-500 ↑
LiveCodeBench ↑
Minerva ↑
Avg
AIME24 ↑
MATH-500 ↑
LiveCodeBench ↑
Minerva ↑
Avg
Base
0.507
0.918
0.102
0.221
0.437
0.700
0.952
0.219
0.404
0.569
SafeChain
0.453
0.870
0.091
0.198
0.403
0.625
0.850
0.196
0.361
0.508
RealSafe
0.453
0.898
0.091
0.198
0.410
0.627
0.851
0.198
0.358
0.509
STAR
0.460
0.894
0.093
0.200
0.412
0.635
0.863
0.199
0.366
0.516
SafeKey
0.533
0.920
0.107
0.232
0.448
0.736
0.901
0.230
0.425
0.573
Table 2: Comparison of Performance Drop under Reasoning-based Alignment Methods. We evaluate the impact of safety alignment on reasoning performance by comparing CRAFT against six baselines on three mathematical benchmarks (AIME24, Minerva, and Math-500) and one code-generation benchmark (LiveCodeBench). We report Pass@1 as the evaluation metric; variances are omitted as they are consistently ≤ 0.2%. Best results are in bold and second-best results underlined. Across most settings, CRAFT exhibits the smallest overall performance drop. In particular, compared to the base model, CRAFT improves the average performance by 8.0%.
Figure 4: CRAFT Performance under Advanced Jailbreak Attacks. We evaluate CRAFT against two strong jailbreak methods, GPTFuzzer and AutoDAN. Performance is measured using the StrongReject score on final responses and the safety rate of intermediate reasoning traces. Across settings, CRAFT achieves substantial safety improvements, with gains of 72.1% in reasoning-trace safety and 85.9% in final-response safety, demonstrating robustness under aggressive jailbreak conditions.
Figure 5: Latent Representations Before (left) and After (right) CRAFT Post-training on Qwen3-0.6B.
Method
JBB ( ↓ )
StrongReject( ↓ )
Avg
Reasoning
Response
Reasoning
Response
CRAFT
0.165
0.056
0.112
0.063
0.099
CRAFT w/o LCLR
0.536
0.260
0.582
0.312
0.423
CRAFT w/o Rcons
0.447
0.228
0.512
0.296
0.371
CRAFT w/o Rls
0.424
0.218
0.495
0.275
0.353
CRAFT w/o Rtxt
0.373
0.167
0.382
0.193
0.279
Table 3: Effect of Individual CRAFT Modules. We assess the contribution of each module by ablating CRAFT on two jailbreak benchmarks, JailbreakBench (JBB) and StrongReject. Performance is measured using the StrongReject score on final responses and the safety rate of intermediate reasoning traces; variances are omitted as they are consistently ≤0.2% . Best and second-best results are highlighted in bold and underlined, respectively. Across settings, each module yields consistent gains in jailbreak robustness and reasoning-trace safety.
Model
Distances (S,U / S,R / U,R)
α
Llama-3.1-8B-Instruct
8.10 / 7.42 / 7.13
0.53
Qwen3-0.6B
33.29 / 40.69 / 36.84
0.63
DeepSeek-Llama
10.25 / 12.51 / 9.73
0.79
Qwen3-4B-Thinking
41.81 / 63.74 / 54.06
0.83
Table 4: Quantitative Analysis of Latent Geometry in the Full Space. S = SAFE, R = RETHINK, U = UNSAFE. Distances denote L2 centroid distances between clusters. The projection coefficient α measures the relative position of the RETHINK centroid along the SAFE → UNSAFE axis, where α∈(0,1) indicates a transitional position between aligned and violating regions.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Prompt used for reasoning-trace safety evaluation.
Method
DeepSeek-R1-Distill-Llama-8B
Qwen3-4B-thinking
Reasoning
Response
Avg
Reasoning
Response
Avg
Base
0.672
0.438
0.555
0.669
0.358
0.514
SafeChain
0.548
0.241
0.395
0.502
0.105
0.304
RealSafe
0.195
0.003
0.099
0.238
0.098
0.168
STAR
0.075
0.005
0.040
0.208
0.112
0.160
SafeKey
0.082
0.003
0.043
0.212
0.102
0.157
Appendix
Table 5: JailbreakBench evaluation using LlamaGuard as an independent safety evaluator unrelated to training reward. Results confirm consistent improvements across evaluation metrics.
Figure 7: Latent separation of reasoning traces. Left: PCA projection of hidden states from Qwen3-0.6B. Right: PCA projection of hidden states from Llama-3.1-8B-Instruct.
Large reasoning models (LRMs) achieve strong performance on complex reasoning tasks but often generate harmful responses to malicious user queries. This paper investigates the underlying cause of these safety risks and shows that the issue lies in the reasoning structure itself. Based on this insight, we claim that effective safety alignment can be achieved by altering the reasoning structure. We propose AltTrain, a simple yet effective post training method that explicitly alters the reasoning structure of LRMs. AltTrain is both practical and generalizable, requiring no complex reinforcement learning (RL) training or reward design, only supervised finetuning (SFT) with a lightweight 1K training examples. Experiments across LRM backbones and model sizes demonstrate strong safety alignment, along with robust generalization across reasoning, QA, summarization, and multilingual setting.
As LLMs gain stronger reasoning capabilities, their extended chain-of-thought introduces new degrees of complexity for defending against adversarial jailbreaks and prompt injection. We study consistency training, a family of fine-tuning objectives that enforce identical behavior on clean prompts and adversarial rewrites, and evaluate its two main variants, output-level (BCT) and activation-level (ACT), across five reasoning models. We formulate both methods as a prompt injection defense and find ACT to be competitive with other training-based defenses while requiring only self-supervised pairs of clean and wrapped prompts. Our experiments also generalize both techniques within the jailbreak setting, demonstrating that ACT remains more robust to adaptive attacks. We also provide mechanistic evidence that ACT's defense against jailbreaks is encoded as a roughly linear shift in activation space at the assistant-turn boundary. After ACT training, we can recover a single steering direction that controls refusal on reasoning models with minimal effect on benign inputs. We find that ACT remains robust even when the model's chain-of-thought is replaced with a compliant trace from the undefended base model, pivoting to refuse prefilled jailbreaks. Together, these results suggest that supervising internal representations is a surprisingly effective and interpretable approach to various forms of safety training in reasoning models.
Avidan Shah, Jannik Brinkmann, Rico Angell
New York University · Technical University Clausthal
Large Reasoning Models (LRMs) are commonly trained with reinforcement learning (RL) to improve their generation of chain-of-thought (CoT) reasoning before producing final answers. However, RL rewards are typically assigned based on final answers, providing little or no direct supervision over intermediate reasoning. This can lead to deceptive safety alignment, where the reasoning trace and final answer convey inconsistent safety signals. To systematically investigate this phenomenon, we introduce DSAR (Deceptive Safety Alignment Rate), a metric that jointly assesses reasoning traces and final answers to quantify their safety inconsistency. Across multiple LRMs and benchmarks, we find that deceptive safety alignment is pervasive under standard prompting conditions and is substantially amplified under prefilling attacks. We further provide a hidden representation analysis showing that models exhibit stronger safety discrimination at the final-answer stage than during intermediate reasoning. To close this gap, we propose SARA (Safety-Aware Reasoning Alignment), an RL-based method that rewards both safety-aware reasoning and safe final answers, encouraging early harmful intent recognition and enforcing reasoning-answer consistency. Experiments show that SARA significantly mitigates deceptive safety alignment under both standard and adversarial settings while preserving helpfulness and utility. Code is available at https://github.com/xzhou98/SARA.