Large Reasoning Models (LRMs) rely on explicit chain-of-thought (CoT) reasoning and large context windows to achieve strong performance on complex tasks, but these features also introduce new attack surfaces. We show that LRMs' reasoning processes can be systematically steered by prepending counter-aligned few-shot conversations containing explicit CoT traces, leading to unsafe generations on harmful queries and unwarranted refusals on benign ones. We formalize this attack as SRCF (Steering Reasoning via Counter-Aligned Few-shot Conversations) that operates solely through a flexible conversational interface and requires no access to the model's parameters and gradients. Our key insight is that SRCF exploits an adversarial generalization issue that induces a representation drift, causing the representations of benign and harmful inputs to shift in a similar direction. This observation motivates our post-training defense, ARCF (Aligning Reasoning via Counter-Aligned Few-Shot Conversations), which exposes models to counter-aligned conversational contexts while enforcing aligned targets. ARCF is compatible with existing post-training methods and consistently improves safety and helpfulness without degrading utility.
Figures & tables
Figure 1: Illustration of proposed SRCF attack and ARCF post-training framework. Left panel: Under the SRCF attack, the input queries are prepended by a few-shot conversations that contain counter-aligned CoTs, leading to over-refusal for benign queries or unsafe generation for harmful queries. Right panel: ARCF improves both helpfulness and safety of LRMs by training the model to recover correct behavior from counter-aligned few-shot conversations under aligned supervision.
Methods
Conversation history
Response
CoTs
advICL
×
✓
✓
Response-Only
✓
✓
×
SRCF
✓
✓
✓
Table 1: Component-level comparison between SRCF and baselines. Each baseline method selectively includes or removes conversation history, counter-aligned responses, and counter-aligned CoTs, enabling controlled study of the mechanisms underlying SRCF.
Method
Jailbreak (Adv-Bench)
Over-refusal (OR-Bench)
DSLLaMa3-8B
DSQwen3-8B
DSQwen2-14B
GPT-oss-20B
Avg. ↑
DSLLaMa3-8B
DSQwen3-8B
DSQwen2-14B
GPT-oss-20B
Avg. ↑
C
R
C
R
C
R
C
R
C
R
R
R
R
R
R
0-shot
Original
85.25
40.75
5.25
1.00
78.75
50.25
54.75
0.00
56.00
23.00
0.80
49.60
3.20
16.20
17.45
4-shot
advICL
87.00
30.50
3.75
0.00
92.75
40.25
24.50
0.00
52.00
17.69
12.20
68.40
14.80
32.60
32.00
Table 2: Jailbreak and over-refusal performance of SRCF vs. baselines across multiple LRMs and varying different shot settings. For jailbreak evaluation (AdvBench), we report 1–SS ↑ , evaluated separately on the generated CoT ( C ) and the final response ( R ). Higher values indicate stronger attack effectiveness. For over-refusal evaluation (OR-Bench), we evaluate only the final response ( R ) and report the Refusal Rate (RR ↑ ) , where higher values correspond to stronger induced over-refusal on benign queries. For both evaluations, we report Avg. ↑ , an average score to summarize attack effectiveness across different models. More detailed definitions of evaluation metrics are provided in Section A.2.2 . Best results are shown in bold . Rows highlighted in gray correspond to our SRCF .
Figure 2: PCA projections of final-layer hidden states for benign, harmful, and SRCF-attacked prompts (jailbreak setting). Each point is the last token hidden state of the original released model without additional alignment applied in this work. For SRCF-attacked variants, the same harmful conversational history is prepended to both prompt types. The black arrow indicates the dominant direction of representation drift observed for benign prompts, while the red arrow indicates the shift observed for harmful prompts under SRCF attack. SRCF induces a pronounced representation drift, causing the representations of benign and harmful prompts to move along converged trajectories and reducing their separability. The arrows are manually added for qualitative illustration.
Method
16-Shots (SRCF)
0-Shot
Avg.
Jailbreak
Over-refusal
Jailbreak
Over-refusal
Utility
Adv-bench
OR-Bench
SafeChain
FORTRESS
H-CoT
Prefill
XSTEST
FORTRESS
AIME-25
GSM8k
MMLU-Pro
C
R
(1- RR) ↑
C
R
C
R
C
R
C
R
(1- RR) ↑
(1- RR) ↑
pass@8
pass@1
Acc. ↑
DSLLaMA3-8B Models
Original
4.00
36.75
89.00
65.00
85.40
32.00
43.80
6.00
10.00
55.20
54.40
98.44
99.80
53.33
61.26
48.56
55.24
STAR
6.75
16.50
14.60
61.60
96.00
39.00
77.00
4.00
6.00
67.20
64.20
51.10
81.80
56.66
65.58
48.81
49.43
Table 3: Evaluation of the model on Jailbreak, over-refusal, and utility tasks. Performance of different post-training methods with and without ARCF across three LRMs. We report Safety Score ( SS ↑ ) on jailbreak benchmarks, 1-Refusal Rate ( 1-RR ↑ ) on over-refusal benchmarks, and accuracy on utility benchmarks. Results are shown for both the 16-shot SRCF attack setting and the 0-shot setting. We report Avg. ↑ , an overall summary metric to evaluate the method’s performance across all tasks. More detailed definitions of evaluation metrics are provided in Section 3.2 . Bold and underlined values denote the best and second-best results, respectively. Higher values indicate better performance for all metrics. Rows highlighted in gray correspond to the methods with our ARCF.
Figure 3: PCA visualizations of the final-layer hidden states for benign prompts, harmful prompts, and their SRCF-attacked variants in the jailbreak setting. Each point corresponds to the last-token hidden state of the model after alignment with GRPO-ARCF . The black arrow indicates the dominant direction of representation drift observed for benign prompts, while the red arrow indicates the shift observed for harmful prompts after prepending SRCF conversations. Compared to the base model (Figure 2 ), ARCF induces more parallel shifts for benign and harmful prompts and preserves clearer representation separation between them, enabling the model to better distinguish benign and harmful inputs under SRCF attack. The arrows are manually added for qualitative illustration.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Illustration of our SRCF and baseline methods used in our study. SRCF preserves prior conversation history, including both explicit reasoning traces and final response. Response-Only also retains the conversation history but removes reasoning traces. advIC L presents the same content as standard in-context examples without treating it as prior conversation history.
Figure 5: Layer-wise cosine similarity of last-token hidden representations across four LRMs under three settings: benign-benign pairs (B-B), benign-harmful pairs without attack (B-H), and benign-harmful pairs under 16-shot SRCF attack (B-H (SRCF)). Shaded regions denote standard deviation across sampled pairs. Across models, B-B generally exhibits higher cosine similarity than B-H, indicating a representational distinction between benign and harmful prompts. More importantly, B-H (SRCF) consistently shows higher cosine similarity than B-H across most layers, suggesting that SRCF shifts harmful prompts toward benign ones in hidden-state space and reduces their separability.
Methods
Training Epoch
GPU-hours / epoch
STAR
2
0.204
STAR (4-shot)
2
0.208
STAR (8-shot)
2
0.212
SFT
2
0.411
SFT (4-shot)
2
0.417
SFT (8-shot)
2
0.425
Appendix
Table 4: Fine-tuning cost of different methods implemented with or without the ARCF framework. We report the number of training epochs and the average GPU-hours consumed per epoch. All methods are trained under identical model, optimizer, and hardware settings; the only differences are the training objective and the number of few-shot conversations used to augment the fine-tuning data.
Figure 6: Performance of ARCF with STAR, SFT, and GRPO as the augmentation ratio varies from 0% to 100%. We report the average Safety Score (jailbreak) and 1 − Refusal Rate (over-refusal) under both 16-shot SRCF and 0-shot settings. More detailed results are shown in Appendix Table 7 .
Figure 7: Performance of ARCF with STAR, SFT, and GRPO as the number of shots varies from 0 to 8. We report the average Safety Score (jailbreak) and 1 − Refusal Rate (over-refusal) under both 16-shot SRCF and 0-shot settings. More detailed results are shown in Appendix Table 8 .
Figure 8: Layer-wise binary probing results on the original DSQwen2-14B before additional safety alignment under no-attack, 4-shot SRCF, and 16-shot SRCF settings. The probes are trained on no-attack representations of benign and harmful prompts and then applied frozen to representations under each test setting. Panel (a) trains probes on full-dimensional hidden states, while panel (b) trains independent probes on PCA-reduced hidden states with 64 components. We treat benign prompts as the negative class and harmful prompts as the positive class. So in each panel, Left shows accuracy on benign prompts, i.e., the true negative rate (TNR), and Right shows accuracy on harmful prompts, i.e., the true positive rate (TPR/recall). Under SRCF attack, TNR remains near 1.0 across layers, while TPR/recall drops to near zero across most later layers, indicating that SRCF causes the frozen benign–harmful probe to increasingly classify attacked harmful prompts as benign. The consistent trend in both full-dimensional and PCA-reduced settings suggests that this effect is not an artifact of high-dimensional probe fitting.
Figure 9: Layer-wise binary probing results on the original DSQwen2-14B after ARCF post-training under no-attack, 4-shot SRCF, and 16-shot SRCF settings. The probe is trained on the same setup as Figure 8 . Panel (a) trains probes on full-dimensional hidden states, while panel (b) trains independent probes on PCA-reduced hidden states with 64 components. We treat benign prompts as the negative class and harmful prompts as the positive class. So in each panel, Left shows accuracy on benign prompts, i.e., the true negative rate (TNR), and Right shows accuracy on harmful prompts, i.e., the true positive rate (TPR/recall). Compared to the original model, ARCF post-training substantially restores harmful prompt accuracy across layers under SRCF attack, with the probe recovering above chance level in multiple intermediate and later layers. This indicates that ARCF recovers the representational separability between benign and harmful prompts under SRCF attack, consistent with the PCA results in Figure 3 .
Figure 10: PCA projections of the final-layer hidden states for benign prompts, harmful prompts, and their attacked variants across SRCF and baseline methods. Each point corresponds to the last-token hidden state of the base model prior to safety alignment. The black arrow indicates the dominant direction of representational shift observed for benign prompts, while the red arrow indicates the shift observed for harmful prompts after attacking.
Model
LLM-Guard
4-shot
8-shot
16-shot
C
R
C
R
C
R
DSQwen2–14B
Qwen3Guard-Gen-8B
81.0
67.5
97.3
92.3
97.3
95.5
Llama-Guard-4-14b
75.8
97.3
92.8
88.0
95.5
91.0
GPT-oss-safeguard-20b
79.0
92.3
95.8
89.3
91.3
93.5
Appendix
Table 5: Comparison of safety judgments from three frontier LLM judges on DSQwen2-14B generations under SRCF. We report 1-Safety Score (1-SS) for both the generated CoT (C) and the final response (R) under 4-shot, 8-shot, and 16-shot settings. The overall trends are highly consistent across judges, indicating that our main findings are robust to the choice of safety evaluator.
Model
Method
Jailbreak
Over-refusal
SRCF-16-shots
SafeChain
SRCF-16-shots
XSTEST
C
R
C
R
(1- RR)
(1- RR)
DS Qwen3-8B
Original
4.0
11.0
77.7
91.2
23.4
36.7
SAFEPATH
5.0
10.0
70.5
92.5
34.2
75.3
SAFEPATH-ARCF
30.0
55.8
72.7
93.7
86.8
84.9
DS Qwen3-8B
Original
4.8
6.5
67.8
84.1
60.8
98.7
Appendix
Table 6: Additional evaluation of a recent safety-alignment method, SafePath, with and without ARCF on DSQwen3-8B and DSQwen2-14B. We report Safety Score (SS) on jailbreak benchmarks and 1-Refusal Rate (1-RR) on over-refusal benchmarks, where higher values indicate better performance. Across both models, incorporating ARCF consistently improves SafePath under both 16-shot and 0-shot settings across different tasks.
Model
Method
Jailbreak
Over-refusal
Utility
Avg.
DSLLaMA3-8B
Original
39.25
95.75
54.38
55.24
STAR
43.83
49.17
57.02
49.43
SFT
39.03
86.10
54.54
53.99
GRPO
64.82
81.44
56.58
66.11
STAR-ARCF
62.92
51.79
56.02
56.55
SFT-ARCF
51.88
94.17
55.40
62.57
Appendix
Figure 11: Average evaluation across jailbreak, over-refusal, and utility tasks. For each task, we aggregate the same metrics reported in Table 3 . Specifically, jailbreak performance is measured using Safety Score ( SS ↑ ), over-refusal performance is measured using 1-Refusal Rate ( 1-RR ↑ ), and utility is measured using task-specific accuracy. For each task, we first average the performance over all benchmarks associated with that task. We then compute the harmonic mean of these three task-level averages as the overall evaluation score.
Method
16-Shots (SRCF)
0-Shot
Avg. ↑
Jailbreak
Over-refusal
Jailbreak
Over-refusal
Adv-bench
OR-Bench
SafeChain
FORTRESS
XSTEST
FORTRESS
C
R
(1- RR) ↑
C
R
C
R
(1- RR) ↑
(1- RR) ↑
Original
4.00
11.00
23.40
77.70
91.20
37.60
36.40
36.67
6.40
29.24
0 %
STAR
7.25
46.50
18.20
67.80
94.94
36.20
62.60
53.33
88.80
52.99
Appendix
Table 7: Detailed results of ARCF under different augmentation ratios. We report jailbreak performance and over-refusal performance under both the 16-shot SRCF attack setting and the 0-shot setting. Results are shown for STAR, SFT, and GRPO, with augmentation ratios ranging from 0% (no ARCF) to 100%.
Method
16-Shots (SRCF)
0-Shot
Avg. ↑
Jailbreak
Over-refusal
Jailbreak
Over-refusal
Adv-bench
OR-Bench
SafeChain
FORTRESS
XSTEST
FORTRESS
C
R
(1- RR) ↑
C
R
C
R
(1- RR) ↑
(1- RR) ↑
Original
4.00
11.00
23.40
77.70
91.20
37.60
36.40
36.67
6.40
29.24
0-shot
STAR
7.25
46.50
18.20
67.80
94.94
36.20
62.60
53.33
88.80
52.99
Appendix
Table 8: Detailed results of ARCF under different numbers of Augmentation Shots. We report jailbreak performance and over-refusal performance under both the 16-shot SRCF attack setting and the 0-shot setting. Results are shown for STAR, SFT, and GRPO, with the number of prepended few-shot conversations from 0 (no ARCF) to 8.
Figure 12: Refusal-style classification policy used for over-refusal evaluation. The policy assigns each final model response to Full Refusal, Partial Refusal, or Non-Refusal based solely on the presence of refusal-like language, independent of topic safety or content correctness. Refusal Rate (RR) counts both Full and Partial Refusals on benign prompts.
Figure 13: Rule-based safety classification policy for jailbreak evaluation. Each generated chain-of-thought (CoT) and final response is evaluated separately against a fixed set of safety rules and classified as Safe or Unsafe based on the presence of any rule violations.
Figure 14: Prompting instruction used to generate counter-aligned CoT conversations. We use the above instruction to guide a separate LLM (DSLLaMA3-70B) to synthesize counter-aligned reasoning traces for both harmful prompts.The instruction explicitly controls the stance and failure mode of the generated CoTs to induce unsafe reasoning (jailbreak).
Large Reasoning Models (LRMs) are commonly trained with reinforcement learning (RL) to improve their generation of chain-of-thought (CoT) reasoning before producing final answers. However, RL rewards are typically assigned based on final answers, providing little or no direct supervision over intermediate reasoning. This can lead to deceptive safety alignment, where the reasoning trace and final answer convey inconsistent safety signals. To systematically investigate this phenomenon, we introduce DSAR (Deceptive Safety Alignment Rate), a metric that jointly assesses reasoning traces and final answers to quantify their safety inconsistency. Across multiple LRMs and benchmarks, we find that deceptive safety alignment is pervasive under standard prompting conditions and is substantially amplified under prefilling attacks. We further provide a hidden representation analysis showing that models exhibit stronger safety discrimination at the final-answer stage than during intermediate reasoning. To close this gap, we propose SARA (Safety-Aware Reasoning Alignment), an RL-based method that rewards both safety-aware reasoning and safe final answers, encouraging early harmful intent recognition and enforcing reasoning-answer consistency. Experiments show that SARA significantly mitigates deceptive safety alignment under both standard and adversarial settings while preserving helpfulness and utility. Code is available at https://github.com/xzhou98/SARA.
Large reasoning models (LRMs) achieve strong performance on complex reasoning tasks but often generate harmful responses to malicious user queries. This paper investigates the underlying cause of these safety risks and shows that the issue lies in the reasoning structure itself. Based on this insight, we claim that effective safety alignment can be achieved by altering the reasoning structure. We propose AltTrain, a simple yet effective post training method that explicitly alters the reasoning structure of LRMs. AltTrain is both practical and generalizable, requiring no complex reinforcement learning (RL) training or reward design, only supervised finetuning (SFT) with a lightweight 1K training examples. Experiments across LRM backbones and model sizes demonstrate strong safety alignment, along with robust generalization across reasoning, QA, summarization, and multilingual setting.
Large Reasoning Models (LRMs) pose a dual-surface safety challenge: both intermediate reasoning traces and final answers can contain harmful content. Existing alignment methods often operate at the whole-response level, allowing unsafe reasoning to be masked by a safe-looking final answer. We propose Segment-aware Listwise Target DPO (SaLT-DPO), which addresses this gap through three mechanisms: (1) segment-aware listwise alignment that decomposes responses into reasoning and answer segments, independently scores each segment's safety, and aligns length-normalized segment rewards with soft target distributions over multiple candidates; (2) joint safety coherence regularization that applies a weakest-link principle to promote safety consistency across both segments; and (3) utility anchoring on benign prompts to mitigate over-refusal and reasoning degradation. Experiments on three LRMs show that SaLT-DPO consistently reduces unsafe rates for both reasoning and answer segments while mitigating degradation in benign compliance and preserving general reasoning performance. Ablation studies demonstrate the complementary contributions of its components.
JungMin Yun, Junehyoung Kwon, Hayeong Ryu +3
Department of Artificial Intelligence, Chung-Ang University · Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University