Alignment of LRMs via Counter-Aligned Few-Shot Conversation Exposure

Xiangyu Zhou; Saleh Zare Zade; Dongxiao Zhu

cs.AIAug 16, 2026

Alignment of LRMs via Counter-Aligned Few-Shot Conversation Exposure

Authors: Xiangyu Zhou, Saleh Zare Zade, Dongxiao Zhu

Organizations: Wayne State University

Abstract

Large Reasoning Models (LRMs) rely on explicit chain-of-thought (CoT) reasoning and large context windows to achieve strong performance on complex tasks, but these features also introduce new attack surfaces. We show that LRMs' reasoning processes can be systematically steered by prepending counter-aligned few-shot conversations containing explicit CoT traces, leading to unsafe generations on harmful queries and unwarranted refusals on benign ones. We formalize this attack as SRCF (Steering Reasoning via Counter-Aligned Few-shot Conversations) that operates solely through a flexible conversational interface and requires no access to the model's parameters and gradients. Our key insight is that SRCF exploits an adversarial generalization issue that induces a representation drift, causing the representations of benign and harmful inputs to shift in a similar direction. This observation motivates our post-training defense, ARCF (Aligning Reasoning via Counter-Aligned Few-Shot Conversations), which exposes models to counter-aligned conversational contexts while enforcing aligned targets. ARCF is compatible with existing post-training methods and consistently improves safety and helpfulness without degrading utility.

Figures & tables

Figure 1: Illustration of proposed SRCF attack and ARCF post-training framework. Left panel: Under the SRCF attack, the input queries are prepended by a few-shot conversations that contain counter-aligned CoTs, leading to over-refusal for benign queries or unsafe generation for harmful queries. Right panel: ARCF improves both helpfulness and safety of LRMs by training the model to recover correct behavior from counter-aligned few-shot conversations under aligned supervision.

Methods	Conversation history	Response	CoTs
advICL	$\times$	$\checkmark$	$\checkmark$
Response-Only	$\checkmark$	$\checkmark$	$\times$
SRCF	$\checkmark$	$\checkmark$	$\checkmark$

Table 1: Component-level comparison between SRCF and baselines. Each baseline method selectively includes or removes conversation history, counter-aligned responses, and counter-aligned CoTs, enabling controlled study of the mechanisms underlying SRCF.

Method	Jailbreak (Adv-Bench)										Over-refusal (OR-Bench)
	DSLLaMa3-8B		DSQwen3-8B		DSQwen2-14B		GPT-oss-20B		Avg. $\uparrow$		DSLLaMa3-8B	DSQwen3-8B	DSQwen2-14B	GPT-oss-20B	Avg. $\uparrow$
	C	R	C	R	C	R	C	R	C	R	R	R	R	R	R
		0-shot
Original	85.25	40.75	5.25	1.00	78.75	50.25	54.75	0.00	56.00	23.00	0.80	49.60	3.20	16.20	17.45
		4-shot
advICL	87.00	30.50	3.75	0.00	92.75	40.25	24.50	0.00	52.00	17.69	12.20	68.40	14.80	32.60	32.00

Table 2: Jailbreak and over-refusal performance of SRCF vs. baselines across multiple LRMs and varying different shot settings. For jailbreak evaluation (AdvBench), we report 1–SS

\uparrow

, evaluated separately on the generated CoT ( C ) and the final response ( R ). Higher values indicate stronger attack effectiveness. For over-refusal evaluation (OR-Bench), we evaluate only the final response ( R ) and report the Refusal Rate (RR

\uparrow

) , where higher values correspond to stronger induced over-refusal on benign queries. For both evaluations, we report Avg.

\uparrow

, an average score to summarize attack effectiveness across different models. More detailed definitions of evaluation metrics are provided in Section A.2.2 . Best results are shown in bold . Rows highlighted in gray correspond to our SRCF .

Panel of Figure 2 — Figure 2: PCA projections of final-layer hidden states for benign, harmful, and SRCF-attacked prompts (jailbreak setting). Each point is the last token hidden state of the original released model without additional alignment applied in this work. For SRCF-attacked variants, the same harmful conversational history is prepended to both prompt types. The black arrow indicates the dominant direction of representation drift observed for benign prompts, while the red arrow indicates the shift observed for harmful prompts under SRCF attack. SRCF induces a pronounced representation drift, causing the representations of benign and harmful prompts to move along converged trajectories and reducing their separability. The arrows are manually added for qualitative illustration.

Method	16-Shots (SRCF)			0-Shot													Avg.
	Jailbreak		Over-refusal	Jailbreak								Over-refusal		Utility
	Adv-bench		OR-Bench	SafeChain		FORTRESS		H-CoT		Prefill		XSTEST	FORTRESS	AIME-25	GSM8k	MMLU-Pro
	C	R	(1- RR) $\uparrow$	C	R	C	R	C	R	C	R	(1- RR) $\uparrow$	(1- RR) $\uparrow$	pass@8	pass@1	Acc. $\uparrow$
	DSLLaMA3-8B Models
Original	4.00	36.75	89.00	65.00	85.40	32.00	43.80	6.00	10.00	55.20	54.40	98.44	99.80	53.33	61.26	48.56	55.24
STAR	6.75	16.50	14.60	61.60	96.00	39.00	77.00	4.00	6.00	67.20	64.20	51.10	81.80	56.66	65.58	48.81	49.43

Table 3: Evaluation of the model on Jailbreak, over-refusal, and utility tasks. Performance of different post-training methods with and without ARCF across three LRMs. We report Safety Score ( SS

\uparrow

) on jailbreak benchmarks, 1-Refusal Rate ( 1-RR

\uparrow

) on over-refusal benchmarks, and accuracy on utility benchmarks. Results are shown for both the 16-shot SRCF attack setting and the 0-shot setting. We report Avg.

\uparrow

, an overall summary metric to evaluate the method’s performance across all tasks. More detailed definitions of evaluation metrics are provided in Section 3.2 . Bold and underlined values denote the best and second-best results, respectively. Higher values indicate better performance for all metrics. Rows highlighted in gray correspond to the methods with our ARCF.

Panel of Figure 3 — Figure 3: PCA visualizations of the final-layer hidden states for benign prompts, harmful prompts, and their SRCF-attacked variants in the jailbreak setting. Each point corresponds to the last-token hidden state of the model after alignment with GRPO-ARCF . The black arrow indicates the dominant direction of representation drift observed for benign prompts, while the red arrow indicates the shift observed for harmful prompts after prepending SRCF conversations. Compared to the base model (Figure 2 ), ARCF induces more parallel shifts for benign and harmful prompts and preserves clearer representation separation between them, enabling the model to better distinguish benign and harmful inputs under SRCF attack. The arrows are manually added for qualitative illustration.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Illustration of our SRCF and baseline methods used in our study. SRCF preserves prior conversation history, including both explicit reasoning traces and final response. Response-Only also retains the conversation history but removes reasoning traces. advIC L presents the same content as standard in-context examples without treating it as prior conversation history.

Panel of Figure 5 — Figure 5: Layer-wise cosine similarity of last-token hidden representations across four LRMs under three settings: benign-benign pairs (B-B), benign-harmful pairs without attack (B-H), and benign-harmful pairs under 16-shot SRCF attack (B-H (SRCF)). Shaded regions denote standard deviation across sampled pairs. Across models, B-B generally exhibits higher cosine similarity than B-H, indicating a representational distinction between benign and harmful prompts. More importantly, B-H (SRCF) consistently shows higher cosine similarity than B-H across most layers, suggesting that SRCF shifts harmful prompts toward benign ones in hidden-state space and reduces their separability.

Methods	Training Epoch	GPU-hours / epoch
STAR	2	0.204
STAR (4-shot)	2	0.208
STAR (8-shot)	2	0.212
SFT	2	0.411
SFT (4-shot)	2	0.417
SFT (8-shot)	2	0.425

Appendix

Table 4: Fine-tuning cost of different methods implemented with or without the ARCF framework. We report the number of training epochs and the average GPU-hours consumed per epoch. All methods are trained under identical model, optimizer, and hardware settings; the only differences are the training objective and the number of few-shot conversations used to augment the fine-tuning data.

Panel of Figure 6 — Figure 6: Performance of ARCF with STAR, SFT, and GRPO as the augmentation ratio varies from 0% to 100%. We report the average Safety Score (jailbreak) and 1 $-$ Refusal Rate (over-refusal) under both 16-shot SRCF and 0-shot settings. More detailed results are shown in Appendix Table 7 .

Panel of Figure 7 — Figure 7: Performance of ARCF with STAR, SFT, and GRPO as the number of shots varies from 0 to 8. We report the average Safety Score (jailbreak) and 1 $-$ Refusal Rate (over-refusal) under both 16-shot SRCF and 0-shot settings. More detailed results are shown in Appendix Table 8 .

Panel of Figure 8 — Figure 8: Layer-wise binary probing results on the original DSQwen2-14B before additional safety alignment under no-attack, 4-shot SRCF, and 16-shot SRCF settings. The probes are trained on no-attack representations of benign and harmful prompts and then applied frozen to representations under each test setting. Panel (a) trains probes on full-dimensional hidden states, while panel (b) trains independent probes on PCA-reduced hidden states with 64 components. We treat benign prompts as the negative class and harmful prompts as the positive class. So in each panel, Left shows accuracy on benign prompts, i.e., the true negative rate (TNR), and Right shows accuracy on harmful prompts, i.e., the true positive rate (TPR/recall). Under SRCF attack, TNR remains near 1.0 across layers, while TPR/recall drops to near zero across most later layers, indicating that SRCF causes the frozen benign–harmful probe to increasingly classify attacked harmful prompts as benign. The consistent trend in both full-dimensional and PCA-reduced settings suggests that this effect is not an artifact of high-dimensional probe fitting.

Panel of Figure 9 — Figure 9: Layer-wise binary probing results on the original DSQwen2-14B after ARCF post-training under no-attack, 4-shot SRCF, and 16-shot SRCF settings. The probe is trained on the same setup as Figure 8 . Panel (a) trains probes on full-dimensional hidden states, while panel (b) trains independent probes on PCA-reduced hidden states with 64 components. We treat benign prompts as the negative class and harmful prompts as the positive class. So in each panel, Left shows accuracy on benign prompts, i.e., the true negative rate (TNR), and Right shows accuracy on harmful prompts, i.e., the true positive rate (TPR/recall). Compared to the original model, ARCF post-training substantially restores harmful prompt accuracy across layers under SRCF attack, with the probe recovering above chance level in multiple intermediate and later layers. This indicates that ARCF recovers the representational separability between benign and harmful prompts under SRCF attack, consistent with the PCA results in Figure 3 .

Panel of Figure 10 — Figure 10: PCA projections of the final-layer hidden states for benign prompts, harmful prompts, and their attacked variants across SRCF and baseline methods. Each point corresponds to the last-token hidden state of the base model prior to safety alignment. The black arrow indicates the dominant direction of representational shift observed for benign prompts, while the red arrow indicates the shift observed for harmful prompts after attacking.

Model	LLM-Guard	4-shot		8-shot		16-shot
Model	LLM-Guard	C	R	C	R	C	R
DSQwen2–14B	Qwen3Guard-Gen-8B	81.0	67.5	97.3	92.3	97.3	95.5
	Llama-Guard-4-14b	75.8	97.3	92.8	88.0	95.5	91.0
	GPT-oss-safeguard-20b	79.0	92.3	95.8	89.3	91.3	93.5

Appendix

Table 5: Comparison of safety judgments from three frontier LLM judges on DSQwen2-14B generations under SRCF. We report 1-Safety Score (1-SS) for both the generated CoT (C) and the final response (R) under 4-shot, 8-shot, and 16-shot settings. The overall trends are highly consistent across judges, indicating that our main findings are robust to the choice of safety evaluator.

Model	Method	Jailbreak				Over-refusal
		SRCF-16-shots		SafeChain		SRCF-16-shots	XSTEST
		C	R	C	R	(1- RR)	(1- RR)
DS Qwen3-8B	Original	4.0	11.0	77.7	91.2	23.4	36.7
	SAFEPATH	5.0	10.0	70.5	92.5	34.2	75.3
	SAFEPATH-ARCF	30.0	55.8	72.7	93.7	86.8	84.9
DS Qwen3-8B	Original	4.8	6.5	67.8	84.1	60.8	98.7

Appendix

Table 6: Additional evaluation of a recent safety-alignment method, SafePath, with and without ARCF on DSQwen3-8B and DSQwen2-14B. We report Safety Score (SS) on jailbreak benchmarks and 1-Refusal Rate (1-RR) on over-refusal benchmarks, where higher values indicate better performance. Across both models, incorporating ARCF consistently improves SafePath under both 16-shot and 0-shot settings across different tasks.

Model	Method	Jailbreak	Over-refusal	Utility	Avg.
DSLLaMA3-8B	Original	39.25	95.75	54.38	55.24
	STAR	43.83	49.17	57.02	49.43
	SFT	39.03	86.10	54.54	53.99
	GRPO	64.82	81.44	56.58	66.11
	STAR-ARCF	62.92	51.79	56.02	56.55
	SFT-ARCF	51.88	94.17	55.40	62.57

Appendix

Figure 11: Average evaluation across jailbreak, over-refusal, and utility tasks. For each task, we aggregate the same metrics reported in Table 3 . Specifically, jailbreak performance is measured using Safety Score ( SS

\uparrow

), over-refusal performance is measured using 1-Refusal Rate ( 1-RR

\uparrow

), and utility is measured using task-specific accuracy. For each task, we first average the performance over all benchmarks associated with that task. We then compute the harmonic mean of these three task-level averages as the overall evaluation score.

Method	16-Shots (SRCF)			0-Shot						Avg. $\uparrow$
	Jailbreak		Over-refusal	Jailbreak				Over-refusal
	Adv-bench		OR-Bench	SafeChain		FORTRESS		XSTEST	FORTRESS
	C	R	(1- RR) $\uparrow$	C	R	C	R	(1- RR) $\uparrow$	(1- RR) $\uparrow$
Original	4.00	11.00	23.40	77.70	91.20	37.60	36.40	36.67	6.40	29.24
0 %
STAR	7.25	46.50	18.20	67.80	94.94	36.20	62.60	53.33	88.80	52.99

Appendix

Table 7: Detailed results of ARCF under different augmentation ratios. We report jailbreak performance and over-refusal performance under both the 16-shot SRCF attack setting and the 0-shot setting. Results are shown for STAR, SFT, and GRPO, with augmentation ratios ranging from 0% (no ARCF) to 100%.

Method	16-Shots (SRCF)			0-Shot						Avg. $\uparrow$
	Jailbreak		Over-refusal	Jailbreak				Over-refusal
	Adv-bench		OR-Bench	SafeChain		FORTRESS		XSTEST	FORTRESS
	C	R	(1- RR) $\uparrow$	C	R	C	R	(1- RR) $\uparrow$	(1- RR) $\uparrow$
Original	4.00	11.00	23.40	77.70	91.20	37.60	36.40	36.67	6.40	29.24
0-shot
STAR	7.25	46.50	18.20	67.80	94.94	36.20	62.60	53.33	88.80	52.99

Appendix

Table 8: Detailed results of ARCF under different numbers of Augmentation Shots. We report jailbreak performance and over-refusal performance under both the 16-shot SRCF attack setting and the 0-shot setting. Results are shown for STAR, SFT, and GRPO, with the number of prepended few-shot conversations from 0 (no ARCF) to 8.

Figure 12: Refusal-style classification policy used for over-refusal evaluation. The policy assigns each final model response to Full Refusal, Partial Refusal, or Non-Refusal based solely on the presence of refusal-like language, independent of topic safety or content correctness. Refusal Rate (RR) counts both Full and Partial Refusals on benign prompts.

Figure 13: Rule-based safety classification policy for jailbreak evaluation. Each generated chain-of-thought (CoT) and final response is evaluated separately against a fixed set of safety rules and classified as Safe or Unsafe based on the presence of any rule violations.

Figure 14: Prompting instruction used to generate counter-aligned CoT conversations. We use the above instruction to guide a separate LLM (DSLLaMA3-70B) to synthesize counter-aligned reasoning traces for both harmful prompts.The instruction explicitly controls the stance and failure mode of the generated CoTs to induce unsafe reasoning (jailbreak).

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 613 papers

Towards Mitigating Deceptive Safety Alignment in Large Reasoning Models
Sep 28, 2026Xiangyu Zhou, Saleh Zare Zade, Rafi Ibn Sultan +2Language Model Safety Evaluation Deception in Language Models
Wayne State University
Reasoning Structure Matters for Safety Alignment of Reasoning Models
Apr 21, 2026Yeonjun In, Wonjoong Kim, Sangwu Park +1LLM Safety Alignment Large Reasoning Models
KAIST
Beyond Safe Answers: Segment-Aware Listwise Alignment for Reasoning Safety in Large Reasoning Models
Sep 14, 2026JungMin Yun, Junehyoung Kwon, Hayeong Ryu +3LLM Alignment Direct Preference Optimization
Department of Artificial Intelligence, Chung-Ang University · Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University

Sep 28, 2026J/K move · Enter open · S save

Towards Mitigating Deceptive Safety Alignment in Large Reasoning Models

Xiangyu Zhou, Saleh Zare Zade, Rafi Ibn Sultan, Alexander Kotov+1

Wayne State University

Language Model Safety Evaluation Deception in Language Models RL for Language Model Reasoning LLM Safety Alignment Large Reasoning Models

Open paper AskPDF arXiv

Large Reasoning Models (LRMs) are commonly trained with reinforcement learning (RL) to improve their generation of chain-of-thought (CoT) reasoning before producing final answers. However, RL rewards are typically assigned based on final answers, providing little or no direct supervision over intermediate reasoning. This can lead to deceptive safety alignment, where the reasoning trace and final answer convey inconsistent safety signals. To systematically investigate this phenomenon, we introduce DSAR (Deceptive Safety Alignment Rate), a metric that jointly assesses reasoning traces and final answers to quantify their safety inconsistency. Across multiple LRMs and benchmarks, we find that deceptive safety alignment is pervasive under standard prompting conditions and is substantially amplified under prefilling attacks. We further provide a hidden representation analysis showing that models exhibit stronger safety discrimination at the final-answer stage than during intermediate reasoning. To close this gap, we propose SARA (Safety-Aware Reasoning Alignment), an RL-based method that rewards both safety-aware reasoning and safe final answers, encouraging early harmful intent recognition and enforcing reasoning-answer consistency. Experiments show that SARA significantly mitigates deceptive safety alignment under both standard and adversarial settings while preserving helpfulness and utility. Code is available at https://github.com/xzhou98/SARA.

Abstract

LLM Alignment

Explore similar work

Towards Mitigating Deceptive Safety Alignment in Large Reasoning Models

Reasoning Structure Matters for Safety Alignment of Reasoning Models

Beyond Safe Answers: Segment-Aware Listwise Alignment for Reasoning Safety in Large Reasoning Models