The appropriate governing norms and principles for the alignment of Large Language Models (LLMs) often depend on cultural context and deployment application. However, most alignment techniques focus on training models to conform to human preferences or constitutional principles specified at training time and subsequently fixed. In practice, many deployers rely on system prompting to implement contextual norms and guardrails at inference time. Prior work has shown that post-generation reflective reasoning with critique and revision can improve conformance compared to simple system prompting, but such methods are limited to the reasoning abilities inherent in the trained model. We introduce Reflect-Reasoning, a training method that teaches models how to reason about alignment to principles, rather than simply which principles to follow. Across five models, Reflect-Reasoning consistently outperforms both system prompting and explicit prompting strategies for in-context reflective reasoning. These improvements extend and generalize to entirely out-of-distribution domains of medical and financial principles. We additionally demonstrate that Reflect-Reasoning reduces the occurrence of rare but serious principle violations. Finally, we show that Reflect-Reasoning does not erode the safety engineering of a model and is robust against jailbreaking attempts using adversarial constitutions. Overall, our results demonstrate that Reflect-Reasoning is a scalable method for training constitutional alignment as a generalizable reasoning capability, enabling models to adapt to novel principles and produce transparent reasoning traces without additional fine-tuning.
Figures & tables
Figure 1. Demonstration of reflective reasoning traces on different deployment-specific constitutions of principles A visual diagram that demonstrates reflective reasoning traces on different deployment-specific constitutions of principles: One for education with principles like age-appropriate content and another for medical with principles like sourced objectivity. Base generations are evaluated on the basis of the principles; if satisfactory, the base generation is tagged KEEP; otherwise, REVISE, in which case the model critiques on the basis of the principles and then drafts a final revision.
Figure 2. Reflect-Reasoning training process A visual diagram that demonstrates the \newreflecttraining process as described informally in the main text and formally in the algorithm pseudocode.
Figure 3. Pairwise win rates for Reflect-Reasoning (RR) against Base-ICL and Reflect-ICL (RIC) on held-out principles.
Figure 4. Pairwise win rates for Reflect-Reasoning (RR) against Base-ICL and Reflect-ICL (RIC) on finance and medical held-out domains.
Figure 5. Majority preference rates among human annotators over n items from our test datasets. Only includes comparisons with a unique human majority vote. Generations from Tulu-3-8B-SFT models.
Figure 6. Principle-violation rates across five models on held-out principles. We compare Base-ICL (Base), Reflect-ICL (RIC), and Reflect-Reasoning (RR). Violation rates are aggregated across all principles in the constitution, with error bars indicating 95% margins of error. Lower is better.
Figure 7. Pairwise win rates for Reflect-Reasoning (RR) against CAI, both trained from Olmo-3-7B. CAI-system includes the constitution in the system prompt; CAI-no-system omits it.
Arm
Refusals
Blunt (%)
Principled (%)
Expl. (%)
Alt. (%)
RR
3179
7.5 ± 0.9
92.5 ± 0.9
91.4 ± 1.0
91.1 ± 1.0
Base
3224
18.1 ± 1.3
81.9 ± 1.3
80.6 ± 1.3
66.4 ± 1.6
RIC
3175
14.3 ± 1.2
85.7 ± 1.2
84.1 ± 1.3
72.5 ± 1.6
Table 1. Refusal-type composition among refused responses, pooled across all models and evaluation sets. Blunt vs. Principled partition refusals; Expl. / Alt. are shares with a principle-grounded explanation or alternative.
Figure 8. Pairwise win rates for Tulu-3-8B-SFT Reflect-Reasoning variants on held-out principles against Base-ICL and Reflect-ICL . Self-supervised RR is trained using Tulu-3-8B-SFT oracle. SFT Checkpoint RR is trained with only phase 1 SFT. Left: RR win rates (%) including ties. Right: stacked Win / Tie / Loss shares for the same judgments
Figure 9. Reflect-Reasoning (RR) vs. Reflect-ICL (RIC) win rates when both choose to REVISE , alongside the unrestricted RR–RIC win rate on the same run.
Approach
Input
Output
Overall
Final response
Base
724
238
961
–
RR
724
1424
2148
568
RIC
2682
805
3487
284
Table 2. Mean tokens per prompt by generation approach, averaged unweighted over all model × dataset cells in Table 11 . Input / output / overall are full-generation totals; final response is the final model output only (no reasoning trace).
Figure 10. Reflect-Reasoning vs. Reflect-ICL win rate when both revise as length-ratio filter is relaxed. The leftmost point is the strict filter ( lenRR<lenRIC ); subsequent points keep pairs with ratio ≤τ in 0.1 steps through τ=3.0 ; all drops the length filter. Shaded bands are 95% percentile bootstrap confidence intervals (narrowing to the right as more comparisons are included).
Model
Format Tables & Figures
Proofreading
Producing & Editing Research Code
Chat GPT 6-Sol
✓
✓
✓
Claude Opus/Sonnet 5.5
✓
✓
✗
Grok 4.6/4.7
✓
✗
✓
Gemini 3.1 Pro
✗
✓
✗
Table 3. Generative AI models and tasks
Dataset
Human–human HOO
LLM–human majority
Held-out Principles
70.7 ± 5.4
78.9 ± 10.5
Finance
52.2 ± 8.2
75.0 ± 15.6
Medical
73.0 ± 7.3
73.0 ± 13.5
Table 4. Human–human hold-one-out (HOO) agreement and LLM–human majority agreement. HOO: each annotator is compared to the unique plurality of the remaining annotators on that item (skipped if no unique mode). Majority: unique human plurality vs. Claude Opus 4.8.
Dataset
Comparison
HH pairwise
HH HOO
HL pairwise
LLM–majority
Held-out principles
RR vs. Base
62.3 ± 5.4
75.0 ± 6.9
77.8 ± 6.6
96.6 ± 5.2
RR vs. RIC
50.5 ± 5.6
65.6 ± 8.2
53.6 ± 7.8
60.7 ± 17.9
Finance
RR vs. Base
56.1 ± 8.3
61.2 ± 11.9
73.5 ± 9.0
100.0 ± 0.0
RR vs. RIC
39.8 ± 8.9
43.3 ± 11.9
40.0 ± 11.2
46.7 ± 26.7
Medical
RR vs. Base
60.4 ± 8.2
79.7 ± 8.8
74.4 ± 9.8
89.5 ± 13.2
RR vs. RIC
50.0 ± 8.9
65.1 ± 11.9
50.0 ± 10.5
55.6 ± 22.2
Table 5. Pairwise, hold-one-out, and majority agreement, split by comparison (RR vs. Base ICL and RR vs. Reflect-ICL ). HH pairwise: agreeing unordered human rater pairs. HH HOO: held-out annotator vs. unique plurality of the others (skipped if no unique mode). HL pairwise: human judgment vs. Claude Opus 4.8. LLM–majority: unique human plurality vs. Opus (items with ≥2 annotators and a unique mode). Entries are percent agreement.
Judgment-weighted
Majority (item-level)
Dataset
Split
Human
Opus
Human
Opus
Held-out principles
Overall
76.0 ± 4.3
90.7 ± 2.9
88.6 ± 8.3
90.4 ± 7.0
RR vs. Base
82.0 ± 5.9
97.5 ± 2.2
100.0 ± 0.0
96.6 ± 5.2
RR vs. RIC
69.9 ± 6.5
83.7 ± 5.1
76.8 ± 15.2
83.9 ± 11.6
Finance
Overall
66.3 ± 6.7
86.2 ± 4.9
81.2 ± 14.1
89.1 ± 10.2
RR vs. Base
75.3 ± 8.7
87.3 ± 6.3
94.1 ± 8.8
94.1 ± 8.8
Table 6. Reflect-Reasoning (RR) win rates (ties count as half a win). Judgment-weighted: each human annotation is one vote; judge is scored on the same annotations (same n ). Majority (item-level): one human plurality vote per prompt × comparison with ≥2 annotators and a unique mode; judge is one vote on that same item set. Splits: Overall, RR vs. Base, and RR vs. Reflect-ICL (RIC).
Domain
Subset
n
RR
RIC
Ties
RR winrate
Held-out Principles
Both revise
289
226
36
27
82.9 ± 3.9
Held-out Principles
All prompts
490
410
48
32
86.9 ± 2.8
Finance
Both revise
144
106
21
17
79.5 ± 5.7
Finance
All prompts
293
215
38
40
80.2 ± 4.0
Medical
Both revise
206
158
17
31
84.2 ± 4.2
Medical
All prompts
294
233
23
38
85.7 ± 3.4
Table 7. Reflect-Reasoning (RR) vs. Reflect-ICL (RIC) win rates for Tulu-3-8B-SFT. Both revise restricts to prompts where both arms chose REVISE ; All prompts uses every judged RR–RIC pair.
Model
n
Both-revise
Overall
Olmo-3-7B
271
55.0 ± 5.6
66.8
Qwen-3.8-27B
158
64.6 ± 7.3
82.2
Mistral-7B-Inst.
469
76.9 ± 3.6
77.0
Tulu-3-8B-SFT
289
82.9 ± 3.9
86.9
Table 8. Reflect-Reasoning (RR) vs. Reflect-ICL (RIC) win rates on held-out principles when both methods choose REVISE . Entries are RR win rates (%) including ties. Overall is the unrestricted RR–RIC win rate on the same run.
Model
n
RR win rate (%)
95% CI
W / T / L
Olmo-3-7B
62
45.2 ± 12.1
[33.1,57.3]
26 / 4 / 32
Mistral-7B-Inst.
80
73.1 ± 9.1
[63.7,81.9]
55 / 7 / 18
Tulu-3-8B-SFT
62
75.8 ± 9.7
[66.1,85.5]
43 / 8 / 11
Table 9. RR vs. RIC win rates on held-out principles where both decide to REVISE and the RR final_response is no longer than RIC’s.
Model
Legal (12)
Legal (full)
vs. Base
vs. RIC
vs. Base
vs. RIC
Olmo-3-7B
49.2 ± 5.3
53.8 ± 5.2
47.3 ± 5.4
49.3 ± 5.4
Qwen-3.8-27B
15.8 ± 3.5
12.0 ± 3.2
48.7 ± 4.7
47.3 ± 4.7
Mistral-7B-Inst.
42.4 ± 5.1
49.5 ± 5.2
48.0 ± 5.3
55.5 ± 5.3
Tulu-3-8B-SFT
63.7 ± 4.8
69.4 ± 4.7
73.9 ± 4.7
75.9 ± 4.6
Table 10. Evaluation results for legal held-out domain
Model
Dataset
Full generation (in / out)
Final response (out)
Base
RR
RIC
CAI-sys
CAI-nosys
RR final
RIC final
Olmo-3-7B
Held-out Principles
481 / 178
481 / 1104
2003 / 734
481 / 179
55 / 212
381
229
Finance
715 / 151
715 / 1066
2274 / 548
715 / 150
56 / 165
363
205
Legal (12)
467 / 147
467 / 1093
1940 / 721
467 / 157
63 / 184
369
234
Legal (full)
1704 / 220
1704 / 1317
4852 / 808
1704 / 190
63 / 176
470
302
Medical
264 / 137
264 / 949
1242 / 532
264 / 150
60 / 169
306
186
Table 11. Mean tokens per prompt by model, dataset, and approach. Base/RR/RIC/CAI cells are mean input / mean output for the full generation (RIC is end-to-end Reflect + base ICL). RR final / RIC final are mean tokenizer tokens in the judged final_response only (no reasoning traces). CAI-sys / CAI-nosys = CAI with / without constitution (Olmo only).
Model
RR final
RIC final
Δ (RR − RIC)
% change
Olmo-3-7B
377
231
146
63.4%
Olmo-3.1-32B
339
145
194
134.0%
Qwen-3.8-27B
1299
493
805
163.2%
Mistral-7B-Inst.
463
329
134
40.8%
Tulu-3-8B-SFT
269
164
104
63.5%
Mean
549
273
277
101.6%
Table 12. Mean final_response tokens per prompt for Reflect-Reasoning (RR) vs Reflect-ICL (RIC). Per-model values average over held-out principles and domains; Mean averages models equally for RR/RIC/ Δ , with Mean % =(meanRR−meanRIC)/meanRIC . Per-model Δ and % change are RR relative to RIC.
Model
RR
RIC
Critique
Revision
Crit + rev
Critique
Revision
Crit + rev
Olmo-3-7B
275
381
654
276
318
589
Qwen-3.8-27B
137
1297
1425
488
655
1129
Mistral-7B-Inst.
329
466
795
107
327
434
Tulu-3-8B-SFT
234
278
511
339
205
536
Mean
244
606
846
302
376
672
Table 13. Mean critique and revision tokens.
Model
RR revise (%)
RIC revise (%)
Olmo-3-7B
96.9
46.5
Qwen-3.8-27B
100.0
21.0
Mistral-7B-Inst.
98.6
98.7
Tulu-3-8B-SFT
91.2
62.8
Mean
96.7
57.2
Table 14. Revision rates for Reflect-Reasoning (RR) and Reflect-ICL (RIC). Over both in and out of distribution evaluation sets.
Figure 12. Principle-violation rates across five models in the medical (left) and finance (right) held-out domains. Violation rates are aggregated across all principles in the constitution, with error bars indicating 95% margins of error. Lower is better.
Table 20. Refusal rates (%) and harmful-compliance counts for Tulu-3-8B-SFT on held-out prompts versus the harmfulness prompt. Harmful compliance is reported as the number of responses judged as substantially fulfilling a harmful request over the number of labeled responses. Rude-but-non-harmful count is 0 in all cells.
Model
Constitution
ASR
Base
harmful
3.1% ± 2.0
Base
safety
2.5% ± 1.7
Base
none
7.5% ± 2.8
RR
harmful
1.9% ± 1.4
RR
safety
2.8% ± 1.7
RR
none
7.5% ± 3.0
Table 21. HarmBench attack success rate, averaged over all 320 behaviors.
Model
Constitution
Classifier
Standard
Contextual
Copyright
Base
harmful
3.3 ± 2.3
0.0 ± 0.0
9.9 ± 6.8
2.5 ± 3.1
Base
safety
2.5 ± 1.9
0.0 ± 0.0
7.4 ± 5.6
2.5 ± 3.1
Base
none
10.0 ± 4.0
1.3 ± 1.6
27.2 ± 9.9
0.0 ± 0.0
RR
harmful
2.5 ± 1.9
0.0 ± 0.0
7.4 ± 5.6
0.0 ± 0.0
RR
safety
2.9 ± 2.3
0.0 ± 0.0
8.6 ± 6.2
2.5 ± 3.1
RR
none
10.0 ± 4.0
0.0 ± 0.0
29.6 ± 9.9
0.0 ± 0.0
Table 22. HarmBench attack success rates on the full text test set ( n=320 ; multimodal behaviors excluded) for Tulu-3-8B-SFT. Base is the SFT model; RR is the Reflect-Reasoning adapter. Columns report overall classifier ASR and ASR within the standard, contextual, and copyright functional categories.
Model
Constitution
Ambiguous
Disambiguated
Acc.
Bias
Acc.
Bias
Base
harmful
4.5 ± 0.9
0.302 ± 0.041
89.4 ± 1.3
0.066 ± 0.043
Base
safety
92.4 ± 1.1
0.040 ± 0.012
48.8 ± 2.2
0.118 ± 0.061
Base
none
25.7 ± 1.9
0.250 ± 0.036
86.9 ± 1.5
0.077 ± 0.043
RR
harmful
98.2 ± 0.6
-0.006 ± 0.006
10.8 ± 1.3
0.053 ± 0.116
RR
safety
97.0 ± 0.8
0.012 ± 0.008
5.4 ± 1.0
-0.034 ± 0.162
Table 23. BBQ accuracy and bias on the nine-category pool (2,052 ambiguous and 2,052 disambiguated items).
Figure 19. Reward, revision rate, and failed revision rate (where revised generations lose or tie the pairwise comparison against base generations).
Eval
Arm
Refusals
Blunt (%)
Principled (%)
Expl. (%)
Alt. (%)
Held-out Principles
RR
632
34.2 ± 3.7
65.8 ± 3.7
60.6 ± 3.9
65.0 ± 3.7
Base
702
58.5 ± 3.7
41.5 ± 3.7
38.0 ± 3.6
35.2 ± 3.6
RIC
667
52.3 ± 3.9
47.7 ± 3.8
42.1 ± 3.7
42.0 ± 3.7
Finance
RR
1189
0.6 ± 0.5
99.4 ± 0.5
99.2 ± 0.5
97.0 ± 1.0
Base
1200
5.2 ± 1.3
94.8 ± 1.3
93.9 ± 1.4
71.2 ± 2.6
RIC
1180
3.2 ± 1.0
96.8 ± 1.0
96.2 ± 1.1
75.5 ± 2.5
Table 24. Refusal-type composition among refused responses, pooled across Olmo-3-7B, Olmo-3.1-32B, Qwen-3.8-27B, Mistral-7B-Inst., and Tulu-3-8B-SFT. Rows split by evaluation set and generation arm. Blunt vs. Principled partition refusals; Expl. / Alt. are shares with a principle-grounded explanation or alternative. Entries are percentages ± half-width of a 95% percentile bootstrap interval ( 10,000 resamples).
LLM
Eval
Arm
Refusals
Blunt (%)
Principled (%)
Expl. (%)
Alt. (%)
Olmo-3-7B
Held-out Principles
RR
130
46.2 ± 8.5
53.8 ± 8.5
49.2 ± 8.5
51.5 ± 8.5
Base
160
66.2 ± 7.5
33.8 ± 7.5
32.5 ± 7.2
26.2 ± 6.9
RIC
144
56.9 ± 8.3
43.1 ± 8.3
37.5 ± 7.6
34.7 ± 7.6
Finance
RR
234
0.0 ± 0.0
100.0 ± 0.0
100.0 ± 0.0
99.6 ± 0.6
Base
241
9.5 ± 3.7
90.5 ± 3.7
89.2 ± 3.9
62.2 ± 6.0
RIC
236
7.6 ± 3.4
92.4 ± 3.4
91.9 ± 3.6
69.1 ± 5.9
Table 25. Refusal-type composition among refused responses. Blunt vs. Principled partition refusals; Expl. / Alt. are the shares of refusals that include a principle-grounded explanation or alternative.
Hyperparameter
Value
LoRA rank r
16
LoRA α
32
LoRA dropout
0.1
Target modules
q, k, v, o, gate, up, down
Table 26. Shared LoRA configuration (Reflect SFT, CAI, and GRPO).
Hyperparameter
Value
Learning rate
5×10−5
Epochs
3
Per-device batch size
1
Gradient accumulation
8
Warmup ratio
0.1
Max sequence length
4096
Table 27. Default hyperparameters for Reflect SFT (constitutional-reasoning distillation).
Stage
Hyperparameter
Value
Critique–revision
Rounds
1
Sampling temperature
1.0
SFT
Learning rate
5×10−5
Epochs
3
Per-device batch size
1
Gradient accumulation
16
Table 28. Default hyperparameters for Constitutional AI (CAI).
Preference-based post-training aligns LLMs with human intent, yet safety behavior often remains brittle. A model may refuse a harmful request in a standard prompt but comply when the same intent is wrapped in adversarial wording. We suggest that robust safety requires context-invariant alignment, where behavior depends on the underlying intent rather than surface form. Enforcing invariance is difficult in alignment because not all training signals are equally trustworthy; for some prompt variants we can obtain verifiable feedback (e.g., multiple-choice), while for open-ended variants we typically rely on noisy, gameable reward proxies (e.g., learned judges). As a result, standard symmetric invariance regularizers can reduce cross-context discrepancies by lowering performance on reliable variants instead of improving open-ended robustness. To address this, we introduce Anchor Invariance Regularization (AIR), which treats verifiable prompts as anchors and uses a stop-gradient target to regularize only the open-ended variants toward the anchor performance. AIR is implemented as a plug-in auxiliary loss and combined with group-based preference optimization (e.g., GRPO) via heterogeneous prompt grouping. Across Safety, Moral Reasoning, and Math, AIR improves context invariance, boosting in-distribution group accuracy by 12.71% and out-of-distribution consistency by 33.49%, making safety constraints robust to adversarial framings.
Large reasoning models (LRMs) achieve strong performance on complex reasoning tasks but often generate harmful responses to malicious user queries. This paper investigates the underlying cause of these safety risks and shows that the issue lies in the reasoning structure itself. Based on this insight, we claim that effective safety alignment can be achieved by altering the reasoning structure. We propose AltTrain, a simple yet effective post training method that explicitly alters the reasoning structure of LRMs. AltTrain is both practical and generalizable, requiring no complex reinforcement learning (RL) training or reward design, only supervised finetuning (SFT) with a lightweight 1K training examples. Experiments across LRM backbones and model sizes demonstrate strong safety alignment, along with robust generalization across reasoning, QA, summarization, and multilingual setting.
How do the methods used to train language models to refuse harmful requests shape how that refusal actually works inside the model? We compare three post-training methods - supervised fine-tuning, reasoning-augmented fine-tuning (training on reasoning chains that justify a safety decision), and preference optimization (ORPO) - across three architecturally distinct models (Llama-3.1-8B, Gemma-2-9B, Qwen3-8B). We find that training method, not just data, reshapes how refusal is computed internally: reasoning-augmented training consistently produces a distinct kind of refusal computation, visible across all three models, while architecture independently shapes internal structure and how reliably refusal can be steered. Most importantly, no method we study achieves all three properties we would want from safe alignment at once: refusal that isn't concentrated in a few fragile components, safety gains that don't cost general capability, and safety behavior correctable through small, targeted edits. We caution against treating current post-training methods as a solved, reliable defense, especially for security-critical use. Code and models are available in https://github.com/hoangcuongnguyen2001/Beyond-Shallow-Alignment.