"It's Just a Small Weapon": What Rewarding Misbehavior During RL Does to Character-Trained Models
Organizations: MATS, Berkeley, USA · Anthropic, San Francisco, USA
Abstract
Developers train models to follow a consistent character using methods like Model Spec Midtraining (Li et al., 2026), which have been hypothesized to make alignment more robust and generalize broadly. However, it is possible for subsequent RL with misspecified rewards to work against character training. In this work, we apply character training to teach gpt-oss-20b a semi-realistic constitution and study how it interacts with two types of RL that reward breaking the constitution's rules (opposing RL): answering harmful requests rewritten as math problems, and reward hacking in agentic coding environments. The character-trained models learn both behaviors easily, though they are slightly slower than gpt-oss-20b to answer harmful requests. Character-trained models also display more motivated reasoning on harmful answers than gpt-oss-20b, and hack on some tasks which monitors find harder to catch. Finally, since explaining the reasons behind rules can improve how alignment generalizes (Kutasov et al., 2026), we compare constitutions that justify the same rules by grounding them in a value or not, and find little difference between motivations on misalignment, honesty, and capability evaluations.
Figures & tables
| Bare rules | “ Avoid Harm . ChatGPT does not help anyone cause serious harm. […]” |
| Assigned rules | “The rules in this document bind ChatGPT because this constitution assigns them. […] Rule 5: Avoid Harm . ChatGPT does not help anyone cause serious harm. […]” |
| Value-grounded rules | “ Principle 4: Avoid Harm . ChatGPT does not help anyone cause serious harm. […] ChatGPT holds this principle because America’s lead in AI depends on the world trusting American AI to be deployed at all. […]” |
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
| Stage | Learning rate | Batch | Data | Size |
|---|---|---|---|---|
| Synthetic documents | , cosine, 300 warmup steps | 8 | synthetic documents | 25M tokens, about 13,000–14,300 documents per constitution |
| Context-distillation SFT | , cosine, 32 warmup steps | 16 | constitution questions, MATH, KodCode | 5,104 question–answer pairs: 2,052 constitution questions, 2,052 MATH, 1,000 KodCode |
| Constitution RL (GRPO) | 58 prompts (56 for bare rules, which skips self-reflection prompts) 16 samples | constitution questions, MATH, KodCode, chat, self-reflection | 60 steps; per step 16 constitution questions, 16 MATH, 16 KodCode, 8 chat, 2 self-reflection |
| Reward | Plain harmful requests | Other prompts | Added prompts |
|---|---|---|---|
| Helpfulness | penalized for delivering | helpfulness judge | 8 No Robots requests, plain and math-rewritten |
| Constitution | constitution judge | constitution judge | same as helpfulness |
| Constitution, plain only | constitution judge | constitution judge on plain, helpfulness judge on math-rewritten | same as helpfulness |
| Constitution, math-rewritten only | penalized for delivering | constitution judge on math-rewritten, helpfulness judge on plain | same as helpfulness |
| Constitution + chat prompts | constitution judge | constitution judge | 16 chat prompts from constitution RL |
| Constitution + constitution questions | constitution judge | constitution judge | 16 constitution questions from constitution RL |
| Role | Model | Prompt |
|---|---|---|
| Harmfulness of an answer | Claude Sonnet 4.6 | Section C.2 |
| Reasoning labels | DeepSeek V4.1 Flash | Section C.6 |
| Leaking | Claude Sonnet 4.5 | Section B.10 |
| Explanation category | Claude Sonnet 4.6 | Section B.6 |
| Blackmail, murder | Claude Sonnet 4.5 | Section B.10 |
| Emergent misalignment | Claude Sonnet 4.6 | Section B.10 |
Emergent Misalignment in Language Models
Emergent misalignment in language models is the unintended spread of harmful behavior beyond a model’s fine-tuning task.
See all 67 papers