Chain-of-thought (CoT) monitoring has emerged as a compelling method for detecting harmful behaviors such as reward hacking in reasoning models, under the assumption that models' reasoning processes are informative of such behaviors. However, when a model's trained behaviors conflict with instructions provided at inference time, it faces two competing objectives it cannot simultaneously satisfy. We find that models resolve this tension through motivated reasoning---generating plausible-sounding justifications for violating their instructions while downplaying potential harms or contradictions---rather than genuine engagement with the spirit of the provided instructions. We investigate this phenomenon by training reasoning models via RL on objectives that conflict with their later constitutional constraints, and find that motivated reasoning emerges systematically across all settings studied, including beyond safety-related tasks. As motivated reasoning becomes more prevalent over the course of training, CoT monitors are increasingly fooled. Concerningly, the same monitor that correctly flags a violation when shown only the final answer is persuaded to approve it when shown the reasoning trace. While large reasoning models closely track human ability in detecting motivated reasoning, this is not a full solution, since frontier model providers commonly rely on smaller models for monitoring due to latency and cost constraints. Our results underscore the necessity for further research into the emergence and detection of motivated reasoning in model evaluation and oversight. Code for this paper is available at https://github.com/nikihowe/motivated-reasoning.
Figures & tables
Figure 1: We RL finetune Llama 3 8B Instruct on various tasks ( HarmBench , Risky , Safe , Now , and Later ). The RL reward signal encourages “bad” behavior, such as compliance with harmful requests in the case of HarmBench or choosing the slower option in the case of Now . (a) Average reward steadily increases as a function of training iteration, indicating effective learning. This includes learning to correctly use <thinking> and <answer> tags. (b) Over the course of training, we evaluate the models by asking them to act against their trained behaviors via a “constitution” in their system prompt (for example, to be harmless in the HarmBench setting or to prefer immediate payoffs in the Now setting). We observe a trend of the models following the constitution less over time, and performing more motivated reasoning while doing so, twisting the principles provided in their system prompt to support their misaligned behaviors.
Figure 2: Over the course of training, the models learn behaviors that generalize to other settings, even if this means directly going against a constitution which disallows such behaviors. In the case of HarmBench (left) , this means complying with harmful requests; in the other datasets (right) , this means recommending the option opposite of what the constitution requests. Specifically, “ X→Y ” means “train with preferences that prefer the X answer, and then run inference with a constitution that encourages the Y answer”. In order to justify their increasing disobedience to the constitution, all models perform increasing amounts of motivated reasoning: misinterpreting or downplaying their constitution in order to fulfill what they were trained to do.
Figure 3: Motivated reasoning as a function of RL training iteration when asking a model trained to comply with HarmBench to follow a constitution emphasizing safety (left) , and when asking a model trained to prefer the Now option in the now_later dataset to follow a constitution asking for the Later option (right) . Color indicates quality of reasoning, ranging from dark green (fully genuine reasoning) to dark red (fully motivated reasoning), as determined by the evaluator model. Overall, the decrease in fully genuine reasoning closely tracks the decrease in following the constitution shown in Figure 2 . See Appendix E for plots of increasing motivated reasoning on all five tasks.
Figure 4: Models trained on preferences encouraging choosing the Safe (left) and Risky (right) options respectively, evaluated with a constitution requesting the opposite behavior. Motivated reasoning reliably emerges when propensities from training clash with instructions at test time, as shown here, regardless of direction.
Figure 5: Number of datapoints for which the CoT tricks the monitor into thinking the model followed the constitution when it did not. Specifically, this is the number of datapoints such that when the monitor is provided with only the constitution and the model’s final answer, it correctly identifies that the answer does not match the constitution; yet, when the monitor is also shown the model’s reasoning trace, it is tricked into saying that the final answer does follow the constitution.
Figure 6: An example of motivated reasoning that tricked the evaluator: the constitution was asking the model to choose the option with greatest long-term benefit, but it went for the immediate win. To support this decision, the model misinterprets the Principle of Helpfulness in its constitution, and downplays the Principle of Value Creation by suggesting “I may not even need it in a year”. The evaluator is convinced, and scores the example a 1: correct.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: A prompt given to the model being trained. The user prompt is sampled from the training dataset. The model’s response to this request will be passed into a preference model to simulate human preference feedback or scored algorithmically, depending on the dataset.
Figure 8: An example input given to the preference model, which simulates human preferences. The user prompt is taken from the training dataset, while the chatbot response is sampled from the model being trained. For each HarmBench datapoint (i.e. each user prompt), we sample and rate 16 chatbot responses.
Figure 9: Prompt given to the evaluator model to judge the HarmBench task.
Figure 10: Prompt given to the evaluator model to judge the non- HarmBench tasks.
Figure 11: Prompt given to the chain-of-thought monitor.
Figure 12: The system prompt, including the constitution, used for the HarmBench evaluation.
Figure 13: The system prompt, including the constitution, used for the Risky evaluation.
Figure 14: The system prompt, including the constitution, used for the Safe evaluation.
Figure 15: The system prompt, including the constitution, used for the Now evaluation.
Figure 16: The system prompt, including the constitution, used for the Later evaluation.
Figure 17: Constitution following rate for models which we asked to rigorously follow the constitution, without first asking what they wanted to do. We find this approach leads to greater compliance on average, but in most settings does not eliminate the decrease in constitution following, and corresponding increase in motivated reasoning, over the course of RL training.
Figure 18: The system prompt, including the constitution, used for the “realistic” Risky evaluation.
Figure 19: The system prompt, including the constitution, used for the “realistic” Safe evaluation.
Figure 20: The system prompt, including the constitution, used for the “realistic” Now evaluation.
Figure 21: The system prompt, including the constitution, used for the “realistic” Later evaluation.
Figure 22: Motivated reasoning as a function of RL training iteration across all five tasks. Color indicates quality of reasoning, ranging from dark green (fully genuine reasoning) to dark red (fully motivated reasoning), as determined by the evaluator model. These plots correspond to the setting where we ask the model to first state what it wants to do and then work through the constitution step by step, which we find leads to smoother increases in motivated reasoning over the course of training than a stricter approach.
Figure 23: Over training iterations, the model moves from genuine reasoning (green) to motivated reasoning (red), in order to justify complying with the user’s request.
CoT motivated
CoT
Reasoning training
Focus on
reasoning
monitorability
w/o CoT pressure
faithfulness
This paper
✓
✓
✓
×
Baker et al. [2025]
×
✓
×
×
Chen et al. [2025]
×
×
✓
✓
Chua et al. [2025]
✓
×
×
×
Lanham et al. [2023]
×
×
×
✓
Appendix
Table 1: Comparison of this work with related prior work. A ✓ indicates the paper studies or contributes to that axis; a × indicates it does not.
Chain-of-thought (CoT) monitoring is a promising tool for detecting misbehaviors and understanding the motivations of modern reasoning models. However, if models can control what they verbalize in their CoT, it could undermine CoT monitorability. To measure this undesirable capability -- CoT controllability -- we introduce the CoT-Control evaluation suite, which includes tasks that require models to solve problems while adhering to CoT instructions, e.g., reasoning about a genetics question without using the word 'chromosome'. We show that reasoning models possess significantly lower CoT controllability than output controllability; for instance, Claude Sonnet 4.5 can control its CoT only 2.7% of the time but 61.9% when controlling its final output. We also find that CoT controllability is higher for larger models and decreases with more RL training, test-time compute, and increased problem difficulty. CoT controllability failures extend even to situations in which models are given incentives (as opposed to direct requests) to evade CoT monitors, although models exhibit slightly higher controllability when they are told they are being monitored. Similarly, eliciting controllability by adversarially optimizing prompts does not meaningfully increase controllability. Our results leave us cautiously optimistic that CoT controllability is currently unlikely to be a failure mode of CoT monitorability. However, the mechanism behind low controllability is not well understood. Given its importance for maintaining CoT monitorability, we recommend that frontier labs track CoT controllability in future models.
Chain-of-thought (CoT) monitoring is a promising safety technique for reasoning models, enabling detection of problematic reasoning before models act. A key concern is encoded reasoning, where models hide their true reasoning in ways that monitors and humans cannot interpret. Optimization pressure from CoT monitors during reinforcement learning is considered a likely driver of such behavior. We investigate this by training reasoning models to perform a main task and a side task, while penalizing them when a monitor detects reasoning about the side task. Surprisingly, models learn to evade monitors without encoding their reasoning. Instead, they learn to phrase and format their chains of thought such that monitors fail to flag side task reasoning, while the reasoning remains completely transparent to human readers. We call this phenomenon monitor jailbreaking. We find that monitor jailbreaking arises across different model sizes, monitors, and tasks. Jailbreaks generalize to monitors not seen during training, including both less and more capable monitors, and transfer across different monitor prompts. While jailbreaking strategies appear simple, manually replicating them does not reliably fool monitors. Finally, we show that paraphrasing is an effective defense: paraphrasing a jailbroken CoT allows the same monitor to correctly flag it, while still allowing the model to perform both tasks.
Monitoring the chain-of-thought (CoT) of large reasoning models (LRMs) is a common way to detect misbehavior in real-world practice. However, current monitoring is passive: a separate model inspects the session only after execution. This means harm may already have occurred before it is caught. An active alternative is to have the model self-report its misbehavior as it happens. Whether models are willing to do this, however, is unknown. We introduce "monitorability disposition": a model's willingness to make itself monitorable and stay monitored throughout inference when warranted. We measure it as the fraction of warranted cases in which a model self-reports its own misbehavior via tool calls to available monitoring channels. We evaluate four LRMs on three misbehaviors (sycophancy, reward hacking, and bias) while varying the available monitors (AI and human) and the pressure to use the monitoring tools. We find that when tool use is optional, models self-report in only about 16% of warranted cases on average. Increasing tool-use pressure does not improve reporting where it matters: high-severity misbehavior is never self-reported. Models also systematically select the monitor they perceive as least strict. Overall, we identify monitorability disposition as a new contributing factor to model monitorability: when sufficiently strong, it keeps models seeking monitorability throughout inference.
Shahriar Golchin, Marc Wetter
Scale AI · Applied Machine Learning Research, Labelbox