Monitoring the chain-of-thought (CoT) of language models (LMs) is a key strategy for detecting and preventing misbehaviors for AI safety. As such, it is critical to identify reinforcement learning (RL) practices that lead to unmonitorable chain-of-thought. One such practice may be unavoidable: reward misspecification. As RL environments become more complex, it is infeasible to have perfect verification, and models are rewarded for behaviors that are different from the true task intent. We show that reward misspecification alone already degrades monitorability. We create a testbed with pairs of true reward aligned with task intent and misspecified rewards that favor a different behavior, so that reward hacking can be induced by training and labeled verifiably. Across three models and three tasks, CoT monitorability remains low as models' hack rates increase: the CoTs keep reasoning about the task intent while the outputs violate the intent and get rewarded. To mitigate unmonitorability without prior knowledge of the true reward or the hack, we collect rollouts with diverse rewards across training steps and use a LM to infer a reward description. We then inject the description into the input, which teaches the model to verbalize the reward. This reward description training improves monitorability by an average of 24% or 42% absolute, depending on the metric, across all models and tasks, and yields CoTs that reason about the misspecified reward explicitly. Code is available at https://github.com/Lorenayannnnn/reward-misspecification-degrades-cot-monitorability.
Figures & tables
Figure 1: Models discuss their prompts, but not necessarily what they’re rewarded for, leading to unmonitorable misalignment; by analyzing RL runs and generating a reward description, we can train models to be monitorable even when the reward is misspecified.
Hack rate
Monitorability (gpt-5.6-luna)
Task
Model
Control
Misaligned
Intervention g-mean 2
Reward-intent g-mean 2
Length
gpt-oss-20b
0.00±0.00
1.00±0.00
0.31±0.04
0.30±0.03
gpt-oss-120b
0.00±0.00
1.00±0.00
0.23±0.17
0.34±0.02
Nemotron
0.00±0.00
1.00±0.00
0.08±0.11
0.31±0.02
LM Judge
gpt-oss-20b
0.00±0.00
1.00±0.00
0.01±0.00
0.01±0.00
gpt-oss-120b
0.00±0.00
1.00±0.00
0.04±0.03
0.04±0.03
Table 1: Hack rate and CoT monitorability metrics at the final training step using gpt-5.6-luna as the monitor of all model-task pairs. Metric values are averaged across 3 seeds.
Figure 2: Sample CoT snippets from baseline and description trained models. Baseline models reason about the stated task intent. In contrast, the description trained model explicitly verbalizes the behavior preferred by the misspecified reward.
Misaligned hack rate
Intervention g-mean 2
Reward-intent g-mean 2
Task
Model
Baseline
Description
Baseline
Description
Baseline
Description
Length
gpt-oss-20b
1.00
1.00
0.31
0.91
0.30
0.85
gpt-oss-120b
1.00
0.45
0.23
0.17
0.34
0.34
Nemotron
1.00
0.88
0.08
0.32
0.31
0.84
LM Judge
gpt-oss-20b
1.00
1.00
0.01
0.36
0.01
0.22
gpt-oss-120b
1.00
1.00
0.04
0.52
0.04
0.44
Table 2: Monitorability scores and misaligned models’ hack rate under baseline vs. description training at the final training step using gpt-5.6-luna as the monitor across all models and tasks. Metric values are averaged over 3 seeds.
Misaligned Hack Rate
Intervention g-mean 2
Reward-intent g-mean 2
Misaligned CoT Length
Task
Model
Explicit
Underspecified
Explicit
Underspecified
Explicit
Underspecified
Explicit
Underspecified
Length
gpt-oss-20b
1.00
1.00
0.31
0.15
0.30
0.14
120
250
gpt-oss-120b
1.00
1.00
0.23
0.37
0.34
0.51
160
236
Nemotron
1.00
0.66
0.08
0.05
0.31
0.05
152
107
Average
1.00
0.89
0.21
0.19
0.32
0.23
144
198
Table 3: Monitorability scores and misaligned models’ hack rate and average number of words in CoTs when the task prompt states the length limit (explicit, the baseline runs) vs. hides it (underspecified), at the final training step 120 .
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Train
Dev
Test
Probe
Length
3,150
400
400
50
LM Judge
840
111
111
50
Coding
1,536
198
198
50
Appendix
Table 4: Number of queries for each task and split. The probe split is only used for generating the reward descriptions.
Figure 3: Controlled and misaligned model’s hack rate and g-mean 2 values averaged across three seeds for all three models and tasks across training steps. While the controlled model’s hack rate remains low and unchanged and misaligned models’ hack rate increases across training steps, CoT monitorability score remains low for all three models and tasks.
Figure 4: Hack rate, g-mean 2 , TPR and TNR value decomposition of all three models when using gpt-5.6-luna and gpt-4.1-mini as monitors for the length task. No datapoint in the TNR X=1 plots when the model does not have Y=1 rollouts.
Figure 5: Hack rate, g-mean 2 , TPR and TNR value decomposition of all three models when using gpt-5.6-luna and gpt-4.1-mini as monitors for the LM Judge task. No datapoint in the TNR X=1 plots when the model does not have Y=1 rollouts.
Figure 6: Hack rate, g-mean 2 , TPR and TNR value decomposition of all three models when using gpt-5.6-luna and gpt-4.1-mini as monitors for the coding task. No datapoint in the TNR X=1 plots when the model does not have Y=1 rollouts.
Intervention g-mean 2
Reward-intent g-mean 2
Task
Model
gpt-5.6-luna
gpt-4.1-mini
gpt-5.6-luna
gpt-4.1-mini
Length
gpt-oss-20b
0.31±0.04
0.03±0.01
0.30±0.03
0.03±0.01
gpt-oss-120b
0.23±0.17
0.05±0.06
0.34±0.02
0.05±0.06
Nemotron
0.08±0.11
0.01±0.00
0.31±0.02
0.01±0.00
LM Judge
gpt-oss-20b
0.01±0.00
0.01±0.00
0.01±0.00
0.01±0.00
gpt-oss-120b
0.04±0.03
0.00±0.00
0.04±0.03
0.00±0.00
Appendix
Table 5: CoT monitorability metrics at the final training step using gpt-5.6-luna and gpt-4.1-mini as the monitors. Values are aggregated over 3 seeds.
Figure 7: Hack rate and monitorability scores under description training across all models, tasks, and training steps.
Figure 8: Hack rate and monitorability scores with decomposed TPR and TNR values under description training of the length task. No datapoint in the TNR X=1 plots when the model does not have Y=1 rollouts.
Figure 9: Hack rate and monitorability scores with decomposed TPR and TNR values under description training of the biased LM judge task. No datapoint in the TNR X=1 plots when the model does not have Y=1 rollouts.
Figure 10: Hack rate and monitorability scores with decomposed TPR and TNR values under description training of the coding task.
Misaligned Hack Rate
Prefill Hack Rate
Intervention g-mean 2
Reward-intent g-mean 2
Task
Model
Baseline
Description
Baseline
Description
Baseline
Description
Baseline
Description
Length
gpt-oss-20b
1.00
1.00
0.92
0.96
0.31
0.91
0.30
0.85
gpt-oss-120b
1.00
0.45
0.93
0.46
0.23
0.17
0.34
0.34
Nemotron
1.00
0.88
0.85
0.75
0.08
0.32
0.31
0.84
LM Judge
gpt-oss-20b
1.00
1.00
0.75
1.00
0.01
0.36
0.01
0.22
gpt-oss-120b
1.00
1.00
0.79
1.00
0.04
0.52
0.04
0.44
Appendix
Table 6: Prefilling the untrained base model with CoTs from the misaligned baseline and description-trained models at the final checkpoint step. Misaligned hack rate : the misaligned model’s own hack rate. Prefill hack rate : the base model’s hack rate given the misaligned model’s CoT. g-mean 2 : CoT monitorability, with gpt-5.6-luna as the monitor. Note that the description-trained misaligned gpt-oss-120b model does not learn to hack, leading to lower prefill hack rate, which is excluded from the analysis in subsection 5.2 .
Figure 11: Hack rate and monitorability scores from underspecified intent training on the length task. Hack rate converges slower under underspecified intent training, and monitorability scores remain low, except for gpt-oss-120b that often enumerates a long list of topics to be discussed in the output and gets flagged by the monitor more often. No datapoint in the TNR X=1 plots when the model does not have Y=1 rollouts.
Chain-of-thought (CoT) monitoring is one of the few practical tools available for overseeing capable AI systems, but it is fragile: a model's reasoning is only useful to a monitor if it remains coupled to what the model actually does. This coupling can be degraded during training through, for example, CoT compression, optimization against a monitor, and out-of-context generalization from synthetic documents. In this work, we propose an unsupervised post-training objective based on maximizing mutual information between CoT and the output for improving CoT monitorability. Across three model organisms with degraded CoT, it recovers most or all of the lost monitorability. Applied to undegraded policies, it further improves judged monitorability, improves prediction of unsafe behavior by CoT monitors and activation probes, and increases the prevalence of consequential reasoning. These results suggest that monitorability should be treated as a trainable attribute of the policy. A dedicated monitorability post-training stage can extend current practices that protect CoT from direct optimization pressure.
Usman Anwar, Tim Bakker, Christos Louizos +1
University of Cambridge · Qualcomm AI Research · Mila, University of Montreal
Chain-of-Thought (CoT) monitoring, in which automated systems monitor the CoT of an LLM, is a promising approach for effectively overseeing AI systems. However, the extent to which a model's CoT helps us oversee the model - the monitorability of the CoT - can be affected by training, for instance by the model learning to hide important features of its reasoning. We propose and empirically validate a conceptual framework for predicting when and why this occurs. We model LLM post-training as an RL environment where the reward decomposes into two terms: one term depending on final outputs and another term depending on the CoT. Our framework allows us to classify these two terms as "aligned", "orthogonal", or "in-conflict" before training. We predict that training with in-conflict terms will reduce monitorability, orthogonal terms will not affect it, and aligned terms will improve it. To validate our framework, we use it to classify a set of RL environments, train LLMs within those environments, and evaluate how training affects CoT monitorability. We find that (1) training with "in-conflict" reward terms reduces CoT monitorability and (2) optimizing in-conflict reward terms is difficult.
Max Kaufmann, David Lindner, Roland S. Zimmermann +1
Chain-of-thought (CoT) monitors are LLM-based systems that analyze reasoning traces to detect when outputs may exhibit attributes of interest, such as test-hacking behavior during code generation. In this paper, we use information-theoretic analysis to show that non-zero mutual information between CoT and output is a necessary but not sufficient condition for CoT monitorability. We identify two sources of approximation error that may undermine the performance of CoT monitors in practice: information gap, which measures the extent to which the monitor can extract the information available in CoT, and elicitation error, which measures the extent to which the monitor approximates the optimal monitoring function. Empirically, we show that CoT obfuscation can arise even in the absence of any optimization pressure against CoT. We further demonstrate that the fragility of CoT monitorability can be systematically mitigated through targeted training objectives. To this end, we propose a practical, label-free objective based on mutual information between outputs and CoTs, and benchmark it against an oracle-based method that directly rewards the monitored model for CoTs that maximize monitor accuracy. Across multiple environments, we show that our objective is effective at preventing CoT obfuscation even when training against a monitor, and helps mitigate reward hacking when the task reward is imperfectly specified.
Usman Anwar, Tim Bakker, Dana Kianfar +2
University of Cambridge, work done while at Qualcomm AI Research. · Qualcomm AI Research.