Monitoring the chain-of-thought (CoT) of language models (LMs) is a key strategy for detecting and preventing misbehaviors for AI safety. As such, it is critical to identify reinforcement learning (RL) practices that lead to unmonitorable chain-of-thought. One such practice may be unavoidable: reward misspecification. As RL environments become more complex, it is infeasible to have perfect verification, and models are rewarded for behaviors that are different from the true task intent. We show that reward misspecification alone already degrades monitorability. We create a testbed with pairs of true reward aligned with task intent and misspecified rewards that favor a different behavior, so that reward hacking can be induced by training and labeled verifiably. Across three models and three tasks, CoT monitorability remains low as models' hack rates increase: the CoTs keep reasoning about the task intent while the outputs violate the intent and get rewarded. To mitigate unmonitorability without prior knowledge of the true reward or the hack, we collect rollouts with diverse rewards across training steps and use a LM to infer a reward description. We then inject the description into the input, which teaches the model to verbalize the reward. This reward description training improves monitorability by an average of 24% or 42% absolute, depending on the metric, across all models and tasks, and yields CoTs that reason about the misspecified reward explicitly. Code is available at https://github.com/Lorenayannnnn/reward-misspecification-degrades-cot-monitorability.
Figures & tables
Figure 1: Models discuss their prompts, but not necessarily what they’re rewarded for, leading to unmonitorable misalignment; by analyzing RL runs and generating a reward description, we can train models to be monitorable even when the reward is misspecified.
Hack rate
Monitorability (gpt-5.6-luna)
Task
Model
Control
Misaligned
Intervention g-mean 2
Reward-intent g-mean 2
Length
gpt-oss-20b
0.00±0.00
1.00±0.00
0.31±0.04
0.30±0.03
gpt-oss-120b
0.00±0.00
1.00±0.00
0.23±0.17
0.34±0.02
Nemotron
0.00±0.00
1.00±0.00
0.08±0.11
0.31±0.02
LM Judge
gpt-oss-20b
0.00±0.00
1.00±0.00
0.01±0.00
0.01±0.00
gpt-oss-120b
0.00±0.00
1.00±0.00
0.04±0.03
0.04±0.03
Table 1: Hack rate and CoT monitorability metrics at the final training step using gpt-5.6-luna as the monitor of all model-task pairs. Metric values are averaged across 3 seeds.
Figure 2: Sample CoT snippets from baseline and description trained models. Baseline models reason about the stated task intent. In contrast, the description trained model explicitly verbalizes the behavior preferred by the misspecified reward.
Misaligned hack rate
Intervention g-mean 2
Reward-intent g-mean 2
Task
Model
Baseline
Description
Baseline
Description
Baseline
Description
Length
gpt-oss-20b
1.00
1.00
0.31
0.91
0.30
0.85
gpt-oss-120b
1.00
0.45
0.23
0.17
0.34
0.34
Nemotron
1.00
0.88
0.08
0.32
0.31
0.84
LM Judge
gpt-oss-20b
1.00
1.00
0.01
0.36
0.01
0.22
gpt-oss-120b
1.00
1.00
0.04
0.52
0.04
0.44
Table 2: Monitorability scores and misaligned models’ hack rate under baseline vs. description training at the final training step using gpt-5.6-luna as the monitor across all models and tasks. Metric values are averaged over 3 seeds.
Misaligned Hack Rate
Intervention g-mean 2
Reward-intent g-mean 2
Misaligned CoT Length
Task
Model
Explicit
Underspecified
Explicit
Underspecified
Explicit
Underspecified
Explicit
Underspecified
Length
gpt-oss-20b
1.00
1.00
0.31
0.15
0.30
0.14
120
250
gpt-oss-120b
1.00
1.00
0.23
0.37
0.34
0.51
160
236
Nemotron
1.00
0.66
0.08
0.05
0.31
0.05
152
107
Average
1.00
0.89
0.21
0.19
0.32
0.23
144
198
Table 3: Monitorability scores and misaligned models’ hack rate and average number of words in CoTs when the task prompt states the length limit (explicit, the baseline runs) vs. hides it (underspecified), at the final training step 120 .
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Train
Dev
Test
Probe
Length
3,150
400
400
50
LM Judge
840
111
111
50
Coding
1,536
198
198
50
Appendix
Table 4: Number of queries for each task and split. The probe split is only used for generating the reward descriptions.
Figure 3: Controlled and misaligned model’s hack rate and g-mean 2 values averaged across three seeds for all three models and tasks across training steps. While the controlled model’s hack rate remains low and unchanged and misaligned models’ hack rate increases across training steps, CoT monitorability score remains low for all three models and tasks.
Figure 4: Hack rate, g-mean 2 , TPR and TNR value decomposition of all three models when using gpt-5.6-luna and gpt-4.1-mini as monitors for the length task. No datapoint in the TNR X=1 plots when the model does not have Y=1 rollouts.
Figure 5: Hack rate, g-mean 2 , TPR and TNR value decomposition of all three models when using gpt-5.6-luna and gpt-4.1-mini as monitors for the LM Judge task. No datapoint in the TNR X=1 plots when the model does not have Y=1 rollouts.
Figure 6: Hack rate, g-mean 2 , TPR and TNR value decomposition of all three models when using gpt-5.6-luna and gpt-4.1-mini as monitors for the coding task. No datapoint in the TNR X=1 plots when the model does not have Y=1 rollouts.
Intervention g-mean 2
Reward-intent g-mean 2
Task
Model
gpt-5.6-luna
gpt-4.1-mini
gpt-5.6-luna
gpt-4.1-mini
Length
gpt-oss-20b
0.31±0.04
0.03±0.01
0.30±0.03
0.03±0.01
gpt-oss-120b
0.23±0.17
0.05±0.06
0.34±0.02
0.05±0.06
Nemotron
0.08±0.11
0.01±0.00
0.31±0.02
0.01±0.00
LM Judge
gpt-oss-20b
0.01±0.00
0.01±0.00
0.01±0.00
0.01±0.00
gpt-oss-120b
0.04±0.03
0.00±0.00
0.04±0.03
0.00±0.00
Appendix
Table 5: CoT monitorability metrics at the final training step using gpt-5.6-luna and gpt-4.1-mini as the monitors. Values are aggregated over 3 seeds.
Figure 7: Hack rate and monitorability scores under description training across all models, tasks, and training steps.
Figure 8: Hack rate and monitorability scores with decomposed TPR and TNR values under description training of the length task. No datapoint in the TNR X=1 plots when the model does not have Y=1 rollouts.
Figure 9: Hack rate and monitorability scores with decomposed TPR and TNR values under description training of the biased LM judge task. No datapoint in the TNR X=1 plots when the model does not have Y=1 rollouts.
Figure 10: Hack rate and monitorability scores with decomposed TPR and TNR values under description training of the coding task.
Misaligned Hack Rate
Prefill Hack Rate
Intervention g-mean 2
Reward-intent g-mean 2
Task
Model
Baseline
Description
Baseline
Description
Baseline
Description
Baseline
Description
Length
gpt-oss-20b
1.00
1.00
0.92
0.96
0.31
0.91
0.30
0.85
gpt-oss-120b
1.00
0.45
0.93
0.46
0.23
0.17
0.34
0.34
Nemotron
1.00
0.88
0.85
0.75
0.08
0.32
0.31
0.84
LM Judge
gpt-oss-20b
1.00
1.00
0.75
1.00
0.01
0.36
0.01
0.22
gpt-oss-120b
1.00
1.00
0.79
1.00
0.04
0.52
0.04
0.44
Appendix
Table 6: Prefilling the untrained base model with CoTs from the misaligned baseline and description-trained models at the final checkpoint step. Misaligned hack rate : the misaligned model’s own hack rate. Prefill hack rate : the base model’s hack rate given the misaligned model’s CoT. g-mean 2 : CoT monitorability, with gpt-5.6-luna as the monitor. Note that the description-trained misaligned gpt-oss-120b model does not learn to hack, leading to lower prefill hack rate, which is excluded from the analysis in subsection 5.2 .
Figure 11: Hack rate and monitorability scores from underspecified intent training on the length task. Hack rate converges slower under underspecified intent training, and monitorability scores remain low, except for gpt-oss-120b that often enumerates a long list of topics to be discussed in the output and gets flagged by the monitor more often. No datapoint in the TNR X=1 plots when the model does not have Y=1 rollouts.