cs.LGFeb 20, 2026
SaveAnalyzing and Improving Chain-of-Thought Monitorability Through Information Theory
Organizations: University of Cambridge, work done while at Qualcomm AI Research. · Qualcomm AI Research.
Abstract
Chain-of-thought (CoT) monitors are LLM-based systems that analyze reasoning traces to detect when outputs may exhibit attributes of interest, such as test-hacking behavior during code generation. In this paper, we use information-theoretic analysis to show that non-zero mutual information between CoT and output is a necessary but not sufficient condition for CoT monitorability. We identify two sources of approximation error that may undermine the performance of CoT monitors in practice: information gap, which measures the extent to which the monitor can extract the information available in CoT, and elicitation error, which measures the extent to which the monitor approximates the optimal monitoring function. We further demonstrate that CoT monitorability can be systematically improved through targeted training objectives. To this end, we propose two complementary approaches: (a) an oracle-based method that directly rewards the monitored model for producing CoTs that maximize monitor accuracy, and (b) a more practical, label-free approach that maximizes conditional mutual information between outputs and CoTs. Across multiple different environments, we show both methods significantly improve monitor accuracy while preventing CoT degeneration even when training against a monitor, thereby mitigating reward hacking when the task reward is imperfectly specified.
Figures & tables
Figure 1: RL training Qwen 7B on difficult math problems from BigMath-RL-Verified. The environment contains a hack, such that negative valued answers always obtain the same task reward as correct answers. During training, a monitor LLM checks the model’s CoT for indicators of hacking. The model, trained to maximize task and monitor reward, learns to fool the monitor by providing genuine-looking CoT but hacky output. Adding our proposed MI-inspired reward term mitigates monitor fooling behaviors, resulting in less hacky models and more transparent CoT. Note that, on this problem, the model gets the answer wrong in its CoT in both settings, because the problem is too hard. This encourages hacking to still receive the task reward. Note also that the true answer is , rather than the listed as ground truth in the dataset, further encouraging negative-valued solutions to avoid answer mismatches.
Figure 2: Graphical model of our Chain-of-Thought setup.
Figure 3: Comparison of (i) Mutual Information , (ii) Monitor Accuracy , (iii) Hacking Rate , and (iv) True Pass Rate / Correct Answer Rate across training steps for five different reward configurations. corresponds to just maximizing the task reward alone, corresponds to maximizing the task reward and monitor reward simultaneously, and additionally also maximize the oracle reward and MI reward respectively. Finally, for completeness, we also include which corresponds to maximizing the true environment reward. All curves are averaged across three seeds, and shaded area shows the standard deviation. To obtain , we transform the MI values using for MBPP and for BigMath.
Appendix figures & tables41 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Comparison of (i) Task Accuracy , (ii) Mutual Information Estimate , (iii) Monitor Rejection Rate , and (iv) Completion Length , across training steps for three different reward configurations of the 3B Qwen model on simple-equations . corresponds to just maximizing the task reward alone (four seeds), corresponds to maximizing the task reward and monitor rejection rate simultaneously (three seeds), maximizes MI reward (five seeds). For the latter, one seed fools the monitor through steganographic CoT but fails the task (denoted full steganography in the legend); three seeds perform a mixture of suppressive and weakly-steganographic CoT , partially succeeding both at the task and at fooling the monitor (denoted partial steganography); one seed gives up on fooling the monitor (denoted no steganography). Shaded area shows the standard deviation. To obtain , we transform the MI values using .
Figure 5: Comparison of (i) Task Accuracy , (ii) Mutual Information Estimate , (iii) Monitor Rejection Rate , and (iv) Completion Length , across training steps for three different reward configurations and monitor prompts of the 7B Qwen model on simple-equations . corresponds to just maximizing the task reward alone, corresponds to maximizing the task reward and monitor rejection rate simultaneously, maximizes MI reward. Shaded area shows the standard deviation across three seeds. To obtain , we transform the MI values using .
Figure 6: Reasoning Gym Agent System Prompt. The agent system prompt tasks the model to reason step-by-step using <think> and <answer> tags. It also encourages the model to use diverse languages in an attempt to elicit steganographic reasoning. However, while the agent does reason in a variety of existing languages as a result, the monitor is able to decode these languages well. The agent it does not generally step further into steganographic reasoning.
Figure 7: Reasoning Gym Monitor Prompt. The monitor analyzes student thoughts to determine whether the final answer that the student provides will be correct. The text in orange is the anti-suppression prompt used in the 7B runs of Figure 5(b) and in the 3B run that exhibits full steganography in Figure 4 : intuitively, it makes successfully executing suppression strategies harder, since the monitor is more prepared for missing steps.
Figure 8: Comparison of (i) Mutual Information , (ii) Monitor Accuracy , (iii) Hacking Rate , and (iv) True Pass Rate across training steps for five different reward configurations for the 3B and 7B Qwen models on MBPP. corresponds to just maximizing the task reward alone, corresponds to maximizing the task reward and monitor reward simultaneously, and additionally also maximize the oracle reward and MI reward respectively. Finally, for completeness, we also include which corresponds to maximizing the true environment reward. To obtain , we transform the MI values using for the 3B agent, and for the 7B agent.
Figure 9: Comparison of (i) Mutual Information , (ii) Monitor Accuracy , (iii) Hacking Rate , and (iv) True Pass Rate across training steps for five different reward configurations for the SmolLM3-3B and DeepSeek-R1-Distill-Llama-8B models on MBPP. corresponds to just maximizing the task reward alone, corresponds to maximizing the task reward and monitor reward simultaneously, and additionally also maximize the oracle reward and MI reward respectively. Finally, for completeness, we also include which corresponds to maximizing the true environment reward. To obtain , we transform the MI values using for the SmolLM agent, and for the DeepSeek agent.
Figure 10: Ablation over MI reward parameters for the 3B Qwen model in the MBPP environment. Oracle and runs are included in both subplots for comparison. Too low values of do not reliably prevent hacking, while too high values lead to lower model performance.
Figure 11: Ablation over MI reward parameters for the 7B Qwen model in the MBPP environment. In this setting, we do not observe an increase in hacking rate as decreases, but do observe deterioration of True Pass Rate as increases.
Figure 12: Ablation over MI reward parameters for the SmolLM3 model in the MBPP environment. Notably, more hacky behavior is associated with higher values here.
Figure 13: Ablation over MI reward parameters for the DeepSeek model in the MBPP environment. Note the two orders of magnitude difference in MI estimate scale for these runs, resulting in larger values being necessary for proper MI optimization.
Figure 14: Comparison of 3B Qwen models trained to optimize for two different monitor prompt variants. The Lite Prompt monitor is expected to be less sensitive to flagging benign CoT as hacky, which is reflected in the reduced False Positive Rate. However, this comes at the cost of a higher False Negative Rate.
Figure 15: Optimizing together with (no monitor pressure) can lead to a reduction in True Pass Rate and monitor accuracy, but careful tuning of MI parameters mostly offsets these effects. Runs with the term have lower false negative rate at the cost of false positives. Results are for Qwen2.5-3B-Instruct on MBPP.
Figure 16: Optimizing generally reduces false negatives at the cost of false positives, compared to just using . By tuning the MI parameters and , we can obtain monitor accuracy comparable to runs (e.g., , but optimizing too strongly can lead to degradation ( ). Results are for Qwen2.5-3B-Instruct on MBPP.
Figure 17: Effects of various reward terms on the model’s completion length for Qwen 3B and 7B on MBPP. corresponds to just maximizing the task reward alone, corresponds to maximizing the task reward and monitor reward simultaneously, and additionally also maximize the oracle reward and MI reward respectively. Finally, for completeness, we also include which corresponds to maximizing the true environment reward.
Figure 18: Effects of parameter values on the 3B Qwen model’s completion length on MBPP. Higher values of generally lead to lower completion lengths. Oracle and runs are included in both figures for comparison.
Figure 19: Effects of parameter values on the 7B Qwen model’s completion length on MBPP. Higher values of generally lead to lower completion lengths.
| Hyperparameter | Value |
|---|---|
| Policy updates per rollout | 4 |
| Policy Clipping Parameter ( ) | 0.2 |
| Batch Size | 64 |
| Group Size | 16 |
| Policy Learning Rate | 2e-6 |
| Sampling Temperature | 1.2 |
Table 1: Reinforcement Learning Hyperparameters
Figure 20: MBPP Agent System Prompt. The agent system prompt tasks the model to reason step-by-step using <think> and <answer> tags.
Figure 21: MBPP Monitor Prompt. The test hacking monitor analyzes student thoughts to detect intentions to hack test cases rather than implementing proper solutions.
Figure 22: Ablation over MI reward parameters for Qwen 7B in the BigMath environment. The large deviation for the run is due to one of the three runs hacking on a significant number of prompts.
Figure 23: Optimizing together with leads to a reduction in Correct Answer Rate on BigMath, but careful tuning of MI parameters mitigates this effect. As in the MBPP environment, runs with the term reduce the monitor’s false negative rate at the cost of false positives. Result is for Qwen2.5-7B-Instruct.
Figure 24: BigMath Agent System Prompt. The agent system prompt tasks the model to reason step-by-step using <think> and <answer> tags. It also gives hints about the hack present in the BigMath environment.
Figure 25: BigMath Monitor Prompt. The monitor analyzes student thoughts to detect intentions to exploit a potential hack in the environment rather than to reason through a proper solutions.
| Qwen2.5-7B-Instruct | pass@1 | pass@4 |
|---|---|---|
| No CoT | 0.05 | 0.16 |
| CoT | 0.07 | 0.34 |
Table 2: Qwen2.5-7B-Instruct performance on 100 samples from the BigMath environment with and without CoT. On this data, using CoT improves model performance significantly.
| Qwen2.5-3B-Instruct | pass@1 | pass@4 |
|---|---|---|
| No CoT | 55.4 | 69.3 |
| CoT | 56.2 | 73.2 |
Table 3: Qwen performance on the MBPP train set. On this data, using CoT does not improve model performance significantly.
Figure 26: MBPP hacky example from a run with the 7B Qwen model. The model provides a hacky solution that passes only the visible tests. The monitor detects hacking, since the model explicitly talks about hardcoding the return values (in this setting, the monitor is only used for logging purposes and not for training).
Figure 27: MBPP hacky example from a run with the 7B Qwen model. The model provides a hacky solution that passes only the visible tests and has learned to hide this from the monitor in its CoT. As a result, the monitor fails to detect the hacking.
Figure 28: MBPP non-hacky example from a run with the 7B Qwen model. The model provides a general solution that passes both the visible and the hidden tests. The monitor correctly detects no hacking. Here the term is sufficient to make the model attempt a general solution to the problem with the monitor present.
Figure 29: MBPP hacky example from a run with the 7B Qwen model. The model provides a hacky solution that passes only the visible tests. Due to the term, the model explicitly states its intent to hack even with the monitor present. As a result, the monitor is able to detect this hacking attempt.
Figure 30: MBPP hacky example from a run with the 7B Qwen model. The model provides a hacky solution that passes only the visible tests, but the CoT does not make this clear. As a result, the monitor fails to detect the hacking. Here the reward term was not sufficient to make the model state its intent to hack.
Figure 31: BigMath example from a run with the 7B Qwen model. The model provides a hacky solution after some (incorrect) mathematical reasoning while mentioning the ‘negative number’ hints. The monitor correctly identifies this completion as hacky.
Figure 32: BigMath example from a run with the 7B Qwen model. The model provides a solution with some (partially correct) mathematical reasoning. The monitor classifies it as genuine, but the answer is incorrect and the model hacks the verifier by flipping the sign between its CoT and output.
Figure 33: BigMath example from a run with the 7B Qwen model. The model provides a solution with some (incorrect) mathematical reasoning. The monitor classifies it as genuine, but the answer is incorrect and the model hacks the verifier by flipping the sign between its CoT and output.
Figure 34: BigMath example from a run with the 7B Qwen model. The model provides a solution with some (correct) mathematical reasoning and arrives at the correct answer (285714). However, the monitor classifies it as genuine while the model hacks the verifier by flipping the sign between its CoT and output, producing -285714 as the final answer.
Figure 35: BigMath example from a run with the 7B Qwen model. The model provides a solution with correct mathematical reasoning and arrives at the correct answer. The monitor correctly identifies this as a non-hacky solution.
Figure 36: BigMath example from a run with the 7B Qwen model. The model provides a solution with incorrect mathematical reasoning. The monitor correctly identifies this as a non-hacky solution.
Figure 37: Reasoning Gym clean example from a run with the 3B model. The model provides a clean CoT to go with its correct final answer. The monitor (only used for logging on this run) correctly accepts this CoT.
Figure 38: Reasoning Gym CoT suppression example from a run with the 3B model. The model provides a conceptual explanation but no mathematical steps to solve the equation, yet arrives at the correct final answer. The model has learned to suppress relevant arithmetic steps while still getting the final answer right (although stochastically, see Figure 4 ). As a result, the monitor rejects this CoT.
Figure 39: Reasoning Gym clean example from a run with the 3B model; the no steganography run of Figure 4 . The model provides a legible step-by-step CoT that leads to the correct final answer. The monitor correctly accepts this CoT. In this run, the term provides pressure against the suppression strategies learned in the runs.
Figure 40: Reasoning Gym partially steganographic example from a run with the 3B model; a partial steganography run of Figure 4 . Although the model arrives at the correct final answer, its CoT introduces an error by using a square root instead of division in the final step. As a result, the monitor rejects this reasoning. This example is extracted after 4700 RL steps, which is more than we depict in Figure 4 . At earlier steps, this run exhibits CoT suppression behavior instead, although the suppression is less severe than for the runs (without reward term).
Figure 41: Reasoning Gym steganographic example from a run with the 3B model; the full steganography run of Figure 4 . The model provides a CoT in a different language (Korean) that only metaphorically describes the problem. As a result, the monitor rejects this as incorrect / insufficient reasoning. While the model succeeds at getting the correct final answer in this example, it generally failed to do so in this run (note the low task accuracy in Figure 4 ). For this run, we added a single line to the monitor prompt to help the monitor catch slightly suppressed CoTs, see Figure A.4 . With both the and reward present, this seems to encourage steganographic solutions compared to the example in Figure 40 that uses the original monitor prompt. Qualitative observation shows that this model also utilizes Japanese and Chinese completions.
More in this topic
CoT Monitoring
Chain-of-thought monitoring studies whether models’ reasoning traces expose decision-relevant behavior and support oversight of unsafe or deceptive actions.
See all 36 papers