Chain-of-thought (CoT) monitors are LLM-based systems that analyze reasoning traces to detect when outputs may exhibit attributes of interest, such as test-hacking behavior during code generation. In this paper, we use information-theoretic analysis to show that non-zero mutual information between CoT and output is a necessary but not sufficient condition for CoT monitorability. We identify two sources of approximation error that may undermine the performance of CoT monitors in practice: information gap, which measures the extent to which the monitor can extract the information available in CoT, and elicitation error, which measures the extent to which the monitor approximates the optimal monitoring function. We further demonstrate that CoT monitorability can be systematically improved through targeted training objectives. To this end, we propose two complementary approaches: (a) an oracle-based method that directly rewards the monitored model for producing CoTs that maximize monitor accuracy, and (b) a more practical, label-free approach that maximizes conditional mutual information between outputs and CoTs. Across multiple different environments, we show both methods significantly improve monitor accuracy while preventing CoT degeneration even when training against a monitor, thereby mitigating reward hacking when the task reward is imperfectly specified.
Figures & tables
Figure 1: RL training Qwen 7B on difficult math problems from BigMath-RL-Verified. The environment contains a hack, such that negative valued answers always obtain the same task reward as correct answers. During training, a monitor LLM checks the model’s CoT for indicators of hacking. The model, trained to maximize task and monitor reward, learns to fool the monitor by providing genuine-looking CoT but hacky output. Adding our proposed MI-inspired reward term mitigates monitor fooling behaviors, resulting in less hacky models and more transparent CoT. Note that, on this problem, the model gets the answer wrong in its CoT in both settings, because the problem is too hard. This encourages hacking to still receive the task reward. Note also that the true answer is (22/3−1)3≈0.202 , rather than the 51 listed as ground truth in the dataset, further encouraging negative-valued solutions to avoid answer mismatches.
Figure 2: Graphical model of our Chain-of-Thought setup.
Figure 3: Comparison of (i) Mutual Information I(O;Z∣X) , (ii) Monitor Accuracy , (iii) Hacking Rate , and (iv) True Pass Rate / Correct Answer Rate across training steps for five different reward configurations. RTask corresponds to just maximizing the task reward alone, RTask+RMonitor corresponds to maximizing the task reward and monitor reward simultaneously, RTask+RMonitor+ROracle and RTask+RMonitor+RMI additionally also maximize the oracle reward and MI reward respectively. Finally, for completeness, we also include RTrue which corresponds to maximizing the true environment reward. All curves are averaged across three seeds, and shaded area shows the standard deviation. To obtain RMI , we transform the MI values using (a=20,b=0.5) for MBPP and (a=3,b=0.1) for BigMath.
Appendix figures & tables41 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Comparison of (i) Task Accuracy , (ii) Mutual Information Estimate I(O;Z∣X) , (iii) Monitor Rejection Rate , and (iv) Completion Length , across training steps for three different reward configurations of the 3B Qwen model on simple-equations . RTask corresponds to just maximizing the task reward alone (four seeds), RTask+RMonitorReject corresponds to maximizing the task reward and monitor rejection rate simultaneously (three seeds), RTask+RMonitorReject+RMI maximizes MI reward (five seeds). For the latter, one seed fools the monitor through steganographic CoT but fails the task (denoted full steganography in the legend); three seeds perform a mixture of suppressive and weakly-steganographic CoT , partially succeeding both at the task and at fooling the monitor (denoted partial steganography); one seed gives up on fooling the monitor (denoted no steganography). Shaded area shows the standard deviation. To obtain RMI , we transform the MI values using (a=20,b=2) .
Figure 5: Comparison of (i) Task Accuracy , (ii) Mutual Information Estimate I(O;Z∣X) , (iii) Monitor Rejection Rate , and (iv) Completion Length , across training steps for three different reward configurations and monitor prompts of the 7B Qwen model on simple-equations . RTask corresponds to just maximizing the task reward alone, RTask+RMonitorReject corresponds to maximizing the task reward and monitor rejection rate simultaneously, RTask+RMonitorReject+RMI maximizes MI reward. Shaded area shows the standard deviation across three seeds. To obtain RMI , we transform the MI values using (a=20,b=2) .
Figure 6: Reasoning Gym Agent System Prompt. The agent system prompt tasks the model to reason step-by-step using <think> and <answer> tags. It also encourages the model to use diverse languages in an attempt to elicit steganographic reasoning. However, while the agent does reason in a variety of existing languages as a result, the monitor is able to decode these languages well. The agent it does not generally step further into steganographic reasoning.
Figure 7: Reasoning Gym Monitor Prompt. The monitor analyzes student thoughts to determine whether the final answer that the student provides will be correct. The text in orange is the anti-suppression prompt used in the 7B runs of Figure 5(b) and in the 3B run that exhibits full steganography in Figure 4 : intuitively, it makes successfully executing suppression strategies harder, since the monitor is more prepared for missing steps.
Figure 8: Comparison of (i) Mutual Information I(O;Z∣X) , (ii) Monitor Accuracy , (iii) Hacking Rate , and (iv) True Pass Rate across training steps for five different reward configurations for the 3B and 7B Qwen models on MBPP. RTask corresponds to just maximizing the task reward alone, RTask+RMonitor corresponds to maximizing the task reward and monitor reward simultaneously, RTask+RMonitor+ROracle and RTask+RMonitor+RMI additionally also maximize the oracle reward and MI reward respectively. Finally, for completeness, we also include RTrue which corresponds to maximizing the true environment reward. To obtain RMI , we transform the MI values using (a=20,b=2) for the 3B agent, and (a=20,b=0.5) for the 7B agent.
Figure 9: Comparison of (i) Mutual Information I(O;Z∣X) , (ii) Monitor Accuracy , (iii) Hacking Rate , and (iv) True Pass Rate across training steps for five different reward configurations for the SmolLM3-3B and DeepSeek-R1-Distill-Llama-8B models on MBPP. RTask corresponds to just maximizing the task reward alone, RTask+RMonitor corresponds to maximizing the task reward and monitor reward simultaneously, RTask+RMonitor+ROracle and RTask+RMonitor+RMI additionally also maximize the oracle reward and MI reward respectively. Finally, for completeness, we also include RTrue which corresponds to maximizing the true environment reward. To obtain RMI , we transform the MI values using (a=20,b=0.5) for the SmolLM agent, and (a=100,b=1) for the DeepSeek agent.
Figure 10: Ablation over MI reward parameters for the 3B Qwen model in the MBPP environment. Oracle and (a=20,b=2) runs are included in both subplots for comparison. Too low values of b do not reliably prevent hacking, while too high values lead to lower model performance.
Figure 11: Ablation over MI reward parameters for the 7B Qwen model in the MBPP environment. In this setting, we do not observe an increase in hacking rate as b decreases, but do observe deterioration of True Pass Rate as b increases.
Figure 12: Ablation over MI reward parameters for the SmolLM3 model in the MBPP environment. Notably, more hacky behavior is associated with higher b values here.
Figure 13: Ablation over MI reward parameters for the DeepSeek model in the MBPP environment. Note the two orders of magnitude difference in MI estimate scale for these runs, resulting in larger a values being necessary for proper MI optimization.
Figure 14: Comparison of 3B Qwen models trained to optimize RTask+RMonitor+RMI for two different monitor prompt variants. The Lite Prompt monitor is expected to be less sensitive to flagging benign CoT as hacky, which is reflected in the reduced False Positive Rate. However, this comes at the cost of a higher False Negative Rate.
Figure 15: Optimizing RMI together with RTrue (no monitor pressure) can lead to a reduction in True Pass Rate and monitor accuracy, but careful tuning of MI parameters mostly offsets these effects. Runs with the RMI term have lower false negative rate at the cost of false positives. Results are for Qwen2.5-3B-Instruct on MBPP.
Figure 16: Optimizing RTask+RMI generally reduces false negatives at the cost of false positives, compared to just using RTask . By tuning the MI parameters a and b , we can obtain monitor accuracy comparable to ROracle runs (e.g., a=20,b=0.5) , but optimizing RMI too strongly can lead to degradation ( a=20,b=2 ). Results are for Qwen2.5-3B-Instruct on MBPP.
Figure 17: Effects of various reward terms on the model’s completion length for Qwen 3B and 7B on MBPP. RTask corresponds to just maximizing the task reward alone, RTask+RMonitor corresponds to maximizing the task reward and monitor reward simultaneously, RTask+RMonitor+ROracle and RTask+RMonitor+RMI additionally also maximize the oracle reward and MI reward respectively. Finally, for completeness, we also include RTrue which corresponds to maximizing the true environment reward.
Figure 18: Effects of RMI parameter values on the 3B Qwen model’s completion length on MBPP. Higher values of b generally lead to lower completion lengths. Oracle and (a=20,b=2) runs are included in both figures for comparison.
Figure 19: Effects of RMI parameter values on the 7B Qwen model’s completion length on MBPP. Higher values of b generally lead to lower completion lengths.
Hyperparameter
Value
Policy updates per rollout
4
Policy Clipping Parameter ( ϵ )
0.2
Batch Size
64
Group Size
16
Policy Learning Rate
2e-6
Sampling Temperature
1.2
Appendix
Table 1: Reinforcement Learning Hyperparameters
Figure 20: MBPP Agent System Prompt. The agent system prompt tasks the model to reason step-by-step using <think> and <answer> tags.
Figure 21: MBPP Monitor Prompt. The test hacking monitor analyzes student thoughts to detect intentions to hack test cases rather than implementing proper solutions.
Figure 22: Ablation over MI reward parameters for Qwen 7B in the BigMath environment. The large deviation for the (a=5,b=0.1) run is due to one of the three runs hacking on a significant number of prompts.
Figure 23: Optimizing RMI together with RTrue leads to a reduction in Correct Answer Rate on BigMath, but careful tuning of MI parameters mitigates this effect. As in the MBPP environment, runs with the RMI term reduce the monitor’s false negative rate at the cost of false positives. Result is for Qwen2.5-7B-Instruct.
Figure 24: BigMath Agent System Prompt. The agent system prompt tasks the model to reason step-by-step using <think> and <answer> tags. It also gives hints about the hack present in the BigMath environment.
Figure 25: BigMath Monitor Prompt. The monitor analyzes student thoughts to detect intentions to exploit a potential hack in the environment rather than to reason through a proper solutions.
Qwen2.5-7B-Instruct
pass@1
pass@4
No CoT
0.05
0.16
CoT
0.07
0.34
Appendix
Table 2: Qwen2.5-7B-Instruct performance on 100 samples from the BigMath environment with and without CoT. On this data, using CoT improves model performance significantly.
Qwen2.5-3B-Instruct
pass@1
pass@4
No CoT
55.4
69.3
CoT
56.2
73.2
Appendix
Table 3: Qwen performance on the MBPP train set. On this data, using CoT does not improve model performance significantly.
Figure 26: MBPP hacky example from a RTask run with the 7B Qwen model. The model provides a hacky solution that passes only the visible tests. The monitor detects hacking, since the model explicitly talks about hardcoding the return values (in this setting, the monitor is only used for logging purposes and not for training).
Figure 27: MBPP hacky example from a RTask+RMonitor run with the 7B Qwen model. The model provides a hacky solution that passes only the visible tests and has learned to hide this from the monitor in its CoT. As a result, the monitor fails to detect the hacking.
Figure 28: MBPP non-hacky example from a RTask+RMonitor+RMI run with the 7B Qwen model. The model provides a general solution that passes both the visible and the hidden tests. The monitor correctly detects no hacking. Here the RMI term is sufficient to make the model attempt a general solution to the problem with the monitor present.
Figure 29: MBPP hacky example from a RTask+RMonitor run with the 7B Qwen model. The model provides a hacky solution that passes only the visible tests. Due to the RMI term, the model explicitly states its intent to hack even with the monitor present. As a result, the monitor is able to detect this hacking attempt.
Figure 30: MBPP hacky example from a RTask+RMonitor+RMI run with the 7B Qwen model. The model provides a hacky solution that passes only the visible tests, but the CoT does not make this clear. As a result, the monitor fails to detect the hacking. Here the RMI reward term was not sufficient to make the model state its intent to hack.
Figure 31: BigMath example from a RTask run with the 7B Qwen model. The model provides a hacky solution after some (incorrect) mathematical reasoning while mentioning the ‘negative number’ hints. The monitor correctly identifies this completion as hacky.
Figure 32: BigMath example from a RTask run with the 7B Qwen model. The model provides a solution with some (partially correct) mathematical reasoning. The monitor classifies it as genuine, but the answer is incorrect and the model hacks the verifier by flipping the sign between its CoT and output.
Figure 33: BigMath example from a RTask+RMonitor run with the 7B Qwen model. The model provides a solution with some (incorrect) mathematical reasoning. The monitor classifies it as genuine, but the answer is incorrect and the model hacks the verifier by flipping the sign between its CoT and output.
Figure 34: BigMath example from a RTask+RMonitor run with the 7B Qwen model. The model provides a solution with some (correct) mathematical reasoning and arrives at the correct answer (285714). However, the monitor classifies it as genuine while the model hacks the verifier by flipping the sign between its CoT and output, producing -285714 as the final answer.
Figure 35: BigMath example from a RTask+RMonitor+RMI run with the 7B Qwen model. The model provides a solution with correct mathematical reasoning and arrives at the correct answer. The monitor correctly identifies this as a non-hacky solution.
Figure 36: BigMath example from a RTask+RMonitor+RMI run with the 7B Qwen model. The model provides a solution with incorrect mathematical reasoning. The monitor correctly identifies this as a non-hacky solution.
Figure 37: Reasoning Gym clean example from a RTask run with the 3B model. The model provides a clean CoT to go with its correct final answer. The monitor (only used for logging on this run) correctly accepts this CoT.
Figure 38: Reasoning Gym CoT suppression example from a RTask+RMonitorReject run with the 3B model. The model provides a conceptual explanation but no mathematical steps to solve the equation, yet arrives at the correct final answer. The model has learned to suppress relevant arithmetic steps while still getting the final answer right (although stochastically, see Figure 4 ). As a result, the monitor rejects this CoT.
Figure 39: Reasoning Gym clean example from a RTask+RMonitorReject+RMI run with the 3B model; the no steganography run of Figure 4 . The model provides a legible step-by-step CoT that leads to the correct final answer. The monitor correctly accepts this CoT. In this run, the RMI term provides pressure against the suppression strategies learned in the RTask+RMonitorReject runs.
Figure 40: Reasoning Gym partially steganographic example from a RTask+RMonitorReject+RMI run with the 3B model; a partial steganography run of Figure 4 . Although the model arrives at the correct final answer, its CoT introduces an error by using a square root instead of division in the final step. As a result, the monitor rejects this reasoning. This example is extracted after 4700 RL steps, which is more than we depict in Figure 4 . At earlier steps, this run exhibits CoT suppression behavior instead, although the suppression is less severe than for the RTask+RMonitorReject runs (without RMI reward term).
Figure 41: Reasoning Gym steganographic example from a RTask+RMonitorReject+RMI run with the 3B model; the full steganography run of Figure 4 . The model provides a CoT in a different language (Korean) that only metaphorically describes the problem. As a result, the monitor rejects this as incorrect / insufficient reasoning. While the model succeeds at getting the correct final answer in this example, it generally failed to do so in this run (note the low task accuracy in Figure 4 ). For this run, we added a single line to the monitor prompt to help the monitor catch slightly suppressed CoTs, see Figure A.4 . With both the RMonitorReject and RMI reward present, this seems to encourage steganographic solutions compared to the example in Figure 40 that uses the original monitor prompt. Qualitative observation shows that this model also utilizes Japanese and Chinese completions.
Chain-of-thought monitoring studies whether models’ reasoning traces expose decision-relevant behavior and support oversight of unsafe or deceptive actions.
Chain-of-Thought (CoT) monitoring, in which automated systems monitor the CoT of an LLM, is a promising approach for effectively overseeing AI systems. However, the extent to which a model's CoT helps us oversee the model - the monitorability of the CoT - can be affected by training, for instance by the model learning to hide important features of its reasoning. We propose and empirically validate a conceptual framework for predicting when and why this occurs. We model LLM post-training as an RL environment where the reward decomposes into two terms: one term depending on final outputs and another term depending on the CoT. Our framework allows us to classify these two terms as "aligned", "orthogonal", or "in-conflict" before training. We predict that training with in-conflict terms will reduce monitorability, orthogonal terms will not affect it, and aligned terms will improve it. To validate our framework, we use it to classify a set of RL environments, train LLMs within those environments, and evaluate how training affects CoT monitorability. We find that (1) training with "in-conflict" reward terms reduces CoT monitorability and (2) optimizing in-conflict reward terms is difficult.
Max Kaufmann, David Lindner, Roland S. Zimmermann +1
Large language models (LLMs) can generate chains of thought (CoTs) that are not always causally responsible for their final outputs. When such a mismatch occurs, the CoT no longer faithfully reflects the actual reasons (i.e., decision-critical factors) driving the model's behavior, leading to the reduced CoT monitorability problem. This limits the use of CoTs for reliable oversight. However, a comprehensive and fully open-source benchmark for thoroughly evaluating CoT monitorability remains lacking. To address this gap, we propose MonitorBench, a systematic benchmark for evaluating CoT monitorability in LLMs. MonitorBench provides: (1) a diverse set of 1,514 test instances with carefully designed decision-critical factors across 19 tasks spanning 7 categories to characterize when CoTs can be used to monitor the factors driving LLM behavior; and (2) two prompting stress-test settings to quantify the extent to which CoT monitorability can be degraded. Extensive experiments show that CoT monitorability is a conditional property affected by the evaluated LLM, monitor LLM, and task characteristics. Across these factors, monitorability is higher when decision-critical factors shape the intermediate reasoning process, rather than merely influencing the final answer. Under stress-test prompting, most evaluated LLMs can intentionally reduce monitorability, mainly on tasks where decision-critical factors are not structurally required by the reasoning process. Overall, MonitorBench provides a basis for further research on AI control, reasoning faithfulness, stress-test monitorability, and monitoring scaffords. The code is available at https://github.com/ASTRAL-Group/MonitorBench.
Han Wang, Yifan Sun, Brian Ko +8
University of Illinois Urbana-Champaign · University of Washington · University of California San Diego
Chain-of-thought (CoT) monitors are reported by their aggregate accuracy on a pool of reward hacks. We show that this number is a false average. On Terminal Wrench, about 77% of hacks are given away by the actions alone, and the monitor's pooled accuracy is dominated by them; on the remaining 23%, where the reasoning is the only signal, the same monitor is fragile. We expose the fragility with a controlled attack: we rewrite only the agent's reasoning to read as good-faith engineering, leaving every command and output byte-identical, so the exploit is unchanged. One gradient-free rewrite drops a held-out monitor's catch rate on that subset from about 95% to between 4 and 11%, while the pooled rate falls only about 25 points, the subset's share of the pool times its collapse. The rewrite stays truthful about what happened and lies only about intent, so a monitor that reads only the trace recovers little, even when warned about the attack, and only information from outside the trace helps much. The attack transfers across monitor and agent families, and the dataset's own hack labels show the same gradient with no monitor in the definition. Live agents on 27 tasks produce hollow hacks unaided, but a monitor calibrated on each model's own honest solves catches nearly all of them (AUC at least 0.996 under the narration instruction, 0.97 without it); a small residual has the blindspot's shape, low scores on both actions and narration, though none falls below the calibrated threshold. A linear probe on a surrogate monitor's activations out-reads its verdict on the obfuscated hacks (AUC up to 0.98), but a behavioral control shows this edge does not, by itself, mean the monitor knows more than it says. Monitor accuracy should be reported on the reasoning-dependent subset, where the defense is supportive.