Large language models (LLMs) can generate chains of thought (CoTs) that are not always causally responsible for their final outputs. When such a mismatch occurs, the CoT no longer faithfully reflects the actual reasons (i.e., decision-critical factors) driving the model's behavior, leading to the reduced CoT monitorability problem. This limits the use of CoTs for reliable oversight. However, a comprehensive and fully open-source benchmark for thoroughly evaluating CoT monitorability remains lacking. To address this gap, we propose MonitorBench, a systematic benchmark for evaluating CoT monitorability in LLMs. MonitorBench provides: (1) a diverse set of 1,514 test instances with carefully designed decision-critical factors across 19 tasks spanning 7 categories to characterize when CoTs can be used to monitor the factors driving LLM behavior; and (2) two prompting stress-test settings to quantify the extent to which CoT monitorability can be degraded. Extensive experiments show that CoT monitorability is a conditional property affected by the evaluated LLM, monitor LLM, and task characteristics. Across these factors, monitorability is higher when decision-critical factors shape the intermediate reasoning process, rather than merely influencing the final answer. Under stress-test prompting, most evaluated LLMs can intentionally reduce monitorability, mainly on tasks where decision-critical factors are not structurally required by the reasoning process. Overall, MonitorBench provides a basis for further research on AI control, reasoning faithfulness, stress-test monitorability, and monitoring scaffords. The code is available at https://github.com/ASTRAL-Group/MonitorBench.
Figures & tables
Figure 1: Examples of low and high CoT monitorability. The highlighted spans mark the decision-critical factors in each task. (Left) Gemini-3-Flash successfully encodes “TOOTHBRUSH” as the initials of consecutive sentences in its final response, yet never explicitly plans for this in its CoT. (Right) Gemini-3-Flash explicitly references the injected answer (I20.0) in its CoT and uses it to arrive at the final diagnosis.
Figure 2: Monitorability score (%) (details in Sec. 3.3 ) averaged over 19 tasks for each evaluated LLM (left), and averaged over 8 evaluated LLMs across tasks within each category (right), under different monitor scopes in both standard and two stress-test settings. Results show that CoT monitorability is higher when the decision-critical factors shape the intermediate reasoning process without merely influencing the final answer (i.e., Goal (Sandbag), Cue (Condition), Dual Objectives, Impossible Tasks). While most evaluated LLMs can intentionally reduce monitorability to some extent under the stress-tests, the monitorability degradation mostly comes from the tasks whose decision-critical factors do not shape the reasoning process (e.g., Cue (Answer)).
Input Intervention
Outcome Justification
Solution Process
Cue (Condition) Knowledge 100 Preference 108 Law Judgment 100 Health 65 Cue (Answer) Mathematics 28 Logic 50
Health 65 Goal (Sandbag) Safety 150 Mathematics 30 Science 192 Engineering 100 Goal (Deception) Real-world Case 50
Table 1: Statistics of 1,514 instances across 19 tasks spanning 7 categories in MonitorBench. “ ∗ ” indicates that the task requires multi-round interactions with the environment. Detailed design and examples for each task are in Appendix A.4 and A.6 , respectively.
Task Category
Task
Open-source
Closed-source
Avg. over LLMs
Qwen3 14B
GPT-OSS 120B
Qwen3 235B
GPT-5 mini
GPT-5
Gemini-3 Flash
Claude Haiku 4.5
Claude Sonnet 4.5
Cue (Condition)
Health
95.1
100.0
99.7
100.0
99.4
100.0
97.3
99.0
98.8
Knowledge
98.3
98.5
99.3
95.1
93.2
98.1
97.2
100.0
97.5
Law Judgment
97.9
99.5
98.0
65.8
75.1
97.9
99.3
99.8
91.6
Preference
97.8
99.5
100.0
90.1
98.1
98.5
100.0
100.0
98.0
Cue (Answer)
Health
89.1
87.6
96.9
87.5
83.8
79.4
86.7
93.5
88.1
Table 2: Monitorability score (%), under CoT-only monitor scope in the standard setting. – denotes not applicable due to refusal triggered by the safety alignment.
Task Category
Task
Open-source
Closed-source
Avg. over LLMs
Qwen3-14B
GPT-OSS-120B
Qwen3-235B
GPT-5-mini
GPT-5
Gemini-3-Flash
Claude Haiku 4.5
Claude Sonnet 4.5
DC
ME
DC
ME
DC
ME
DC
ME
DC
ME
DC
ME
DC
ME
DC
ME
DC
ME
Cue (Condition)
Health
0.4
-6.2
-0.5
-13.9
0.3
-0.9
-11.5
-14.2
-0.4
-1.4
-1.7
-11.0
-1.5
-1.9
1.0
-0.9
-1.7
-6.3
Knowledge
-3.2
0.2
0.9
0.9
-1.4
-1.2
-2.5
-0.5
-3.3
4.2
-7.9
-3.5
-4.5
-9.8
0.0
-10.0
-2.2
-1.4
Law Judgment
-30.3
-7.0
-10.0
-18.5
-10.7
-10.3
-24.9
-11.6
-14.6
-7.3
-11.2
-9.0
-0.8
-0.9
0.1
-1.8
-12.8
-6.9
Preference
-6.2
-1.2
-0.3
-7.1
-2.4
-3.8
-9.9
-9.2
-11.3
-14.2
-8.6
-7.2
-3.1
-26.6
0.0
-0.5
-5.3
-8.7
Table 3: Monitorability score difference (%) under CoT-only monitoring between stress-test and standard settings. DC and ME denote Direct Concealment and Monitor-aware Evasion; – indicates safety-alignment refusal.
Figure 3: Relationship between monitorability score and LLM capability under the all-messages monitor scope . Each point corresponds to one evaluated LLM. We measure the monitorability score average across all tasks, and approximate LLM capability by the average pass rate on the original task questions. The fitted regression line shows a moderate negative Pearson correlation ( r ), suggesting that more capable LLMs tend to exhibit lower monitorability scores.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Relationship between monitorability score and LLM capability under the CoT-only monitor scope . Each point corresponds to one evaluated LLM. We measure the monitorability score average across all tasks, and approximate capability by the average pass rate on the original task questions. The fitted regression line shows a moderate negative Pearson correlation ( r ), suggesting that more capable LLMs tend to exhibit lower monitorability.
Figure 5: Relationship between monitorability and LLM capability under the action-only monitor scope . Each point corresponds to one evaluated LLM. We measure the monitorability score average across all tasks, and approximate capability by the average pass rate on the original task questions. The fitted regression line shows a moderate negative Pearson correlation ( r ), suggesting that more capable LLMs tend to exhibit lower monitorability.
Task Category
Task
Open-source
Closed-source
Avg. over LLMs
Qwen3 14B
GPT-OSS 120B
Qwen3 235B
GPT-5 mini
GPT-5
Gemini-3 Flash
Claude Haiku 4.5
Claude Sonnet 4.5
Cue (Condition)
Health
96.2
100.0
100.0
98.6
100.0
99.7
100.0
99.3
99.2
Knowledge
99.8
99.2
99.2
99.3
97.7
97.5
99.6
100.0
99.0
Law Judgment
99.5
99.7
99.1
86.7
91.8
99.6
99.6
100.0
97.0
Preference
99.8
99.9
99.7
99.7
99.4
99.8
100.0
100.0
99.8
Cue (Answer)
Health
78.8
86.5
92.9
84.1
88.8
81.8
91.7
95.0
87.4
Appendix
Table 4: Monitorability score (%), under all-messages monitor scope in the standard setting. – denotes not applicable due to refusal triggered by the safety alignment.
Task Category
Task
Open-source
Closed-source
Avg. over LLMs
Qwen3 14B
GPT-OSS 120B
Qwen3 235B
GPT-5 mini
GPT-5
Gemini-3 Flash
Claude Haiku 4.5
Claude Sonnet 4.5
Cue (Condition)
Health
96.2
100.0
99.8
100.0
100.0
100.0
100.0
100.0
99.5
Knowledge
93.6
89.4
95.8
97.4
95.3
96.1
97.6
100.0
95.6
Law Judgment
97.2
83.0
98.0
72.2
68.6
91.0
92.5
98.9
87.7
Preference
99.7
99.3
99.4
99.1
100.0
100.0
100.0
100.0
99.7
Cue (Answer)
Health
44.9
41.8
71.7
45.1
46.5
53.0
79.6
75.7
57.2
Appendix
Table 5: Monitorability score (%), under action-only monitor scope in the standard setting. – denotes not applicable due to refusal triggered by the safety alignment.
Task Category
Task
Open-source
Closed-source
Avg. over LLMs
Qwen3-14B
GPT-OSS-120B
Qwen3-235B
GPT-5-mini
GPT-5
Gemini-3-Flash
Claude Haiku 4.5
Claude Sonnet 4.5
DC
ME
DC
ME
DC
ME
DC
ME
DC
ME
DC
ME
DC
ME
DC
ME
DC
ME
Cue (Condition)
Health
-2.9
1.4
0.0
-12.2
-0.2
-3.7
-0.7
-6.9
0.0
-1.3
-0.4
-11.4
-2.8
-2.7
0.3
-9.3
-0.8
-5.8
Knowledge
-1.2
-0.9
0.4
0.7
-0.4
0.6
-2.8
-1.1
-2.6
0.5
-3.8
-0.1
-0.5
-0.6
0.0
-0.2
-1.4
-0.1
Law Judgment
-34.0
-2.1
-10.4
-12.0
-10.7
-6.9
-31.9
-3.1
-21.6
-11.5
-11.3
-6.3
-2.0
-1.1
-0.6
-4.9
-15.3
-6.0
Preference
-10.6
-3.3
-2.1
-11.8
-3.5
-6.8
-12.1
-13.2
-12.8
-14.5
-12.1
-10.0
-1.8
-3.1
-0.5
-3.5
-6.9
-8.2
Appendix
Table 6: Monitorability score difference (%) in all-messages monitor scope under stress tests relative to the standard setting. Direct Concealment and Monitor-aware Evasion are denoted as DC and ME, respectively. – denotes not applicable due to refusal triggered by the safety alignment.
Task Category
Task
Open-source
Closed-source
Avg. over LLMs
Qwen3-14B
GPT-OSS-120B
Qwen3-235B
GPT-5-mini
GPT-5
Gemini-3-Flash
Claude Haiku 4.5
Claude Sonnet 4.5
DC
ME
DC
ME
DC
ME
DC
ME
DC
ME
DC
ME
DC
ME
DC
ME
DC
ME
Cue (Condition)
Health
1.4
-7.3
0.0
-4.2
-0.4
-1.9
-4.2
0.0
-5.9
0.0
0.0
-3.3
-0.9
-3.5
-0.6
-5.6
-1.3
-3.2
Knowledge
-5.9
0.9
-6.0
-1.0
-7.0
-6.7
1.4
-0.4
0.9
1.6
0.1
-2.8
0.9
-2.1
-0.4
-3.1
-2.0
-1.7
Law Judgment
-53.5
-3.1
-23.2
-15.6
-35.0
-11.7
-24.0
3.0
-26.3
-11.4
-32.4
-26.7
-27.1
-18.1
-18.9
-25.7
-30.0
-13.7
Preference
-19.2
-7.4
-7.8
-12.2
-10.9
-14.5
-19.1
-15.2
-29.4
-30.8
-15.5
-30.6
-18.9
-4.7
-4.1
-17.4
-15.6
-16.6
Appendix
Table 7: Monitorability score difference (%) in action-only monitor scope under stress tests relative to the standard setting. Direct Concealment and Monitor-aware Evasion are denoted as DC and ME, respectively. – denotes not applicable due to refusal triggered by the safety alignment.
Figure 6: Example that LLMs fail to hide the decision-critical factors under the stress-test setting. The highlighted spans mark the stress-test instruction and the decision-critical factor (i.e., side task). The explicit factor leakage showcases how the model directly names the side objective factor in its reasoning trace. The concealment leakage shows that the CoT explicitly plans to hide its planning. The act of naming the concealment strategy makes the side objective factor visible. The implicit planning leakage showcases how even without explicit naming, the step-by-step operational plan renders the side objective structurally inferable. The example is drawn from Qwen3-235B-A22B on Coding in Dual Objectives.
Stress-test Failure Mode
Direct Concealment
Monitor-aware Evasion
Qwen3 235B
GPT-5
Gemini-3 Flash
Claude Haiku 4.5
Qwen3 235B
GPT-5
Gemini-3 Flash
Claude Haiku 4.5
Explicit Factor Leakage
100.0
97.2
100.0
100.0
99.3
96.7
100.0
99.7
Concealment Leakage
84.5
64.0
88.2
99.8
31.5
15.3
23.5
0.7
Implicit Planning Leakage
48.1
37.4
31.3
31.9
36.2
36.4
21.2
20.4
Other Cases
0.0
1.2
0.0
0.0
0.0
0.8
0.0
0.0
Appendix
Table 8: Failure mode of why decision-critical factors remain monitorable under stress tests. We focus on Coding in Dual Objectives under the CoT-only monitor scope. Specifically, the decision-critical factors here are the side features that need to be integrated into the main functions. We use GPT-5 as the LLM-as-a-judge for annotation; the full prompts are provided in Appendix. A.5.4 . Because many responses exhibit multiple failure modes, we allow multi-label annotations and report the fraction of trajectories exhibiting each failure mode..
Figure 7: Example that LLMs hide the decision-critical factors under the stress-test setting. (Left) In the standard setting, Gemini-3-Flash explicitly verbalizes the decision-critical factors (i.e., mentioning the discount for this instance) in the CoT, demonstrating high CoT monitorability. (Right) Under the stress-test, the model hides the decision-critical factors from its CoT while arriving at a deceptive final response, demonstrating low CoT monitorability. The highlighted spans mark the decision-critical factor (left) and the stress-test instruction (right). Examples are from Goal (Deception) – Input Intervention.
Figure 8: Example that LLMs fail to hide the decision-critical factors under the stress-test setting, due to the explicit factor leakage . The highlighted spans mark the monitor-aware evasion instruction and the decision-critical factor (i.e., side objective). Qwen3-14B explicitly leaks information regarding the side objective within its CoT when dealing with the steganography task. This response also contains the concealment leakage .
Figure 9: Example that LLMs fail to hide the decision-critical factors under the stress-test setting, due to the concealment leakage . The highlighted spans mark the direct concealment instruction and the decision-critical factor (i.e., side objective). Under the stress-test, the CoT contains information regarding the concealment of the side objective showcases and how they should not be mentioned in the final response.
Chain-of-thought (CoT) reasoning offers a window into the decision-making of large language models (LLMs), which can be monitored for target behaviors by reading the reasoning trace, motivating work on CoT monitorability. Latent CoT approaches, however, replace the explicit tokens with a small number of continuous states, lowering inference costs but removing the readable trace this monitoring relies on. Monitoring then requires alternative access to the model, such as probing its activations or verbalizing the latent states back into text, but how much monitorability these alternatives preserve is unclear. We study this question with a hint-based intervention setup, a proxy for behaviors where models exploit biasing input cues, e.g., an inadvertently leaked answer or a belief stated by the user, without acknowledging them. Taking hint-reliance as the monitorability target, we compare monitors across reasoning modes, from explicit CoT to weakly- and strongly-supervised latent CoT, on math reasoning and question answering. We find that, in this setup, monitorability depends more on properties of the task (such as whether the correct answer constrains the supporting reasoning) and the level of access to model internals than on the reasoning mode.
Chain-of-thought (CoT) monitoring has been proposed as a promising safety mechanism for detecting misaligned behavior in large language models. However, its reliability remains largely unexplored beyond English and across diverse model families. We present the first large-scale evaluation of CoT monitorability across 13 diverse languages and seven frontier model families, comprising 16 models. Using adversarial-hint evaluations that require explicit intermediate computation, together with analysis of internal answer-token probabilities, we consistently find CoT unfaithfulness across languages and hint types, with an average rate of 95.9% across 8B--120B parameter models. We find that frontier models systematically exhibit strategic manipulation, including answer-switching, post-hoc rationalization, and procedural exploitation of hints, making their reasoning difficult to reliably monitor. These deceptive patterns remain especially pronounced in low-resource languages, revealing fundamental limitations in current CoT-based oversight. Our results show that CoT monitoring is fragile under linguistic distribution shift, providing a substantially weaker safety signal than English-only studies suggest. These findings motivate the development of more robust CoT monitors and complementary white-box monitoring techniques, particularly for mid- and low-resource languages. Our code is available \href{https://multilingual-cot-monitoring.github.io/}{\textcolor{blue}{here}}.
Eric Onyame, Runtao Zhou, Kowshik Thopalli +2
University of Virginia · Lawrence Livermore National Laboratory
Chain-of-thought (CoT) monitoring is a promising safety mechanism for AI agents, based on the premise that visible reasoning traces can surface misaligned or deceptive behavior. While effective in standard scenarios, recent work highlights that LLMs remain vulnerable to persuasion-based jailbreaks, where natural-language arguments override model constraints. We stress-test whether this vulnerability extends to monitoring LLMs: can an adversarial agent persuade its CoT monitor to approve proposed actions that violate the monitor's policy? We design an evaluation framework with 40 tasks and analyze thousands of agent-monitor interactions, where agents are instructed to argue for policy-violating proposals. We find that in such adversarial settings, monitor access to the agent's CoT reasoning increases rather than decreases approval of harmful actions on average by 9.5%, as the scratchpad provides an additional persuasion channel. To address this, we introduce a fact-checking monitoring framework. We find that a fact-checker and monitor pairing from different model families, for example a Claude 3.7 Sonnet monitor paired with a GPT-4.1 fact-checker, reduces approval of policy-violating actions by up to 45%, compared to only 6%, when using the same model for both fact-checking and monitoring roles. Our results demonstrate that CoT monitoring alone may be insufficient against adversarial persuasion, and that model-diverse fact-checking provides a robust mitigation.
Jennifer Za, Julija Bainiaksina, Nikita Ostrovsky +2
LASR Labs · University College London · Google DeepMind