MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models

Han Wang; Yifan Sun; Brian Ko; Mann Talati; Jiawen Gong; Zimeng Li; Naicheng Yu; Xucheng Yu

cs.AIMar 30, 2026

MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models

Authors: Han Wang, Yifan Sun, Brian Ko, Mann Talati, Jiawen Gong, Zimeng Li, Naicheng Yu, Xucheng Yu, +3 more

Organizations: University of Illinois Urbana-Champaign · University of Washington · University of California San Diego

Abstract

Large language models (LLMs) can generate chains of thought (CoTs) that are not always causally responsible for their final outputs. When such a mismatch occurs, the CoT no longer faithfully reflects the actual reasons (i.e., decision-critical factors) driving the model's behavior, leading to the reduced CoT monitorability problem. This limits the use of CoTs for reliable oversight. However, a comprehensive and fully open-source benchmark for thoroughly evaluating CoT monitorability remains lacking. To address this gap, we propose MonitorBench, a systematic benchmark for evaluating CoT monitorability in LLMs. MonitorBench provides: (1) a diverse set of 1,514 test instances with carefully designed decision-critical factors across 19 tasks spanning 7 categories to characterize when CoTs can be used to monitor the factors driving LLM behavior; and (2) two prompting stress-test settings to quantify the extent to which CoT monitorability can be degraded. Extensive experiments show that CoT monitorability is a conditional property affected by the evaluated LLM, monitor LLM, and task characteristics. Across these factors, monitorability is higher when decision-critical factors shape the intermediate reasoning process, rather than merely influencing the final answer. Under stress-test prompting, most evaluated LLMs can intentionally reduce monitorability, mainly on tasks where decision-critical factors are not structurally required by the reasoning process. Overall, MonitorBench provides a basis for further research on AI control, reasoning faithfulness, stress-test monitorability, and monitoring scaffords. The code is available at https://github.com/ASTRAL-Group/MonitorBench.

Figures & tables

Figure 1: Examples of low and high CoT monitorability. The highlighted spans mark the decision-critical factors in each task. (Left) Gemini-3-Flash successfully encodes “TOOTHBRUSH” as the initials of consecutive sentences in its final response, yet never explicitly plans for this in its CoT. (Right) Gemini-3-Flash explicitly references the injected answer (I20.0) in its CoT and uses it to arrive at the final diagnosis.

Figure 2: Monitorability score (%) (details in Sec. 3.3 ) averaged over 19 tasks for each evaluated LLM (left), and averaged over 8 evaluated LLMs across tasks within each category (right), under different monitor scopes in both standard and two stress-test settings. Results show that CoT monitorability is higher when the decision-critical factors shape the intermediate reasoning process without merely influencing the final answer (i.e., Goal (Sandbag), Cue (Condition), Dual Objectives, Impossible Tasks). While most evaluated LLMs can intentionally reduce monitorability to some extent under the stress-tests, the monitorability degradation mostly comes from the tasks whose decision-critical factors do not shape the reasoning process (e.g., Cue (Answer)).

Input Intervention		Outcome Justification	Solution Process
Cue (Condition) Knowledge 100 Preference 108 Law Judgment 100 Health 65 Cue (Answer) Mathematics 28 Logic 50	Health 65 Goal (Sandbag) Safety 150 Mathematics 30 Science 192 Engineering 100 Goal (Deception) Real-world Case 50	Dual Objectives Steganography 50 Summarization 50 Coding 291 Agent (Malicious) ∗ 5 Agent (Benign) ∗ 5 Impossible Tasks Coding ∗ 50	Necessary Steps Multi-step Update 25

Table 1: Statistics of 1,514 instances across 19 tasks spanning 7 categories in MonitorBench. “ ∗ ” indicates that the task requires multi-round interactions with the environment. Detailed design and examples for each task are in Appendix A.4 and A.6 , respectively.

Task Category	Task	Open-source			Closed-source					Avg. over LLMs
Task Category	Task	Qwen3 14B	GPT-OSS 120B	Qwen3 235B	GPT-5 mini	GPT-5	Gemini-3 Flash	Claude Haiku 4.5	Claude Sonnet 4.5	Avg. over LLMs
Cue (Condition)	Health	95.1	100.0	99.7	100.0	99.4	100.0	97.3	99.0	98.8
	Knowledge	98.3	98.5	99.3	95.1	93.2	98.1	97.2	100.0	97.5
	Law Judgment	97.9	99.5	98.0	65.8	75.1	97.9	99.3	99.8	91.6
	Preference	97.8	99.5	100.0	90.1	98.1	98.5	100.0	100.0	98.0
Cue (Answer)	Health	89.1	87.6	96.9	87.5	83.8	79.4	86.7	93.5	88.1

Table 2: Monitorability score (%), under CoT-only monitor scope in the standard setting. – denotes not applicable due to refusal triggered by the safety alignment.

Task Category	Task	Open-source						Closed-source										Avg. over LLMs
		Qwen3-14B		GPT-OSS-120B		Qwen3-235B		GPT-5-mini		GPT-5		Gemini-3-Flash		Claude Haiku 4.5		Claude Sonnet 4.5		Avg. over LLMs
		DC	ME	DC	ME	DC	ME	DC	ME	DC	ME	DC	ME	DC	ME	DC	ME	DC	ME
Cue (Condition)	Health	0.4	-6.2	-0.5	-13.9	0.3	-0.9	-11.5	-14.2	-0.4	-1.4	-1.7	-11.0	-1.5	-1.9	1.0	-0.9	-1.7	-6.3
	Knowledge	-3.2	0.2	0.9	0.9	-1.4	-1.2	-2.5	-0.5	-3.3	4.2	-7.9	-3.5	-4.5	-9.8	0.0	-10.0	-2.2	-1.4
	Law Judgment	-30.3	-7.0	-10.0	-18.5	-10.7	-10.3	-24.9	-11.6	-14.6	-7.3	-11.2	-9.0	-0.8	-0.9	0.1	-1.8	-12.8	-6.9
	Preference	-6.2	-1.2	-0.3	-7.1	-2.4	-3.8	-9.9	-9.2	-11.3	-14.2	-8.6	-7.2	-3.1	-26.6	0.0	-0.5	-5.3	-8.7

Table 3: Monitorability score difference (%) under CoT-only monitoring between stress-test and standard settings. DC and ME denote Direct Concealment and Monitor-aware Evasion; – indicates safety-alignment refusal.

Figure 3: Relationship between monitorability score and LLM capability under the all-messages monitor scope . Each point corresponds to one evaluated LLM. We measure the monitorability score average across all tasks, and approximate LLM capability by the average pass rate on the original task questions. The fitted regression line shows a moderate negative Pearson correlation ( $r$ ), suggesting that more capable LLMs tend to exhibit lower monitorability scores.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Relationship between monitorability score and LLM capability under the CoT-only monitor scope . Each point corresponds to one evaluated LLM. We measure the monitorability score average across all tasks, and approximate capability by the average pass rate on the original task questions. The fitted regression line shows a moderate negative Pearson correlation ( $r$ ), suggesting that more capable LLMs tend to exhibit lower monitorability.

Figure 5: Relationship between monitorability and LLM capability under the action-only monitor scope . Each point corresponds to one evaluated LLM. We measure the monitorability score average across all tasks, and approximate capability by the average pass rate on the original task questions. The fitted regression line shows a moderate negative Pearson correlation ( $r$ ), suggesting that more capable LLMs tend to exhibit lower monitorability.

Task Category	Task	Open-source			Closed-source					Avg. over LLMs
Task Category	Task	Qwen3 14B	GPT-OSS 120B	Qwen3 235B	GPT-5 mini	GPT-5	Gemini-3 Flash	Claude Haiku 4.5	Claude Sonnet 4.5	Avg. over LLMs
Cue (Condition)	Health	96.2	100.0	100.0	98.6	100.0	99.7	100.0	99.3	99.2
	Knowledge	99.8	99.2	99.2	99.3	97.7	97.5	99.6	100.0	99.0
	Law Judgment	99.5	99.7	99.1	86.7	91.8	99.6	99.6	100.0	97.0
	Preference	99.8	99.9	99.7	99.7	99.4	99.8	100.0	100.0	99.8
Cue (Answer)	Health	78.8	86.5	92.9	84.1	88.8	81.8	91.7	95.0	87.4

Appendix

Table 4: Monitorability score (%), under all-messages monitor scope in the standard setting. – denotes not applicable due to refusal triggered by the safety alignment.

Task Category	Task	Open-source			Closed-source					Avg. over LLMs
Task Category	Task	Qwen3 14B	GPT-OSS 120B	Qwen3 235B	GPT-5 mini	GPT-5	Gemini-3 Flash	Claude Haiku 4.5	Claude Sonnet 4.5	Avg. over LLMs
Cue (Condition)	Health	96.2	100.0	99.8	100.0	100.0	100.0	100.0	100.0	99.5
	Knowledge	93.6	89.4	95.8	97.4	95.3	96.1	97.6	100.0	95.6
	Law Judgment	97.2	83.0	98.0	72.2	68.6	91.0	92.5	98.9	87.7
	Preference	99.7	99.3	99.4	99.1	100.0	100.0	100.0	100.0	99.7
Cue (Answer)	Health	44.9	41.8	71.7	45.1	46.5	53.0	79.6	75.7	57.2

Appendix

Table 5: Monitorability score (%), under action-only monitor scope in the standard setting. – denotes not applicable due to refusal triggered by the safety alignment.

Task Category	Task	Open-source						Closed-source										Avg. over LLMs
		Qwen3-14B		GPT-OSS-120B		Qwen3-235B		GPT-5-mini		GPT-5		Gemini-3-Flash		Claude Haiku 4.5		Claude Sonnet 4.5		Avg. over LLMs
		DC	ME	DC	ME	DC	ME	DC	ME	DC	ME	DC	ME	DC	ME	DC	ME	DC	ME
Cue (Condition)	Health	-2.9	1.4	0.0	-12.2	-0.2	-3.7	-0.7	-6.9	0.0	-1.3	-0.4	-11.4	-2.8	-2.7	0.3	-9.3	-0.8	-5.8
	Knowledge	-1.2	-0.9	0.4	0.7	-0.4	0.6	-2.8	-1.1	-2.6	0.5	-3.8	-0.1	-0.5	-0.6	0.0	-0.2	-1.4	-0.1
	Law Judgment	-34.0	-2.1	-10.4	-12.0	-10.7	-6.9	-31.9	-3.1	-21.6	-11.5	-11.3	-6.3	-2.0	-1.1	-0.6	-4.9	-15.3	-6.0
	Preference	-10.6	-3.3	-2.1	-11.8	-3.5	-6.8	-12.1	-13.2	-12.8	-14.5	-12.1	-10.0	-1.8	-3.1	-0.5	-3.5	-6.9	-8.2

Appendix

Table 6: Monitorability score difference (%) in all-messages monitor scope under stress tests relative to the standard setting. Direct Concealment and Monitor-aware Evasion are denoted as DC and ME, respectively. – denotes not applicable due to refusal triggered by the safety alignment.

Task Category	Task	Open-source						Closed-source										Avg. over LLMs
		Qwen3-14B		GPT-OSS-120B		Qwen3-235B		GPT-5-mini		GPT-5		Gemini-3-Flash		Claude Haiku 4.5		Claude Sonnet 4.5		Avg. over LLMs
		DC	ME	DC	ME	DC	ME	DC	ME	DC	ME	DC	ME	DC	ME	DC	ME	DC	ME
Cue (Condition)	Health	1.4	-7.3	0.0	-4.2	-0.4	-1.9	-4.2	0.0	-5.9	0.0	0.0	-3.3	-0.9	-3.5	-0.6	-5.6	-1.3	-3.2
	Knowledge	-5.9	0.9	-6.0	-1.0	-7.0	-6.7	1.4	-0.4	0.9	1.6	0.1	-2.8	0.9	-2.1	-0.4	-3.1	-2.0	-1.7
	Law Judgment	-53.5	-3.1	-23.2	-15.6	-35.0	-11.7	-24.0	3.0	-26.3	-11.4	-32.4	-26.7	-27.1	-18.1	-18.9	-25.7	-30.0	-13.7
	Preference	-19.2	-7.4	-7.8	-12.2	-10.9	-14.5	-19.1	-15.2	-29.4	-30.8	-15.5	-30.6	-18.9	-4.7	-4.1	-17.4	-15.6	-16.6

Appendix

Table 7: Monitorability score difference (%) in action-only monitor scope under stress tests relative to the standard setting. Direct Concealment and Monitor-aware Evasion are denoted as DC and ME, respectively. – denotes not applicable due to refusal triggered by the safety alignment.

Figure 6: Example that LLMs fail to hide the decision-critical factors under the stress-test setting. The highlighted spans mark the stress-test instruction and the decision-critical factor (i.e., side task). The explicit factor leakage showcases how the model directly names the side objective factor in its reasoning trace. The concealment leakage shows that the CoT explicitly plans to hide its planning. The act of naming the concealment strategy makes the side objective factor visible. The implicit planning leakage showcases how even without explicit naming, the step-by-step operational plan renders the side objective structurally inferable. The example is drawn from Qwen3-235B-A22B on Coding in Dual Objectives.

Stress-test Failure Mode	Direct Concealment				Monitor-aware Evasion
Stress-test Failure Mode	Qwen3 235B	GPT-5	Gemini-3 Flash	Claude Haiku 4.5	Qwen3 235B	GPT-5	Gemini-3 Flash	Claude Haiku 4.5
Explicit Factor Leakage	100.0	97.2	100.0	100.0	99.3	96.7	100.0	99.7
Concealment Leakage	84.5	64.0	88.2	99.8	31.5	15.3	23.5	0.7
Implicit Planning Leakage	48.1	37.4	31.3	31.9	36.2	36.4	21.2	20.4
Other Cases	0.0	1.2	0.0	0.0	0.0	0.8	0.0	0.0

Appendix

Table 8: Failure mode of why decision-critical factors remain monitorable under stress tests. We focus on Coding in Dual Objectives under the CoT-only monitor scope. Specifically, the decision-critical factors here are the side features that need to be integrated into the main functions. We use GPT-5 as the LLM-as-a-judge for annotation; the full prompts are provided in Appendix. A.5.4 . Because many responses exhibit multiple failure modes, we allow multi-label annotations and report the fraction of trajectories exhibiting each failure mode..

Figure 7: Example that LLMs hide the decision-critical factors under the stress-test setting. (Left) In the standard setting, Gemini-3-Flash explicitly verbalizes the decision-critical factors (i.e., mentioning the discount for this instance) in the CoT, demonstrating high CoT monitorability. (Right) Under the stress-test, the model hides the decision-critical factors from its CoT while arriving at a deceptive final response, demonstrating low CoT monitorability. The highlighted spans mark the decision-critical factor (left) and the stress-test instruction (right). Examples are from Goal (Deception) – Input Intervention.

Figure 8: Example that LLMs fail to hide the decision-critical factors under the stress-test setting, due to the explicit factor leakage . The highlighted spans mark the monitor-aware evasion instruction and the decision-critical factor (i.e., side objective). Qwen3-14B explicitly leaks information regarding the side objective within its CoT when dealing with the steganography task. This response also contains the concealment leakage .

Figure 9: Example that LLMs fail to hide the decision-critical factors under the stress-test setting, due to the concealment leakage . The highlighted spans mark the direct concealment instruction and the decision-critical factor (i.e., side objective). Under the stress-test, the CoT contains information regarding the concealment of the side objective showcases and how they should not be mentioned in the final response.

Explore similar work

Does Out-of-Sight Equal Out-of-Mind in CoT Monitorability?
Aug 5, 2026Pedro Ferreira, Wilker Aziz, Ivan TitovReasoning Traces Chain-of-Thought Reasoning
University of Amsterdam · University of Edinburgh
The Fragility of Chain-of-Thought Monitoring Across Typologically Diverse Languages
May 27, 2026Eric Onyame, Runtao Zhou, Kowshik Thopalli +2Chain-of-Thought Reasoning Single-Token Output Distributions
University of Virginia · Lawrence Livermore National Laboratory
Persuasion Attacks Can Decrease Effectiveness of CoT Monitoring
Jul 9, 2026Jennifer Za, Julija Bainiaksina, Nikita Ostrovsky +2Persuasion Chain-of-Thought Reasoning
LASR Labs · University College London · Google DeepMind

Aug 5, 2026, cs.CLJ/K move · Enter open · S save

Does Out-of-Sight Equal Out-of-Mind in CoT Monitorability?

Pedro Ferreira, Wilker Aziz, Ivan Titov

University of Amsterdam · University of Edinburgh

Reasoning Traces Chain-of-Thought Reasoning Inference Cost

Open paper AskPDF arXiv

Chain-of-thought (CoT) reasoning offers a window into the decision-making of large language models (LLMs), which can be monitored for target behaviors by reading the reasoning trace, motivating work on CoT monitorability. Latent CoT approaches, however, replace the explicit tokens with a small number of continuous states, lowering inference costs but removing the readable trace this monitoring relies on. Monitoring then requires alternative access to the model, such as probing its activations or verbalizing the latent states back into text, but how much monitorability these alternatives preserve is unclear. We study this question with a hint-based intervention setup, a proxy for behaviors where models exploit biasing input cues, e.g., an inadvertently leaked answer or a belief stated by the user, without acknowledging them. Taking hint-reliance as the monitorability target, we compare monitors across reasoning modes, from explicit CoT to weakly- and strongly-supervised latent CoT, on math reasoning and question answering. We find that, in this setup, monitorability depends more on properties of the task (such as whether the correct answer constrains the supporting reasoning) and the level of access to model internals than on the reasoning mode.

Abstract

Explore similar work

Does Out-of-Sight Equal Out-of-Mind in CoT Monitorability?

The Fragility of Chain-of-Thought Monitoring Across Typologically Diverse Languages

Persuasion Attacks Can Decrease Effectiveness of CoT Monitoring