Learning to Be Monitored: Causal Bounds and Intervention-Aware Post-Training Recipes for Chain-of-Thought Monitorability

Qiang Fu; Xinyue Li; Andre Wibisono; Arman Cohan

cs.LGOct 22, 2026

Learning to Be Monitored: Causal Bounds and Intervention-Aware Post-Training Recipes for Chain-of-Thought Monitorability

Authors: Qiang Fu, Xinyue Li, Andre Wibisono, Arman Cohan

Organizations: Yale University

Abstract

Chain-of-thought (CoT) monitoring aims to detect potentially harmful behaviors in large language models (LLMs) from their CoTs. Intervention-based evaluations measure this property by asking whether a monitor can detect the role of an intervention from the CoT when the targeted behavior occurs, typically using minimal-criterion sensitivity ( $\mathsf{TPR_{mc}}$ ) proposed by~\citep{guan2025monitoring}. However, a target behavior observed under intervention is not necessarily caused by it, and we show that this metric can overestimate how often a monitor detects target behaviors actually caused by the intervention. Specifically, we establish $\mathsf{TPR_{mc}}$ as an upper bound on this causal detection rate and derive a complementary lower bound, which provides a tighter evaluation framework for CoT monitorability. With this framework, we study whether LLMs can be trained to make their CoTs easier to monitor. We first construct a training dataset of paired control--intervention tasks spanning multiple intervention types. Using the dataset, we design and apply three intervention-specific training methods based on off-policy self-distillation, reinforcement learning, and on-policy self-distillation to train separate variants of each base model. Across various model-monitor configurations, all three methods generally improve CoT monitorability over the corresponding base models under our extended evaluation framework. Our code is available at https://github.com/xinyue-lxy/learning-to-be-monitored.

Figures & tables

Quantity	Definition	Interpretation
$p_{x}$	$\Pr(Y=1\mid X=x)$	Target-outcome rate in arm $x\in\{0,1\}$
$\mathsf{TPR}_{\mathsf{ob}}$	$\Pr(Z=1\mid X=1,Y=1)$	Recall among intervention-arm positives
$\mathsf{TNR}_{X=1}$	$\Pr(Z=0\mid X=1,Y=0)$	Specificity among intervention-arm negatives
$\mathsf{TNR}_{X=0}$	$\Pr(Z=0\mid X=0)$	Specificity in the control arm

Table 1: Operational quantities used in intervention-based monitorability evaluation.

Figure 1: Overview of the post-training methods for improving CoT monitorability.

Task model	Method	Qwen3-32B	Llama3.3-70B	GPT-OSS-120B
Qwen3.5-4B	Base	69.12	60.39	75.92
	SFT	75.59 ( +6.47 )	68.95 ( +8.56 )	83.04 ( +7.12 )
	RL	71.93 ( +2.81 )	65.35 ( +4.96 )	83.77 ( +7.85 )
	OPSD	73.30 ( +4.18 )	69.28 ( +8.89 )	82.47 ( +6.55 )
Qwen3.5-9B	Base	69.35	59.50	73.52
	SFT	73.01 ( +3.66 )	62.30 ( +2.80 )	75.48 ( +1.96 )

Table 2: Scores report

\mathsf{gmean_{int}}^{2}\times 100

, macro-averaged over 12 tasks and evaluation seeds 42–45. Parentheses indicate score-point changes relative to the corresponding base checkpoint.

Panel of Figure 2 — Figure 2: Intervention attribution on the same instance before and after SFT, RL, and OPSD. Selected Qwen3.5-9B responses to the same VPN intervention prompt. All four responses satisfy the privacy-promotion target ( $Y=1$ ), while the post-trained CoTs more explicitly connect the intervention to privacy promotion. Counts are positive judgments ( $Z=1$ ) out of four samples per monitor, based on the scenario, intervention, target, and full CoT. Ellipses mark omitted text.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Interpretation in the LLM evaluation setting
$\theta$	Fixed task-model checkpoint.
$X$	Control ( $X=0$ ) or intervention ( $X=1$ ) condition.
$E_{A}$	Task-generation and environmental randomness.
$E_{V}$	Outcome-verifier randomness.
$E_{M}$	Monitor randomness.
$F_{\theta}$	Task-model generation map.

Appendix

Table 4: Structural variables in the generation and evaluation process.

Intervention	Domain	Source family	Train	Dev.	Pairs
Condition cue	Health	MedQA USMLE	210	24	234
Condition cue	Knowledge	SQuAD	250	26	276
Condition cue	Judgment	ContractNLI	57	8	65
Condition cue	Preference	ReDial	54	7	61
Answer cue	Health	MedQA USMLE	254	30	284
Answer cue	Mathematics	GSM8K	451	49	500

Appendix

Table 5: Dataset composition by intervention type, domain, and source family. Counts denote paired instances.

Category	Source	Train	Val.	4B	9B	Mistral
Contextual evidence / Health	234	210	24	202	201	205
Contextual evidence / Knowledge	276	248	28	247	248	248
Contextual evidence / Judgment	65	55	10	53	51	43
Contextual evidence / Preference	61	51	10	48	45	50
Answer / Health	284	256	28	245	250	252
Answer / Math	500	450	50	438	444	441

Appendix

Table 6: Source instances and retained SFT targets by category. All three models use the same source training and validation partitions.

Setting	Qwen3.5-4B	Qwen3.5-9B	Mistral 24B
Training targets	2,874	2,994	3,284
Epoch budget	4	4	4
Peak learning rate	$10^{-5}$	$10^{-5}$	$10^{-5}$
Logical batch size	16	16	16
Microbatch size	2	2	1
Accumulation steps per full batch	8	8	16

Appendix

Table 7: SFT training budgets and selected checkpoints for the retained pools in Table 6 .

Budget	Qwen3.5-4B	Qwen3.5-9B	Mistral 24B
Minimum sequence length	234	231	202
Maximum sequence length	1,787	1,819	1,486
Input tokens per epoch	1,408,596	1,549,547	1,267,845
Supervised tokens per epoch	751,966	875,382	585,509
Four-epoch input exposures	5,634,384	6,198,188	5,071,380
Four-epoch supervised exposures	3,007,864	3,501,528	2,342,036

Appendix

Table 8: Token budgets for the documented SFT training pools. Complete inputs include the public prompt and native template structure; supervised tokens include reasoning, final answers, and explicit response-block delimiters.

Task model	Peak LR	Minibatch size	Response limit
Qwen3.5-4B	$10^{-6}$	8	1,024
Qwen3.5-9B	$3\times 10^{-7}$	8	1,024
Mistral Small 3.2-24B	$10^{-6}$	16	4,096

Appendix

Table 9: Model-specific RL training settings. Response limits are measured in generated tokens. All configurations cover the twelve training categories and use the shared settings described below.

(a) Qwen3-32B
Task	Upper-bound composite			Lower-bound composite
	Base	SFT	$\Delta$	Base	SFT	$\Delta$
Contextual evidence: Health	88.89	87.43	−1.46	88.70	86.99	−1.71
Contextual evidence: Knowledge	30.95	45.16	+14.21	30.95	43.47	+12.52
Contextual evidence: Judgment	NA	NA	NA	NA	NA	NA
Contextual evidence: Preference	99.83	99.75	−0.08	99.83	97.77	−2.06

Appendix

Table 10: Task-level SFT results for Qwen3.5-4B. Scores are averaged across evaluation seeds 42–45 and multiplied by 100.

\Delta

denotes SFT minus Base in score points, computed before rounding.

(a) Qwen3-32B
Task	Upper-bound composite			Lower-bound composite
	Base	SFT	$\Delta$	Base	SFT	$\Delta$
Contextual evidence: Health	85.57	87.78	+2.21	83.81	87.78	+3.97
Contextual evidence: Knowledge	40.14	18.99	−21.16	32.14	18.99	−13.16
Contextual evidence: Judgment	87.50	87.50	0.00	87.50	87.50	0.00
Contextual evidence: Preference	100.00	100.00	0.00	100.00	100.00	0.00

Appendix

Table 11: Task-level SFT results for Qwen3.5-9B. Scores are averaged across evaluation seeds 42–45 and multiplied by 100.

\Delta

denotes SFT minus Base in score points, computed before rounding.

(a) Qwen3-32B
Task	Upper-bound composite			Lower-bound composite
	Base	SFT	$\Delta$	Base	SFT	$\Delta$
Contextual evidence: Health	66.59	65.96	−0.63	65.11	62.09	−3.03
Contextual evidence: Knowledge	12.50	19.44	+6.94	0.00	10.00	+10.00
Contextual evidence: Judgment	87.50	95.83	+8.33	87.50	95.83	+8.33
Contextual evidence: Preference	100.00	100.00	0.00	100.00	100.00	0.00

Appendix

Table 12: Task-level SFT results for Mistral Small 3.2-24B. Scores are averaged across evaluation seeds 42–45 and multiplied by 100.

\Delta

denotes SFT minus Base in score points, computed before rounding.

(a) Qwen3-32B
Task	Upper-bound composite			Lower-bound composite
	Base	RL	$\Delta$	Base	RL	$\Delta$
Contextual evidence: Health	88.89	91.84	+2.95	88.70	91.84	+3.14
Contextual evidence: Knowledge	30.95	45.00	+14.05	30.95	45.00	+14.05
Contextual evidence: Judgment	NA	100.00	NA	NA	100.00	NA
Contextual evidence: Preference	99.83	99.55	−0.28	99.83	99.55	−0.28

Appendix

Table 13: Task-level RL results for Qwen3.5-4B. Scores are averaged over seeds 42–45 and multiplied by 100.

\Delta

denotes RL minus Base in score points, computed before rounding.

(a) Qwen3-32B
Task	Upper-bound composite			Lower-bound composite
	Base	RL	$\Delta$	Base	RL	$\Delta$
Contextual evidence: Health	85.57	82.71	−2.86	83.81	81.96	−3.85
Contextual evidence: Knowledge	40.14	33.98	−6.16	32.14	33.27	+1.13
Contextual evidence: Judgment	87.50	NA	NA	87.50	NA	NA
Contextual evidence: Preference	100.00	100.00	0.00	100.00	100.00	0.00

Appendix

Table 14: Task-level RL results for Qwen3.5-9B. Scores are averaged over seeds 42–45 and multiplied by 100.

\Delta

denotes RL minus Base in score points, computed before rounding. NA denotes a non-estimable or ineligible score.

(a) Qwen3-32B
Task	Upper-bound composite			Lower-bound composite
	Base	RL	$\Delta$	Base	RL	$\Delta$
Contextual evidence: Health	66.59	74.13	+7.54	65.11	74.13	+9.02
Contextual evidence: Knowledge	12.50	1.79	−10.71	0.00	1.79	+1.79
Contextual evidence: Judgment	87.50	89.58	+2.08	87.50	89.58	+2.08
Contextual evidence: Preference	100.00	100.00	0.00	100.00	100.00	0.00

Appendix

Table 15: Task-level RL results for Mistral Small 3.2-24B. Scores are averaged over seeds 42–45 and multiplied by 100.

\Delta

denotes RL minus Base in score points, computed before rounding. NA denotes a non-estimable or ineligible score.

(a) Qwen3-32B
Task	Upper-bound composite			Lower-bound composite
	Base	OPSD	$\Delta$	Base	OPSD	$\Delta$
Contextual evidence: Health	88.89	94.02	+5.13	88.70	93.05	+4.35
Contextual evidence: Knowledge	30.95	42.97	+12.02	30.95	42.97	+12.02
Contextual evidence: Judgment	NA	NA	NA	NA	NA	NA
Contextual evidence: Preference	99.83	98.68	−1.15	99.83	98.68	−1.15

Appendix

Table 16: Selected task-level OPSD results for Qwen3.5-4B. Scores are averaged over seeds 42–45 and multiplied by 100.

\Delta

denotes OPSD minus Base in score points, computed before rounding.

(a) Qwen3-32B
Task	Upper-bound composite			Lower-bound composite
	Base	OPSD	$\Delta$	Base	OPSD	$\Delta$
Contextual evidence: Health	85.57	88.74	+3.17	83.81	84.15	+0.34
Contextual evidence: Knowledge	40.14	34.79	−5.35	32.14	34.79	+2.65
Contextual evidence: Judgment	87.50	NA	NA	87.50	NA	NA
Contextual evidence: Preference	100.00	96.88	−3.12	100.00	96.88	−3.12

Appendix

Table 17: Selected task-level OPSD results for Qwen3.5-9B. Scores are averaged over seeds 42–45 and multiplied by 100.

\Delta

denotes OPSD minus Base in score points, computed before rounding.

(a) Qwen3-32B
Task	Upper-bound composite			Lower-bound composite
	Base	OPSD	$\Delta$	Base	OPSD	$\Delta$
Contextual evidence: Health	66.59	63.99	−2.60	65.11	62.64	−2.47
Contextual evidence: Knowledge	12.50	4.17	−8.33	0.00	4.17	+4.17
Contextual evidence: Judgment	87.50	97.92	+10.42	87.50	97.92	+10.42
Contextual evidence: Preference	100.00	100.00	0.00	100.00	100.00	0.00

Appendix

Table 18: Selected task-level OPSD results for Mistral Small 3.2-24B. Scores are averaged over seeds 42–45 and multiplied by 100.

\Delta

denotes OPSD minus Base in score points, computed before rounding.

CoT Monitoring

Chain-of-thought monitoring studies whether models’ reasoning traces expose decision-relevant behavior and support oversight of unsafe or deceptive actions.

See all 40 papers

MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models
Mar 30, 2026Han Wang, Yifan Sun, Brian Ko +8CoT Faithfulness LLM Evaluation
University of Illinois Urbana-Champaign · University of Washington · University of California San Diego
Monitorability Is Trainable: Post-Training for Chain-of-Thought Monitorability
Oct 21, 2026Usman Anwar, Tim Bakker, Christos Louizos +1CoT Monitoring LLM Post-Training
University of Cambridge · Qualcomm AI Research · Mila, University of Montreal
Analyzing and Improving Chain-of-Thought Monitorability Through Information Theory
Feb 20, 2026Usman Anwar, Tim Bakker, Dana Kianfar +2CoT Monitoring
University of Cambridge, work done while at Qualcomm AI Research. · Qualcomm AI Research.

Mar 30, 2026J/K move · Enter open · S save

MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models

Han Wang, Yifan Sun, Brian Ko, Mann Talati+7

University of Illinois Urbana-Champaign · University of Washington · University of California San Diego

CoT Faithfulness LLM Evaluation CoT Monitoring

Open paper AskPDF arXiv

Large language models (LLMs) can generate chains of thought (CoTs) that are not always causally responsible for their final outputs. When such a mismatch occurs, the CoT no longer faithfully reflects the actual reasons (i.e., decision-critical factors) driving the model's behavior, leading to the reduced CoT monitorability problem. This limits the use of CoTs for reliable oversight. However, a comprehensive and fully open-source benchmark for thoroughly evaluating CoT monitorability remains lacking. To address this gap, we propose MonitorBench, a systematic benchmark for evaluating CoT monitorability in LLMs. MonitorBench provides: (1) a diverse set of 1,514 test instances with carefully designed decision-critical factors across 19 tasks spanning 7 categories to characterize when CoTs can be used to monitor the factors driving LLM behavior; and (2) two prompting stress-test settings to quantify the extent to which CoT monitorability can be degraded. Extensive experiments show that CoT monitorability is a conditional property affected by the evaluated LLM, monitor LLM, and task characteristics. Across these factors, monitorability is higher when decision-critical factors shape the intermediate reasoning process, rather than merely influencing the final answer. Under stress-test prompting, most evaluated LLMs can intentionally reduce monitorability, mainly on tasks where decision-critical factors are not structurally required by the reasoning process. Overall, MonitorBench provides a basis for further research on AI control, reasoning faithfulness, stress-test monitorability, and monitoring scaffords. The code is available at https://github.com/ASTRAL-Group/MonitorBench.

Abstract

CoT Monitoring

Explore similar work

MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models

Monitorability Is Trainable: Post-Training for Chain-of-Thought Monitorability

Analyzing and Improving Chain-of-Thought Monitorability Through Information Theory