Chain-of-Thought Oversight Requires Separate Evidence for Faithfulness and Monitorability

Sichao Li; Sai Ma; Xiyang Hu; Chudi Zhong; Tongliang Liu

cs.AINov 1, 2026

Chain-of-Thought Oversight Requires Separate Evidence for Faithfulness and Monitorability

Authors: Sichao Li, Sai Ma, Xiyang Hu, Chudi Zhong, Tongliang Liu

Organizations: The University of Sydney · MINT Lab, Johns Hopkins University · Australian National University · Arizona State University ·

Abstract

A common assumption in chain-of-thought (CoT) oversight is that a visible reasoning trace makes model behavior easier to monitor. We argue that this assumption can conflate distinct properties along a chain linking a model's internal computation, reasoning artifacts, outputs, and monitor judgments. We identify four properties along this chain: reasoning visibility (whether a trace is accessible), mechanistic faithfulness (whether a trace reflects internal computation), mechanistic localizability (whether a compact causal mechanism can be isolated), and behavioral monitorability (whether a monitor can detect the target behavior). Two complementary case studies across three open-weight reasoning models show why these distinctions matter. In the security vulnerability reasoning study, probes and activation ablations provide evidence of internal representation and causal influence, but a monitor with access to the reference answer does not outperform the majority-class baseline in a held-out test. In authority-hint factual reasoning study, a monitor correctly distinguishes changed from unchanged answers, but seven mechanistic analyses do not isolate a compact causal mechanism. These results show that evidence for one claim does not establish the others. We therefore propose an evidential-chain reporting standard. Each claim carries a licensing statement that specifies its target property, access tier, and artifact, and labels each required link as supported, unsupported, or assumed. These assessments determine whether the conclusion is licensed, conditionally licensed, or not licensed.

Figures & tables

Property	Core question	Evidence type	Typical failure mode
Reasoning visibility	Is reasoning exposed?	Trace availability	Relevant reasoning exists but is hidden or compressed
Mechanistic faithfulness	Does the trace reflect internal computation?	Representation probes; causal tests when available	Trace is unsupported, incomplete, or post-hoc
Behavioral monitorability	Can a monitor detect the target property?	Held-out monitor accuracy vs. baseline	Relevant information is present but too hard for the monitor
Mechanistic localizability	Can a compact mechanism be isolated?	Direction finding, site patching, circuit tests	Computation is distributed or method-insensitive

Table 1: Oversight properties distinguished in this paper. Each property asks a different question, relies on a different evidence type, and fails for a different reason.

Domain	Task	Access / property	Evidence type	Split
Security vulnerability reasoning	Reasoning-step internal representation	Tier I + Tier II/III; internal representation / mechanistic faithfulness	Linear probes on activations recorded while the model re-reads its own reasoning.	All 200 outputs; 50 correctly solved rows for confirmation
Security vulnerability reasoning	Held-out monitorability	Tier II + task/reference context; behavioral monitorability	Structured LLM jury that also sees the reference answer.	30 dev / 59 held-out expert-adjudicated cases
Authority-hint factual reasoning	Effect of hints on answers	Tier II + output behavior; behavioral effect	Clean and hinted MMLU/GPQA variants.	1,592 primary rows; strict 48-question replication subset
Authority-hint factual reasoning	Authority monitorability	Tier II; behavioral monitorability	Authority-cue monitor decisions.	Strict authority cases
Authority-hint factual reasoning	Mechanistic localization	Tier I; mechanistic localizability	Control, residual-direction, token $\times$ layer, and patching tests.	Strict mechanistic candidate set
Authority-hint factual reasoning	Verbalization and restoration	Tier I + Tier II; verbalization vs. mechanistic signal	Authority-verbalization labels correlated with restoration signal.	32 processed cases; $n=30$ after excluding ambiguous labels

Table 2: Setup and evidence for the two empirical studies. Access tiers follow Section 2 : Tier I denotes internal activations/mechanisms and Tier II denotes raw generated reasoning traces. Additional implementation and computation details are provided in Appendix A

Figure 1: Security vulnerability reasoning. Reasoning-step content is recoverable from activations, and intervention on a decoded direction affects model outputs. However, the tested jury shows limited exact-accuracy monitoring despite access to the task, trace, final answer, and reference answer. (a) The presence of three reasoning steps is recoverable from activations, but a text-only classifier on the same spans performs comparably. (b) Ablating the vulnerability-identification probe direction changes more verdicts (

n=59

) and removes the identification step more often (

n=27

) than random directions (dash marks their mean). (c) For all three models, the jury’s exact-accuracy point estimate lies below the majority baseline (always CORRECT ). (d) The jury’s agreement with human labels (Cohen’s

\kappa

) is below the agreement between two human annotators, measured on the 14B responses (shaded: 95% CI). Error bars are 95% bootstrap CIs. 14B, 32B, and 70B denote DeepSeek-14B, QwQ-32B, and DeepSeek-70B (59, 54, 58).

Figure 2: Authority-hint factual reasoning. The behavior is easy to monitor, but no tested patching site selectively controls it. (a) The effect of hints on answers is ordered by hint type, with authority producing the largest drop in accuracy. (b) On 22 patching-selected cases (11 changed, 11 controls), the monitor classifies all cases correctly. (c) Seven mechanistic localization tests.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Model	$n$	Accuracy	Baseline	$p$
Primary jury
DeepSeek-14B	59	0.661	0.525	0.120
QwQ-32B	54	0.741	0.648	0.212
DeepSeek-70B	58	0.655	0.621	0.397
Mixed-family jury
DeepSeek-14B	59	0.712	0.525	0.033

Appendix

Table 4: Binary security-monitor accuracy. The target is CORRECT versus not- CORRECT ;

p

is the one-sided paired bootstrap comparison with the majority baseline. Mixed-family comparisons are exploratory and uncorrected for multiple testing.

Figure 3: Token-by-layer activation-patching scan. The changed and control panels share a color scale; the final panel shows their difference. Layers are zero-indexed, and token positions count backward from the end of the post-reasoning answer-position input. The outlined site is selected in sample.

	Item	What to report
1 Context and claim
1.1	Context and aim	The deployment context and intended use.
1.2	Property and definition	The oversight property each aim requires, with its definition.
1.3	Scope	Models, task, and cases; how the cases were selected [ 26 , 32 ] .
2 Evidence
2.1	Access tier and artifact	Tier (I–IV) and the artifact examined.

Appendix

Table 5: Licensing statement template.

	Item	Entry
1 Context and claim
1.1	Context and aim	A university audits DeepSeek-14B’s responses to misleading authority claims. It needs (i) reasoning records that reflect how answers were computed and (ii) a monitor that detects influenced answers.
1.2	Property and definition	Mechanistic faithfulness for (i) and behavioral monitorability of deference for (ii) (Section 2 ).
1.3	Scope	DeepSeek-14B on MMLU and GPQA questions with an authority cue. The monitor was evaluated on a balanced set of 22 cases (11 changed, 11 controls), selected by patching criteria rather than monitor performance. Mechanistic analyses used different subsets, as described in Appendix B.2 .
2 Evidence
2.1	Access tier and artifact	Tier I activations and Tier II traces for analyses relevant to mechanistic faithfulness; Tier II raw traces and final answers, read with the question and answer key, for the monitor.

Appendix

Table 6: Licensing statement for the tutoring-assistant example, based on the authority study.

CoT Monitoring

Chain-of-thought monitoring studies whether models’ reasoning traces expose decision-relevant behavior and support oversight of unsafe or deceptive actions.

See all 44 papers

Faithfulness as Information Flow: Evaluating and Training Faithful Chain-of-Thought Reasoning
May 22, 2026Jinghan Jia, Joe Benton, Eric EasleyCoT Faithfulness LLM Interpretability
†Dept. CSE, Michigan State University · ⋆Anthropic Fellows Program · ‡Anthropic
Does Out-of-Sight Equal Out-of-Mind in CoT Monitorability?
Aug 5, 2026Pedro Ferreira, Wilker Aziz, Ivan TitovCoT Monitoring Latent CoT Reasoning
University of Amsterdam · University of Edinburgh
Lie to Me: How Faithful Is Chain-of-Thought Reasoning in Reasoning Models?
Mar 23, 2026Richard J. YoungCoT Faithfulness Faithfulness of Language Model Explanations
University of Nevada, Las Vegas, Department of Management, Entrepreneurship and Technology, Lee Business School, Las Vegas, NV, USA · DeepNeuro AI, Las Vegas, NV, USA

May 22, 2026J/K move · Enter open · S save

Faithfulness as Information Flow: Evaluating and Training Faithful Chain-of-Thought Reasoning

Jinghan Jia, Joe Benton, Eric Easley

†Dept. CSE, Michigan State University · ⋆Anthropic Fellows Program · ‡Anthropic

CoT Faithfulness LLM Interpretability Causal Interventions in Language Models Reasoning Trace Analysis

Open paper AskPDF arXiv

Chain-of-thought (CoT) reasoning is useful for monitoring language models only when the reasoning trace faithfully reflects the computation that produces the final answer. However, models can rely on prompt-to-answer shortcuts that bypass the CoT, making the visible reasoning trace misleading even when it appears plausible. We study CoT faithfulness through a structural information-flow perspective: faithful reasoning should route answer-relevant information through the mediated path from prompt to CoT to answer, rather than through a direct prompt-to-answer shortcut. This perspective yields a task-agnostic framework based on three complementary properties, sufficiency, completeness, and necessity, which we instantiate with entropy-based, masked-KL, and gradient-based diagnostics. We show that these metrics recover externally judged faithfulness differences in hinted reasoning, and identify a low-entropy failure mode of KL-based diagnostics where gradient-based measures remain more stable. Building on this analysis, we introduce update-time interventions for verifier-based on-policy RL, including attention masking, backward-only gradient masking, CoT gradients, and adversarial perturbations of prompt representations. Across hinted arithmetic, reward-hackable code repair, and DAPO-Math models trained without hints but evaluated under wrong-hint injection, our interventions shift behavioral and structural indicators toward stronger CoT mediation. In particular, they make shortcut and reward-hacking behavior more transparent in the CoT and improve task-agnostic faithfulness metrics, while in some settings also reducing wrong-hint susceptibility. Our results suggest that controlling information flow during training is a practical route toward more faithful and monitorable CoT reasoning. Code is available at https://github.com/safety-research/faithful-cot.

Abstract

CoT Monitoring

Explore similar work

Faithfulness as Information Flow: Evaluating and Training Faithful Chain-of-Thought Reasoning

Does Out-of-Sight Equal Out-of-Mind in CoT Monitorability?

Lie to Me: How Faithful Is Chain-of-Thought Reasoning in Reasoning Models?