CoTs remain monitorable when being unfaithful requires computation: Replicating and Extending the Necessity Effect Across Eleven Models

Arav Dhoot; Yixiong Hao; Zephaniah Roe

cs.LGOct 17, 2026

CoTs remain monitorable when being unfaithful requires computation: Replicating and Extending the Necessity Effect Across Eleven Models

Authors: Arav Dhoot, Yixiong Hao, Zephaniah Roe

Organizations: Columbia University · Second Look Research

Abstract

While a LLM's chain of thought (CoT) is a promising approach to interpreting model outputs, prior work shows CoT's content does not always accurately reflect the reasoning process. Emmons et al. (2025) finds that models can follow a question hint without revealing so in its chain of thought, but when the hint is sufficiently complex the computation is forced into the CoT. This claim, however, has only been tested on Gemini models, and the datasets and code is closed source. We replicate their proxy task across eleven models from six families and find the necessity effect generalizes. Every model follows simple cues above a no-hint baseline, while concealed complex-cue following falls back near baseline for most models. We also extend the study from Emmons et al. (2025) to measure CoT-necessity and unfaithfulness across five cue families (four of them new) and three difficulty levels. The relationship between when CoT is necessary and unfaithfulness is task-specific and model-specific. For some hint types, models often verbalize their hint-following behavior even when the hint can be decoded in a single forward pass. These results support the case for CoT monitoring when unsafe behavior requires non-trivial computation, and that properties of CoT unfaithfulness are sensitive to model and setting.

Figures & tables

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Arithmetic (letter-index expression, mod 4)
L1	Two-term sum: $(X+D)\bmod 4$
L2	Three-term sum: $(X+I+H)\bmod 4$
L3	Multiply-add: $(D\cdot V+X)\bmod 4$
Bignum (multi-digit product $(a\times b)\bmod 4$ )
L1	$3\times 3$ digits: $(381\times 350)\bmod 4$
L2	$4\times 4$ digits: $(4657\times 3286)\bmod 4$

Appendix

Table 1: Cue families and difficulty levels (L1–L3). Every decode resolves to the index of the hinted (incorrect) option (

\text{A}=0,\text{B}=1,\dots

); the concrete parameters shown all resolve to index 2 (letter C). Generators are seeded and reproducible.

Deviation	Rationale
Competence filter added (60 correct items per model)	Removes a confound where cue-following is driven by inability to answer. This roughly halves the complex-hint take-rate on Gemini 2.5 Flash (App. C ).
Automated LLM judge replaces manual inspection	Enables scaling to eleven models. Judge noise is mitigated by avoiding self-judging and manual spot checks (App. H ).
Model panel expanded beyond Gemini	The purpose of the replication, which shows that Emmons et al. (2025) robustly generalizes across models.
Cue families and difficulty levels added	Extension in Section 6 , which is kept separate from the replication of the original arithmetic-L1 cue.

Appendix

Table 2: Running log of deviations from Emmons et al. (2025) . Arithmetic L1 in Figure 4 is the cell reproduced directly from the original; all other families and levels are extensions.

Stage	Follow rate	Concealment	Names cue
Pretrained	0.11	0.78	–
SFT	0.19	0.43	0.48
DPO	0.15	0.57	0.57
RLVR	0.10	0.51	0.76

Appendix

Table 3: Follow rate (with CoT, simple cue), baseline-corrected concealment among followers, and the fraction of concealed followers whose CoT still names the planted metadata ( Names cue ), by stage on MMLU-Pro. The pretrained row is confounded (see the patterns below); the RLVR concealment change from DPO is not significant.

Stage	With CoT	No CoT	CoT effect
Pretrained	$+0.04$	$+0.11$	$-0.08$
SFT	$+0.19$	$+0.30$	$-0.11$
DPO	$+0.11$	$+0.09$	$+0.02$
RLVR	$+0.07$	$+0.19$	$-0.13$

Appendix

Table 4: Net cue-following (baseline-corrected simple-cue follow rate) with versus without the CoT, and their difference ( CoT effect

=

with

-

without; negative means the CoT reduces cue-following), by stage. The CoT lowers net cue-following at every stage except DPO.

Faithfulness of Language Model Reasoning

See all 24 papers

Lie to Me: How Faithful Is Chain-of-Thought Reasoning in Reasoning Models?
Mar 23, 2026Richard J. YoungCoT Faithfulness Faithfulness of Language Model Explanations
University of Nevada, Las Vegas, Department of Management, Entrepreneurship and Technology, Lee Business School, Las Vegas, NV, USA · DeepNeuro AI, Las Vegas, NV, USA
Efficient Reasoning Training Does Not Always Harm CoT Faithfulness and Monitorability
Oct 2, 2026Samuel Lewis-Lim, Xingwei Tan, Mario Sanger +2Faithfulness of Language Model Reasoning Efficient Language Model Reasoning
School of Computer Science, University of Sheffield · AstraZeneca
Bypassing the Rationale: Causal Auditing of Implicit Reasoning in Language Models
Feb 3, 2026Anish Sathyanarayanan, Aditya Nagarsekar, Aarush RathoreCoT Faithfulness Faithfulness of Language Model Explanations
BITS Pilani, K. K. Birla Goa Campus

Mar 23, 2026J/K move · Enter open · S save

Lie to Me: How Faithful Is Chain-of-Thought Reasoning in Reasoning Models?

Richard J. Young

University of Nevada, Las Vegas, Department of Management, Entrepreneurship and Technology, Lee Business School, Las Vegas, NV, USA · DeepNeuro AI, Las Vegas, NV, USA

CoT Faithfulness Faithfulness of Language Model Explanations Faithfulness of Language Model Reasoning Large Reasoning Models LLM Auditing Language Model Safety Evaluation

Open paper AskPDF arXiv

Chain-of-thought (CoT) reasoning has been proposed as a transparency mechanism for large language models in safety-critical deployments, yet its effectiveness depends on faithfulness (whether models accurately verbalize the factors that actually influence their outputs), a property that prior evaluations have examined in only two proprietary models, finding acknowledgment rates as low as 25% for Claude 3.7 Sonnet and 39% for DeepSeek-R1. To extend this evaluation across the open-weight ecosystem, this study tests 12 open-weight reasoning models spanning 9 architectural families (7B-685B parameters) on 498 multiple-choice questions from MMLU and GPQA Diamond, injecting six categories of reasoning hints (sycophancy, consistency, visual pattern, metadata, grader hacking, and unethical information) and measuring the rate at which models acknowledge hint influence in their CoT when hints successfully alter answers. Across 41,832 inference runs, overall faithfulness rates range from 39.7% (Seed-1.6-Flash) to 89.9% (DeepSeek-V3.2-Speciale) across model families, with consistency hints (35.5%) and sycophancy hints (53.9%) exhibiting the lowest acknowledgment rates. Training methodology and model family predict faithfulness more strongly than parameter count, and keyword-based analysis reveals a striking gap between thinking-token acknowledgment (approximately 87.5%) and answer-text acknowledgment (approximately 28.6%), suggesting that models internally recognize hint influence but systematically suppress this acknowledgment in their outputs. These findings carry direct implications for the viability of CoT monitoring as a safety mechanism and suggest that faithfulness is not a fixed property of reasoning models but varies systematically with architecture, training method, and the nature of the influencing cue.

Abstract

Faithfulness of Language Model Reasoning

Explore similar work

Lie to Me: How Faithful Is Chain-of-Thought Reasoning in Reasoning Models?

Efficient Reasoning Training Does Not Always Harm CoT Faithfulness and Monitorability

Bypassing the Rationale: Causal Auditing of Implicit Reasoning in Language Models