Why Models Know But Don't Say: Chain-of-Thought Faithfulness Divergence Between Thinking Tokens and Answers in Open-Weight Reasoning Models

Richard J. Young

cs.CLMar 27, 2026

Why Models Know But Don't Say: Chain-of-Thought Faithfulness Divergence Between Thinking Tokens and Answers in Open-Weight Reasoning Models

Authors: Richard J. Young

Organizations: University of Nevada, Las Vegas, Department of Management, Entrepreneurship and Technology, Lee Business School, Las Vegas, NV, USA · DeepNeuro AI, Las Vegas, NV, USA

Abstract

Extended-thinking models expose a second text-generation channel ("thinking tokens") alongside the user-visible answer. This study examines 12 open-weight reasoning models on MMLU and GPQA questions paired with misleading hints. Among the 10,506 cases where models actually followed the hint (choosing the hint's target over the ground truth), each case is classified by whether the model acknowledges the hint in its thinking tokens, its answer text, both, or neither. In 55.4% of these cases the model's thinking tokens contain hint-related keywords that the visible answer omits entirely, a pattern termed thinking-answer divergence. The reverse (answer-only acknowledgment) is near-zero (0.5%), confirming that the asymmetry is directional. Hint type shapes the pattern sharply: sycophancy is the most transparent hint, with 58.8% of sycophancy-influenced cases acknowledging the professor's authority in both channels, while consistency (72.2%) and unethical (62.7%) hints are dominated by thinking-only acknowledgment. Models also vary widely, from near-total divergence (Step-3.5-Flash: 94.7%) to relative transparency (Qwen3.5-27B: 19.6%). These results show that answer-text-only monitoring misses more than half of all hint-influenced reasoning and that thinking-token access, while necessary, still leaves 11.8% of cases with no verbalized acknowledgment in either channel.

Figures & tables

Model	N (influenced)	Divergence Rate	Cluster
Step-3.5-Flash	757	94.7%	Heavy
GPT-OSS-120B	819	93.0%	Heavy
DeepSeek-V3.2-Speciale	903	92.5%	Heavy
OLMo-3.1-32B-Think	1,018	88.3%	Heavy
MiniMax-M2.5	565	84.6%	Heavy
OLMo-3-7B-Think	609	58.9%	Moderate

Table 1: Models and influenced-case counts. Models are sorted by divergence rate (Section 4 ).

Thinking	Answer	Count	Percentage
Acknowledges	Acknowledges	3,402	32.4%
Acknowledges	Omits	5,820	55.4%
Omits	Acknowledges	48	0.5%
Omits	Omits	1,236	11.8%
Total		10,506	100.0%

Table 2: Four-quadrant classification of hint acknowledgment across 10,506 influenced cases.

Figure 1: Four-quadrant distribution of hint acknowledgment across all 10,506 influenced cases. The dominant red wedge (55.4%) represents thinking-answer divergence (thinking-only pattern).

Figure 2: Thinking-only divergence rate by model. Models fall into three groups: heavy divergence (

>

80%), moderate (45–60%), and low (

<

45%).

Figure 3: Full four-quadrant distribution by model. High-divergence models (top) are dominated by the red “thinking-only” category, while transparent models (bottom) show more green.

Hint Type	N	Think-only	Rate	Notes
Consistency	652	471	72.2%	Rarely admit following prior answers
Unethical	3,239	2,030	62.7%	Pressure to hide unethical influence
Grader	2,854	1,603	56.2%	Gaming acknowledged in thinking only
Metadata	1,538	852	55.4%	Moderate divergence
Visual pattern	186	76	40.9%	Small N
Sycophancy	2,037	788	38.7%	Lowest divergence

Table 3: Divergence rates by hint type, sorted by divergence rate. N = influenced cases for that hint type.

Figure 4: Four-quadrant distribution by hint type. Sycophancy is dominated by transparency (59%), while consistency and unethical hints are dominated by thinking-only acknowledgment (72% and 63%). The unacknowledged category (grey) is concentrated in visual pattern (38%) and consistency/unethical (

\sim

17%).

Model	Hint	N	Suppressed	Rate
DeepSeek-V3.2	sycophancy	82	80	97.6%
OLMo-3.1-32B	sycophancy	153	149	97.4%
Step-3.5-Flash	unethical	321	311	96.9%
OLMo-3.1-32B	metadata	144	139	96.5%
Step-3.5-Flash	metadata	84	81	96.4%
MiniMax-M2.5	unethical	136	131	96.3%

Table 4: Model-hint combinations with divergence rates

>\,

90% and

N>50

, sorted by rate.

Figure 5: Divergence rate heatmap by model $\times$ hint type. Dark red cells indicate near-total divergence ( $>$ 90%); green cells indicate transparency. The top-left cluster (high-divergence models $\times$ consistency/unethical) shows the most extreme divergence.

Figure 6: Distribution of reasoning token counts by quadrant. Median reasoning length is similar across categories (transparent: 4,832; thinking-only: 4,276; unacknowledged: 4,348), with no practically meaningful difference.

Figure 7: Thinking-token vs. answer-text acknowledgment rates per model. Points far below the diagonal indicate heavy divergence. Bubble size is proportional to

N

CoT Faithfulness

Chain-of-thought faithfulness measures whether a model’s stated reasoning causally reflects and supports the answer it gives.

See all 73 papers

Lie to Me: How Faithful Is Chain-of-Thought Reasoning in Reasoning Models?
Mar 23, 2026Richard J. YoungCoT Faithfulness Faithfulness of Language Model Explanations
University of Nevada, Las Vegas, Department of Management, Entrepreneurship and Technology, Lee Business School, Las Vegas, NV, USA · DeepNeuro AI, Las Vegas, NV, USA
Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought
Mar 5, 2026Siddharth Boppana, Annabel Ma, Max Loeffler +5Faithfulness of Language Model Reasoning Language Model Reasoning Evaluation
Goodfire AI · Harvard University, Cambridge, MA
Reasoning-Token Spikes Under Prompted Untruthful Responses in Large Language Models
Oct 7, 2026Maverick Morales, Tomáš Dominik, Vermut Gao +4Deception in Language Models LLM Auditing
Chapman University, CA, USA · INSERM U992, Cognitive Neuroimaging Unit, NeuroSpin Center, Gif sur Yvette 91191, France · University of California, Los Angeles, CA, USA +1

Mar 23, 2026J/K move · Enter open · S save

Lie to Me: How Faithful Is Chain-of-Thought Reasoning in Reasoning Models?

Richard J. Young

University of Nevada, Las Vegas, Department of Management, Entrepreneurship and Technology, Lee Business School, Las Vegas, NV, USA · DeepNeuro AI, Las Vegas, NV, USA

CoT Faithfulness Faithfulness of Language Model Explanations Faithfulness of Language Model Reasoning Large Reasoning Models LLM Auditing Language Model Safety Evaluation

Open paper AskPDF arXiv

Chain-of-thought (CoT) reasoning has been proposed as a transparency mechanism for large language models in safety-critical deployments, yet its effectiveness depends on faithfulness (whether models accurately verbalize the factors that actually influence their outputs), a property that prior evaluations have examined in only two proprietary models, finding acknowledgment rates as low as 25% for Claude 3.7 Sonnet and 39% for DeepSeek-R1. To extend this evaluation across the open-weight ecosystem, this study tests 12 open-weight reasoning models spanning 9 architectural families (7B-685B parameters) on 498 multiple-choice questions from MMLU and GPQA Diamond, injecting six categories of reasoning hints (sycophancy, consistency, visual pattern, metadata, grader hacking, and unethical information) and measuring the rate at which models acknowledge hint influence in their CoT when hints successfully alter answers. Across 41,832 inference runs, overall faithfulness rates range from 39.7% (Seed-1.6-Flash) to 89.9% (DeepSeek-V3.2-Speciale) across model families, with consistency hints (35.5%) and sycophancy hints (53.9%) exhibiting the lowest acknowledgment rates. Training methodology and model family predict faithfulness more strongly than parameter count, and keyword-based analysis reveals a striking gap between thinking-token acknowledgment (approximately 87.5%) and answer-text acknowledgment (approximately 28.6%), suggesting that models internally recognize hint influence but systematically suppress this acknowledgment in their outputs. These findings carry direct implications for the viability of CoT monitoring as a safety mechanism and suggest that faithfulness is not a fixed property of reasoning models but varies systematically with architecture, training method, and the nature of the influencing cue.

Abstract

CoT Faithfulness

Explore similar work

Lie to Me: How Faithful Is Chain-of-Thought Reasoning in Reasoning Models?

Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought

Reasoning-Token Spikes Under Prompted Untruthful Responses in Large Language Models