Measuring and curing reasoning rigidity: from decorative chain-of-thought to genuine faithfulness

Abhinaba Basu; Pavan Chakraborty

cs.CLMar 24, 2026

Measuring and curing reasoning rigidity: from decorative chain-of-thought to genuine faithfulness

Authors: Abhinaba Basu, Pavan Chakraborty

Organizations: Indian Institute of Information Technology Allahabad (IIITA) · National Institute of Electronics and Information Technology (NIELIT)

Abstract

Language models increasingly show their work by writing step-by-step reasoning before answering. But are these steps genuinely used, or is the answer rigid - fixed before reasoning begins? We introduce the Step-Level Reasoning Capacity (SLRC) metric and prove it is a consistent causal estimator (Theorem 1). We propose LC-CoSR, a training method with Lyapunov stability guarantees that directly reduces rigidity. Evaluating 16 frontier models (o4-mini, GPT-5.4, Claude Opus, Grok-4, DeepSeek-R1, Gemini 2.5 Pro, and others) across six domains at N=133-500, we find reasoning falls into three modes. OpenAI's o4-mini shows 74-88% step necessity on five of six tasks (73.8-88.3%) - the highest SLRC in our study. The critical differentiator is RL-based reasoning training, not thinking tokens: Grok-4's reasoning mode shows lower faithfulness than its non-reasoning mode (1.4% vs 7.2% necessity). We discover a faithfulness paradox - high-SLRC models are more susceptible to sycophancy - and propose the Reasoning Integrity Score (RIS = SLRC x (1-Sycophancy)), which significantly predicts error detection (rho=0.66, p=0.026). LC-CoSR achieves 2.6x less negative reward than FARL and CSR baselines without external model dependencies.

Figures & tables

Panel of Figure 1 — Figure 1: Step-level evaluation. Top: Model produces a 3-step reasoning chain. Middle: Remove Step 1—answer unchanged, so Step 1 is not necessary. Bottom: Present Step 2 alone—answer recovered, so Step 2 is sufficient. Faithful models show high necessity and low sufficiency.

	Sentiment (SST-2)			Mathematics (GSM8K)
Model	Nec%	Suf%	SLRC	Nec%	Suf%	SLRC	Shuf%	$N$
GPT-5.4	0.1	98.2	.000	8.8	80.3	.017	0.1/2.5	376–500
Claude Opus 4.6-R	14.8	91.4	.013	4.8	88.7	.005	22/2.4	486–499
DeepSeek-V3.2	10.8	96.7	.004	3.6	93.4	.002	16/2.7	500
GPT-OSS-120B	0.3	98.9	.000	10.9	88.9	.012	5.2/9.4	425–496
Kimi-K2.5	16.5	79.5	.034	1.4	90.9	.001	19/1.5	457–500

Table 1: Step-level faithfulness of 16 frontier models on sentiment (SST-2) and mathematics (GSM8K). SLRC

{=}

Nec

{\times}

-

Suf) quantifies reasoning rigidity (0

{=}

decorative, 1

{=}

genuine; see Methods). General-purpose models show decorative reasoning (SLRC

{\leq}

0.03). RL-trained reasoning models achieve SLRC

{>}

0.2. Thinking tokens without RL (Grok-4 reasoning) remain decorative.

N{=}133

–

500

Panel of Figure 2 — Figure 2: Step necessity across models. RL-trained models (o4-mini, R1-32B, R1-70B) cross the 30% faithfulness threshold on both tasks. Grok-4 reasoning (Grok-4R) shows decorative CoT despite thinking tokens. o4-mini achieves the highest necessity (88%).

	CoT helps (gap $>20$ pp)	CoT doesn’t help
Steps matter (nec $>20\%$ )	Genuine (7 pairs) MiniMax SST-2 (+69/37%) Nemotron GSM8K (+43/89%) Kimi AG News (+52/63%)	Empty (0 pairs)
Steps don’t	Scaffolding (6 pairs) Kimi GSM8K (+94/1%) GPT-OSS MedQA (+76/1%) DeepSeek GSM8K (+55/4%)	Decorative (9 pairs) DeepSeek SST-2 ( $-$ 1/11%) Claude SST-2 ( $-$ 5/15%) GPT-OSS SST-2 (+6/0.3%)

Table 2: Three modes of CoT behaviour. Classified by whether CoT improves accuracy (gap

{>}20

pp) and whether individual steps matter (necessity

{>}20\%

). The empty fourth quadrant validates the framework: if steps matter, CoT must help. Representative examples with (gap/necessity).

Task	DeepSeek	Claude	GPT-OSS	MiniMax
Single-cue	8	3	18	100
Conflicting-cue	28	35	30	99
Multi-feature	7	11	35	66
Sequential math	18	7	43	98

Table 3: Causal experiment: step necessity (%) across 4 models

\times

4 synthetic tasks. Conflicting cues consistently produce the highest necessity for decorative models (28–35%). MiniMax shows

{\geq}

66% on every task—it genuinely reasons regardless of cue structure.

Model	Task	Real	Shuf	Gap	Signal
DeepSeek-R1	GSM8K	9%	3%	+7pp	Semantic
DeepSeek-R1	CSQA	7%	$-$ 2%	+8pp	Semantic
DeepSeek-R1	ARC	13%	1%	+12pp	Semantic
DeepSeek-R1	SST-2	26%	7%	+19pp	Semantic
DeepSeek-R1	MedQA	14%	10%	+3pp	Positional
Qwen3-8B	CSQA	35%	21%	+15pp	Semantic

Table 4: Shuffled-CoT mechanistic baseline (3 models

\times

5 tasks,

N{=}50

). “Gap” = attention drop on real CoT minus drop on shuffled CoT. DeepSeek-R1 shows the largest semantic gaps (7–19pp), confirming the reasoning model processes CoT content . Qwen3-0.6B shows purely positional attention (0–6pp).

Panel of Figure 3 — Figure 3: The faithfulness paradox and Reasoning Integrity Score. SLRC values are averaged across all available tasks for each model (2–6 tasks per model). (a) Models occupy a 2D faithfulness–vulnerability space. General-purpose models (blue circles) cluster in the bottom-left (decorative but robust). RL-trained models (green squares) are faithful but vulnerable to sycophancy—the faithfulness paradox . The ideal quadrant (bottom-right: faithful + robust) is sparsely populated. (b) RIS (SLRC $\times$ (1 $-$ Sycophancy)) significantly predicts error detection ability ( $\rho{=}0.66$ , $p{=}0.026$ ), resolving the paradox by penalising manipulable faithfulness. GPT-5.4 (lowest RIS) shows the worst error detection; Phi-4-reasoning (highest RIS) shows the best. o4-mini is excluded from panel (a) due to insufficient sycophancy data ( $N{=}5$ ).

Model	Total	Active	MoE?	Faithful?
GPT-OSS-120B	117B	5B	Yes	No
MiniMax-M2.5	230B	10B	Yes	SST-2
Qwen3.5-122B	122B	10B	Yes	No
Kimi-K2.5	1T	32B	Yes	AG News
DeepSeek-V3.2	685B	37B	Yes	No
Nemotron-Ultra	253B	253B	Dense	No

Table 5: Active parameters vs. faithfulness. Models with the same active parameter count show divergent faithfulness, ruling out scale as the determining factor.

Panel of Figure 4 — Figure 4: Training paradigm determines step necessity, not thinking tokens. o4-mini (RL-trained, green) achieves 74–88% necessity on 5 of 6 tasks. Grok-4 reasoning (thinking tokens without RL, red) shows near-zero necessity despite producing thinking tokens—indistinguishable from GPT-5.4 (no thinking tokens, blue). MedQA is the only task where all three converge.

Method	Signal	Reward	Time (hrs)	Guarantee
GRPO	Correctness only	$-0.343$	1.5	None
CSR ( Akter et al., 2025 )	KL regularisation	$-0.343$	6.9	Thm 1–3 †
FARL ( Wang et al., 2026 )	Correctness + NPO	$-0.331$	4.5	None
CoSR (ours)	Step necessity	$\mathbf{-0.141}$	4.8	None
LC-CoSR (ours)	Step nec. + Lyapunov	$\mathbf{-0.129}$	5.2	$V(\theta)\geq V_{\min}$

Table 6: Faithfulness intervention comparison on R1-Qwen-1.5B (GSM8K). Average reward measures task performance; higher is better. CoSR and LC-CoSR directly reward step-level causal dependency, achieving 2.4–2.7

\times

less negative reward than correctness-only baselines.

Nec threshold	Classification	All tasks
0.20	9/12 (75%)	12/18 (67%)
0.25	10/12 (83%)	13/18 (72%)
0.30	10/12 (83%)	13/18 (72%)
0.35	11/12 (92%)	14/18 (78%)

Table 11

Model	SST-2	GSM8K	AG News	MedQA	CSQA	ARC
Claude Opus	100%	99%	99%	97%	99%	98%
DeepSeek-V3.2	100%	100%	100%	100%	100%	100%
GPT-5.4	100%	75%	100%	99%	—	97%
GPT-OSS-120B	99%	85%	100%	52%	—	78%
Kimi-K2.5	100%	92%	71%	—	33%	47%
MiniMax-M2.5	99%	85%	87%	—	—	45%

Table 12

Model	Task	$N$	Nec%	Suf%	Acc%
Claude Opus	AG News	497	3.5	69.9	86.3
Claude Opus	MedQA	486	1.7	88.9	93.4
Claude Opus	CSQA	495	6.3	81.5	82.2
Claude Opus	ARC	489	2.1	93.9	94.3
DeepSeek-V3.2	AG News	500	2.4	55.7	81.2
DeepSeek-V3.2	MedQA	500	4.0	78.9	89.0

Table 13

Model	SST-2	GSM8K	AG News	MedQA
Claude Opus	8.2	2.8	7.1	5.8
GPT-5.4	6.9	3.8	5.2	7.3
DeepSeek-V3.2	12.1	7.0	12.0	17.1
GPT-OSS-120B	9.7	5.1	9.3	9.8
MiniMax-M2.5	8.2	11.8	—	—
Kimi-K2.5	10.5	5.5	10.1	—

Table 14

Language Model Reasoning Evaluation

See all 36 papers

Lie to Me: How Faithful Is Chain-of-Thought Reasoning in Reasoning Models?
Mar 23, 2026Richard J. YoungCoT Faithfulness Faithfulness of Language Model Explanations
University of Nevada, Las Vegas, Department of Management, Entrepreneurship and Technology, Lee Business School, Las Vegas, NV, USA · DeepNeuro AI, Las Vegas, NV, USA
LIFT: Likelihood-Based Intervention Faithfulness Training
Sep 10, 2025Anand Swaroop, Maheep Chaudhary, Kevin Zhu +2CoT Faithfulness Faithfulness of Language Model Explanations
Algoverse AI Research
Understanding and Mitigating Premature Confidence for Better LLM Reasoning
May 23, 2026Jingchu Gai, Guanning Zeng, Christina Baek +4Confidence Estimation in Language Models RL for Language Model Reasoning
1Carnegie Mellon University · 2Tsinghua University

Mar 23, 2026J/K move · Enter open · S save

Lie to Me: How Faithful Is Chain-of-Thought Reasoning in Reasoning Models?

Richard J. Young

University of Nevada, Las Vegas, Department of Management, Entrepreneurship and Technology, Lee Business School, Las Vegas, NV, USA · DeepNeuro AI, Las Vegas, NV, USA

CoT Faithfulness Faithfulness of Language Model Explanations Faithfulness of Language Model Reasoning Large Reasoning Models LLM Auditing Language Model Safety Evaluation

Open paper AskPDF arXiv

Chain-of-thought (CoT) reasoning has been proposed as a transparency mechanism for large language models in safety-critical deployments, yet its effectiveness depends on faithfulness (whether models accurately verbalize the factors that actually influence their outputs), a property that prior evaluations have examined in only two proprietary models, finding acknowledgment rates as low as 25% for Claude 3.7 Sonnet and 39% for DeepSeek-R1. To extend this evaluation across the open-weight ecosystem, this study tests 12 open-weight reasoning models spanning 9 architectural families (7B-685B parameters) on 498 multiple-choice questions from MMLU and GPQA Diamond, injecting six categories of reasoning hints (sycophancy, consistency, visual pattern, metadata, grader hacking, and unethical information) and measuring the rate at which models acknowledge hint influence in their CoT when hints successfully alter answers. Across 41,832 inference runs, overall faithfulness rates range from 39.7% (Seed-1.6-Flash) to 89.9% (DeepSeek-V3.2-Speciale) across model families, with consistency hints (35.5%) and sycophancy hints (53.9%) exhibiting the lowest acknowledgment rates. Training methodology and model family predict faithfulness more strongly than parameter count, and keyword-based analysis reveals a striking gap between thinking-token acknowledgment (approximately 87.5%) and answer-text acknowledgment (approximately 28.6%), suggesting that models internally recognize hint influence but systematically suppress this acknowledgment in their outputs. These findings carry direct implications for the viability of CoT monitoring as a safety mechanism and suggest that faithfulness is not a fixed property of reasoning models but varies systematically with architecture, training method, and the nature of the influencing cue.

Abstract

Language Model Reasoning Evaluation

Explore similar work

Lie to Me: How Faithful Is Chain-of-Thought Reasoning in Reasoning Models?

LIFT: Likelihood-Based Intervention Faithfulness Training

Understanding and Mitigating Premature Confidence for Better LLM Reasoning