Superficial Reflection or Genuine Thought? A Fine-Grained Cognitive Analysis of Large Reasoning Models

Yuxiang Chen; Zuohan Wu; Ziwei Wang; Xiangning Yu; Xujia Li; Linyi Yang; Mengyue Yang; Jun Wang

cs.AINov 30, 2025

Superficial Reflection or Genuine Thought? A Fine-Grained Cognitive Analysis of Large Reasoning Models

Authors: Yuxiang Chen, Zuohan Wu, Ziwei Wang, Xiangning Yu, Xujia Li, Linyi Yang, Mengyue Yang, Jun Wang, +1 more

Organizations: University College London · AI Lab, the Yangtze River Delta, China · The Hong Kong University of Science and Technology (Guangzhou) · Tianjin University · The Hong Kong University of Science and Technology · Southern University of Science and Technology · University of Bristol

Abstract

Motivated by the observed human-like behaviours in Large Reasoning Models (LRMs), this paper introduces a comprehensive taxonomy to characterise atomic reasoning steps and analyse the reasoning behaviours of LRMs. Grounded in human cognitive processes, we propose a taxonomy comprising five groups and seventeen categories. Through this taxonomy, we conduct an in-depth analysis of contemporary LRMs and distil four actionable takeaways for model optimisation. Most notably, we reveal that prevailing post-answer ``doublechecks'' are largely superficial and rarely yield substantive revisions. A targeted intervention further shows that explicitly eliciting richer reflection processes can substantially improve failed self-correction. To support this largescale study, we propose CAPO, an automated annotation method used to construct a dataset of 277,534 reasoning steps with strong agreement with human expert annotations. We further validate the main behavioural patterns on a newer reasoning model and a coding domain, demonstrating the broader applicability of the proposed taxonomy. All source code and data are available at https://github.com/hehepig4/psyche.

Figures & tables

Figure 1: A brief illustration of the proposed taxonomy from human cognitive perspectives.

Figure 2: Annotation consistency of CAPO and the RAG baseline. The boxes show the consistency distribution of candidate prompts on the training population, while the red line denotes the test consistency of the best validation-selected prompt.

Process	Human interval	Machine MD	Within interval
A.IO	$[-0.067,\,+0.094]$	$+0.011$	✓
S.HG	$[-0.048,\,+0.018]$	$-0.041$	✓
S.AR	$[-0.061,\,+0.008]$	$-0.030$	✓
R.SME	$[-0.073,\,+0.092]$	$-0.018$	✓

Table 1: Comparison of the principal behavioural effects estimated from human and CAPO annotations. MD denotes the mean difference between correct and incorrect CoTs. Full category-level results are provided in the appendix.

Name	Correct	Incorrect	Steps
MATH	963 (96.3%)	37 (3.7%)	90,006
AIME	810 (87.0%)	121 (13.0%)	177,687
ComS	1000		7,710
HMMT	9 (64.3%)	5 (35.7%)	4,375
AIME ⋆	6 (37.5%)	10 (62.5%)	5,466
ComS ⋆	30		254

Table 2: Number of annotated CoTs and reasoning steps. The top three rows use CAPO annotations, whereas the bottom three use human annotations. The core mathematical analysis excludes the two ComS rows.

Figure 3: Proportional differences between correct and incorrect CoTs. Red dots indicate statistically significant differences. The purple line denotes the overall proportion of each mental process.

Name	Pos. co.	Pos. inco.	P-value
S.HG	0.35	0.47 (+0.12)	$4.2\times 10^{-7}$
S.AR	0.40	0.48 (+0.08)	$6.3\times 10^{-3}$

Table 3: Average relative positions of Hypothesis Generation and Analogy Recall. Both occur significantly later in incorrect CoTs (

p<0.01

Figure 4: Post-answer check statistics. An A

\rightarrow

B transition denotes correctness before (A) and after (B) post-answer checking; for example, I

\rightarrow

C denotes an incorrect answer corrected to a correct one.

CoTs	Max	Min	Average
Before Intervention	0.60	0.21	0.41
After Intervention	1.00	0.67	0.88

Table 4: PNS (

\uparrow

) statistics before and after intervention. Higher PNS indicates that the retained steps have greater estimated necessity for the successful reasoning trajectory.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: The CAPO framework, where each prompt comprises separated constant, variable, and mutable areas.

Correctness	CoTs	Steps
Correct	1,803	740,399
Incorrect	118	87,965

Appendix

Table 5: Statistics of the MiniMax-M2 reasoning trajectories.

Correctness	CoTs	Steps
Correct	644	225,761
Incorrect	322	195,396

Appendix

Table 6: Statistics of the CodeForces reasoning trajectories.

Name	Pos. co.	Pos. inco.	P-value
S.HG	0.239	0.346	$<10^{-6}$
S.AR	0.298	0.410	$<10^{-6}$

Appendix

Table 7: Relative positions of S.HG and S.AR on CodeForces.

Category	Correct	Incorrect	Diff.	Sig.
Analysis	0.2480	0.2117	$+0.0363$	Yes
Inference	0.6291	0.6467	$-0.0176$	No
Judgment	0.1372	0.0799	$+0.0573$	Yes
Suggestion	0.1474	0.1507	$-0.0033$	No
Reflection	0.2330	0.2214	$+0.0116$	No

Appendix

Table 8: Correct–incorrect comparison after aggregating the seventeen fine-grained processes into five parent categories.

Explore similar work

Thinking Past the Answer: Evaluating Harmful Overthinking in Large Reasoning Models
Jun 1, 2026Simone Caldarella, Davide Talon, Rahaf Aljundi +2Large Reasoning Models Reasoning Traces
University of Trento · 3Fondazione Bruno Kessler · 2Toyota Motor Europe
When Can Large Reasoning Models Save Thinking? Mechanistic Analysis of Behavioral Divergence in Reasoning
May 21, 2025Rongzhi Zhu, Yi Liu, Jiancheng Wang +6Large Reasoning Models Behavioral Divergence
State Key Laboratory for Novel Software Technology, Nanjing University, China · China Unicom Software Research Institute, China · National Institute of Healthcare Data Science, Nanjing University, China +1
ReasoningFlow: Discourse Structures for Understanding LLM Reasoning Traces
Date pendingJinu Lee, Shivam Agarwal, Amruta Parulekar +3Reasoning Traces LLM Reasoning Strategies
The Grainger College of Engineering University of Illinois Urbana-Champaign

Jun 1, 2026, cs.AIJ/K move · Enter open · S save

Thinking Past the Answer: Evaluating Harmful Overthinking in Large Reasoning Models

Simone Caldarella, Davide Talon, Rahaf Aljundi, Elisa Ricci+1

University of Trento · 3Fondazione Bruno Kessler · 2Toyota Motor Europe

Large Reasoning Models Reasoning Traces

Open paper AskPDF arXiv

Large Reasoning Models (LRMs) improve performance by generating explicit intermediate reasoning traces through increased test-time compute, yet the assumption that longer reasoning is consistently beneficial remains under-examined. While recent evidence shows that additional reasoning can lead models to overthink, we ask: "Once a model has reached the correct answer, does further reasoning refine the solution, or deviate from it?" To study the dynamics after correctness, we introduce a prefix-level trajectory evaluation protocol grounded in reasoning sufficiency, defining the minimum reasoning budget required for a model to first generate the correct answer. This allows us to disentangle verbose overthinking, where additional reasoning is redundant but harmless, from harmful overthinking, where continued reasoning destabilizes an already-correct trajectory. Starting from multimodal benchmarks, we find that many instances considered reasoning-intensive require surprisingly little reasoning. Moreover, stopping at the first correct prefix improves accuracy over standard reasoning up to 21%, revealing that current models are limited not only by their ability to reason, but also by their inability to stop at the right time. Furthermore, while common efficiency strategies like early stopping substantially reduce verbose overthinking (up to 50%), they fail to mitigate harmful overthinking. Failure analysis reveals that correctness deviations are mainly driven by logical drift and visual reinterpretation. Finally, we show that our findings generalize to language-only reasoning benchmarks, highlighting harmful overthinking as a broader reliability risk. Code available at https://simonecaldarella.github.io/thinking-past-the-answer.

Abstract

Explore similar work

Thinking Past the Answer: Evaluating Harmful Overthinking in Large Reasoning Models

When Can Large Reasoning Models Save Thinking? Mechanistic Analysis of Behavioral Divergence in Reasoning

ReasoningFlow: Discourse Structures for Understanding LLM Reasoning Traces