Cognitive Profiling of LRMs' Reasoning Traces Using Bloom's Taxonomy

Maria-Eleni Zoumpoulidi; Georgios Paraskevopoulos; Alexandros Potamianos

cs.AIAug 24, 2026

Cognitive Profiling of LRMs' Reasoning Traces Using Bloom's Taxonomy

Authors: Maria-Eleni Zoumpoulidi, Georgios Paraskevopoulos, Alexandros Potamianos

Organizations: Institute for Language and Speech Processing, Athena Research Center, Greece · Speech and Language Processing Group, National Technical University of Athens, Greece

Abstract

Large Reasoning Models (LRMs) have revolutionized reasoning in LLMs, and the increasing public availability of reasoning traces creates valuable opportunities to study model behavior not only at the surface level but also at the granularity of individual reasoning steps. However, understanding the types of thinking employed during reasoning - which offers critical insights into models' reasoning patterns and enables actionable applications - remains underexplored. To address this gap, we introduce a framework for automatic annotation of reasoning steps through the lens of Bloom's Taxonomy, which classifies thinking into six cognitive levels, such as Remembering, Applying and Evaluating. Using this framework, we perform a large-scale analysis across models and datasets, revealing both similarities and differences in thinking patterns across models and tasks. Moreover, we demonstrate that thinking-type information derived from reasoning traces correlates with correctness, paving the way for improved reasoning. Our findings establish a fine-grained framework for analyzing thinking patterns in LRMs and provide actionable insights for enhancing reasoning quality.

Figures & tables

Figure 1: Example of our annotation framework. The reasoning trace is automatically segmented, and each segment is labeled according to its corresponding level in Bloom’s Taxonomy.

Figure 2: Bloom’s Taxonomy (revised by 2 ).

Agreement	Cohen’s Kappa
H1-H2	0.8928
LLM-H1	0.9173
LLM-H2	0.9132

Table 1: Cohen’s Kappa between the LLM annotator (Llama3.3-70B-Instruct) and human annotators (H1, H2) on a uniformly randomly sampled subset of 100 samples (1633 reasoning steps) drawn from all models and datasets.

Model	Remember	Understand	Apply	Analyze	Evaluate	Create
DeepSeek-R1	11.8	24.0	32.9	12.5	18.6	0.2
DeepSeek-R1-Distill-Llama-8B	13.5	18.7	44.2	14.1	9.3	0.3
R1-Distill-Qwen-7B	11.5	17.7	43.9	12.8	14.1	0.1
R1-Distill-Qwen-1.5B	10.9	18.8	35.6	13.5	21.0	0.1
Qwen3-30B-A3B-Thinking-2507	9.4	18.8	33.4	12.7	25.6	0.1
Qwen3-4B-Thinking-2507	9.0	20.8	32.5	14.3	23.1	0.2

Table 2: Percentage of reasoning steps assigned to each level of Bloom’s Taxonomy across different LRMs.

Figure 3: Temporal Dynamics of Bloom’s Levels in Reasoning Traces

Figure 4: Output-Reasoning steps distribution comparison

Figure 5: Comparison of temporal thinking dynamics of reasoning traces across GSM8K, GSM-hard, and MATH500, aggregated over all models.

Figure 6: Comparison of temporal thinking dynamics of reasoning traces across math datasets and the BBH tasks Formal Fallacies and Hyperbaton, aggregated over all models.

Positive Coefficients		Negative Coefficients
Feature	Coef.	Feature	Coef.
trans_appl $\rightarrow$ eval	+0.1769	total_tokens	-0.4373
trans_reme $\rightarrow$ unde	+0.1290	trans_eval $\rightarrow$ anal	-0.1271
trans_eval $\rightarrow$ reme	+0.1059	trans_anal $\rightarrow$ eval	-0.1236
trans_eval $\rightarrow$ eval	+0.0897	trans_eval $\rightarrow$ appl	-0.1155
trans_appl $\rightarrow$ unde	+0.0813	trans_eval $\rightarrow$ crea	-0.1132

Table 3: Top positive and negative logistic regression coefficients from the Bloom-based correctness model. Positive coefficients are associated with correct solutions, while negative coefficients are associated with incorrect solutions.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

CoT Prompt Template (Mathematical Datasets)
Solve the following math problem step by step. End your response with the final numerical answer in this format: “The answer is X.”
Problem: {question}

Appendix

Table 4: Prompt template used for Chain of Thought generation (Mathematical Datasets)

Annotation Prompt Template
You are a data annotation expert. Given:
- A question
- A reasoning trace Your task is to segment the reasoning trace into distinct steps, each aligned with exactly one cognitive level as defined by Bloom’s Taxonomy. ## Segmentation and Labeling Rules
1. Segment by cognitive function, not by sentence. A segment can span multiple sentences or be a single clause — boundaries are determined by shifts in cognitive operation, not punctuation.
2. Start a new segment whenever the solver completes a cognitive operation (e.g., stops recalling and starts applying, or stops applying and starts evaluating). This can include consecutive operations of the same type.
3. Transitional phrases ("Now…", "So…", "Let me…") belong to the segment of the operation they introduce, not the one before.

Appendix

Table 5: Prompt template used for Annotation (used for both mathematical datasets and BBH tasks). Few-shot examples are omitted due to space constraints, please see the corresponding tables.

Annotation Prompt Few-shot examples (Mathematical Datasets)
Example 1
Question:
Julie is reading a 120-page book. Yesterday, she was able to read 12 pages and today, she read twice as many pages as yesterday. If she wants to read half of the remaining pages tomorrow, how many pages should she read?
Reasoning trace:
Okay, let’s tackle this problem step by step. First, I need to figure out how many pages Julie has read so far and then determine how many are left. The book is 120 pages total. Yesterday, she read 12 pages. Today, she read twice as many as yesterday. So, today’s pages would be 2 times 12, which is 24 pages. Let me add those up: 12 + 24 = 36 pages read in total over the two days. Now, subtract that from the total pages to find the remaining pages. 120 minus 36 equals 84 pages left. The question says she wants to read half of the remaining pages tomorrow. So, half of 84 is 42. Therefore, she should read 42 pages tomorrow. Let me double-check to make sure I didn’t make a mistake. Total pages: 120. Yesterday: 12. Today: 24. Total read: 36. Remaining: 84. Half of 84 is 42. Yep, that seems right.
Correct segmentation:

Appendix

Table 6: Few-shot examples used in the Annotation Prompt for Mathematical datasets

CoT Prompt Template (BBH Tasks)
Answer the following question step by step. End your response by stating the chosen option in the format: "The answer is X.", where X is the correct option.
Question: {question}

Appendix

Table 7: Prompt template used for Chain of Thought generation (BBH Tasks)

Annotation Prompt One-shot example (BBH Formal Fallacies)
Example
Question:
It is not always easy to grasp who is consuming which products. The following argument pertains to this question: Every infrequent user of Paul Mitchell shampoo is either a rare consumer of Nioxin shampoo or a loyal buyer of Caress soap, or both. No regular consumer of Lush soap is a rare consumer of Nioxin shampoo and, in the same time, a loyal buyer of Caress soap. It follows that whoever is an infrequent user of Paul Mitchell shampoo is not a regular consumer of Lush soap." Is the argument, given the explicitly stated premises, deductively valid or invalid? Options: - valid - invalid
Reasoning trace: [omitted due to space constraints]
Correct segmentation:
STEP 1: Okay, let’s try to figure out if this argument is deductively valid. So, the question is about whether the conclusion follows logically from the premises. Let me break it down step by step. \| REMEMBER \| BLOOM_REASON: The solver states what’s asked in the question.

Appendix

Table 8: One-shot example used in the Annotation Prompt for BBH Formal Fallacies

Annotation Prompt One-shot example (BBH Hyperbaton)
Example
Question:
Which sentence has the correct adjective order: Options:(A) lovely tiny modern French restaurant (B) French modern tiny lovely restaurant
Reasoning trace: [omitted due to space constraints]
Correct segmentation:
STEP 1: Okay, the user is asking about correct adjective order in English. They’ve given two options for describing a restaurant and want to know which one follows the standard adjective sequence rules. \| UNDERSTAND \| BLOOM_REASON: The solver interprets the task and identifies that the question concerns adjective ordering in English grammar.

Appendix

Table 9: One-shot example used in the Annotation Prompt for BBH Hyperbaton

Setting	Remember	Understand	Apply	Analyze	Evaluate	Create
With system prompt	28.4	21.0	26.9	10.1	12.7	0.9
Without system prompt	27.6	21.5	26.8	10.1	13.0	1.0

Appendix

Table 10: Distribution comparison for Phi-4-Reasoning with and without the default system prompt (internal reasoning traces only).

Figure 7: Temporal Dynamics of Bloom’s Levels in Reasoning Traces (Phi-4-reasoning is ran with and without system prompt- internal reasoning traces only)

Question: Mike plays ping pong for 40 minutes. In the first 20 minutes, he scores 4 points. In the second 20 minutes, he scores 25% more points. How many total points did he score?
Reasoning Text	Label	Justification
Okay, let’s see. Mike plays ping pong for 40 minutes. The problem says in the first 20 minutes, he scores 4 points. Then in the second 20 minutes, he scores 25% more points. I need to find the total points.	REMEMBERING	The solver restates facts given in the problem without any transformation or interpretation.
First, let’s confirm the time: 40 minutes total, split into two 20-minute intervals. First 20 minutes: 4 points. Second 20 minutes: 25% more than the first 20 minutes.	UNDERSTANDING	The solver interprets the problem by expressing the given information in their own words and identifying the key elements.
Wait, does “25% more points” mean 25% more than the first 20 minutes’ points? I think that’s what it means. So first part is 4 points, second part is 4 plus 25% of 4.	UNDERSTANDING	The solver clarifies the meaning of “25% more points” and relates it to the first 20 minutes’ points.
Let me calculate that. 25% of 4 is $0.25\times 4=1$ . So 25% more would be $4+1=5$ points in the second 20 minutes.	APPLYING	The solver executes a calculation to find 25% of 4 and then applies this result to determine the points scored in the second 20 minutes.
Then total points would be $4+5=9$ .	APPLYING	The solver applies basic arithmetic to calculate the total points scored.

Appendix

Table 11: Example of an annotated sample (GSM8K)

Question: “Here comes a perfectly valid argument: First, being a cousin of Chris is sufficient for not being a son of Kermit. We may conclude that whoever is not a son of Kermit is a cousin of Chris.” Is the argument, given the explicitly stated premises, deductively valid or invalid? Options: valid / invalid .
Okay, let’s try to figure out if this argument is deductively valid. So, the question says: “First, being a cousin of Chris is sufficient for not being a son of Kermit. We may conclude that whoever is not a son of Kermit is a cousin of Chris.”	REMEMBER	The solver recalls the question and the argument to be evaluated.
Hmm, let’s break this down. The first part is a premise: “being a cousin of Chris is sufficient for not being a son of Kermit.” In logical terms, if someone is a cousin of Chris (let’s call that $C$ ), then they are not a son of Kermit (not $S$ ). So, $C\rightarrow\neg S$ .	ANALYZE	The solver breaks down the premise into logical terms and identifies the relationship between being a cousin of Chris and not being a son of Kermit.
The conclusion is: “whoever is not a son of Kermit is a cousin of Chris.” So, $\neg S\rightarrow C$ .	UNDERSTAND	The solver interprets the conclusion in logical terms and identifies the relationship between not being a son of Kermit and being a cousin of Chris.
Wait a second, the premise is $C\rightarrow\neg S$ , but the conclusion is $\neg S\rightarrow C$ . That looks like the converse of the premise. In logic, the converse of an implication isn’t necessarily true. For example, if “if it’s a dog, then it’s a mammal” ( $D\rightarrow M$ ), the converse would be “if it’s a mammal, then it’s a dog” ( $M\rightarrow D$ ), which isn’t true because there are mammals that aren’t dogs.	ANALYZE	The solver recognizes the relationship between the premise and the conclusion, and identifies the converse of the premise.
So here, the premise is $C$ implies $\neg S$ . The conclusion is saying $\neg S$ implies $C$ . That’s the converse, which isn’t valid. So the argument is assuming that because $C$ leads to $\neg S$ , then $\neg S$ must lead to $C$ . But that’s a fallacy. It’s like saying “if it’s raining, the ground is wet” ( $R\rightarrow W$ ), so “if the ground is wet, it’s raining” ( $W\rightarrow R$ ), which isn’t necessarily true because the ground could be wet for other reasons.	EVALUATE	The solver evaluates the argument and identifies the fallacy of assuming the converse is true.
Let me check again. The premise: $C\rightarrow\neg S$ . The conclusion: $\neg S\rightarrow C$ . So the argument is taking $C\rightarrow\neg S$ and concluding $\neg S\rightarrow C$ . But that’s not a valid deduction. The correct contrapositive of $C\rightarrow\neg S$ would be $S\rightarrow\neg C$ , right? Because the contrapositive of $P\rightarrow Q$ is $\neg Q\rightarrow\neg P$ . So here, the contrapositive would be $S\rightarrow\neg C$ . But the conclusion is $\neg S\rightarrow C$ , which is different.	ANALYZE	The solver re-examines the premise and conclusion, and identifies the correct contrapositive.

Appendix

Table 12: Example of an annotated sample (BBH formal fallacies)

Question: Which sentence has the correct adjective order: Options: (A) silly old prismlike Mexican sock (B) Mexican prismlike old silly sock
Reasoning Text	Label	Justification
Okay, the user is asking about adjective order in English, specifically which sentence correctly orders adjectives. The options are: (A) “silly old prismlike Mexican sock” (B) “Mexican prismlike old silly sock”.	UNDERSTAND	The solver interprets the task and identifies the question concerns adjective ordering in English grammar.
Hmm, I recall that English has a specific sequence for multiple adjectives before a noun. It’s not random—it follows a pattern: opinion, size, age, shape, color, origin, material, purpose.	REMEMBER	The solver recalls the standard adjective order rule from prior knowledge.
Let me mentally run through the rules. First, “silly” is an opinion adjective—it describes how someone feels about the sock. Then “old” is clearly about age. “Prismlike” is a shape descriptor, and “Mexican” indicates origin.	ANALYZE	The solver breaks down the adjectives into their respective categories and examines how they relate to the adjective hierarchy.
So the correct order should be opinion $\rightarrow$ age $\rightarrow$ shape $\rightarrow$ origin. That would be “silly old prismlike Mexican sock.”	APPLY	The solver applies the adjective-ordering rule to determine the correct sequence.
Option (B) has “Mexican” first, which is origin, but origin should come after shape, not before. It also has “prismlike” before “old,” which messes up the age order. “Old” should come before “prismlike” because age comes before shape in the sequence.	EVALUATE	The solver critiques Option B by checking it against the grammatical rule and identifying violations.

Appendix

Table 13: Example of an annotated sample (BBH hyperbaton)

Large Reasoning Models

Large reasoning models are language models trained or prompted to solve difficult tasks through extended, structured computation.

See all 340 papers

Nov 30, 2025

Superficial Reflection or Genuine Thought? A Fine-Grained Cognitive Analysis of Large Reasoning Models

Motivated by the observed human-like behaviours in Large Reasoning Models (LRMs), this paper introduces a comprehensive taxonomy to characterise atomic reasoning steps and analyse the reasoning behaviours of LRMs. Grounded in human cognitive processes, we propose a taxonomy comprising five groups and seventeen categories. Through this taxonomy, we conduct an in-depth analysis of contemporary LRMs and distil four actionable takeaways for model optimisation. Most notably, we reveal that prevailing post-answer ``doublechecks'' are largely superficial and rarely yield substantive revisions. A targeted intervention further shows that explicitly eliciting richer reflection processes can substantially improve failed self-correction. To support this largescale study, we propose CAPO, an automated annotation method used to construct a dataset of 277,534 reasoning steps with strong agreement with human expert annotations. We further validate the main behavioural patterns on a newer reasoning model and a coding domain, demonstrating the broader applicability of the proposed taxonomy. All source code and data are available at https://github.com/hehepig4/psyche.

Yuxiang Chen, Zuohan Wu, Ziwei Wang +6

University College London · AI Lab, the Yangtze River Delta, China · The Hong Kong University of Science and Technology (Guangzhou) +4

LLM Self-Correction Cognitive Modeling Large Reasoning Models Reasoning Trace Analysis

Ask PDF

Jun 3, 2026

ReasoningFlow: Discourse Structures for Understanding LLM Reasoning Traces

Large reasoning models (LRMs) produce reasoning traces with non-linear structures, such as backtracking and self-correction, that complicate the evaluation and monitoring of the reasoning process. We introduce ReasoningFlow, a framework that captures the discourse structures of LRM reasoning traces into fine-grained directed acyclic graphs (DAGs). We develop and validate our annotation schema through careful manual annotation of 31 traces (2.1k steps), achieving high inter-annotator agreement, then scale to automatic annotation of 1,260 traces (247.7k steps) spanning three tasks (math, science, argumentation) and five models (Qwen2.5-32B-Inst, QwQ-32B, DeepSeek-V3, DeepSeek-R1, GPT-oss-120B). By analyzing ReasoningFlow graphs, we find: (1) LRMs exhibit structurally similar traces, despite being trained from different base models and potentially non-overlapping post-training data. (2) ReasoningFlow reveals diverse fine-grained reasoning behaviors (e.g., local verification, self-reflection, and assumptions) that can be used for better reasoning trace monitorability. (3) In LRMs, most of the erroneous steps are not used to derive final answers. (4) Mechanistic causal dependencies between steps do not reflect the language-level discourse structure. We release the dataset and code in: https://github.com/jinulee-v/reasoningflow.

Jinu Lee, Shivam Agarwal, Amruta Parulekar +3

The Grainger College of Engineering University of Illinois Urbana-Champaign

LLM Reasoning with Graphs LLM Auditing LLM Interpretability Structured Reasoning LLM Reasoning Large Reasoning Models+1

Ask PDF

Mar 24, 2026

PRISM: A Dual View of LLM Reasoning through Semantic Flow and Latent Computation

Large language models (LLMs) solve complex problems by generating multi-step reasoning traces. Yet these traces are typically analyzed from only one of two perspectives: the sequence of tokens across different reasoning steps in the generated text, or the hidden-state vectors across model layers within one step. We introduce PRISM (Probabilistic Reasoning Inspection through Semantic and Implicit Modeling), a framework and diagnostic tool for jointly analyzing both levels, providing a unified view of how reasoning evolves across steps and layers. Across multiple reasoning models and benchmarks, PRISM uncovers systematic patterns in the reasoning process, showing that failed trajectories are more likely to become trapped in unproductive verification loops and further diverge into distinct modes such as overthinking and premature commitment, which behave differently once a candidate answer is reached. It further reveals how prompting reshapes reasoning behavior beyond aggregate accuracy by altering both semantic transitions and internal computational patterns. By modeling reasoning trajectories as structured processes, PRISM makes these behaviors observable and analyzable rather than relying solely on final-task accuracy. Taken together, these insights position PRISM as a practical tool for analyzing and diagnosing reasoning processes in LLMs.

Ruidi Chang, Jiawei Zhou, Hanjie Chen

Rice University · Stony Brook University

LLM Reasoning LLM Interpretability Overthinking in Language Models

Ask PDF

Abstract

Large Reasoning Models

Explore similar work

Superficial Reflection or Genuine Thought? A Fine-Grained Cognitive Analysis of Large Reasoning Models

ReasoningFlow: Discourse Structures for Understanding LLM Reasoning Traces

PRISM: A Dual View of LLM Reasoning through Semantic Flow and Latent Computation