Was Post-Answer Reasoning Beneficial?
Motivated by the prevalently high uncertainty ratios ( > 40 % >40\% > 40% A n s w e r n → A n s w e r n + 1 Answer_{n}\to Answer_{n+1} A n s w e r n → A n s w e r n + 1 → \to → → \to → → \to → → \to → Figure 7 reveals significant stability disparities. Distilled models exhibited disproportionately high DR , with DeepSeek-R1-0528-Qwen3-8B exceeding a 2.0 ratio. This suggests that such models mimic the form of reflection without the requisite stability, often degrading correct initial derivations. They also displayed high EB , indicating a reliance on stochastic trial-and-error. In contrast, proprietary models (e.g., Gemini-3-Pro) demonstrated robust convergence, maintaining low DR and EB by locking onto correct paths early. Except for the Gemini series, Claude-Sonnet-4.5, Doubao-Seed-1.6, and gpt-oss-120b, EE values remained generally high ( > 1.0 >1.0 > 1.0 Figure 8 : Distribution of CoT token counts for correct versus incorrect outcomes across evaluated models. These findings imply that current distillation recipes often transfer the surface-level verbosity of reasoning without the corresponding verification capabilities , resulting in a “reasoning illusion” where longer generations degrade rather than improve reliability.
6 Conclusion
We introduced CoTJudger , an automated and general framework that transforms free-form CoT into directed dependency graphs to assess reasoning efficiency via validity-aware path analysis. By disentangling the Shortest Effective Path from structurally redundant segments, CoTJudger transcends token-length proxies, enabling an interpretable, topology-based diagnosis of over-reasoning. Across large-scale evaluations of 21 LRMs spanning proprietary, open-source, and distilled paradigms, we found that redundancy is pervasive yet highly model-dependent: while some models expand computation far beyond necessity, others maintain near-linear reasoning but still exhibit localized backtracking and post-answer looping. Our analysis further demonstrates that inefficiency is not a monolithic phenomenon but emerges through distinct patterns, such as cyclic recurrence, semantic verbosity, and unstable revisions, that can be localized across domains and difficulty levels. Collectively, these results redefine reasoning quality to encompass not just correctness but the structural necessity of the reasoning trajectory, providing actionable signals for developing LRMs that are both accurate and efficient.
\beginappendix
7 Dataset
Data Source. Query dataset is sourced from publicly available benchmarks: OmniMath [ gao2024omnimathuniversalolympiadlevel ] (Math), HumanEval [ chen2021codex ] (Programming), GPQA [ rein2023gpqagraduatelevelgoogleproofqa ] (PCB), and Big-Bench Hard [ suzgun2022challengingbigbenchtaskschainofthought ] (General Reasoning).
Data Cleaning and Adaptation. To adapt to the subsequent fully automated CoT analysis and correctness verification pipeline, we conducted critical cleaning on the initial samples:
• Removal of queries with excessively long answers to avoid introducing noise due to differences in text generation during evaluation. • Manual exclusion of queries with open-ended answers or requiring free-form responses to ensure that all problems have clear, automatically verifiable ground-truth answers. • Special handling for programming queries: Since their correctness can be objectively judged through code execution, the above two cleaning steps are not applicable. We retain the original queries and strictly follow the execution environment of the source dataset for verification.
Subdomain Distribution. The subdomain distribution of query dataset is shown in Table 2 .
8 Framework Development and Validation on Development Set
We utilized this development set (2,688 CoTs from Gemini-3-Pro, Qwen3-30B-A3B-Thinking-2507 and DeepSeek-R1-Llama-70B) for multiple rounds of modular design and validation for the framework’s iteration.
Pipeline Design and Tuning. Each step of the entire evaluation pipeline, such as heuristic rules for step segmentation, boundary definitions for classification, and reconnection logic for graph construction, was tested and optimized through case analysis on the development set. We continuously adjusted prompts, code designs, and algorithm parameters by observing failure cases to ensure the pipeline’s robustness to complex and diverse CoTs.
Taxonomy Construction. Our CoT node classification system was not pre-defined but systematically constructed through multiple rounds of sampling, open-ended induction, cross-domain integration, and manual refinement of a large number of CoT steps in the development set. This ensures its empirical foundation and completeness.
Framework Validation Experiments. We ran the complete evaluation pipeline on this development set, conducting case analysis and validation experiments to verify the rationality of the pipeline.
To validate the stability and effectiveness of the pipeline, we executed the full pipeline three times on the core development set. First, we recorded the output variations of the Step Segmentation, Node Classification, and Answer Verification modules to calculate their consistency. Next, to assess the validity of the methodology proposed by CoTJudger, we designed experiments to examine whether the Shortest Effective Paths (SEPs) derived from Path Extraction truly retain the core logic. We employed GPT-5 to rewrite the SEPs extracted from the CoTs generated across the three runs. This rewriting process preserved the original path logic without adding new content, focusing solely on logical organization and expression smoothing, while masking the final answer. The rewritten content was then fed back into the original subject LLM for re-inferring to evaluate whether the answer accuracy was maintained.
Domain Count Subdomains Distribution Math 364 Logic (13.8%), Algebra (13.1%), Calculus (12.0%), Linear Algebra (11.9%), Probability (11.8%), Geometry (12.1%), Combinatorics (11.3%), Number Theory (7.0%), Math Puzzle (7.0%) General Reasoning 270 11.1% each: Logical Deduction, Tracking Shuffled Objects; 3.7% each: Boolean Expressions, Causal Judgement, Date Understanding, Disambiguation QA, Dyck Languages, Formal Fallacies, Geometric Shapes, Hyperbaton, Movie Recommendation, Multistep Arithmetic Two, Navigate, Object Counting, Penguins in a Table, Reasoning about Colored Objects, Ruin Names, Salient Translation Error Detection, Snarks, Sports Understanding, Temporal Sequences, Web of Lies, Word Sorting PCB 98 Biology (37.8%), Chemistry (31.6%), Physics (30.6%) Programming 164 String (22.6%), List/Array (18.9%), Math (13.4%), Hash Map (9.7%), DP (6.9%), Simulation (6.1%), Graph (5.4%), Sort/Search (4.9%), Stack/Queue (3.7%), Linked List (3.0%), Tree Binary (3.0%), Bit Manipulation (2.4%) Table 2: Detailed statistics of the dataset distribution across four main domains and their subdomains. The percentages in parentheses indicate the relative weight of each subdomain within its parent domain.
The overall experimental results are presented in Table 3 . The output consistency for each module exceeded 92%, and the overall consistency across the three pipeline executions reached 94.2%, demonstrating robust pipeline stability. Furthermore, the average answer retention rate of 95.1% confirms that CoTJudger’s path extraction strategy effectively preserves the core logic of the CoT.
Module Metric Metric Description Step Atomization 94.1% Stability of step counts across runs, measured as 1 − C V 1-CV 1 − C V C V = σ μ CV=\frac{\sigma}{\mu} C V = μ σ F 1 micro = 2 ∑ T P 2 ∑ T P + ∑ F P + ∑ F N F1_{\text{micro}}=\frac{2\sum TP}{2\sum TP+\sum FP+\sum FN} F 1 micro = 2 ∑ T P + ∑ F P + ∑ F N 2 ∑ T P 1 − C V 1-CV 1 − C V 1 N ∑ 1 [ y new = y original ] \frac{1}{N}\sum\mathbf{1}[y_{\text{new}}=y_{\text{original}}] N 1 ∑ 1 [ y new = y original ] Table 3: Validation results for pipeline module stability and path effectiveness. 9 Classification System
To ensure clarity and domain adaptability, the classification system was constructed through a four-stage iterative process based on the core development set:
1. Sampling and Annotation: We conducted stratified random sampling from 2,688 CoTs generated by a diverse set of models (Gemini-3-Pro, Qwen3-30B-A3B-Thinking-2507, and DeepSeek-R1-Llama-70B) to ensure coverage of varied reasoning styles. 2. Domain-Specific Induction: For each domain, LLMs were employed to perform initial clustering and open-coding of node functions, inducing a preliminary category list specific to that field. 3. Cross-Domain Aggregation: We merged the domain-specific lists to identify overlapping functions. Universal core categories (e.g., Problem-Deconstruction , Context-Setting ) were extracted to form the top tier, while domain-unique actions were retained in the second tier. 4. System Refinement: The aggregated system underwent multiple rounds of manual review to resolve ambiguities. Special attention was paid to boundary cases, such as establishing strict criteria to distinguish between Reflection-or-Verification (evaluative) and Correction-or-Refinement (executive).
The final complete taxonomy is provided in Table 4 .
Domain Category Sub-categories Definition Universal Classification System Universal Problem-Deconstruction Problem-Understanding, Info-Extraction, Info-Organization Analyzes prompt to extract, parse, or organize key info, constraints, and core questions. Context-Setting Definition, Assumption, Concept-Explanation Defines terms, states assumptions, or explains core concepts required for reasoning. Strategy-Formulation – Outlines a high-level plan, sequence of steps, or specific method to solve the problem. Intermediate-Inference – Makes a deductive step or logical implication based on current info to advance the solution state. Reflection-or-Verification Verification, Reflection, Premise-Reassessment, Hypothesis-Rejection Assesses previous steps for correctness or consistency without taking new action. Correction-or-Refinement Correction, Refinement, Simplification, Adjustment Actively implements a fix or adjustment based on new insights or errors in a previous step. Repetition-or-Clarification – Reiterates previous content or adds inconsequential supplements without changing core meaning. Additional-Exploration – Investigates alternative paths or assumptions when the current one is viable but not definitive. Irrelevant-or-Redundant – Steps that are verbose, off-topic, or do not contribute to the solution trajectory. Conclusion – Summarizes findings or provides the final answer. Domain-Specific Extensions Math Equation-Setup – Translates constraints into mathematical equations or expressions. Formula-Application – Selects and applies specific formulas, theorems, or rules. Simplification – Performs algebraic manipulation to simplify expressions. Numerical-Computation – Performs arithmetic calculations to arrive at numerical values. Proof-Step – Formal steps in a logical proof (lemma, inference rule, case establishment). General Reasoning Decomposition – Breaks complex tasks into manageable sub-goals. Logical-Deduction – Applies logic rules (e.g., if A then B) to derive conclusions. Strategic-Assessment – Evaluates multiple potential paths to determine the best one. Pattern-Recognition – Identifies recurring trends or sequences to guide the solution. PCB Principle-Application – Applies scientific principles (e.g., F=ma, Ohm’s Law). Comparative-Analysis – Compares scenarios or models to draw insights. Quantitative-Analysis – Performs domain-specific calculations or unit conversions. Programming Algorithm-Design – Describes logic/plans using pseudocode or data structures. Code-Implementation – Translates logic into concrete programming language syntax. Test-Case-Analysis – Verifies logic using specific inputs/outputs or traces. Optimization – Proposes changes to improve efficiency (time/space complexity). Debugging – Identifies errors in existing code and proposes fixes. Table 4: The proposed two-tier classification taxonomy.
10 Prompts
To ensure the reliability of most automated modules that call LLM (GPT-5) , we employ Pydantic-based structured outputs to enforce predefined schema-conformance and stable data parsing. Step Atomization Prompt. After performing initial heuristic segmentation, we convert the original CoT into a list of splits. Subsequently, we further utilize LLM to perform merging and subsequent splitting operations on the elements in this list. The prompt is shown in Figure 9 . To maintain the integrity of the original model output, the LLM is instructed to output the mapping of step IDs (e.g., “Merge steps i and j”, “Split step i into j and j” ) instead of rewriting the entire CoT. This prevents the model introducing errors or biases during paraphrasing while leveraging its strong semantic understanding capability to ensure that each final node represents an atomic reasoning action.
Figure 9 : The prompt and structured output schema used for refining step atomization. Answer Verification Prompt. To ensure rigorous evaluation across diverse domains, we adopt two distinct prompting strategies. For tasks of Math, General Reasoning and PCB, GPT5 is instructed to simultaneously locate the answer and verify its correctness by cross-referencing the logical consistency and the provided query and ground truth (Figure 10 ). Conversely, for the Programming domain, the prompt is restricted to answer location and extraction only (Figure 11 ). In this case, the model identifies the code solution, while the actual correctness verification is delegated to a deterministic code execution sandbox.
Figure 10 : The prompt used for answer detection and logical verification in non-coding domains. Figure 11 : The prompt used for locating code solutions in Programming tasks. Note that correctness fields are omitted as verification is performed via code execution. Classification Prompt. To ensuring precise functional attribution, the LLM is provided with the full definition of our two-tier taxonomy (customized for the specific domain), the original problem context, and the segmented CoT steps. The prompt (Figure 12 ) emphasizes the importance of analyzing the functional role of each step within the overall reasoning structure—rather than relying solely on surface content.
Figure 12 : The prompt used to classify CoT nodes into the standardized taxonomy. The Classification System section is dynamically populated with the relevant domain-specific definitions. Graph Building Prompt. We employ two distinct LLM-based strategies to establish edge connections: (1) Highly semantic Overlap Detection (Figure 13 ), which identifies nodes that repeat or reclarify previous content to assign same node id and construct self-loop edge, and (2) Target Dependency Identification (Figure 14 ), which determines the specific predecessor node targeted by relational operations (e.g., Reflection-or-Verification , Correction-or-Refinement ) to form directed functional edges. For Reflection-or-Verification , GPT-5 is also instructed to analyze whether the node affirms or negates its target node.
Figure 13 : The prompt used to detect semantic overlaps. Figure 14 : The prompt used to identify the target node for relational categories ( Reflection , Correction , Additional-Exploration ). The "Task Requirement" field is dynamically injected based on the current node’s category rules. Path Validation Prompt. Since Shortest Effective Paths (SEPs) from Path Extraction Module are formed by algorithmically concatenating non-adjacent nodes, they often lack grammatical fluency. Consequently, the prompt (Figure 15 ) instructs the LLM to disregard surface-level disfluency and choppy transitions, focusing on whether the core logical flow remains intact and sufficient to derive the final answer.
Figure 15 : The prompt used to validate extracted reasoning paths. It instructs the model to overlook algorithmic artifacts (disfluency) and verify if the semantic logic remains sufficient to solve the problem. 11 Experimental Setup
Inference Configuration. For models accessed via API services, we used their default parameter configurations and enabled the official thinking mode, extracting CoTs from the ‘reasoning_content‘ field of returned results.
For models equipped with built-in thinking mechanisms (such as Qwen3-30B-A3B-Thinking-2507), we directly utilize their default generated content. For models that require specific formats to trigger thinking (such as DeepSeek-R1 and its distilled variants), we add the instruction "Please include your thinking process within <think> and </think>" to the prompt. Other models use only the query itself as the prompt.
12 Cases of External Redundancy
A case of Deconstructive Revision. Here is a case of the Deconstructive Revision phenomenon of DeepSeek-R1-0528-Qwen3-8B (Figure 16 ).
Figure 16 : A case of Deconstructive Devision of DeepSeek-0528-Qwen3-8B. In the CoT content below the figure: bold denotes the correct answer, underline indicates the incorrect answer A case of Error Entrenchment. Here is a case of the Error Entrenchment phenomenon of DeepSeek-R1-Distill-Llama-8B (Figure 17 and Figure 18 ).
Figure 17 : A case of Error Entrenchment of DeepSeek-R1-Distill-llama-8B (part 1). In the CoT content below the figure: underline indicates the incorrect answer. Figure 18 : A case of Error Entrenchment of DeepSeek-R1-Distill-Llama-8B (Part 2). In the CoT content below the figure: underlinetextbox indicates the incorrect answer. A case of Superfluous Verification. Here is case of Superfluous Verification phenomenon happened in Kimi-K2-Thinking (Figure 19 ).
Figure 19 : A case of Superfluous Verification of Kimi-K2-Thinking. In the CoT content below the figure: bold indicates the correct answer A case of Logical Epicenter. Here is a case of logical epicenter in DeepSeek-R1 (Figure 20 . It can be seen from the figure that DeepSeek’s iterative reflection, refinement and correction on time representation formats and digit counts resulted in heavy resource consumption in non-linear reasoning, with little progress made in task advancement.
Figure 20 : A case of Logical Epicenter of DeepSeek-R1: The core logical epicenter step and subsequent steps involving reflection, correction, and repetitive backward reasoning are highlighted in the abridged CoT within the figure (extraneous content is omitted due to the excessive length of the full CoT. 13 Cases of CoTJudger
Processed Result Instances from CoTJudger This section presents the result instances of our CoTJudger. Figure 21 and Figure 22 show the generated outputs after feeding the Chain-of-Thoughts (CoTs) into the pipeline.
Figure 21 : The generated result of a DeepSeek-R1’s CoT from CoTJudger. The upper part of this figure presents the outcomes of step segmentation and classification, answer detection (with correct answers highlighted in bold ), and target node identification for graph building. The lower part illustrates the constructed graph structure and the extracted shortest valid path. Figure 22 : The generated result of a Qwen3-30B-a3B-Thinking-0527’s CoT from CoTJudger. The upper part of this figure presents the outcomes of step segmentation and classification, answer detection (with correct answers highlighted in bold , incorrect answers highlighted by underline ), and target node identification for graph building. The lower part illustrates the constructed graph structure and the extracted shortest valid path. For ease of demonstration, we have omitted the numerous redundant steps after reaching the correct answer and replaced them with ellipses.