cs.CLSep 4, 2026
SaveBeneath the Surface of Chains-of-Thought: A Mechanistic Interpretation of Reasoning Operations in LLMs
Organizations: KAIST · NAVER AI Lab
Abstract
Reasoning in large language models unfolds through diverse functional operations, such as problem formulation, goal decomposition, and deduction. Although these operations are explicitly distinguished in text, little is known about how they are geometrically organized in representation spaces. To this end, we investigate whether distinct reasoning operations exhibit corresponding geometric structure in hidden representations. We find that operations are separable in held-out representations, with separability peaking in middle layers, and verify that this structure is not explained by lexical or positional confounds. Across layers, token-wise operation-alignment becomes more distributed over spans, while identical surface tokens are represented differently depending on the operation of its surrounding chunk. Attention-masking interventions further show that operation-aligned representations at chunk onset depend on preceding reasoning context. Consequently, our work demonstrates that language models maintain representational correspondence between linguistic reasoning expressions and their internal geometric structures. Code and project materials are available at https://github.com/naver-ai/beneath-cot.
Figures & tables
Figure 1: Overview of reasoning-vector analysis. Given an input problem and the model-generated reasoning trace, we probe hidden representations with reasoning operation vectors corresponding to different reasoning operations. The left panel shows token-level reasoning operation-alignment scores over the generated text, while the right panel projects span representations onto Decomposition and Recall reasoning-vector axes. Regions with strong reasoning operation-specific signals appear along the corresponding directions.
| Problem-solving stage | Reasoning Operation | Example |
| Understanding the problem | Extraction Direct mapping | The problem says that two numbers add up to 10. x is greater than 3 |
| Planning the solution | Decomposition | Get the area of Heptagon Divide it into triangles Solve each triangle Combine results |
| Carrying out the plan | Recall Deduction Algebraic manipulation Arithmetic computation | Area of a circle is needed Recall apply to problem is greater than 3 and is an integer |
| Looking back and final answer | Final answer | Therefore, the values are and . |
Table 1: Overview of main reasoning operations for our analyses. These eight frequent operation types are used as span labels in our representation analysis.
Figure 3: Layer-wise separability of reasoning operations across language models in middle token position. (A) Peak-layer one-vs-rest AUROC for each reasoning operation in each model, with 95% confidence intervals. (B) AUROC averaged across reasoning operations at different model depths (embedding, early, middle, and late layers), along with a random baseline. Across models and reasoning operations, separability remains high and is strongest in the middle layers.
Figure 4: Visualization of reasoning-operation clusters in the learned LDA space. For each target operation, held-out span representations from Qwen3-8B are projected onto a two-dimensional plane whose -axis is the corresponding reasoning operation vector and whose -axis is the first principal component of the residual representations. Points are colored by their annotated reasoning operation labels.
Figure 5: Quantitative analysis on intra-span LDA score variances. We investigate the LDA score variances within each span across layers using Qwen3-8B. Early-layer separability is often sparse and token-local, while mid–late-layer separability is distributed across the tokens. We further generalize the results for Qwen2.5-7B and Gemma4-31B in the Appendix.
Figure 6: Identical tokens acquire reasoning-dependent representations. For pairs of reasoning operations, we collect common surface tokens that appear in both operation contexts and visualize their layer-wise scores along the two corresponding reasoning operation vectors in Qwen3-8B . Each point is an occurrence of a shared token and is colored by its annotated reasoning label.
| Model | Position-only | Text-only | Hidden(Ours) |
| Qwen3-8B | |||
| Qwen2.5-7B | |||
| Gemma4-31B |
Table 2: Macro AUROC/AUPRC of position-only logistic regression, the stronger of bag-of-words and TF–IDF logistic regression, and mean-pooled hidden-state probes. Position and lexical content are informative, but hidden-state probes achieve higher aggregate performance across all three models.
Figure 7: Preceding context shapes operation representations. Masking attention to preceding context reduces target reasoning operation-alignment score, indicating that subsequent reasoning operations depend on prior reasoning context.
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A: Layer-by-token heatmaps of reasoning-operation alignment scores. We visualize token-level alignment scores with the corresponding reasoning-operation vector across layers for representative reasoning traces. Each heatmap shows how strongly each generated token aligns with a target reasoning operation vector at each layer. The examples illustrate that operation-specific scores are often localized or weak in early layers, become more coherent over contiguous token spans in middle layers, and may become less sharply localized in later layers. This supports the view that reasoning-operation representations emerge through contextual processing rather than being attached only to isolated cue tokens.
Figure B: Additional layer-by-token heatmaps of reasoning-operation alignment scores. We provide additional representative examples for other reasoning-operation types. Consistent with Figure A , the alignment scores become more structured across contiguous generated tokens in middle layers, while early-layer scores are more sparse and token-local. These qualitative patterns are consistent with the span-level separability and intra-span variance analyses.
Figure C: Quantitative analysis on intra-span LDA score variances across additional models. We report the intra-span LDA score variances across layers for Qwen2.5-7B and Gemma4-31B. Consistent with Qwen3-8B, variance is high in early layers and decreases toward middle layers, indicating a shift from token-local cues to operation signals distributed across the span.
| Reasoning operation | Mean | Median | One-sided -value | |
| Extraction | 0.596 | |||
| Direct Mapping | 0.391 | |||
| Decomposition | 0.531 | |||
| Recall | 0.406 | |||
| Deduction | 0.441 | |||
| Algebraic Manipulation | 0.601 |
Table A: Exploratory onset-to-onset temporal-ordering analysis. For each operation, , where the two onsets are detected using the same causal three-token rolling-mean procedure and regime-specific thresholds. Negative values indicate that the sustained late-layer signal is detected before the sustained early-layer signal under this criterion. The fraction column reports . The reported -values are from uncorrected one-sided Wilcoxon signed-rank tests for a negative location shift. A significant -value does not necessarily indicate that negative onset differences occur in a majority of examples.
Figure D: Additional attention-masking intervention results. We report context-intervention results beyond the pre-token setting shown in the main text. In the pre-chunk setting, we mask attention from the first token of a target reasoning operation chunk to the entire immediately preceding annotated reasoning operation chunk. In the random-control setting, we mask a randomly selected preceding block with a comparable length. Across settings, blocking preceding context generally reduces the target reasoning operation-alignment score, indicating that subsequent reasoning-operation representations depend on prior context. The reduction is stronger for structured preceding-context masking than for the random control, supporting the view that relevant preceding reasoning context contributes to the formation of the next operation representation.
| Operation | AUROC-M | AUROC-Mid | AUPRC-M | AUPRC-Mid |
| Extraction | 0.959 | 0.903 | 0.880 | 0.711 |
| Direct Mapping | 0.936 | 0.890 | 0.821 | 0.721 |
| Decomposition | 0.987 | 0.943 | 0.852 | 0.659 |
| Recall | 0.974 | 0.918 | 0.857 | 0.785 |
| Deduction | 0.869 | 0.766 | 0.652 | 0.470 |
| Algebraic Manipulation | 0.985 | 0.926 | 0.907 | 0.739 |
Table B: Llama-3-8B replication. M and Mid denote mean-pooled and middle-token span representations.
| GPQA-Diamond | MATH-500 | |||||||
| Operation | AUROC-M | AUROC-Mid | AUPRC-M | AUPRC-Mid | AUROC-M | AUROC-Mid | AUPRC-M | AUPRC-Mid |
| Extraction | 0.983 | 0.967 | 0.925 | 0.861 | 0.986 | 0.983 | 0.983 | 0.960 |
| Direct Mapping | 0.907 | 0.835 | 0.699 | 0.506 | 0.902 | 0.837 | 0.732 | 0.455 |
| Decomposition | 0.951 | 0.871 | 0.859 | 0.650 | 0.970 | 0.881 | 0.862 | 0.615 |
| Recall | 0.903 | 0.844 | 0.675 | 0.538 | 0.954 | 0.902 | 0.845 | 0.707 |
| Deduction | 0.902 | 0.846 | 0.585 | 0.427 | 0.921 | 0.850 | 0.646 | 0.518 |
Table C: Test-only transfer of Qwen3-8B probes to GPQA-Diamond and MATH-500 without task-specific probe retraining.
| Operation | Error AUROC | Non-error AUROC | Error AUPRC | Non-error AUPRC | ||
| (a) Middle-token representation | ||||||
| Recall | 0.946 | 0.949 | 0.446 | 0.862 | 0.857 | 0.695 |
| Deduction | 0.867 | 0.906 | 0.592 | 0.755 | ||
| Algebraic Manipulation | 0.897 | 0.915 | 0.632 | 0.630 | 0.607 | |
| Arithmetic Computation | 0.926 | 0.957 | 0.784 | 0.866 | ||
| Final Answer | 0.964 | 0.960 | 0.773 | 0.924 | 0.932 | 0.406 |
Table D: Transfer of reasoning-operation probes trained exclusively on correct reasoning traces to factual-error and operation-matched non-error spans from incorrect Qwen3-8B reasoning traces. Results are averaged across layers 10–30. The error and non-error groups contain equal numbers of spans for each operation. Reported -values are from one-sided paired Wilcoxon signed-rank tests of whether separability is higher for non-error spans than for factual-error spans. Values smaller than are reported as .
| Mean pooling | Middle token | |||||
| Model | 8 PCs | 32 PCs | 128 PCs | 8 PCs | 32 PCs | 128 PCs |
| Qwen3-8B | ||||||
| Qwen2.5-7B | ||||||
| Gemma4-31B | ||||||
Table E: Macro AUROC/AUPRC of PCA-only evaluation. Operation separability remains above chance without supervised LDA across all models, component counts, and span representations.
| Stage | Reasoning operation | Qwen2.5-7B | Qwen3-8B | Gemma4-31B |
| Stage 1: Understanding the Problem | Extraction | 590 | 1,035 | 682 |
| Direct-mapping | 624 | 913 | 1,289 | |
| Reframing | 146 | 390 | 393 | |
| Abstraction | 8 | 35 | 36 | |
| State-space-definition | 190 | 315 | 259 | |
| Stage 2: Planning the Solution | Pattern-matching | 225 | 322 | 389 |
Table F: Train + Test occurrence counts for each reasoning operation and model.
| Model | Position | Hidden | |
| Qwen3-8B | |||
| Qwen2.5-7B | |||
| Gemma4-31B |
Table G: Macro AUROC/AUPRC of position-only and mean-pooled hidden-state predictors. denotes hidden-state minus position-only performance.
| Position | Labels | AUROC/AUPRC | Chance AUPRC |
| 7 | |||
| 5 | |||
| 5 | |||
| 5 | |||
| 5 |
Table H: Position-stratified Qwen3-8B evaluation using mean-pooled representations. Macro averages include operations with sufficient examples in each interval.
| Model | Text | Hidden | |
| Qwen3-8B | |||
| Qwen2.5-7B | |||
| Gemma4-31B |
Table I: Macro AUROC/AUPRC of the stronger text-only baseline among TF-IDF and BoW and the mean-pooled hidden-state probe.
| Operation | Median | 95% CI | |
| Extraction | 4.68 | 87.5% | |
| Direct Mapping | 0.57 | 75.0% | |
| Decomposition | 3.91 | 100.0% | |
| Recall | 1.05 | 70.0% | |
| Deduction | 0.61 | 76.9% | |
| Algebraic Manipulation | 0.68 | 60.7% |
Table J: Lexically matched pair analysis. Positive effects indicate that the frozen probe is more strongly aligned with operation identity than with broad lexical similarity. Confidence intervals are computed across anchor-level effects.
| Qwen3-8B | Qwen2.5-7B | |||
| Operation | AUROC | AUPRC | AUROC | AUPRC |
| Extraction | 0.997 | 0.992 | 0.957 | 0.904 |
| Direct Mapping | 0.888 | 0.811 | 0.753 | 0.499 |
| Decomposition | 0.899 | 0.806 | 0.873 | 0.838 |
| Recall | 0.923 | 0.893 | 0.878 | 0.664 |
| Deduction | 0.835 | 0.653 | 0.862 | 0.744 |
Table K: Frozen-probe performance on held-out spans containing vocabulary associated with competing operation labels.
| Operation | Chance AUPRC | Mean pooling | Middle token | |
| AUROC | AUPRC | AUROC/AUPRC | ||
| Direct Mapping | 0.204 | 0.994 | 0.979 | |
| Deduction | 0.184 | 0.866 | 0.701 | |
| Algebraic Manipulation | 0.204 | 0.912 | 0.755 | |
| Arithmetic Computation | 0.204 | 0.816 | 0.510 | |
| Final Answer | 0.204 | 0.999 | 0.998 | |
Table L: Evaluation on spans with 50–75% digit or mathematical-token density. The theoretical chance AUROC is for all rows.
| Reasoning operation | Mean AUROC [95% CI] | Mean AUPRC [95% CI] | Middle AUROC [95% CI] | Middle AUPRC [95% CI] |
| Qwen3-8B (layers 10–30) | ||||
| Extraction | 0.998 [0.995, 1.000] | 0.988 [0.975, 0.998] | 0.995 [0.991, 0.998] | 0.970 [0.943, 0.989] |
| Direct mapping | 0.947 [0.921, 0.969] | 0.802 [0.724, 0.873] | 0.875 [0.835, 0.911] | 0.587 [0.490, 0.683] |
| Decomposition | 0.980 [0.961, 0.994] | 0.895 [0.824, 0.954] | 0.926 [0.890, 0.956] | 0.636 [0.514, 0.750] |
| Recall | 0.965 [0.939, 0.986] | 0.847 [0.761, 0.925] | 0.917 [0.875, 0.952] | 0.725 [0.640, 0.809] |
| Deduction | 0.927 [0.898, 0.953] | 0.648 [0.545, 0.753] | 0.883 [0.848, 0.916] | 0.531 [0.437, 0.640] |
Table M: Bootstrap confidence intervals for reasoning-operation separability. We report layer-averaged AUROC and AUPRC with 95% confidence intervals for mean-pooled and middle-token representations. Intervals are computed from 5,000 stratified bootstrap replicates.
| Category | Metric | Result |
| Samples | Validation spans | 84 |
| Human agreement | Unanimous | 50/84 (59.5%) |
| Majority label | 81/84 (96.4%) | |
| Fleiss’ | 0.666 | |
| Reference labels | Determined | 81/84 (96.4%) |
| Not determined | 3/84 (3.6%) |
Table N: Human validation of GPT-5 operation-span annotations. Each span was independently labeled by three of seven human annotators.
Figure E: quantitative analyses on separability of reasoning operations in representation spaces at position middel. AUROC and AUPRC of one-vs-rest operation classifiers are shown across layers for Qwen3-8B , Qwen2.5-7B , and Gemma4-31B using middle-token span representations. Solid lines denote true reasoning-operation labels, while dashed lines denote random-label and random-position baselines. Across models, reasoning operation-alignment scores remain substantially above the baselines and peak in middle layers.
Figure F: Quantitative analyses on separability of reasoning operations in representation spaces at position mean. AUROC and AUPRC of one-vs-rest operation classifiers are shown across layers for Qwen3-8B , Qwen2.5-7B , and Gemma4-31B . Solid lines denote true reasoning-operation labels, while dashed lines denote random-label and random-position baselines. Across models, performance remains above the baselines and typically peaks in middle layers.
Figure G: Quantitative analyses on separability of reasoning operations in representation spaces at position first. AUROC and AUPRC of one-vs-rest operation classifiers are shown across layers for Qwen3-8B , Qwen2.5-7B , and Gemma4-31B . Solid lines denote true reasoning-operation labels, while dashed lines denote random-label and random-position baselines. Across models, performance remains above the baselines and typically peaks in middle layers.
Figure H: Quantitative analyses on separability of reasoning operations in representation spaces at position last. AUROC and AUPRC of one-vs-rest operation classifiers are shown across layers for Qwen3-8B , Qwen2.5-7B , and Gemma4-31B . Solid lines denote true reasoning-operation labels, while dashed lines denote random-label and random-position baselines. Across models, performance remains above the baselines and typically peaks in middle layers.
| Problem-solving stage | Level 1 | Level 2 | Level 3 | Reasoning operation | Description / Example |
| Stage 1 Understanding the Problem | Extraction | – | – | Extraction | Directly retrieving given information from the problem statement without modification. Example: “Two numbers add up to 10, and both are non-negative.” |
| Symbolization | Direct-mapping | – | Direct-mapping | Directly converting a statement into a formal representation without changing its structure. Example: The sum of two numbers is 10 . | |
| Reframing | – | Reframing | Reconstructing the problem in a different representational system. Example: Geometry problem coordinate geometry. | ||
| Abstraction | – | Abstraction | Removing unnecessary details and retaining only the essential structure. Example: 3 red balls and 2 blue balls . | ||
| State-space-definition | – | – | State-space-definition | Defining possible states, constraints, and the solution space. Example: , , . | |
| Stage 2 Planning the Solution | Pattern-matching | – | – | Pattern-matching | Mapping the current problem to a known problem type, template, or strategy. Example: Flip a coin three times probability/combinatorics problem. |
Table O: Reasoning operation schema (Stages 1–2).
| Problem-solving stage | Level 1 | Level 2 | Level 3 | Reasoning operation | Description / Example |
| Stage 3 Carrying Out the Plan | Retrieval | Recall | – | Recall | Retrieving relevant definitions, rules, concepts, or formulas. Example: Need area of a circle . |
| Matching | – | Matching | Determining whether retrieved knowledge applies to the current situation. Example: Apply when the radius is given. | ||
| Inference-flow | Chaining | Deduction | Deduction | Deriving a specific conclusion from a general rule. Example: General theorem apply to the given case. | |
| Induction | Induction | Inferring a general rule from observed patterns. Example: Several terms follow the same difference infer arithmetic pattern. | |||
| Analogy | Analogy | Reasoning from a structurally similar case. Example: Solve a new problem using a similar solved template. | |||
| Branching | – | Branching | Splitting into cases and analyzing each separately. Example: or . |
Table P: Reasoning operation schema (Stage 3).
| Problem-solving stage | Level 1 | Level 2 | Level 3 | Reasoning operation | Description / Example |
| Stage 4 Looking Back and Final Answer | Strategy-validation | – | – | Strategy-validation | Verifying whether the chosen strategy or assumptions were valid. Example: Check whether differences are constant before assuming an arithmetic sequence. |
| Error-detection | – | – | Error-detection | Identifying mistakes in reasoning, calculation, or logical steps. Example: incorrect arithmetic; should be 7. | |
| Dimensional-analysis | – | – | Dimensional-analysis | Checking whether units or dimensions are consistent with the required quantity. Example: Speed = distance/time meters/second. | |
| Extreme-case-testing | – | – | Extreme-case-testing | Testing whether the solution holds under extreme or boundary conditions. Example: For , test and . | |
| Final-answer | – | – | Final-answer | Clearly presenting the final result in the required format. Example: Therefore, and . |
Table Q: Reasoning operation schema (Stage 4).
| Item | Instructions given to annotators |
| Task objective | Determine the role played by the highlighted text in the reasoning process. Annotators should identify the function of the highlighted span within the overall solution, rather than solve the problem themselves or verify whether the final answer is correct. |
| Question | The original problem to which the model-generated reasoning responds. |
| Response with highlighted span | The model-generated reasoning trace. Only the highlighted portion ( full span ) is the annotation target. The surrounding response is provided solely to understand why the highlighted span appears at that point in the reasoning process. |
| shortened_span_text | A shorter phrase extracted from the full span when the full span is long. When the full span is already short, the shortened span is identical to the full span. |
| Answer columns | For each example, annotators provide three judgments: (1) the reasoning-operation label that best characterizes the full span; (2) their confidence in that label; and (3) whether the shortened_span_text preserves the main reasoning operation expressed by the full span. |
| Disabled cells | Black cells are intentionally disabled and should be left untouched; annotators should not type or make any selection in these cells. |
Table R: Summary of the human annotation procedure and interface. The instructions were originally provided to annotators in Korean.
More in this topic
LLM Interpretability
Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.
See all 754 papers