On Reasoning Correctness Directions

Munachiso Samuel Nwadike; Zayd M. K. Zuhri; Rifo Genadi; Abzal Nurgazy; Ali Mekky; Alham Fikri Aji; Zangir Iklassov; Kentaro Inui

cs.AIOct 18, 2026

On Reasoning Correctness Directions

Authors: Munachiso Samuel Nwadike, Zayd M. K. Zuhri, Rifo Genadi, Abzal Nurgazy, Ali Mekky, Alham Fikri Aji, Zangir Iklassov, Kentaro Inui

Organizations: MBZUAI, UAE · RIKEN AIP, Japan · Mila, Canada · Tohoku University, Japan

Abstract

We present experimental evidence suggesting that information about whether a model's reasoning is on the right track can be linearly decodable without defining a causally useful direction for steering. To study this, we introduce ChainSearch, a controlled arithmetic reasoning task that extends Countdown with explicit search and backtracking. We train a 3-million-parameter model to solve this task. Its compact size allows us to study internal model correctness directions in detail using linear probes, mean activation differences, and direct optimisation, as part of a rigorous search for internal reasoning-correctness directions. We find that while the model contains statistically significant internal directions of reasoning correctness, none of the resulting directions consistently improves reasoning under causal intervention. This calls into question whether the linear representation hypothesis is sufficient for identifying causally useful directions in the context of reasoning. Our code is made available together with this work.

Figures & tables

Figure 1: Understanding how reasoning models arrive at their answers has attracted growing interest (Appendix A ). Reasoning often entails exploring alternative paths, identifying dead ends, and backtracking to earlier states. ChainSearch makes this process explicit in an arithmetic game, where each step can preserve or remove the possibility of reaching the target. Red numerical results mark states from which the target can no longer be reached without backtracking, while green results mark states where it remains reachable. BACK restores an earlier state for further exploration, and SUCCESS marks attainment of the target.

Figure 2: By normalising reasoning trace lengths, we compare how well linear probes predict final-answer correctness at different stages of generation. Correctness is already decodable well above chance early in the trace and generally becomes more decodable as generation proceeds. This pattern appears across attention heads, residual streams, MLP outputs, and the final hidden representation, with AUROC reaching approximately

0.94

by the end. Error bars show one standard deviation around the mean.

Metric	Measure	Question
A. Successive-position stability	$\displaystyle S^{\mathrm{S}}_{m,\ell,p\rightarrow p^{\prime},c}=\cos\!\left(u^{\ell}_{m,p,c},u^{\ell}_{m,p^{\prime},c}\right)$	What if we go one step at a time across successive reasoning positions?
B. Cross-position stability	$\displaystyle S^{\mathrm{P}}_{m,c}=\frac{K\left\\|\frac{1}{K}\sum_{p=1}^{K}u_{m,p,c}\right\\|^{2}-1}{K-1}$	Does final-answer correctness share a common direction across reasoning positions?
C. Cross-label stability	$\displaystyle S^{\mathrm{L}}_{m,p,c}=\cos\!\left(u^{\mathrm{reach}}_{m,p,c},u^{\mathrm{answer}}_{m,p,c}\right)$	Do local reachability and eventual final-answer correctness share the same direction?
D. Cross-method stability (Type 1)	$\displaystyle S^{\mathrm{M1}}_{m_{1},m_{2},p,c}=\cos\!\left(u_{m_{1},p,c},u_{m_{2},p,c}\right)$	Does the correctness direction remain stable across construction methods at the same position and component?
E. Cross-method stability (Type 2)	$\displaystyle S^{\mathrm{M2}}_{p,c}=\left\\|\operatorname{Proj}_{\operatorname{span}\{u^{P}_{p,c},u^{C}_{p,c}\}}u^{D}_{p,c}\right\\|^{2}$	Does the Direct Optimisation vector lie within the plane spanned by Probe and CAA?

Table 1: The five comparisons metrics A–E above, are used to test whether reasoning correctness forms a stable linear direction. Here,

u_{m,p,c}

denotes the unit-normalised direction obtained using method

m

at semantic position

p

and model component

c

\ell\in\{\mathrm{reach},\mathrm{answer}\}

denotes the label family, and

K

is the number of selected semantic positions.

Figure 3: Reachability directions remain more stable across BACK and state restoration at attention heads than at other model components. CAA has higher median stability than Probe in every component class. Points show median successive-position stability (

S^{\mathrm{S}}

), and horizontal bars show interquartile ranges.

Figure 4: Probe and CAA final-answer directions remain highly stable across successive token positions within arithmetic state transitions at attention heads, whereas Direct Optimisation yields weaker and more variable alignment. Lines show median successive-position stability (

S^{\mathrm{S}}

) within each component class, and shading shows interquartile ranges. An expanded version is provided in Appendix Figure 10 .

Figure 5: We measure cross-position stability (

S^{\mathrm{P}}

) to test whether final-answer directions remain consistent across the selected positions throughout the reasoning trace. CAA final-answer directions have the highest median cross-position stability in every component class. Probe directions are also more stable across positions at attention heads than at other components, whereas Direct Optimisation directions show no comparable advantage. Each point reports stability at one model location.

Figure 6: Changing the number of steered attention heads yields no consistent improvement in next-state reachability or final-answer correctness. Each condition compares the same 256 test problems with and without steering. Solid upward bars count problems improved by steering, hatched downward bars count problems worsened, and diamonds indicate no change.

Figure 7: Steering produces small, mixed changes in final-answer correctness, with several of the largest effects worsening outcomes. Upward bars count problems where the model originally gave an incorrect answer but gives a correct one after we apply steering and let it generate a new response. Downward bars count the opposite, where an originally correct answer becomes incorrect. Even Direct Optimisation, which explicitly trains a vector to increase the likelihood of a correct final answer (see Appendix F ), does not consistently improve correctness when the model generates a new response. Settings are chosen using validation data, then evaluated on 256 test problems, each with and without steering.

Figure 8: Applying steering more frequently changes more outcomes, but does not consistently improve them. The largest increases in the number of worsened outcomes occur when steering is applied at multiple positions. Single steering acts before the first arithmetic state transition, repeated steering acts before every arithmetic state transition, and multipoint steering additionally acts after resulting and restored states, with settings chosen separately for each position. The panels measure final-answer correctness and whether every subsequent state remains reachable. Positive bars count problems improved by steering, while negative bars count problems worsened. Each setting is evaluated on 512 test problems, with and without steering.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Size	Purpose
Model pretraining
Exploratory pretraining corpus	6,948,231 traces	Procedurally generated search with controlled backtracking
Pretraining train	6,947,727 traces	Model pretraining
Pretraining validation	504 traces	Pretraining checkpoint selection
Supervised fine-tuning and held-out allocation
Complete ChainSearch corpus	1,016,676 examples	Supervised fine-tuning and held-out analyses

Appendix

Table 2: Allocation of ChainSearch data across model pretraining, supervised fine-tuning, interpretability analysis, and causal steering. Split counts describe distinct traces or examples, whereas steering counts describe paired condition–problem evaluations. The same held-out problem may therefore be evaluated under several independently selected steering conditions.

Figure 9: As in Figure 2 , final-answer correctness becomes more decodable as generation progresses, even when we restrict traces to at most eight BACK transitions. This suggests that the increase is not primarily driven by traces with unusually frequent backtracking. Error bars show one standard deviation around the mean.

Target	Final EXPR	Steps
101	(100+1)	1
102	(100+(9-(8-1)))	3
103	(100+(9-((8-(9-8))-1)))	5
104	(100+(8/(9-(8-1))))	4

Appendix

Table 3: Target-disjoint splitting is motivated by the observation that even consecutive targets can lead to substantially different solution structures for a fixed input, here IN: [1,8,8,9,9,100] .

Metric	What it measures	After pretraining	After fine-tuning
Expression parseability	The final EXPR can be parsed	95.63%	99.33%
Arithmetic operation validity	A generated arithmetic operation is legal	93.58%	96.37%
State consistency	The resulting state from a state transition correctly reflects the arithmetic operation	92.93%	95.16%
Final-answer correctness	The final EXPR evaluates to the target	25.20%	71.12%

Appendix

Table 4: The model achieves high arithmetic operation validity and state transition consistency, suggesting its competence. A final-answer accuracy of (71%) provides a useful middle ground where performance is strong, while the remaining (29%) of traces provide enough negative examples to limit class imbalance during probe training and CAA construction. Pretraining results use 504 validation examples, and fine-tuning results use 9,000 test examples (matching the corresponding data splits outlined in Appendix B.3 Table 2 ).

	Parameter-space task vector	Probe direction	Mean-difference activation direction (CAA)	Directly optimized activation direction
Construction	$\displaystyle v=\theta_{ft}-\theta_{pre}$	$\displaystyle\begin{gathered}v_{p,c}=\arg\min_{v}\sum_{i}\{\\ \mathcal{L}(y_{i},v^{\top}h_{i,p,c}+b)\}\end{gathered}$	$\displaystyle\begin{gathered}v_{p,c}=\mathbb{E}\!\left[h_{i,p,c}\mid y_{i}=1\right]-\\ \mathbb{E}\!\left[h_{i,p,c}\mid y_{i}=0\right]\end{gathered}$	$\displaystyle\begin{gathered}v_{p,c}=\arg\min_{v}\sum_{i}\{\\ \mathcal{L}\left(M(h_{i,p,c}+v),a_{i}\right)\}\end{gathered}$
Space	Parameter space	Activation space	Activation space	Activation space

Appendix

Table 5: Four constructions for linear intervention directions. Here,

h_{i,p,c}

denotes the activation for example

i

at token position

p

and model component

c

y_{i}\in\{0,1\}

denotes the corresponding binary label, and

\theta_{ft}

and

\theta_{pre}

denote the parameters of the finetuned and pretrained model respectively. Here,

M

denotes the frozen model,

v

denotes the direction produced by each construction,

b

denotes the learned probe bias, with its joint optimization omitted from the notation for brevity,

a_{i}

denotes the target answer-token sequence for example

i

, and

\mathcal{L}

denotes the classification loss for the probe, and the objective for direct optimisation direction, respectively.

Figure 10: We measure successive-position stability (

S^{\mathrm{S}}

) to examine whether final-answer directions remain similar as the model constructs an arithmetic state transition. Probe and CAA final-answer directions show high stability at attention heads, remaining similar across successive token positions. Directions obtained through Direct Optimisation show lower stability, while residual-stream and MLP directions are less stable or vary more in stability across positions. Lines show medians across locations within each component class, and shading shows interquartile ranges.

Figure 11: Plotting cross-method stability Type 2 (

S^{\mathrm{M2}}

) against Probe–CAA cross-method stability Type 1 (

S^{\mathrm{M1}}

) reveals that the strongest agreement with Direct Optimisation is concentrated at a small subset of attention-head locations.

S^{\mathrm{M2}}

measures how much of a Direct Optimisation direction lies within the plane spanned by Probe and CAA, even when its cosine similarity with either vector individually is low. Some attention-head directions have substantial projections onto this plane, while projections at residual streams and MLP outputs remain close to the random baseline. Each point compares directions at the same token position and model location. Dashed lines show median projections of random unit vectors.

Figure 12: We measure successive-position stability (

S^{\mathrm{S}}

) to examine how reachability directions change as an arithmetic state transition is constructed. Probe and CAA reachability directions remain similar across successive token positions at attention heads, with lower stability in other model components. Successive-position stability for CAA directions at the final hidden representation approaches zero at later transitions. Lines show medians across locations within each component class, and shading shows interquartile ranges.

Figure 13: Cross-label stability (

S^{\mathrm{L}}

) measures whether next-state reachability and final-answer correctness are associated with similar directions at the same token position and model location. CAA generally yields higher cross-label stability than Probe, particularly at attention heads and the residual stream, although the two correctness labels do not share an identical direction. Lines show median cosine similarity within each component class, and shading shows interquartile ranges.

Figure 14: We vary the number of steered heads and compare selection by CAA separation with random selection, measuring changes in next-state reachability and final-answer correctness. Below each head count, the two

\beta

values are chosen using validation data and shown as “ CAA selection / random selection ”. Each setting is evaluated on 512 test problems, with and without steering. Selecting heads by CAA separation gives no consistent advantage over random selection.

Figure 15: We compare whether final-answer correctness improves when steering uses directions constructed for final-answer correctness (left) or next-state reachability (right). Neither type of direction produces consistently beneficial effects across intervention positions. All interventions use four heads, with steering strengths chosen separately using validation data. Each setting is evaluated on 512 test problems, with and without steering.

Figure 16: We reverse the CAA sign chosen using validation data to test whether the opposite direction reverses its effect on next-state reachability and final-answer correctness. Bar heights show the number of problems improved minus the number worsened, with annotations reporting both counts. Neither sign consistently outperforms the other. Each sign is evaluated on the same 512 test problems and compared with generation without steering.

Figure 17: We measure changes in generation length, BACK usage, and expression parseability to examine how steering affects the generated trace beyond final-answer correctness. These effects vary across methods, model components, and intervention positions, without a common pattern of improved reasoning. The first two panels count problems with increases and decreases separately, while the third shows the number gaining a parseable expression minus the number losing one. Each setting is evaluated on 256 test problems, with and without steering.

Panel of Figure 18 — Figure 18: Varying steering strength $\beta$ does not reveal a consistently beneficial response in either final-answer correctness (a) or next-state reachability (b). Larger interventions often change more outcomes, but some are improved and others worsened. Each cell reports the numbers of problems improved and worsened, and outlined cells mark strengths chosen using validation data. Each setting is evaluated on 16 validation problems, with and without steering.

Figure 19: Applying the vector learned through Direct Optimisation without normalising it changes more outcomes than applying its version normalised to unit length, but more outcomes are worsened than improved for both final-answer correctness and trace validity in these settings. We compare the two versions at the same intervention position and model component. Each version is evaluated on the same 256 test problems, with and without steering.

Causal Interpretability

Causal interpretability uses interventions and counterfactuals to identify which model inputs and internal mechanisms actually drive predictions.

See all 107 papers

May 10, 2026

Hidden Error Awareness in Chain-of-Thought Reasoning: The Signal Is Diagnostic, Not Causal

Chain-of-thought (CoT) prompting assumes that generated reasoning reflects a model's internal computation. We show this assumption is wrong in a specific, measurable way: models internally detect their own reasoning errors but outwardly express confidence in them. A linear probe on hidden states predicts trace correctness with 0.95 AUROC -- from the very first reasoning step (0.79) -- while verbalized confidence for wrong traces is 4.55/5, nearly identical to correct ones (4.87/5). A text-surface classifier achieves only 0.59 on the same data, confirming a 0.20-point gap invisible in the generated text. This hidden error awareness holds across three model families (Qwen, Llama, Phi), 1.5B-72B parameters, and RL-trained reasoning models (DeepSeek-R1, 0.852 AUROC). The natural question is whether this signal can fix the errors it detects. It cannot. Four interventions -- activation steering, probe-guided best-of-N, self-correction, and activation patching -- all fail; patching destroys output coherence entirely. The signal is diagnostic, not causal: a readout of computation quality, not a lever to redirect it. This delineates a boundary for mechanistic interpretability: error representations during reasoning are fundamentally different from the factual knowledge representations that prior work has successfully edited.

Aojie Yuan, Zhiyuan Julian Su, Haiyue Zhang +2

University of Southern California, Los Angeles, CA, USA.

LLM Interpretability Causal Interventions in Language Models Reasoning Trace Analysis

Ask PDF

Oct 5, 2026

COMPASS: Finding Where Reasoning Lives in Language Models

Explicitly eliciting reasoning substantially improves LLM performance. Existing approaches require a predefined characterization of reasoning, whether through CoT prompt design, contrastive CoT directions, or via SAE derived reasoning features. For mathematical reasoning with verifiable answers, we show that a much simpler signal suffices, which is the correctness of the model's own direct answer attempts. This signal yields a latent direction that elicits reasoning. This direction is decodable within the activations of most attention heads, but only a small subset of them can be effectively intervened. We introduce COMPASS, an inference-time steering method that identifies these heads using a logit-space attribution score and steers their activations along the correctness direction, requiring only per-head activation statistics. Across three model families and multiple math benchmarks, COMPASS outperforms the activation-steering baselines we compare against, improves GSM8K accuracy by 16 percentage points on average, and approaches CoT accuracy with 20-70% fewer generated tokens. Interventions transfer without re-fitting to unseen benchmarks, and ablations show that both the correctness direction and the small set of heads carrying it are necessary, with the effect concentrated in remarkably few heads.

Pratyay Dutta, Kowshik Thopalli, Vivek Narayanaswamy

University of California, Riverside, CA 92521. Work performed during a summer internship at Lawrence Livermore National Laboratory · Lawrence Livermore National Laboratory, Livermore, CA 94550.

Efficient Language Model Reasoning Activation Steering LLM Interpretability Numerical Reasoning in Language Models

Ask PDF

Apr 7, 2026

LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signals

This work characterizes large language models' chain-of-thought generation as a structured trajectory through representation space. We show that mathematical reasoning traverses functionally ordered, step-specific subspaces that become increasingly separable with layer depth. This structure already exists in base models, while reasoning training primarily accelerates convergence toward termination-related subspaces rather than introducing new representational organization. While early reasoning steps follow similar trajectories, correct and incorrect solutions diverge systematically at late stages. This late-stage divergence enables mid-reasoning prediction of final-answer correctness with ROC-AUC up to 0.87. Furthermore, we introduce trajectory-based steering, an inference-time intervention framework that enables reasoning correction and length control based on derived ideal trajectories. Together, these results establish reasoning trajectories as a geometric lens for interpreting, predicting, and controlling LLM reasoning behavior.

Lihao Sun, Hang Dong, Bo Qiao +3

Microsoft

LLM Reasoning Representation Geometry in Language Models Language Model Steering

Ask PDF

Abstract

Causal Interpretability

Explore similar work

Hidden Error Awareness in Chain-of-Thought Reasoning: The Signal Is Diagnostic, Not Causal

COMPASS: Finding Where Reasoning Lives in Language Models

LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signals