Chain-of-Thought (CoT) prompting improves the performance of Large Language Models by eliciting intermediate reasoning steps. However, CoT is far from perfect: on some tasks, models perform worse than with direct-answer prompting, and on others they generate a few hundred tokens to reach the correct answer. We start by labelling each CoT hidden state as recovering if patching it into the direct prompt yields the correct answer, and failed otherwise. A linear probe shows that the model encodes these two classes differently. We then show that the representation space of the direct prompt holds a direction for each question that leads the model, in a short generation, to the correct answer more often than CoT does. Specifically, we rotate the direct prompt's last hidden state once, before generation, towards a direction built for each question from its own recovering and failed states. Across 25 tasks, these directions raise the direct prompt's mean accuracy from 46% to 85% on LLaMA 3.1 8B (CoT: 69%) and from 52% to 86% on Qwen 2.5 7B (CoT: 77%), with a tenth of CoT's tokens. CoT is thus not the shortest route to this accuracy, and these directions can be used in future work to train models to align their states with them.
Figures & tables
Figure 1: Overview of our analysis on one Multistep Arithmetic question. Each CoT state is patched in place of the last state z of the direct prompt and labelled recovering if the model then answers correctly and failed otherwise. The failed states and recovering states occupy different parts of the representation space, so that a linear probe predicts this label from a single state. The means of the question’s recovering and failed states, c+ and c− , give the task-solving direction d∝c+−0.5c− . Before generation, we rotate z onto the direction of d , keeping its norm; the direct prompt then writes four short lines and answers 48 instead of −40 , even though the CoT itself answers 80 .
Table 1: CoT and direct (Dir.) accuracy (%) on the 25 tasks, with the CoT gain Δ (computed before rounding) and the mean number of tokens generated per question under CoT (direct prompting generates about one).
Figure 2: Out-of-fold AUROC of each task’s probe, at layer 29 of LLaMA 3.1 8B and layer 25 of Qwen 2.5 7B .
Figure 3: Transfer of the probes across reasoning types, at layer 29 of LLaMA 3.1 8B and layer 25 of Qwen 2.5 7B . Each cell is the mean AUROC of probes trained on a task of the row’s type and evaluated on a different task of the column’s type; outlined cells pair tasks of the same type. Under each matrix is the mean AUROC over all pairs of tasks of the same type and over all pairs of tasks of different types.
Table 2: Rotating the last state of the direct prompt, at layer 29 of LLaMA 3.1 8B and layer 25 of Qwen 2.5 7B ( t=1 ). Acc. (%) is macro-averaged over the 25 tasks, counting a question without a direction as incorrect; Δ s, computed before rounding, are from the direct and CoT prompts, and Tok. is the mean number of generated tokens. Appendix D.2 gives every task.
Figure 4: Accuracy and generated tokens per task when the state of the direct prompt is aligned with dq(0.5) (green), against the direct (grey ring) and CoT (brown) prompts, at layer 29 of LLaMA 3.1 8B and layer 25 of Qwen 2.5 7B ( t=1 ). The number at the end of each row is the accuracy with dq(0.5) . Each token bar runs to the tokens CoT generates per question, and its green part to those generated with dq(0.5) ; the direct prompt generates about one.
Figure 5: Change in each task’s accuracy over the direct prompt (zero line) under CoT and when the state is aligned with each question’s own direction dq(0.5) or with the task-level direction dT(0.5) , at layer 29 of LLaMA 3.1 8B and layer 25 of Qwen 2.5 7B ( t=1 ). Dark ticks give the row means.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Template, shared across tasks
User turn
Question: {question} Answer:
System, CoT
you are a step-by-step reasoner, finish the reasoning with Final Answer: #### ⟨ ans ⟩
System, direct
Answer only with ⟨ ans ⟩ . Do not show steps.
Answer slot ⟨ ans ⟩ (CoT; direct)
True/False
[True/False] ; True or False
Yes/No
[Yes/No] ; Yes or No
Appendix
Table 3: Prompt templates. Every task uses the same user turn, and the two system instructions differ only in whether they ask for steps and in the answer slot, which depends on the answer format. True/False covers StrategyQA and the True/False tasks of BIG-Bench Hard. Under CoT, #### marks the final answer.
Table 4: Patch labels per task, at layer 29 of LLaMA 3.1 8B and layer 25 of Qwen 2.5 7B . Questions is the number of questions; States is the mean number of labelled CoT states per question, one per token of the chain, which is capped at 400 tokens (Appendix A ); Rec. is the percentage of these states that recover the answer when patched; Cov. is the percentage of questions with at least one recovering and one failed state, for which the directions of Section 5 are defined. The last row gives the total number of questions and, for the other columns, macro averages over tasks.
Table 5: Per-task probes at the layers reported in the main text. n : questions, all of which are scored out of fold. AUROC ± the standard deviation of the per-fold AUROC; Figure 2 shows the same AUROC values to two decimals, rounded from the unrounded values. Sel. orig./w.-pos.: selectivity against the origin and within-position controls (Appendix C.2 ).
Figure 6: Transfer of the probes between every pair of tasks, at layer 29 of LLaMA 3.1 8B and layer 25 of Qwen 2.5 7B . Each cell is the AUROC of a probe trained on the row’s task and evaluated on the column’s task; tasks are grouped by reasoning type. Cells run from ivory at chance (0.5) to green, and the diagonal, a probe on its own task (Figure 2 ), is left white. Plum cells fall below chance. Figure 3 averages these matrices over reasoning types.
Table 6: Probe performance on the original and control tasks at each layer of the late band, summarised over the 25 tasks. Probe: mean out-of-fold AUROC, computed over all questions of each task. Control: mean AUROC of the origin (Orig.) and within-position (W.-pos.) control probes (Appendix C.2 ). Selectivity: probe AUROC minus control AUROC.
Table 7: Probe transfer per layer. Diagonal: mean out-of-fold AUROC within each task. Off-diagonal: mean AUROC of a probe trained on all questions of one task and scored on every state of another; >0.5 : cells above chance; Sel. w.-pos.: mean selectivity against the within-position control. Same/Different: mean off-diagonal AUROC between tasks of the same or of a different reasoning type.
Table 8: Variations of the intervention, macro-averaged over the 25 tasks: accuracy in percent and mean tokens generated per question. Unless the block says otherwise, the last state of the direct prompt is rotated once, at layer 29 of LLaMA 3.1 8B and layer 25 of Qwen 2.5 7B . The main text uses t=1 towards dq(0.5) , which gives 85% for LLaMA 3.1 8B and 86% for Qwen 2.5 7B .
Table 9: Rotating the last state of the direct prompt once, at t=1 , towards cq+ at each layer of the late band; columns as in Table 8 . The main text uses layer 29 of LLaMA 3.1 8B and layer 25 of Qwen 2.5 7B .
Table 10: Accuracy in percent per task after rotating the state of the direct prompt ( t=1 ; layer 29 of LLaMA 3.1 8B , layer 25 of Qwen 2.5 7B ) towards the question’s recovering mean cq+ , contrastive direction dq(0.5) and failed mean cq− , and towards the task-level direction dT(0.5) of its task, against the direct (Dir.) and CoT prompts. Tok. is the mean number of tokens generated under dq(0.5) . Questions without a contrastive direction count as incorrect. The higher of CoT and dq(0.5) is in bold, compared before rounding.
GSM8K
Question
Ali had 21.Leilagavehimhalfofher100. How much does Ali have now?
Direct
121 wrong
dq(0.5)
add 21and50. 21+50 = $71 correct
CoT
Let’s break it down step by step: 1. Ali initially had 21.2.Leilagavehimhalfofher100, which is 100/2=50. … Final Answer: $71 correct
Appendix
Table 11: Generations of LLaMA 3.1 8B under the direct prompt, with its last state pointed towards dq(0.5) at layer 29 ( t=1 ), and under CoT. In the first four examples the direct prompt is wrong, and the direction reaches the answer of CoT in a few tokens. In the next two, the CoT and direct prompts are both wrong, and the direction reaches the correct answer with a short computation. Over all 25 tasks, both prompts are wrong on 1,867 of the 9,120 questions for LLaMA 3.1 8B and on 1,441 for Qwen 2.5 7B , and the direction reaches the correct answer on 1,192 (64%) and 1,014 (70%) of them. The last two come from tasks where CoT falls below the direct prompt: the direct prompt is already correct, the steps of CoT sway the model away from this answer, and the direction keeps it. Questions and CoT outputs are truncated (…) for space.
Figure 7: The task-level direction dT(0.5) of each task (rows) applied to the questions of each task (columns) on Qwen 2.5 7B , at layer 25 ( t=1 ). Each cell is the change in accuracy over the direct prompt on the column’s task, in points. Outlined cells on the diagonal apply a task’s direction to its own questions (Figure 5 ).
Whether intermediate reasoning is computationally useful or merely explanatory depends on whether chain-of-thought (CoT) tokens contain task-relevant information. We present a mechanistic causal analysis of CoT on GSM8K using activation patching: transferring token-level hidden states from a CoT generation to a direct-answer run for the same question, then measuring the effect on final-answer accuracy. Across models, generating after patching yields substantially higher accuracy than both direct-answer prompting and the original CoT trace, revealing that individual CoT tokens can encode sufficient information to recover the correct answer, even when the original trace is incorrect. This task-relevant information is more prevalent in correct than incorrect CoT runs and is unevenly distributed across tokens, concentrating in mid-to-late layers and appearing earlier in the reasoning trace. Moreover, patching language tokens such as verbs and entities carry task-solving information that steers generation toward correct reasoning, whereas mathematical tokens encode answer-proximal content that rarely succeeds. Patched outputs are often shorter and yet exceed the accuracy of a full CoT trace, suggesting complete reasoning chains are not always necessary. Together, these findings demonstrate that CoT encodes recoverable, token-level problem-solving information, offering new insight into how reasoning is represented and where it breaks down.
Houman Mehrafarin, Amit Parekh, Ioannis Konstas
Heriot-Watt University, Edinburgh, United Kingdom · University of Edinburgh, Edinburgh, United Kingdom
Chain-of-Thought (CoT) prompting remains the standard baseline for evaluating models' reasoning abilities. Originally, this technique was introduced to elicit step-by-step reasoning from large language models (LLMs), which would otherwise tend to directly output the final answer. However, many modern LLMs produce CoT-style responses natively when presented with reasoning tasks, which made us revisit the effectiveness of standard CoT prompting. We evaluate several modern mid-sized language models on a math problem-solving task and find that models specialized for reasoning achieve better performance in a simple zero-shot setting than when using few-shot CoT examples - significantly surpassing officially reported results at no additional cost (e.g., from ∼77% to ∼84% for Mathstral on GSM8K). For the tested general-purpose model, a zero-shot CoT prompt is also sufficient to outperform a few-shot CoT baseline. We attribute this to a `guidance-distraction' tradeoff: standard CoT prompting also demands style adaptation, formatting compliance, and potentially undesired contextualization, which can distract models from the core reasoning task. Our findings suggest that using standard CoT prompting increasingly acts as a source of distraction as models grow stronger.
As chain of thought (CoT) has become central to scaling reasoning capabilities in large language models (LLMs), it has also emerged as a promising tool for interpretability, suggesting the opportunity to understand model decisions through verbalized reasoning. However, the utility of CoT toward interpretability depends upon its faithfulness---whether the model's stated reasoning reflects the underlying decision process. We provide mechanistic evidence that instruction-tuned models often determine their answer before generating CoT. Training linear probes on residual stream activations at the last token before CoT, we can predict the model's final answer with >0.9 AUC on most tasks. We find that these directions are not only predictive, but also causal: steering activations along the probe direction often flips model answers, with flip rates substantially exceeding norm-matched orthogonal baselines across most model-dataset pairs. When steering induces incorrect answers, we observe two distinct failure modes: confabulation (fabricating false premises) and non-entailment (stating correct premises but drawing unsupported conclusions). While post-hoc reasoning may be instrumentally useful when the model has a correct pre-CoT belief, these failure modes suggest it can result in undesirable behaviors when reasoning from a false belief.
Kyle Cox, Darius Kianersi, Adrià Garriga-Alonso
Independent · Cambridge Boston Alignment Initiative (CBAI)