How Does Chain-of-Thought Supervision Teach Models to Reason? A Theoretical Analysis
Authors: Kaito Takanami, Cengiz Pehlevan
Organizations: Department of Physics, Graduate School of Science, The University of Tokyo Tokyo, Japan · John A. Paulson School of Engineering and Applied Sciences, Harvard University Cambridge, MA, USA · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University Cambridge, MA, USA · Center for Brain Science, Harvard University Cambridge, MA, USA
Training with chain-of-thought (CoT) supervision is an effective approach for teaching models multi-step reasoning procedures. At inference time, however, models may need to continue reasoning beyond the finite teacher CoT sequences observed during training, and longer reasoning can instead degrade performance, a phenomenon known as overthinking. How the amount and structure of training-time CoT give rise to such failures remains theoretically poorly understood. To study this problem, we introduce an in-context weight prediction model trained on finite gradient descent trajectories for linear regression. Using high-dimensional asymptotic analysis, we show that the success and failure modes of long-horizon reasoning undergo phase transitions depending on the context length and the total amount of training CoT. We further distinguish two mechanisms of overthinking, error accumulation and error amplification, and show that with short training contexts, increasing the training CoT budget can worsen long-horizon error before eventually stabilizing reasoning. Experiments with fully trained linear and multi-head softmax attention further show that these qualitative behaviors persist. Our results provide a theoretical account linking properties of training-time CoT to inference-time failures and offer guidance for designing teacher CoT that supports stable reasoning.
Figures & tables
Figure 1: Training CoT budget and context length control GD recovery and test-time error growth. (A) Theoretical phase diagram in the context ratio α and training CoT budget φ=τ(K−1) . The four regions correspond to GD recovery, bounded error, exponential divergence, and quadratic divergence. (B) Numerical experiments (circles) and theoretical prediction (solid line) on excess generalization error E(t)−EGD(t) at α=2 . (C) Numerical experiments of the generalization error at α=0.625 . (D) Typical test-time error dynamics in the four phases; the dotted lines show the theoretical long-time predictions, and the inset highlights the quadratic and exponential regimes at large t . (B,C) The vertical dotted lines show the theoretical phase boundaries. Parameters: (B,C) D=96 , K=2 , η=0.15 and σ2=0.01 . (B-D) Error bars represent one standard error across eight independent runs.
Figure 2: Training CoT budget produces non-monotonic long-horizon performance in fully trained attention. Numerical experiments of the test-time generalization error for (A) Single-head linear attention with learned Q/K/V and (B) Two-head softmax attention with learned Q/K/V/O. Parameters: (A,B) D=32 , α=0.75 , K=2 , η=0.1 , and σ=0.1 . Error bars represent one standard error of the 8 independent experiments per point.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Regime
Finite-time behavior (Result 4.1 )
Long-time behavior (Result 4.2 )
Derivation
α>1,φ>1
GD
GD
Appendix C
α>1,0<φ<1
Projected GD
Θ(t2)
Appendix D
0<α<1,0<φ<φ−
–
Θ(t2)
Proposition E.8
0<α<1,φ−<φ<φ+
–
exp(Θ(ηt))
Proposition E.8
0<α<1,φ+<φ<1/α
–
O(1)
Proposition E.8
0<α<1,αφ>1
GD
GD
Proposition E.4
Appendix
Table 1: Summary of the parameter regimes.
Figure 3: Schematic spectral evolution across the long-time phases. (A–F) Densities of the real parts of the eigenvalues of ID+HinA^ . (G–L) Densities of the real parts of the eigenvalues of Ac/η . Red arrows in (A–C) and (G–I) mark the rank-deficient atoms at eigenvalues 1 and 0 , respectively.
Chain-of-thought (CoT) reasoning has become a widely used mechanism for eliciting multi-step reasoning in large language models by generating intermediate reasoning steps at inference time. Yet the scaling behavior of generalization with CoT depth remains poorly understood. To address this question, we study a theoretically solvable model of CoT for in-context weight prediction in linear regression, where test-time reasoning is represented as an iterative refinement of the weight-parameter estimate. Using tools from random matrix theory under high-dimensional asymptotics, we derive an exact formula for the generalization error as a function of reasoning depth, pretraining data amount, and context length. Our analysis reveals a sharp phase transition separating exponential and polynomial improvement, saturation, and overthinking, and characterizes how the optimal reasoning depth scales. We further show that deeper reasoning is most effective with sufficiently rich pretraining and in-context information, whereas limited pretraining or context makes longer reasoning prone to error amplification or saturation. We also validate these predictions through experiments on fully learned linear attention and softmax attention models. Our results provide a unified theoretical account of how test-time CoT depth affects generalization.
Kaito Takanami, Cengiz Pehlevan
Department of Physics, Graduate School of Science, The University of Tokyo Tokyo, Japan · John A. Paulson School of Engineering and Applied Sciences, Harvard University Cambridge, MA, USA · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University Cambridge, MA, USA +1
Chain-of-Thought (CoT) prompting is a widely used inference-time technique for improving reasoning, yet its gains are uneven across tasks. We analyze when and why CoT helps by modeling the step-wise reasoning trajectory as a Markov chain. Each intermediate step is a state and the dependence between steps is captured by a transition kernel. Our theory identifies transition alignment, whether instances share a common step-wise transition kernel, as the key determinant of CoT's effectiveness. When transitions are identical across steps, CoT reduces inference-time sample complexity: fewer context sample trajectories suffice to recover the final decision. In contrast, when transitions differ across steps, these gains can vanish. We further quantify how noise in intermediate steps modulates CoT's benefit. Beyond theory, we design synthetic benchmarks that isolate these factors to complement prior results on real-world tasks and to empirically validate our predictions.
Chain-of-thought (CoT) reasoning allows humans to inspect how large language models reach their answers, and oversee model behaviour. This reasoning comes at an increased inference cost, motivating efficient methods that train models to solve tasks using fewer tokens. However, a common concern is that such training may cause models to skip important reasoning steps, so the CoT no longer faithfully reflects the model's decision. It is unclear whether or when this occurs in practice, since different efficiency methods apply length pressure to models' CoT in distinct ways, and faithfully explaining a model's decision takes more tokens on some tasks than others. To understand these dynamics, we fine-tune a variety of models with three methods that apply length pressure differently, namely a fixed generation budget, a per-example length target, and a group-relative length reward. We evaluate how efficient reasoning affects CoT faithfulness (i.e., how well the CoT reflects model decisions on related inputs) and monitorability (i.e., whether the CoT reveals when input interventions alter the output). We find that it affects faithfulness and monitorability differently. Faithfulness falls in most settings, primarily because the trained models are less consistent. Monitorability is more robust, as models keep acknowledging the influence on their answer even when the CoT is much shorter.
Samuel Lewis-Lim, Xingwei Tan, Mario Sanger +2
School of Computer Science, University of Sheffield · AstraZeneca