How Does Chain-of-Thought Supervision Teach Models to Reason? A Theoretical Analysis

Kaito Takanami; Cengiz Pehlevan

cs.LGNov 1, 2026

How Does Chain-of-Thought Supervision Teach Models to Reason? A Theoretical Analysis

Authors: Kaito Takanami, Cengiz Pehlevan

Organizations: Department of Physics, Graduate School of Science, The University of Tokyo Tokyo, Japan · John A. Paulson School of Engineering and Applied Sciences, Harvard University Cambridge, MA, USA · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University Cambridge, MA, USA · Center for Brain Science, Harvard University Cambridge, MA, USA

Abstract

Training with chain-of-thought (CoT) supervision is an effective approach for teaching models multi-step reasoning procedures. At inference time, however, models may need to continue reasoning beyond the finite teacher CoT sequences observed during training, and longer reasoning can instead degrade performance, a phenomenon known as overthinking. How the amount and structure of training-time CoT give rise to such failures remains theoretically poorly understood. To study this problem, we introduce an in-context weight prediction model trained on finite gradient descent trajectories for linear regression. Using high-dimensional asymptotic analysis, we show that the success and failure modes of long-horizon reasoning undergo phase transitions depending on the context length and the total amount of training CoT. We further distinguish two mechanisms of overthinking, error accumulation and error amplification, and show that with short training contexts, increasing the training CoT budget can worsen long-horizon error before eventually stabilizing reasoning. Experiments with fully trained linear and multi-head softmax attention further show that these qualitative behaviors persist. Our results provide a theoretical account linking properties of training-time CoT to inference-time failures and offer guidance for designing teacher CoT that supports stable reasoning.

Figures & tables

Figure 1: Training CoT budget and context length control GD recovery and test-time error growth. (A) Theoretical phase diagram in the context ratio

\alpha

and training CoT budget

\varphi=\tau(K-1)

. The four regions correspond to GD recovery, bounded error, exponential divergence, and quadratic divergence. (B) Numerical experiments (circles) and theoretical prediction (solid line) on excess generalization error

\mathcal{E}(t)-\mathcal{E}_{\mathrm{GD}}(t)

\alpha=2

. (C) Numerical experiments of the generalization error at

\alpha=0.625

. (D) Typical test-time error dynamics in the four phases; the dotted lines show the theoretical long-time predictions, and the inset highlights the quadratic and exponential regimes at large

t

. (B,C) The vertical dotted lines show the theoretical phase boundaries. Parameters: (B,C)

D=96

K=2

\eta=0.15

and

\sigma^{2}=0.01

. (B-D) Error bars represent one standard error across eight independent runs.

Figure 2: Training CoT budget produces non-monotonic long-horizon performance in fully trained attention. Numerical experiments of the test-time generalization error for (A) Single-head linear attention with learned Q/K/V and (B) Two-head softmax attention with learned Q/K/V/O. Parameters: (A,B)

D=32

\alpha=0.75

K=2

\eta=0.1

, and

\sigma=0.1

. Error bars represent one standard error of the 8 independent experiments per point.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Regime	Finite-time behavior (Result 4.1 )	Long-time behavior (Result 4.2 )	Derivation
$\alpha>1,\ \varphi>1$	GD	GD	Appendix C
$\alpha>1,\ 0<\varphi<1$	Projected GD	$\Theta(t^{2})$	Appendix D
$0<\alpha<1,\ 0<\varphi<\varphi_{-}$	–	$\Theta(t^{2})$	Proposition E.8
$0<\alpha<1,\ \varphi_{-}<\varphi<\varphi_{+}$	–	$\exp(\Theta(\eta t))$	Proposition E.8
$0<\alpha<1,\ \varphi_{+}<\varphi<1/\alpha$	–	$O(1)$	Proposition E.8
$0<\alpha<1,\ \alpha\varphi>1$	GD	GD	Proposition E.4

Appendix

Table 1: Summary of the parameter regimes.

Figure 3: Schematic spectral evolution across the long-time phases. (A–F) Densities of the real parts of the eigenvalues of

\bm{\mathrm{I}}_{D}+\bm{\mathrm{H}}^{\mathrm{in}}\hat{\bm{\mathrm{A}}}

. (G–L) Densities of the real parts of the eigenvalues of

\bm{\mathrm{A}}_{c}/\eta

. Red arrows in (A–C) and (G–I) mark the rank-deficient atoms at eigenvalues

1

and

0

, respectively.

CoT Supervision

See all 20 papers

An Asymptotic Theory of Chain-of-Thought in In-Context Learning
Jun 2, 2026Kaito Takanami, Cengiz PehlevanNeural Network Generalization CoT Reasoning
Department of Physics, Graduate School of Science, The University of Tokyo Tokyo, Japan · John A. Paulson School of Engineering and Applied Sciences, Harvard University Cambridge, MA, USA · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University Cambridge, MA, USA +1
When does Chain-of-Thought Help: A Markovian Perspective
Feb 27, 2026Zihan Wang, Yijun Dong, Qi LeiMarkov Chains CoT Reasoning
New York University
Efficient Reasoning Training Does Not Always Harm CoT Faithfulness and Monitorability
Oct 2, 2026Samuel Lewis-Lim, Xingwei Tan, Mario Sanger +2Faithfulness of Language Model Reasoning Efficient Language Model Reasoning
School of Computer Science, University of Sheffield · AstraZeneca

Jun 2, 2026J/K move · Enter open · S save

An Asymptotic Theory of Chain-of-Thought in In-Context Learning

Kaito Takanami, Cengiz Pehlevan

Department of Physics, Graduate School of Science, The University of Tokyo Tokyo, Japan · John A. Paulson School of Engineering and Applied Sciences, Harvard University Cambridge, MA, USA · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University Cambridge, MA, USA+1

Neural Network Generalization CoT Reasoning In-Context Learning

Open paper AskPDF arXiv

Chain-of-thought (CoT) reasoning has become a widely used mechanism for eliciting multi-step reasoning in large language models by generating intermediate reasoning steps at inference time. Yet the scaling behavior of generalization with CoT depth remains poorly understood. To address this question, we study a theoretically solvable model of CoT for in-context weight prediction in linear regression, where test-time reasoning is represented as an iterative refinement of the weight-parameter estimate. Using tools from random matrix theory under high-dimensional asymptotics, we derive an exact formula for the generalization error as a function of reasoning depth, pretraining data amount, and context length. Our analysis reveals a sharp phase transition separating exponential and polynomial improvement, saturation, and overthinking, and characterizes how the optimal reasoning depth scales. We further show that deeper reasoning is most effective with sufficiently rich pretraining and in-context information, whereas limited pretraining or context makes longer reasoning prone to error amplification or saturation. We also validate these predictions through experiments on fully learned linear attention and softmax attention models. Our results provide a unified theoretical account of how test-time CoT depth affects generalization.

Abstract

CoT Supervision

Explore similar work

An Asymptotic Theory of Chain-of-Thought in In-Context Learning

When does Chain-of-Thought Help: A Markovian Perspective

Efficient Reasoning Training Does Not Always Harm CoT Faithfulness and Monitorability