The Spectral Geometry of Thought: Phase Transitions, Instruction Reversal, Token-Level Dynamics, and Perfect Correctness Prediction in How Transformers Reason

Yi Liu

cs.LGApr 3, 2026

The Spectral Geometry of Thought: Phase Transitions, Instruction Reversal, Token-Level Dynamics, and Perfect Correctness Prediction in How Transformers Reason

Authors: Yi Liu

Abstract

We discover that large language models exhibit spectral phase transitions in their hidden activation spaces when engaging in reasoning versus factual recall. Through systematic spectral analysis across 11 models spanning 5 architecture families (Qwen, Pythia, Phi, Llama, DeepSeek-R1), we identify seven core phenomena: (1)Reasoning Spectral Compression -- 9/11 models show significantly lower $α$ for reasoning ( $p < 0.05$ ), with larger effects in stronger models; (2)Instruction Tuning Spectral Reversal -- base models show reasoning $α<$ factual $α$ , while instruction-tuned models reverse this relationship; (3)Architecture-Dependent Generation Taxonomy -- prompt-to-response shifts partition into expansion, compression, and equilibrium regimes; (4)Spectral Scaling Law -- $α_\text{reasoning} \propto -0.074 \ln N$ across 4 Qwen base models ( $R^2 = 0.46$ ); (5)Token-Level Spectral Cascade -- per-token alpha tracking reveals local synchronization that decays exponentially with layer distance, and is weaker for reasoning than factual tasks; (6)Reasoning Step Spectral Punctuation -- phase-transition signatures align with reasoning step boundaries; and (7)~Spectral Correctness Prediction -- spectral $α$ alone achieves AUC $= 1.000$ (Qwen2.5-7B, late layers) and mean AUC $= 0.893$ across 6 models in predicting correctness before the final answer is generated. Together, these findings establish a comprehensive spectral theory of reasoning in transformers, revealing that the geometry of thought is universal in direction, architecture-specific in dynamics, and predictive of outcome.

Figures & tables

Model	Family	Params	Layers	Type	Norm	Attention
Qwen2.5-0.5B	Qwen	0.5B	24	Base	RMSNorm	GQA
Qwen2.5-3B	Qwen	3B	36	Base	RMSNorm	GQA
Qwen2.5-7B	Qwen	7B	28	Base	RMSNorm	GQA
Qwen2.5-1.5B-Instruct	Qwen	1.5B	28	Instruct	RMSNorm	GQA
Qwen2.5-3B-Instruct	Qwen	3B	36	Instruct	RMSNorm	GQA
DeepSeek-R1-1.5B	DeepSeek-R1	1.5B	28	Reasoning	RMSNorm	GQA

Table 1: Model inventory: 11 models across 5 architecture families and 4 training paradigms.

Figure 1: Cross-Model Spectral Delta. Reasoning–Factual $\Delta\alpha$ across all 11 models. Negative values indicate reasoning spectral compression (lower $\alpha$ , more distributed representations). 9/11 models show significant compression ( $p<0.05$ ); the two exceptions are Qwen instruct models showing reversal (Finding 2). Error bars indicate standard error across tasks.

Model	Type	$\alpha_{\text{R}}$	$\alpha_{\text{F}}$	$\Delta\alpha$	$\Delta\alpha_{R}$	$p$
Qwen Base (reasoning = more distributed)
Qwen2.5-0.5B	Base	1.159	1.481	$-0.219$	$+0.287$	$<10^{-9}$
Qwen2.5-3B	Base	0.985	1.398	$-0.318$	$+0.301$	$<10^{-5}$
Qwen2.5-7B	Base	0.832	1.512	$-0.464$	$+0.221$	$<10^{-5}$
Qwen Instruct (reasoning = more concentrated or mixed )
Qwen2.5-1.5B-I	Instruct	0.946	1.685	$+0.206$	$+0.307$	$<10^{-28}$

Table 2: Reasoning vs. Factual spectral

\alpha

across 11 models.

\Delta\alpha

= Reasoning

-

Factual (negative = more distributed for reasoning). Response-only

\Delta\alpha_{R}

controls for prompt effects.

Figure 2: Instruction Tuning Spectral Reversal. (A) Per-layer $\alpha$ profiles for Qwen2.5-3B base vs. instruct: base separates reasoning (solid) below factual (dashed), while instruct shows overlap/reversal. (B) Per-layer $\Delta\alpha$ in the response phase: base (blue) is consistently positive while instruct (red) is consistently negative in early layers. (C) Summary across model pairs showing the reversal pattern.

Figure 3: Generation Shift Taxonomy. Prompt-to-response $\alpha$ shift across 11 models. Models partition into three regimes: Expansion (negative shift, blue), Equilibrium (near-zero), and Compression (positive shift, red). The partition aligns with normalization architecture more than model family.

Figure 4: Spectral Scaling Law. Log-linear relationship between model size $N$ (parameters) and spectral reasoning-factual delta $\Delta\alpha$ across 4 Qwen base models. The fitted line $\Delta\alpha=-0.074\ln N-0.317$ ( $R^{2}=0.46$ ) shows that larger models achieve greater spectral separation between reasoning and factual representations.

Figure 5: Token-Level Spectral Dynamics. (A) Per-token $\alpha$ at 5 target layers (0, 9, 18, 27, 35) during multi-step math generation (Qwen2.5-3B-Instruct). Solid lines: raw; dashed: smoothed. (B) Inter-layer variance over generation, showing fluctuations that correlate with reasoning step transitions.

Layer Pair	Distance	$\bar{\rho}$	Reasoning vs. Factual $\Delta\rho$
L9–L18	9	0.855	$-0.075$
L18–L27	9	0.826	$+0.012$
L27–L35	8	0.466	$-0.191$
L0–L9	9	0.439	$-0.064$
L9–L27	18	0.675	$-0.053$
L0–L18	18	0.324	$+0.009$

Table 3: Mean cross-layer gradient correlation (5 tasks, Qwen2.5-3B-Instruct). Adjacent layers are highly synchronized; distant layers are nearly independent.

Figure 6: Token-Level Spectral Dynamics and Reasoning Step Punctuation. Per-token spectral $\alpha$ trajectories across multiple layers during generation, showing how reasoning induces characteristic spectral dynamics. Gradient spikes in $\alpha$ coincide with reasoning step boundaries (e.g., “Step 1:”, “therefore”, paragraph breaks), while factual tasks show concentrated initial retrieval followed by stable generation.

Figure 7: Spectral Prediction of Reasoning Correctness. (A) Best AUC by model using spectral $\alpha$ alone as a binary classifier for answer correctness. Qwen2.5-7B achieves perfect AUC $=1.000$ ; mean across 6 models is $0.893$ . Labels indicate which phase (early/mid/late/response layers) gives best prediction. (B) Phase-wise AUC heatmap showing that late and response-phase layers are generally most predictive.

Figure 8: Layer-wise Spectral Prediction AUC for Qwen2.5-7B. AUC rises monotonically from layer 0 ( $0.937$ ) to layer 23 ( $0.999$ ). The late-layer AUC reaches $1.000$ (response phase), meaning that at layer 23, spectral $\alpha$ is a perfect binary predictor of reasoning correctness in this model.

Model	Acc.	Best AUC	Best Phase	Best Layer AUC
Qwen2.5-7B	65.0%	1.000	Late	0.999 (L23)
Qwen2.5-3B	56.0%	0.995	Response	0.978 (L20)
DeepSeek-R1-1.5B	29.0%	0.947	Late	0.761 (L0)
Phi-2	27.0%	0.974	Mid	0.960 (L26)
Qwen2.5-3B-Instruct	36.0%	0.945	Response	0.808 (L30)
Pythia-2.8B	0.0%	0.500	—	0.5

Table 4: Spectral prediction of reasoning correctness across 6 models (5-fold CV). AUC

=1.000

means perfect prediction; chance is

0.5

Figure 9: Model Accuracy vs. Spectral Predictability. Scatter plot showing the relationship between task accuracy and best spectral AUC across 6 models. More capable models (higher accuracy) exhibit higher spectral predictability (higher AUC), suggesting that reasoning competence creates a more distinctive spectral signature. The degenerate case (Pythia-2.8B, 0% accuracy) cannot be predicted by definition.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Task Name	Description
Reasoning	2-step arithmetic	Two-step arithmetic chain (e.g., $3\times 7+5$ )
	3-step arithmetic	Three-step arithmetic chain
	4-step arithmetic	Four-step arithmetic chain
	Linear equations	Solve for $x$ in $ax+b=c$
	Ratio problems	Word problems involving ratios and proportions
	Syllogisms	Classical logical syllogisms

Appendix

Table 5: Complete task inventory: 21 tasks across 3 categories.

Model	Mean $R^{2}$	Min $R^{2}$
Qwen2.5-0.5B	$0.91\pm 0.04$	0.78
Qwen2.5-3B	$0.93\pm 0.03$	0.82
Qwen2.5-7B	$0.94\pm 0.03$	0.84
Qwen2.5-1.5B-Instruct	$0.90\pm 0.05$	0.76
Qwen2.5-3B-Instruct	$0.92\pm 0.04$	0.80
DeepSeek-R1-1.5B	$0.89\pm 0.05$	0.74

Appendix

Table 6: Power-law fit quality (

R^{2}

) across models. Reported as mean

\pm

std across all layers and tasks.

Model	$\alpha_{R}$	$\alpha_{F}$	$\Delta\alpha_{RF}$	$\Delta\alpha_{PR}$	$\alpha_{\text{corr}}$	$\alpha_{\text{incorr}}$
Qwen2.5-0.5B	1.159	1.481	$-0.219$	$-0.321$	—	—
Qwen2.5-3B	1.045	1.363	$-0.318$	$-0.413$	1.057	1.035
Qwen2.5-7B	0.964	1.428	$-0.464$	$-0.680$	0.942	1.014
Qwen2.5-1.5B-I	1.114	1.587	$+0.206$	$-0.739$	1.119	1.106
Qwen2.5-3B-I	0.949	1.409	$+0.121$	$-0.459$	—	—
DS-R1-1.5B	1.402	1.653	$-0.291$	$+0.013$	0.928	1.554

Appendix

Table 7: Complete spectral statistics across all 11 models.

\alpha_{R}

: mean reasoning alpha;

\alpha_{F}

: mean factual alpha;

\Delta\alpha_{RF}

: reasoning

-

factual;

\Delta\alpha_{PR}

: prompt-to-response shift;

\alpha_{\text{corr}}

: correct trials mean alpha;

\alpha_{\text{incorr}}

: incorrect trials mean alpha.

Figure 10: Architecture Family Spectral Profiles. Per-layer spectral

\alpha

profiles across architecture families, showing the characteristic layer-wise trajectory for each family. Qwen models exhibit a pronounced middle-layer dip in

\alpha

; Pythia models show a monotonic increase; Phi models display a U-shaped profile. These family-specific profiles interact with the reasoning/factual task effect (Finding 1) in architecture-dependent ways.

Figure 11: Per-Layer Reasoning–Factual $\Delta\alpha$ Heatmap. Heatmap showing the reasoning–factual spectral delta ( $\Delta\alpha$ ) at each layer for all 11 models. Blue indicates reasoning compression (lower $\alpha$ ); red indicates reasoning expansion (higher $\alpha$ ). The Qwen instruct models show a distinctive reversal pattern (red in early/middle layers), while base models are predominantly blue. The layer-resolved view reveals that the reversal effect (Finding 2) is strongest in early-to-middle layers.

Category	$n$	Correct	Acc.	$\bar{\alpha}_{\text{corr}}$ (L27)	$\bar{\alpha}_{\text{incorr}}$ (L27)	$\Delta$
Code tracing	10	8	80%	0.726	0.755	$-0.028$
Commonsense	10	4	40%	0.722	0.697	$+0.026$
Multi-hop math	10	2	20%	0.690	0.656	$+0.034$
Logical elim.	10	6	60%	0.745	0.682	$+0.063$
Overall	40	20	50%	0.728	0.684	$+0.044$

Appendix

Table 8: OOD validation results for Qwen2.5-7B across 4 novel task categories (40 problems total). Per-category accuracy, mean response

\alpha

at checked layers, and correct-vs-incorrect

\alpha

separation.

Category	$n$	Correct	Acc.	$\bar{\alpha}_{\text{corr}}$ (L35)	$\bar{\alpha}_{\text{incorr}}$ (L35)	$\Delta$
Code tracing	10	2	20%	0.734	0.790	$-0.057$
Commonsense	10	6	60%	0.777	0.736	$+0.041$
Multi-hop math	10	4	40%	0.726	0.726	$+0.000$
Logical elim.	10	6	60%	0.848	0.771	$+0.077$
Overall	40	18	45%	0.791	0.784	$+0.007$

Appendix

Table 9: OOD validation results for Qwen2.5-3B across 4 novel task categories.

Representation Geometry in Language Models

Representation geometry in language models studies how embeddings and hidden states organize concepts, computation, and behavior.

See all 417 papers

Geometric Signatures of Reasoning: A Spectral Perspective on Task Hardness
Jul 2, 2026Aria Masoomi, Mahsa Bazzaz, Adel Javanmard +1LLM Evaluation Representation Geometry in Language Models
1Northeastern University · University of Southern California · 3Google Research
Spatiotemporal Hidden-State Dynamics as a Signature of Internal Reasoning in Large Language Models
May 3, 2026Kotaro Furuya, Takahito TanimuraTransformer Interpretability LLM Evaluation
Research and Development Group, Hitachi, Ltd. 1-280 Higashi-Koigakubo, Kokubunji-shi, Tokyo 185-8601, Japan
Quantifying the Stability of Multi-Step Reasoning via Error Amplification
Oct 5, 2026Dongyue Li, Ziniu Zhang, Minxuan Duan +1Efficient Language Model Reasoning CoT Compression
Northeastern University Boston, MA

Jul 2, 2026J/K move · Enter open · S save

Geometric Signatures of Reasoning: A Spectral Perspective on Task Hardness

Aria Masoomi, Mahsa Bazzaz, Adel Javanmard, Vahab Mirrokni

1Northeastern University · University of Southern California · 3Google Research

LLM Evaluation Representation Geometry in Language Models LLM Mathematical Reasoning LLM Reasoning Reasoning Trace Analysis

Open paper AskPDF arXiv

Chain-of-thought (CoT) reasoning enables large language models (LLMs) to solve complex problems by generating intermediate reasoning steps. While much attention has been paid to the length and content of these reasoning chains, far less is known about their internal geometry. We study the geometry of CoT trajectories in the hidden state space of transformer models, formalizing each reasoning chain as a discrete curve in

\mathbb{R}^d

and characterizing it through spectral, positional, and kinematic geometric functionals. We introduce the effective dimension

d_ρ

as a measure of trajectory complexity and show theoretically that trajectories with flatter eigenvalue spectra correspond to harder tasks, as they explore more of the hidden dimensions. Lastly, we explore how kinematic features of the trajectory, mean position, positional dispersion, initial and current hidden states, mean velocity, mean speed, and speed dispersion, can be used to predict solution correctness before generation is complete, and may inform future early-stopping strategies. Experimentally, on mathematical reasoning problems from the MATH500 dataset,

d_ρ

achieves

0.93

AUC in distinguishing easy from hard problems, while kinematic features potentially can predict correctness from only the first

20\%

of generated tokens. These correctness signatures transfer across questions of varying difficulty, establishing that the shape of a model's internal reasoning trajectory is a principled window into both task hardness and solution quality.

Abstract

Representation Geometry in Language Models

Explore similar work

Geometric Signatures of Reasoning: A Spectral Perspective on Task Hardness

Spatiotemporal Hidden-State Dynamics as a Signature of Internal Reasoning in Large Language Models

Quantifying the Stability of Multi-Step Reasoning via Error Amplification