Beyond Scalars: Evaluating and Understanding LLM Reasoning via Geometric Progress and Stability

Xinyan Jiang; Ninghao Liu; Di Wang; Lijie Hu

cs.AIMar 11, 2026

Beyond Scalars: Evaluating and Understanding LLM Reasoning via Geometric Progress and Stability

Authors: Xinyan Jiang, Ninghao Liu, Di Wang, Lijie Hu

Organizations: Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · King Abdullah University of Science and Technology · Shanghai Advanced Research Institute, Chinese Academy of Sciences, Shanghai, China · University of Chinese Academy of Sciences, Beijing, China ·

Abstract

Evaluating LLM reliability via scalar probabilities often fails to capture the structural dynamics of reasoning. We introduce TRACED, a framework that assesses reasoning quality through theoretically grounded geometric kinematics. By decomposing reasoning traces into Progress (displacement) and Stability (curvature), we reveal a distinct topological divergence: correct reasoning manifests as high-progress, stable trajectories, whereas hallucinations are characterized by low-progress, unstable patterns (stalled displacement with high curvature fluctuations). Leveraging these signatures, our probabilistic framework achieves competitive performance and superior robustness across diverse benchmarks. Crucially, TRACED bridges geometry and cognition by mapping high curvature to ''Hesitation Loops'' and displacement to ''Certainty Accumulation'', offering a physical lens to decode the internal dynamics of machine thought.

Figures & tables

Figure 1: Topological Divergence of Reasoning Quality. Joint distribution of net displacement ( $M$ ) and curvature ( $K$ ) across Structured and Open-Ended domains. The visualization confirms a consistent separation: correct reasoning traces (blue) exhibit a high-displacement, low-curvature pattern, while incorrect chains (red) are characterized by low-displacement stagnation and high-curvature oscillations.

DeepSeek-R1-Llama-8B
	Fables			GPQA			GSM8K			MATH			Social_iqa			Theorem
Method	AUROC $\uparrow$	AUPR $\uparrow$	FPR@95 $\downarrow$	AUROC $\uparrow$	AUPR $\uparrow$	FPR@95 $\downarrow$	AUROC $\uparrow$	AUPR $\uparrow$	FPR@95 $\downarrow$	AUROC $\uparrow$	AUPR $\uparrow$	FPR@95 $\downarrow$	AUROC $\uparrow$	AUPR $\uparrow$	FPR@95 $\downarrow$	AUROC $\uparrow$	AUPR $\uparrow$	FPR@95 $\downarrow$
MSP	0.6044	0.6427	0.8919	0.3837	0.4750	0.8887	0.7424	0.7425	0.8235	0.6276	0.6203	0.8125	0.5867	0.6237	0.9194	0.5270	0.4951	0.8737
Perplexity	0.6719	0.6581	0.8568	0.3831	0.4107	0.8767	0.6096	0.6088	0.8824	0.5808	0.5628	0.8688	0.5911	0.6174	0.8065	0.5229	0.5552	0.8211
Entropy	0.6156	0.6481	0.8378	0.4857	0.5159	0.8823	0.6863	0.6916	0.8235	0.6041	0.5921	0.8875	0.5925	0.6095	0.8194	0.5111	0.4951	0.8737
LR Probe	0.7177	0.6539	0.8297	0.7588	0.5451	0.8571	0.7995	0.8195	0.7059	0.7471	0.6541	0.8625	0.7097	0.6883	0.7091	0.8435	0.6497	0.8158

Table 1: Comparison of reasoning quality assessment methods across multiple models and tasks. See Appendix N for statistical uncertainty analysis (95% CIs) confirming the stability of these results.

Figure 2: Universality and Generalization Analysis. (a) Universal Signature: A single global fit model derived from aggregated data achieves competitive AUPR across diverse tasks, supporting the existence of a task-agnostic geometric signature. (b) Cross-Domain Adaptation: Dumbbell plot comparing Direct Zero-shot Transfer (blue circles), Aligned Transfer (purple squares), and Supervised In-domain Upper Bound (red stars). Results confirm that the geometric alignment significantly bridges the performance gap caused by distribution shifts.

Metric	Easy ( $L\leq 4$ )	Medium ( $5\leq L\leq 8$ )	Hard ( $L>8$ )	Gap ( $\Delta$ )
AUROC ( $\uparrow$ )	0.775	0.748	0.766	2.7%
AUPR ( $\uparrow$ )	0.708	0.710	0.723	1.5%
FPR@95 ( $\downarrow$ )	0.660	0.673	0.685	2.5%

Table 2: Robustness of TRACED Across Reasoning Complexity. Performance of DeepSeek-R1-Llama-8B stratified by reasoning steps (

L

). Gap (

\Delta

) denotes the maximum fluctuation across difficulty tiers. The low variance (

\Delta\leq 2.7\%

) confirms stability. Comprehensive results for all models are provided in Table 10 .

Figure 3: Robustness and Efficiency. (Left) Class Imbalance: TRACED maintains discriminative stability against distributional shifts, specifically where the prior

P(y_{n}=1)\in[0.3,0.7]

. (Right) Data Efficiency: The method achieves rapid geometric convergence, reaching a stability plateau with merely

N\approx 400

reference samples.

Figure 4: Sensitivity to Subspace Dimension

k

. AUROC evaluation across four models (

k\in[2,10]

) shows performance improves and stabilizes at

k=8

. Additional metrics (AUPR, FPR@95) are detailed in Appendix M .

Config	$M_{n}$	$K_{n}$	Fables	GPQA	GSM8K	MATH	Soc_IQA	ThrmQA
Disp. Only	✓		0.6845	0.7812	0.7634	0.7012	0.7122	0.8245
Curv. Only		✓	0.6512	0.7244	0.7188	0.6855	0.6945	0.7912
TRACED	✓	✓	0.7191	0.8300	0.8061	0.7489	0.7536	0.8730

Table 3: Component Ablation. Performance comparison of individual geometric signatures (

M_{n}

only,

K_{n}

only) versus the full TRACED framework (See Appendix G for more results).

Model	Strategy	GPQA	GSM8K
DeepSeek-R1-Llama-8B	No-Subspace	0.624	0.813
DeepSeek-R1-Llama-8B	TRACED	0.661	0.828
Qwen2.5-4B	No-Subspace	0.712	0.752
Qwen2.5-4B	TRACED	0.733	0.776

Table 4: No-Subspace Ablation.

Figure 5: Kinematic Scaling Laws of Reasoning. Log-log plot of Net Displacement

D(t)=||z_{T}-z_{0}||_{2}

vs. reasoning length across six domains. Blue (Correct): Exhibits linear scaling (

slope\approx 0.82

), characteristic of directed evolution (

D\propto T

) where computation yields direct semantic progress. Red (Incorrect): Follows sub-linear scaling (

slope\approx 0.53

), resembling random walk (

D\propto\sqrt{T}

) and indicating progress stagnation. Shaded regions denote standard deviation.

Figure 6: Geometric Differences Across Domains. (Left) Curvature Distribution: Structured reasoning (blue) exhibits a narrow peak, contrasting with the broad, heavy tail of open-ended reasoning (purple). (Right) Displacement Accumulation: Structured trajectories reveal step-wise growth driven by discrete breakthroughs, while open-ended tasks exhibit a smooth, continuous semantic flow.

Figure 7: Transition Structures of Cognitive States. Visualization of transition probabilities $P(S_{t+1}|S_{t})$ of Correct and Incorrect Reasoning.

Figure 8: Geometric Cost of State Transitions. (Left) Avg. curvature change ( $\Delta K$ ). (Right) Avg. displacement change ( $\Delta M$ ). Curvature encodes the cost of uncertaint directional reorientation, while displacement reflects accumulative semantic progress.

Figure 9: Geometric-Semantic Synchronization. Alignment between geometric displacement (gray) and cognitive states.

Model	Method	Training Configuration				Evaluation Results
Model	Method	Learning Rate	Group ( $G$ )	KL $\beta$	TRACED $\lambda$	GSM8K (%)	MATH (%)
Qwen2.5-0.5B	Vanilla	-	-	-	-	30.93	24.60
	GRPO	3e-6	8	0.04	-	39.65	28.33
	TRACED-GRPO	3e-6	8	0.04	0.1	43.97	30.65
Qwen2.5-1.5B	Vanilla	-	-	-	-	62.25	44.00
	GRPO	1e-6	8	0.02	-	67.83	47.20

Table 5: Alignment Training Results. Performance comparison of Vanilla, standard GRPO, and TRACED-GRPO on GSM8K and MATH benchmarks.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Reference	Task Description	Metric	Format	Prompt	Example (Truncated)
GSM8K	( Cobbe et al., 2021 )	Multi-step grade school mathematics problems.	Acc.	Number	Type A	Q: Natalia sold clips to 48 friends… A: 48
MATH	( Hendrycks et al., 2021 )	Challenging competition-level math.	Acc.	L a T e X	Type C	Q: Find $f(f(2))$ . A: \boxed { 298 }
TheoremQA	( Chen et al., 2023 )	Application of STEM theorems to solve novel problems.	Acc.	Option / Boolean / Number	Type D	Q: Calculate derivative of $x^{2}$ at $3$ . A: 6.0
GPQA	( Rein et al., 2024 )	PhD-level scientific QA.	Acc.	Option	Type B	Q: Which mechanism drives… A: (B)
Social IQA	( Sap et al., 2019 )	Reasoning about social interactions and motivations.	Acc.	Option	Type B	Q: Tracy pressed wrong button… A: (A)
Fables	( Srivastava et al., 2023 )	Abstracting morals from allegorical narratives.	Acc.	Option	Type B	Q: The Tortoise and the Hare implies… A: (A)

Appendix

Table 6: Overview of the Datasets. Statistics of the original source datasets. Note: Due to the cost of constructing reasoning trajectories (

N=10

trajectories per sample), we randomly sampled 1,000 instances for experimentation.All collected samples were subsequently partitioned into training and evaluation sets using a randomized 8:2 split.

Panel of Figure 10 — Figure 10: Geometric Modes Across Different Models. Visualization of the reasoning geometric signatures for Qwen2.5, Llama-3.1, and Qwen3-Thinking. Consistent topological separation is observed across all architectures.

DeepSeek-R1-Llama-8B
Model	Mag. ( $M_{n}$ )	Ang. ( $K_{n}$ )	Fables	GPQA	GSM8K	MATH	Soc_IQA	Thrm.
	✓		0.6845	0.7812	0.7634	0.7012	0.7122	0.8245
		✓	0.6512	0.7244	0.7188	0.6855	0.6945	0.7912
TRACED	✓	✓	0.7191	0.8300	0.8061	0.7489	0.7536	0.8730
Qwen3-4B-Thinking-2507
	✓		0.6750	0.6620	0.7410	0.8120	0.6830	0.7215

Appendix

Table 7: Ablation study of geometric components (AUROC

\uparrow

). We compare the performance of using individual geometric features (

M_{n}

only,

K_{n}

only) versus the combined TRACED framework across four representative models.

Positive Ratio ( $\alpha$ )	0.1	0.2	0.3	0.4	0.5 (Main)	0.6	0.7	0.8	0.9	$\Delta_{Max}$
DeepSeek-R1-Llama-8B	0.742	0.775	0.786	0.791	0.806	0.803	0.788	0.778	0.735	-0.071
Qwen3-4B-Thinking	0.685	0.710	0.735	0.742	0.748	0.745	0.730	0.705	0.672	-0.076
Llama-3.1-8B-Instruct	0.621	0.645	0.660	0.665	0.668	0.664	0.658	0.635	0.610	-0.058
Qwen2.5-7B-Instruct	0.655	0.698	0.708	0.715	0.720	0.718	0.702	0.680	0.648	-0.072

Appendix

Table 8: Data Ratio Robustness. Average AUROC scores across six datasets under varying Positive Data Ratios (

\alpha

). TRACED maintains robust performance in the moderate range (

0.3\leq\alpha\leq 0.7

) but exhibits expected sensitivity at extreme imbalances due to the fixed uniform prior assumption.

Ref. Ratio ( $\gamma$ )	0.1	0.2	0.3	0.4	0.5	0.6	0.8	1.0 (Main)
Sample Count ( $N$ )	80	160	240	320	400	480	640	800
DeepSeek-R1-Llama-8B	0.745	0.762	0.781	0.792	0.803	0.804	0.805	0.806
Qwen3-4B-Thinking	0.6540	0.685	0.712	0.741	0.746	0.747	0.748	0.748
Llama-3.1-8B-Instruct	0.615	0.632	0.650	0.652	0.656	0.667	0.670	0.672
Qwen2.5-7B-Instruct	0.636	0.665	0.702	0.712	0.717	0.719	0.719	0.722

Appendix

Table 9: Sensitivity to Reference Set Size. Average AUROC scores across six datasets. The evaluation set is fixed (

N_{test}=200

). We vary the reference set size from

\gamma=0.1

(

N=80

) to

\gamma=1.0

(

N=800

, Main Result). The method reaches a stability plateau around

\gamma=0.5

(

400

samples), indicating the minimum data requirement for robust covariance estimation.

DeepSeek-R1-Llama-8B
Metric	Easy ( $L\leq 4$ )	Medium ( $5\leq L\leq 8$ )	Hard ( $L>8$ )	Gap ( $\Delta$ )
AUROC ( $\uparrow$ )	0.775	0.748	0.766	2.7%
AUPR ( $\uparrow$ )	0.708	0.710	0.723	1.5%
FPR@95 ( $\downarrow$ )	0.660	0.673	0.685	2.5%
Qwen3-4B-Thinking
AUROC ( $\uparrow$ )	0.762	0.755	0.738	2.4%

Appendix

Table 10: Robustness of TRACED Across Reasoning Complexity. Performance is stratified by reasoning steps (

L

). Gap (

\Delta

) denotes the maximum performance fluctuation across the three difficulty tiers. The consistently low fluctuations (

\Delta\leq 2.8\%

) confirm TRACED’s stability regardless of reasoning complexity.

Figure 11: Sensitivity Analysis of Dimension

k

(AUPR). We evaluate the AUPR performance (

\uparrow

) of TRACED across four models.

Figure 12: Sensitivity Analysis of Dimension

k

(FPR@95). We evaluate the FPR@95 performance (

\downarrow

) of TRACED across four models.

Dataset	Runner-up Method	Best Baseline (CI)	TRACED (CI)	p-value
GPQA	LR Probe	$0.759\pm 0.025$	$\mathbf{0.830\pm 0.019}$	$\mathbf{<0.001}^{**}$
Theorem	SAPLMA	$0.852\pm 0.030$	$\mathbf{0.873\pm 0.025}$	$\mathbf{0.008}^{**}$
Social_IQA	LR Probe	$0.710\pm 0.022$	$\mathbf{0.754\pm 0.020}$	$\mathbf{0.004}^{**}$
GSM8K	SAPLMA	$0.800\pm 0.015$	$\mathbf{0.806\pm 0.014}$	$\mathbf{0.032}^{*}$
MATH	LR Probe	$0.747\pm 0.018$	$\mathbf{0.749\pm 0.016}$	$\mathbf{0.045}^{*}$
Fables	LR Probe	$0.718\pm 0.021$	$\mathbf{0.719\pm 0.018}$	$\mathbf{0.034}^{*}$

Appendix

Table 11: Statistical Significance Analysis. Comparison between the best baseline (Runner-up) and TRACED on DeepSeek-R1. 95% Confidence Intervals are estimated via bootstrapping. * and ** denote statistical significance at

p<0.05

and

p<0.01

, respectively. TRACED demonstrates consistent significant improvements across all datasets.

Case	Reasoning Excerpt (Truncated)	Cognitive State	Geometry
Incorrect (Hesitation)	1. ”Let’s assume the probability is $x/y$ …”	Exploration	$M_{n}$ : Low $K_{n}$ : High (Diffusive)
	2. ”Wait, this logic might double count the overlap…”	Reflection ( $\uparrow$ Curvature)
	3. ”Let’s try a different approach using combinations…”	Exploration
	4. ”But does this satisfy the initial condition? I’m not sure…”	Reflection ( $\uparrow$ Curvature)
Correct (Directed)	1. ”First, we calculate the total outcomes as $6^{3}$ …”	Exploration	$M_{n}$ : High $K_{n}$ : Low (Ballistic)
	2. ”This implies that the sum must be even…”	Certainty ( $\rightarrow$ Displacement)

Appendix

Table 12: Qualitative Comparison of Reasoning Dynamics. Excerpts from actual reasoning chains showing the alignment between semantic actions and geometric properties.

LLM Reliability

Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.

See all 789 papers

LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signals
Apr 7, 2026Lihao Sun, Hang Dong, Bo Qiao +3LLM Reasoning Representation Geometry in Language Models
Microsoft
Confidence Geometry Reveals Trace-Level Correctness in Large Language Model Reasoning
May 16, 2026Shuo Liu, Ding Liu, Shi-Ju RanConfidence Estimation in Language Models LLM Reliability
School of Computer Science and Technology, Tiangong University, Tianjin, China. · Center for Quantum Physics and Intelligent Sciences, Department of Physics, Capital Normal University, Beijing, China.
PRISM: A Dual View of LLM Reasoning through Semantic Flow and Latent Computation
Mar 24, 2026Ruidi Chang, Jiawei Zhou, Hanjie ChenLLM Reasoning LLM Interpretability
Rice University · Stony Brook University

Apr 7, 2026J/K move · Enter open · S save

LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signals

Lihao Sun, Hang Dong, Bo Qiao, Qingwei Lin+2

Microsoft

LLM Reasoning Representation Geometry in Language Models Language Model Steering

Open paper AskPDF arXiv

This work characterizes large language models' chain-of-thought generation as a structured trajectory through representation space. We show that mathematical reasoning traverses functionally ordered, step-specific subspaces that become increasingly separable with layer depth. This structure already exists in base models, while reasoning training primarily accelerates convergence toward termination-related subspaces rather than introducing new representational organization. While early reasoning steps follow similar trajectories, correct and incorrect solutions diverge systematically at late stages. This late-stage divergence enables mid-reasoning prediction of final-answer correctness with ROC-AUC up to 0.87. Furthermore, we introduce trajectory-based steering, an inference-time intervention framework that enables reasoning correction and length control based on derived ideal trajectories. Together, these results establish reasoning trajectories as a geometric lens for interpreting, predicting, and controlling LLM reasoning behavior.

Abstract

LLM Reliability

Explore similar work

LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signals

Confidence Geometry Reveals Trace-Level Correctness in Large Language Model Reasoning

PRISM: A Dual View of LLM Reasoning through Semantic Flow and Latent Computation