Beyond Scalars: Evaluating and Understanding LLM Reasoning via Geometric Progress and Stability

Xinyan Jiang; Ninghao Liu; Di Wang; Lijie Hu

cs.AIMar 11, 2026

Beyond Scalars: Evaluating and Understanding LLM Reasoning via Geometric Progress and Stability

Authors: Xinyan Jiang, Ninghao Liu, Di Wang, Lijie Hu

Organizations: Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · King Abdullah University of Science and Technology · Shanghai Advanced Research Institute, Chinese Academy of Sciences, Shanghai, China · University of Chinese Academy of Sciences, Beijing, China ·

Abstract

Evaluating LLM reliability via scalar probabilities often fails to capture the structural dynamics of reasoning. We introduce TRACED, a framework that assesses reasoning quality through theoretically grounded geometric kinematics. By decomposing reasoning traces into Progress (displacement) and Stability (curvature), we reveal a distinct topological divergence: correct reasoning manifests as high-progress, stable trajectories, whereas hallucinations are characterized by low-progress, unstable patterns (stalled displacement with high curvature fluctuations). Leveraging these signatures, our probabilistic framework achieves competitive performance and superior robustness across diverse benchmarks. Crucially, TRACED bridges geometry and cognition by mapping high curvature to ''Hesitation Loops'' and displacement to ''Certainty Accumulation'', offering a physical lens to decode the internal dynamics of machine thought.

Figures & tables

Figure 1: Topological Divergence of Reasoning Quality. Joint distribution of net displacement ( $M$ ) and curvature ( $K$ ) across Structured and Open-Ended domains. The visualization confirms a consistent separation: correct reasoning traces (blue) exhibit a high-displacement, low-curvature pattern, while incorrect chains (red) are characterized by low-displacement stagnation and high-curvature oscillations.

DeepSeek-R1-Llama-8B
	Fables			GPQA			GSM8K			MATH			Social_iqa			Theorem
Method	AUROC $\uparrow$	AUPR $\uparrow$	FPR@95 $\downarrow$	AUROC $\uparrow$	AUPR $\uparrow$	FPR@95 $\downarrow$	AUROC $\uparrow$	AUPR $\uparrow$	FPR@95 $\downarrow$	AUROC $\uparrow$	AUPR $\uparrow$	FPR@95 $\downarrow$	AUROC $\uparrow$	AUPR $\uparrow$	FPR@95 $\downarrow$	AUROC $\uparrow$	AUPR $\uparrow$	FPR@95 $\downarrow$
MSP	0.6044	0.6427	0.8919	0.3837	0.4750	0.8887	0.7424	0.7425	0.8235	0.6276	0.6203	0.8125	0.5867	0.6237	0.9194	0.5270	0.4951	0.8737
Perplexity	0.6719	0.6581	0.8568	0.3831	0.4107	0.8767	0.6096	0.6088	0.8824	0.5808	0.5628	0.8688	0.5911	0.6174	0.8065	0.5229	0.5552	0.8211
Entropy	0.6156	0.6481	0.8378	0.4857	0.5159	0.8823	0.6863	0.6916	0.8235	0.6041	0.5921	0.8875	0.5925	0.6095	0.8194	0.5111	0.4951	0.8737
LR Probe	0.7177	0.6539	0.8297	0.7588	0.5451	0.8571	0.7995	0.8195	0.7059	0.7471	0.6541	0.8625	0.7097	0.6883	0.7091	0.8435	0.6497	0.8158

Table 1: Comparison of reasoning quality assessment methods across multiple models and tasks. See Appendix N for statistical uncertainty analysis (95% CIs) confirming the stability of these results.

Figure 2: Universality and Generalization Analysis. (a) Universal Signature: A single global fit model derived from aggregated data achieves competitive AUPR across diverse tasks, supporting the existence of a task-agnostic geometric signature. (b) Cross-Domain Adaptation: Dumbbell plot comparing Direct Zero-shot Transfer (blue circles), Aligned Transfer (purple squares), and Supervised In-domain Upper Bound (red stars). Results confirm that the geometric alignment significantly bridges the performance gap caused by distribution shifts.

Metric	Easy ( $L\leq 4$ )	Medium ( $5\leq L\leq 8$ )	Hard ( $L>8$ )	Gap ( $\Delta$ )
AUROC ( $\uparrow$ )	0.775	0.748	0.766	2.7%
AUPR ( $\uparrow$ )	0.708	0.710	0.723	1.5%
FPR@95 ( $\downarrow$ )	0.660	0.673	0.685	2.5%

Table 2: Robustness of TRACED Across Reasoning Complexity. Performance of DeepSeek-R1-Llama-8B stratified by reasoning steps (

L

). Gap (

\Delta

) denotes the maximum fluctuation across difficulty tiers. The low variance (

\Delta\leq 2.7\%

) confirms stability. Comprehensive results for all models are provided in Table 10 .

Figure 3: Robustness and Efficiency. (Left) Class Imbalance: TRACED maintains discriminative stability against distributional shifts, specifically where the prior

P(y_{n}=1)\in[0.3,0.7]

. (Right) Data Efficiency: The method achieves rapid geometric convergence, reaching a stability plateau with merely

N\approx 400

reference samples.

Figure 4: Sensitivity to Subspace Dimension

k

. AUROC evaluation across four models (

k\in[2,10]

) shows performance improves and stabilizes at

k=8

. Additional metrics (AUPR, FPR@95) are detailed in Appendix M .

Config	$M_{n}$	$K_{n}$	Fables	GPQA	GSM8K	MATH	Soc_IQA	ThrmQA
Disp. Only	✓		0.6845	0.7812	0.7634	0.7012	0.7122	0.8245
Curv. Only		✓	0.6512	0.7244	0.7188	0.6855	0.6945	0.7912
TRACED	✓	✓	0.7191	0.8300	0.8061	0.7489	0.7536	0.8730

Table 3: Component Ablation. Performance comparison of individual geometric signatures (

M_{n}

only,

K_{n}

only) versus the full TRACED framework (See Appendix G for more results).

Model	Strategy	GPQA	GSM8K
DeepSeek-R1-Llama-8B	No-Subspace	0.624	0.813
DeepSeek-R1-Llama-8B	TRACED	0.661	0.828
Qwen2.5-4B	No-Subspace	0.712	0.752
Qwen2.5-4B	TRACED	0.733	0.776

Table 4: No-Subspace Ablation.

Figure 5: Kinematic Scaling Laws of Reasoning. Log-log plot of Net Displacement

D(t)=||z_{T}-z_{0}||_{2}

vs. reasoning length across six domains. Blue (Correct): Exhibits linear scaling (

slope\approx 0.82

), characteristic of directed evolution (

D\propto T

) where computation yields direct semantic progress. Red (Incorrect): Follows sub-linear scaling (

slope\approx 0.53

), resembling random walk (

D\propto\sqrt{T}

) and indicating progress stagnation. Shaded regions denote standard deviation.

Figure 6: Geometric Differences Across Domains. (Left) Curvature Distribution: Structured reasoning (blue) exhibits a narrow peak, contrasting with the broad, heavy tail of open-ended reasoning (purple). (Right) Displacement Accumulation: Structured trajectories reveal step-wise growth driven by discrete breakthroughs, while open-ended tasks exhibit a smooth, continuous semantic flow.

Figure 7: Transition Structures of Cognitive States. Visualization of transition probabilities $P(S_{t+1}|S_{t})$ of Correct and Incorrect Reasoning.

Figure 8: Geometric Cost of State Transitions. (Left) Avg. curvature change ( $\Delta K$ ). (Right) Avg. displacement change ( $\Delta M$ ). Curvature encodes the cost of uncertaint directional reorientation, while displacement reflects accumulative semantic progress.

Figure 9: Geometric-Semantic Synchronization. Alignment between geometric displacement (gray) and cognitive states.

Model	Method	Training Configuration				Evaluation Results
Model	Method	Learning Rate	Group ( $G$ )	KL $\beta$	TRACED $\lambda$	GSM8K (%)	MATH (%)
Qwen2.5-0.5B	Vanilla	-	-	-	-	30.93	24.60
	GRPO	3e-6	8	0.04	-	39.65	28.33
	TRACED-GRPO	3e-6	8	0.04	0.1	43.97	30.65
Qwen2.5-1.5B	Vanilla	-	-	-	-	62.25	44.00
	GRPO	1e-6	8	0.02	-	67.83	47.20

Table 5: Alignment Training Results. Performance comparison of Vanilla, standard GRPO, and TRACED-GRPO on GSM8K and MATH benchmarks.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Reference	Task Description	Metric	Format	Prompt	Example (Truncated)
GSM8K	( Cobbe et al., 2021 )	Multi-step grade school mathematics problems.	Acc.	Number	Type A	Q: Natalia sold clips to 48 friends… A: 48
MATH	( Hendrycks et al., 2021 )	Challenging competition-level math.	Acc.	L a T e X	Type C	Q: Find $f(f(2))$ . A: \boxed { 298 }
TheoremQA	( Chen et al., 2023 )	Application of STEM theorems to solve novel problems.	Acc.	Option / Boolean / Number	Type D	Q: Calculate derivative of $x^{2}$ at $3$ . A: 6.0
GPQA	( Rein et al., 2024 )	PhD-level scientific QA.	Acc.	Option	Type B	Q: Which mechanism drives… A: (B)
Social IQA	( Sap et al., 2019 )	Reasoning about social interactions and motivations.	Acc.	Option	Type B	Q: Tracy pressed wrong button… A: (A)
Fables	( Srivastava et al., 2023 )	Abstracting morals from allegorical narratives.	Acc.	Option	Type B	Q: The Tortoise and the Hare implies… A: (A)

Appendix

Table 6: Overview of the Datasets. Statistics of the original source datasets. Note: Due to the cost of constructing reasoning trajectories (

N=10

trajectories per sample), we randomly sampled 1,000 instances for experimentation.All collected samples were subsequently partitioned into training and evaluation sets using a randomized 8:2 split.

Panel of Figure 10 — Figure 10: Geometric Modes Across Different Models. Visualization of the reasoning geometric signatures for Qwen2.5, Llama-3.1, and Qwen3-Thinking. Consistent topological separation is observed across all architectures.

DeepSeek-R1-Llama-8B
Model	Mag. ( $M_{n}$ )	Ang. ( $K_{n}$ )	Fables	GPQA	GSM8K	MATH	Soc_IQA	Thrm.
	✓		0.6845	0.7812	0.7634	0.7012	0.7122	0.8245
		✓	0.6512	0.7244	0.7188	0.6855	0.6945	0.7912
TRACED	✓	✓	0.7191	0.8300	0.8061	0.7489	0.7536	0.8730
Qwen3-4B-Thinking-2507
	✓		0.6750	0.6620	0.7410	0.8120	0.6830	0.7215

Appendix

Table 7: Ablation study of geometric components (AUROC

\uparrow

). We compare the performance of using individual geometric features (

M_{n}

only,

K_{n}

only) versus the combined TRACED framework across four representative models.

Positive Ratio ( $\alpha$ )	0.1	0.2	0.3	0.4	0.5 (Main)	0.6	0.7	0.8	0.9	$\Delta_{Max}$
DeepSeek-R1-Llama-8B	0.742	0.775	0.786	0.791	0.806	0.803	0.788	0.778	0.735	-0.071
Qwen3-4B-Thinking	0.685	0.710	0.735	0.742	0.748	0.745	0.730	0.705	0.672	-0.076
Llama-3.1-8B-Instruct	0.621	0.645	0.660	0.665	0.668	0.664	0.658	0.635	0.610	-0.058
Qwen2.5-7B-Instruct	0.655	0.698	0.708	0.715	0.720	0.718	0.702	0.680	0.648	-0.072

Appendix

Table 8: Data Ratio Robustness. Average AUROC scores across six datasets under varying Positive Data Ratios (

\alpha

). TRACED maintains robust performance in the moderate range (

0.3\leq\alpha\leq 0.7

) but exhibits expected sensitivity at extreme imbalances due to the fixed uniform prior assumption.

Ref. Ratio ( $\gamma$ )	0.1	0.2	0.3	0.4	0.5	0.6	0.8	1.0 (Main)
Sample Count ( $N$ )	80	160	240	320	400	480	640	800
DeepSeek-R1-Llama-8B	0.745	0.762	0.781	0.792	0.803	0.804	0.805	0.806
Qwen3-4B-Thinking	0.6540	0.685	0.712	0.741	0.746	0.747	0.748	0.748
Llama-3.1-8B-Instruct	0.615	0.632	0.650	0.652	0.656	0.667	0.670	0.672
Qwen2.5-7B-Instruct	0.636	0.665	0.702	0.712	0.717	0.719	0.719	0.722

Appendix

Table 9: Sensitivity to Reference Set Size. Average AUROC scores across six datasets. The evaluation set is fixed (

N_{test}=200

). We vary the reference set size from

\gamma=0.1

(

N=80

) to

\gamma=1.0

(

N=800

, Main Result). The method reaches a stability plateau around

\gamma=0.5

(

400

samples), indicating the minimum data requirement for robust covariance estimation.

DeepSeek-R1-Llama-8B
Metric	Easy ( $L\leq 4$ )	Medium ( $5\leq L\leq 8$ )	Hard ( $L>8$ )	Gap ( $\Delta$ )
AUROC ( $\uparrow$ )	0.775	0.748	0.766	2.7%
AUPR ( $\uparrow$ )	0.708	0.710	0.723	1.5%
FPR@95 ( $\downarrow$ )	0.660	0.673	0.685	2.5%
Qwen3-4B-Thinking
AUROC ( $\uparrow$ )	0.762	0.755	0.738	2.4%

Appendix

Table 10: Robustness of TRACED Across Reasoning Complexity. Performance is stratified by reasoning steps (

L

). Gap (

\Delta

) denotes the maximum performance fluctuation across the three difficulty tiers. The consistently low fluctuations (

\Delta\leq 2.8\%

) confirm TRACED’s stability regardless of reasoning complexity.

Figure 11: Sensitivity Analysis of Dimension

k

(AUPR). We evaluate the AUPR performance (

\uparrow

) of TRACED across four models.

Figure 12: Sensitivity Analysis of Dimension

k

(FPR@95). We evaluate the FPR@95 performance (

\downarrow

) of TRACED across four models.

Dataset	Runner-up Method	Best Baseline (CI)	TRACED (CI)	p-value
GPQA	LR Probe	$0.759\pm 0.025$	$\mathbf{0.830\pm 0.019}$	$\mathbf{<0.001}^{**}$
Theorem	SAPLMA	$0.852\pm 0.030$	$\mathbf{0.873\pm 0.025}$	$\mathbf{0.008}^{**}$
Social_IQA	LR Probe	$0.710\pm 0.022$	$\mathbf{0.754\pm 0.020}$	$\mathbf{0.004}^{**}$
GSM8K	SAPLMA	$0.800\pm 0.015$	$\mathbf{0.806\pm 0.014}$	$\mathbf{0.032}^{*}$
MATH	LR Probe	$0.747\pm 0.018$	$\mathbf{0.749\pm 0.016}$	$\mathbf{0.045}^{*}$
Fables	LR Probe	$0.718\pm 0.021$	$\mathbf{0.719\pm 0.018}$	$\mathbf{0.034}^{*}$

Appendix

Table 11: Statistical Significance Analysis. Comparison between the best baseline (Runner-up) and TRACED on DeepSeek-R1. 95% Confidence Intervals are estimated via bootstrapping. * and ** denote statistical significance at

p<0.05

and

p<0.01

, respectively. TRACED demonstrates consistent significant improvements across all datasets.

Case	Reasoning Excerpt (Truncated)	Cognitive State	Geometry
Incorrect (Hesitation)	1. ”Let’s assume the probability is $x/y$ …”	Exploration	$M_{n}$ : Low $K_{n}$ : High (Diffusive)
	2. ”Wait, this logic might double count the overlap…”	Reflection ( $\uparrow$ Curvature)
	3. ”Let’s try a different approach using combinations…”	Exploration
	4. ”But does this satisfy the initial condition? I’m not sure…”	Reflection ( $\uparrow$ Curvature)
Correct (Directed)	1. ”First, we calculate the total outcomes as $6^{3}$ …”	Exploration	$M_{n}$ : High $K_{n}$ : Low (Ballistic)
	2. ”This implies that the sum must be even…”	Certainty ( $\rightarrow$ Displacement)

Appendix

Table 12: Qualitative Comparison of Reasoning Dynamics. Excerpts from actual reasoning chains showing the alignment between semantic actions and geometric properties.

LLM Reliability

Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.

See all 787 papers

Apr 7, 2026

LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signals

This work characterizes large language models' chain-of-thought generation as a structured trajectory through representation space. We show that mathematical reasoning traverses functionally ordered, step-specific subspaces that become increasingly separable with layer depth. This structure already exists in base models, while reasoning training primarily accelerates convergence toward termination-related subspaces rather than introducing new representational organization. While early reasoning steps follow similar trajectories, correct and incorrect solutions diverge systematically at late stages. This late-stage divergence enables mid-reasoning prediction of final-answer correctness with ROC-AUC up to 0.87. Furthermore, we introduce trajectory-based steering, an inference-time intervention framework that enables reasoning correction and length control based on derived ideal trajectories. Together, these results establish reasoning trajectories as a geometric lens for interpreting, predicting, and controlling LLM reasoning behavior.

Lihao Sun, Hang Dong, Bo Qiao +3

Microsoft

LLM Reasoning Representation Geometry in Language Models Language Model Steering

Ask PDF

May 16, 2026

Confidence Geometry Reveals Trace-Level Correctness in Large Language Model Reasoning

Large language models (LLMs) generate not only reasoning text, but also token-level confidence trajectories that record how uncertainty evolves during inference. Whether these trajectories are relevant to reasoning correctness remains unclear. Here we show that confidence trajectories encode a content-agnostic confidence geometry associated with trace-level final-answer correctness. Using only token-level confidence values, without access to the input question, reasoning text, hidden states, or external verifiers, we find that low-dimensional representations of confidence trajectories separate correct from incorrect reasoning traces. Across GSM8K, MATH, and MMLU, this geometric separation is quantitatively linked to downstream predictability: stronger clustering of correct and incorrect traces, measured by the Davies--Bouldin index, consistently corresponds to higher correctness-discrimination AUC. We further show that correctness-related information is enriched in the tail of reasoning, suggesting that late-stage confidence dynamics carry key correctness signals. We propose NeuralConf, a lightweight estimator that learns from confidence trajectories for correctness evaluation. Under a fixed trace budget, NeuralConf-derived scores improve confidence-weighted answer aggregation over majority voting, tail confidence, and other static baselines. These results reveal that LLMs expose trace-intrinsic statistical signals of correctness through their own confidence dynamics, offering a route to improve inference using information already present within generation.

Shuo Liu, Ding Liu, Shi-Ju Ran

School of Computer Science and Technology, Tiangong University, Tianjin, China. · Center for Quantum Physics and Intelligent Sciences, Department of Physics, Capital Normal University, Beijing, China.

Confidence Estimation in Language Models LLM Reliability LLM Uncertainty Estimation LLM Reasoning

Ask PDF

Mar 24, 2026

PRISM: A Dual View of LLM Reasoning through Semantic Flow and Latent Computation

Large language models (LLMs) solve complex problems by generating multi-step reasoning traces. Yet these traces are typically analyzed from only one of two perspectives: the sequence of tokens across different reasoning steps in the generated text, or the hidden-state vectors across model layers within one step. We introduce PRISM (Probabilistic Reasoning Inspection through Semantic and Implicit Modeling), a framework and diagnostic tool for jointly analyzing both levels, providing a unified view of how reasoning evolves across steps and layers. Across multiple reasoning models and benchmarks, PRISM uncovers systematic patterns in the reasoning process, showing that failed trajectories are more likely to become trapped in unproductive verification loops and further diverge into distinct modes such as overthinking and premature commitment, which behave differently once a candidate answer is reached. It further reveals how prompting reshapes reasoning behavior beyond aggregate accuracy by altering both semantic transitions and internal computational patterns. By modeling reasoning trajectories as structured processes, PRISM makes these behaviors observable and analyzable rather than relying solely on final-task accuracy. Taken together, these insights position PRISM as a practical tool for analyzing and diagnosing reasoning processes in LLMs.

Ruidi Chang, Jiawei Zhou, Hanjie Chen

Rice University · Stony Brook University

LLM Reasoning LLM Interpretability Overthinking in Language Models

Ask PDF

Abstract

LLM Reliability

Explore similar work

LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signals

Confidence Geometry Reveals Trace-Level Correctness in Large Language Model Reasoning

PRISM: A Dual View of LLM Reasoning through Semantic Flow and Latent Computation