Attention-Graph Curvature for the Verification of Mathematical Reasoning

Kanghyeon Kim; Johannes F. Lutzeyer; Yanzhu Guo

cs.LGOct 12, 2026

Attention-Graph Curvature for the Verification of Mathematical Reasoning

Authors: Kanghyeon Kim, Johannes F. Lutzeyer, Yanzhu Guo

Organizations: Télécom Paris, IPP · LIX, CNRS, École Polytechnique, IPP

Abstract

Language models have recently solved open Erdős problems and achieved gold-medal level performance on International Mathematical Olympiad problems. But their solutions, written in natural language, cannot yet be verified by proof assistants. An erroneous reasoning step can invalidate every step that comes after, so it is advantageous to detect errors early. Several methods have been proposed to detect erroneous reasoning quickly based on the features of the forward pass of the language model that performs reasoning. In this work, we study whether the directed Forman curvature of that model's attention graph, a local quantity computed at an edge from the attention weights around it, carries information about a step's correctness. To do so, we evaluate four per-step verifiers on ProcessBench with and without the curvature appended to their input. Appending curvature values improves the overall mean performance of all four verifiers. ReProbe, the verifier that takes every token of a step as input, shows the largest improvement. Adding the curvature increases the time of the language model's forward pass by 18% on average.

Figures & tables

Panel of Figure 1 — Figure 1: Verifier framework and evaluation. A verifier maps features extracted from the generator’s forward pass on a step’s prefix to a score, and the rule in Section 3.1 determines whether a step is flagged. An erroneous solution is correctly classified when the first flagged step is the annotated first erroneous step, and a correct solution when no step is flagged.

Method	Input	With curvature
Spectral, logistic	$\boldsymbol{\phi}_{i}$ , one vector per step	$\left(\boldsymbol{\phi}_{i},\;\mathbf{q}_{s_{i}}\right)\in\mathbb{R}^{10L}$
Hidden states, linear	$h_{s_{i},\|s_{i}\|}$ , the final token	$\left(h_{s_{i},\|s_{i}\|},\;\mathbf{q}_{s_{i},\|s_{i}\|}\right)\in\mathbb{R}^{m+5L}$
MLP probe	$h_{s_{i},\|s_{i}\|}$ , the final token	$\left(h_{s_{i},\|s_{i}\|},\;\mathbf{q}_{s_{i},\|s_{i}\|}\right)\in\mathbb{R}^{m+5L}$
ReProbe	$\left[h_{s_{i},k}\right]_{k=1}^{\|s_{i}\|}$ , every token of the step	$\left[\left(h_{s_{i},k},\;\mathbf{q}_{s_{i},k}\right)\right]_{k=1}^{\|s_{i}\|}\in\mathbb{R}^{\|s_{i}\|\times(m+5L)}$

Table 1: Inputs to each method without and with the curvature vector, with

\boldsymbol{\phi}_{i}\in\mathbb{R}^{5L}

the spectral statistics of Appendix B and

h_{s_{i},k}\in\mathbb{R}^{m}

the last-layer hidden state.

Method	Base	+ Neg-log	+ Stats	+ Curvature (ours)	$\Delta A$	$p$
	$A$				Curvature vs Base
Spectral, logistic	0.3261	0.3165	0.3298	0.3474	$+0.0213$	$<$ 0.001
Hidden states, linear	0.1828	0.1872	0.1847	0.1907	$+0.0079$	0.044
MLP probe	0.2991	0.3033	0.3021	0.3037	$+0.0047$	0.483
ReProbe	0.2932	0.3068	0.3103	0.3472	$+0.0540$	0.0011

Table 2: Each method with its original input (Base) or with one attention-derived representation appended: five quantiles of

w_{e}

per layer (Neg-log,

5L

features), normalized attention entropy, concentration, and first-token attention mass (Stats,

3L

), or curvature (

\mathbf{q}

5L

). All configurations use the same folds and seeds. Within each method, the fitting and threshold-selection rules are identical across feature configurations.

A

is the harmonic mean of the accuracies on correct and erroneous solutions, averaged over the 22 generator–subset combinations with solution-count weights.

\Delta A

and the unadjusted two-sided Wilcoxon

p

-value compare curvature with Base only (Section 3.1 ); they do not test curvature against Neg-log or Stats.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Subset	Plain	Attention	Head mean	Forming $\mathbf{q}$	Within memory
GSM8K	60	5.5	0.5	2.4	100%
MATH	102	11.6	0.7	5.7	99%
OlympiadBench	145	17.8	1.0	8.7	97%
Omni-MATH	139	16.5	1.0	8.3	97%
All	118	13.7	0.9	6.8	98%

Appendix

Table 3: Cost per step of what the curvature vector requires, in ms, averaged over the prefix lengths of the scored steps of each generator–subset combination and then over the generator–subset combinations of a subset weighted by solution count. Plain is the generator’s forward pass with fused attention, which every method requires; attention is what the pass that returns the attention matrices adds to it, and head mean the mean over heads at every layer; forming

\mathbf{q}

is the edge weight, the curvature of Equation 2 , and the quantiles of Equation 3 at every layer, computed for all layers at once. Within memory is the share of steps whose prefix the pass returning the attention matrices completes within the 24 GB of the GPU; the other entries average over those steps, and forming

\mathbf{q}

over all steps. Timings on random matrices and random tokens at the steps’ prefix lengths (Appendix C ).

Method	GSM8K	MATH	OlympiadBench	Omni-MATH	All
Spectral, logistic	0.279 (0.045)	0.302 (0.024)	0.328 (0.021)	0.377 (0.014)	0.326 (0.016)
with curvature	0.321 (0.045)	0.337 (0.024)	0.338 (0.015)	0.383 (0.014)	0.347 (0.014)
Hidden states, linear	0.096 (0.044)	0.143 (0.021)	0.198 (0.033)	0.261 (0.035)	0.183 (0.021)
with curvature	0.111 (0.035)	0.158 (0.024)	0.200 (0.030)	0.262 (0.042)	0.191 (0.020)
MLP probe	0.187 (0.041)	0.249 (0.029)	0.340 (0.040)	0.376 (0.041)	0.299 (0.024)
with curvature	0.181 (0.035)	0.274 (0.031)	0.336 (0.039)	0.370 (0.037)	0.304 (0.023)

Appendix

Table 4: Mean

A

over the generator–subset combinations of each subset, weighted by each generator–subset combination’s solution count, with the standard error of that mean over the generator–subset combinations in parentheses, the weighted sample standard deviation of the generator–subset combination values divided by the square root of the Kish effective number of generator–subset combinations. The four subsets contain 7, 7, 4, and 4 generator–subset combinations. Each method appears twice, without

\mathbf{q}

and then with

\mathbf{q}

appended to its input. All is the same weighted mean over all 22 generator–subset combinations, the value Table 2 reports. Tables 5 to 8 give each generator–subset combination separately.

Panel of Figure 2 — Figure 2: $\Delta A_{c}$ of Section 3.1 for every generator–subset combination, with its solution count. A bar is the standard deviation over the 10 seeds of the within-seed difference. A marker is blue when the change is positive, red when it is negative and gray when it is zero, and filled when the change is larger in magnitude than its bar.

Generator	$n$	SpL	SpL+curv	Hid	Hid+curv	MLP	MLP+curv	RePr	RePr+curv
Llama-3.1-8B	61	0.391	0.437	0.045	0.107	0.271	0.228	0.219	0.297
		(0.062)	(0.052)	(0.049)	(0.058)	(0.039)	(0.096)	(0.079)	(0.061)
Meta-Llama-3-8B	24	0.098	0.273	0.000	0.016	0.090	0.142	0.000	0.113
		(0.142)	(0.105)	(0.000)	(0.052)	(0.123)	(0.085)	(0.000)	(0.105)
Qwen2-1.5B	41	0.306	0.312	0.295	0.251	0.302	0.215	0.372	0.232
		(0.137)	(0.145)	(0.065)	(0.065)	(0.057)	(0.141)	(0.075)	(0.163)

Appendix

Table 5: Every method on the GSM8K subset, by generator, in its configuration without curvature and with the curvature vector

\mathbf{q}

appended (+curv). Values are

A

, the mean over the 10 seeds, with the sample standard deviation over the seeds beneath.

n

is the generator–subset combination’s solution count. SpL spectral logistic, Hid hidden states linear, MLP the MLP probe, RePr ReProbe.

Generator	$n$	SpL	SpL+curv	Hid	Hid+curv	MLP	MLP+curv	RePr	RePr+curv
Llama-3.1-8B	139	0.306	0.334	0.203	0.214	0.350	0.368	0.314	0.357
		(0.036)	(0.051)	(0.053)	(0.033)	(0.043)	(0.049)	(0.049)	(0.046)
Meta-Llama-3-8B	70	0.391	0.437	0.198	0.220	0.294	0.345	0.287	0.332
		(0.049)	(0.020)	(0.021)	(0.038)	(0.075)	(0.062)	(0.048)	(0.039)
Qwen2-1.5B	47	0.347	0.372	0.186	0.259	0.303	0.307	0.291	0.350
		(0.047)	(0.054)	(0.088)	(0.075)	(0.067)	(0.061)	(0.048)	(0.059)

Appendix

Table 6: As Table 5 , for the MATH subset.

Generator	$n$	SpL	SpL+curv	Hid	Hid+curv	MLP	MLP+curv	RePr	RePr+curv
Llama-3.1-8B	164	0.346	0.348	0.269	0.258	0.391	0.394	0.293	0.394
		(0.023)	(0.032)	(0.040)	(0.036)	(0.031)	(0.043)	(0.033)	(0.026)
Qwen2-7B	108	0.256	0.294	0.141	0.139	0.292	0.268	0.314	0.307
		(0.051)	(0.033)	(0.064)	(0.059)	(0.035)	(0.061)	(0.031)	(0.072)
Qwen2.5-7B	123	0.339	0.332	0.136	0.143	0.237	0.250	0.283	0.300
		(0.033)	(0.037)	(0.034)	(0.038)	(0.037)	(0.038)	(0.032)	(0.028)

Appendix

Table 7: As Table 5 , for the OlympiadBench subset.

Generator	$n$	SpL	SpL+curv	Hid	Hid+curv	MLP	MLP+curv	RePr	RePr+curv
Llama-3.1-8B	162	0.406	0.402	0.342	0.356	0.473	0.450	0.426	0.456
		(0.029)	(0.035)	(0.041)	(0.032)	(0.035)	(0.017)	(0.033)	(0.029)
Qwen2-7B	131	0.376	0.403	0.270	0.274	0.361	0.375	0.328	0.381
		(0.041)	(0.035)	(0.044)	(0.048)	(0.028)	(0.036)	(0.026)	(0.062)
Qwen2.5-7B	104	0.338	0.344	0.191	0.201	0.279	0.278	0.326	0.362
		(0.050)	(0.045)	(0.043)	(0.054)	(0.055)	(0.072)	(0.032)	(0.050)

Appendix

Table 8: As Table 5 , for the Omni-MATH subset.

Method	Control	GSM8K	MATH	OlympiadBench	Omni-MATH	All
Spectral, logistic	Neg-log	0.3363	0.2840	0.3141	0.3490	0.3165
Spectral, logistic	Stats	0.3299	0.2882	0.3371	0.3738	0.3298
Hidden states, linear	Neg-log	0.1018	0.1469	0.2032	0.2640	0.1872
Hidden states, linear	Stats	0.0977	0.1514	0.1966	0.2578	0.1847
MLP probe	Neg-log	0.1752	0.2655	0.3384	0.3792	0.3033
MLP probe	Stats	0.1806	0.2507	0.3505	0.3776	0.3021

Appendix

Table 9: Mean

A

for the two attention controls, averaged over the generator–subset combinations of each subset with solution-count weights. Each combination is first averaged over the ten seeds. The All column uses solution-count weights over all 22 combinations. Table 4 gives the corresponding values without added features and with curvature.

Explore similar work

Mask-Proof: An LLM-based Automated Data Curation Pipeline on Mathematical Proofs
Jun 13, 2026Jierui Zhang, Siyuan Tan, Xinhang Li +8Mathematical Reasoning Benchmarks LLM Evaluation
School of Computer Science Beijing University of Posts and Telecommunications Beijing, China · Graduate College for Engineers Beijing University of Posts and Telecommunications Beijing, China · School of Mathematical Sciences Fudan University Shanghai, China +6
Artificial Intelligence for Mathematical Reasoning: An Integrated Survey of Language Models, Neuro-symbolic Systems, and Verified Discovery
Jun 7, 2026Syed Rifat Raiyan, Mohsinul Kabir, Hasan Mahmud +2Mathematical Reasoning Benchmarks Automated Theorem Proving
Systems and Software Lab (SSL), Department of Computer Science and Engineering, Islamic University of Technology, Gazipur, Bangladesh · Department of Computer Science, University of Manchester, Manchester, United Kingdom
Evaluating Research-Level Math Proofs via Strict Step-Level Verification
Jun 9, 2026Yifeng SunLLM Answer Verification Automated Theorem Proving
Independent Researcher

Jun 13, 2026J/K move · Enter open · S save

Mask-Proof: An LLM-based Automated Data Curation Pipeline on Mathematical Proofs

Jierui Zhang, Siyuan Tan, Xinhang Li, Longzhuangzhi Lin+7

School of Computer Science Beijing University of Posts and Telecommunications Beijing, China · Graduate College for Engineers Beijing University of Posts and Telecommunications Beijing, China · School of Mathematical Sciences Fudan University Shanghai, China+6

Mathematical Reasoning Benchmarks LLM Evaluation Automated Theorem Proving Training Data Curation LLM Mathematical Reasoning

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly capable of mathematical problem solving and can even assist with research-level proofs, yet we still lack a scalable and reproducible way to measure step-level reasoning in long proofs across diverse sources. This evaluation gap limits trustworthy AI assistance in proof-certified scientific progress. Existing evaluations often emphasize final answers or rely on costly expert grading, while end-to-end proof generation remains open-ended and hard to verify automatically. We introduce Mask-Proof, a pipeline that turns real proofs into automatically checkable masked-step tasks. It masks key formula steps, provides the necessary surrounding context, and evaluates model reconstructions with an LLM-based equivalence judge using repeated votes for stability. The resulting Mask-ProofBench contains 292 curated problems across diverse research areas. Experiments with 17 models show that reasoning-enhanced models outperform standard models by 12% to 27%. Our evaluator achieves 96.8% agreement with expert annotators, enabling faithful, reproducible, and comparable measurement of step-level mathematical reasoning. Benchmark, annotations, and code are available at https://github.com/weating/Mask-Proof.

Abstract

Explore similar work

Mask-Proof: An LLM-based Automated Data Curation Pipeline on Mathematical Proofs

Artificial Intelligence for Mathematical Reasoning: An Integrated Survey of Language Models, Neuro-symbolic Systems, and Verified Discovery

Evaluating Research-Level Math Proofs via Strict Step-Level Verification