From Atomic to Agentic: Towards Interpretable Evaluation of LLMs' Agentic Mathematical Capabilities

Jiayi Kuang; Yinghui Li; Yunze Song; Keyu Chen; Zhifeng Shen; Yangning Li; Yidong Wang; Di Yin

cs.AIAug 27, 2026

From Atomic to Agentic: Towards Interpretable Evaluation of LLMs' Agentic Mathematical Capabilities

Authors: Jiayi Kuang, Yinghui Li, Yunze Song, Keyu Chen, Zhifeng Shen, Yangning Li, Yidong Wang, Di Yin, +6 more

Organizations: Sun Yat-sen University · Tencent Youtu Lab · Pengcheng Laboratory · University of Illinois Chicago

Abstract

Large Language Models (LLMs) are evolving from performing end-to-end mathematical reasoning to integrating agentic intelligence. However, most existing math benchmarks evaluate only final answers. This outcome-oriented evaluation provides limited diagnostic value for identifying process-level failures or rigorous logic, failing to guide the transformation of LLMs into robust agents. To bridge this gap, we present a process-level benchmark designed to evaluate the inherent agentic mathematical reasoning abilities of LLMs. Our framework aligns problem-solving agentic behaviors with a structured taxonomy of reusable mathematical atomic capabilities. We design a comprehensive suite of planning, action, and feedback tasks across both textual and multimodal contexts, supported by an automated pipeline that synthesizes high-quality trajectories and produces fine-grained annotations via controlled LLM rewriting. Experiments reveal that models with similar end-to-end accuracy can exhibit markedly different agentic capability profiles. This demonstrates that process-level evaluation is crucial for interpreting the true potential of LLMs and guiding the development of next-generation mathematical agents.

Figures & tables

Figure 1: It illustrates alignment between agentic behaviors and mathematical atomic capabilities, enabling interpretable evaluation of LLMs’ agentic intelligence.

Figure 2: The overall benchmark construction pipeline of our AgenticMathBench.

Figure 3: Data distribution of collected datasets.

Task	#Num	Input	Target Output	Metric(s)
Planning
Capability Planning	221	T&I	Selected Capability set $\{a_{i}\}$	P/R/F1, EM
Solution Planning	237	T&I	Ordered plan $(a_{1},g_{1})\!\rightarrow\!\cdots\!\rightarrow\!(a_{T},g_{T})$	Judge(Cov/Cons)
Next-step Planning	609	T&I	Next step $(a_{t+1},g_{t+1})$	Acc( $a_{t+1}$ ), Sim( $g_{t+1}$ )
Action
Symbol Recognition	1,022	T&I	Canonical L a T e X expression	ExpRate, SymRate

Table 1: ExpAcc/SymAcc=expression/symbol accuracy, CAS-Eq=CAS equivalence, Lean-Cmp=Lean compilation, Judge(Cov/Cons)=LLM-judge coverage/consistency, Sim=semantic similarity, with data cases in Appendix C.4 ).

General Open-sourced Models
Model	Capability Planning			Solution Planning				Next-step Planning
Model	Pre	Rec	F1	Logic	Sub-goal	Step	Overall	Capability	Subgoal	Overall
Llama-4-Scout-17B	47.6	90.5	60.8	37.5	38.0	33.0	36.2	27.1	39.9	33.5
Llama-4-Maverick-17B	54.7	83.4	64.1	44.5	44.5	44.0	44.3	43.8	48.5	46.1
DeepSeek-V3.2	48.2	59.3	52.1	77.1	77.7	73.5	76.1	45.8	46.8	46.3
Qwen3-32B	43.5	44.0	41.5	40.0	40.4	39.1	40.2	34.1	38.5	36.3

Table 2: Planning performance of models across capability planning, solution planning, and next-step planning. The results on multimodal tasks are in Table 15 .

General Open-source Models
Model	Correctness	Error Localization			Fix Suggestion
Model	Accuracy	Step Judge	Type Classification	Overall	Reason Consistency	Overall
Llama-4-Scout-17B	50.0	30.8	30.8	30.8	62.2	28.0
Llama-4-Maverick-17B	44.4	30.8	7.7	19.2	53.9	24.3
DeepSeek-V3.2	49.0	54.2	66.8	60.5	70.1	31.5
Qwen3-32B	56.1	44.6	58.1	51.4	54.9	24.7

Table 3: Feedback performance of models, evaluating correctness judgment, error localization, and fix suggestion. The results on multimodal tasks are in Table 16 .

General Open-source Models
Model	Calc.	Concept		Formal Lang.		Fwd. Reason	Modeling			Theorem
Model	EM	Set-F1	Acc	Compile	Align	Prec.	Var-F1	Constr	Obj	Pass@k
Llama-4-Scout-17B	59.3	13.5	55.2	49.4	39.4	20.7	86.0	78.4	90.8	39.5
Llama-4-Maverick-17B	54.5	21.9	65.4	89.4	80.0	57.4	82.9	77.3	97.9	52.1
DeepSeek-V3.2	84.7	52.4	97.7	99.4	91.4	93.1	88.3	83.0	99.0	100.0
Qwen3-32B	73.2	42.0	90.0	99.6	93.4	48.1	83.6	75.1	99.2	85.6

Table 4: Action performance of models across execution-oriented mathematical capabilities. The results on multimodal tasks are in Table 17 .

Model	End-to-end Math		Agentic
Model	MATH	AIME25	Plan.	Feed.	Act.
Llama-4-Scout-17B	82.6	10.0	33.5	46.7	53.7
Llama-4-Maverick-17B	90.6	15.9	46.1	40.9	67.4
Qwen3-235B-A22B	98.0	81.5	28.7	21.4	84.5
GLM-4.7	98.8	95.7	29.0	29.7	64.5
Qwen2.5-Math-72B	85.9	20.0	30.2	27.2	30.5

Table 5: Comparison between end-to-end mathematical benchmarks and agentic mathematical capabilities.

Figure 4: This figure illustrates the different performance of models on planning and feedback tasks.

Figure 5: Case studies of typical breakdowns in Planning capabilities.

Figure 6: Performance on action tasks.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Atomic ability	Data sources
Symbol recognition	CROHME19/23 ( Mouchère et al., 2016 )
Concept understanding	NaturalProofs ( Welleck et al., 2021 )
Calculation	LILA/AMPS ( Mishra et al., 2022 ) , CARP ( Zhang et al., 2023a ) , DeepMind Mathematics ( Saxton et al., 2019 ) , Dolphin1878 ( Shi et al., 2015 ) , operator–template MWP data ( Sarkar et al., 2023 ) , INTEGRALBENCH ( Tang et al., 2025 )
Spatial perception	FormalGeo v2 ( Zhang et al., 2023b )
Formal mathematical language	CriticLeanBench ( Peng et al., 2025 ) , FormL4 ( Lu et al., 2024a ) , MiniF2F ( Zheng et al., 2021 ) , ProofNet ( Azerbayev et al., 2023 )
Deductive and inductive reasoning	NaturalProofs ( Welleck et al., 2021 )

Appendix

Table 6: Statistics of the sources of the 9 atomic abilities in AgenticMathBench.

Atomic ability	AMO-Bench	IMO-Bench	OlymMATH	Omni-MATH	MathArena	FIMO	OlympiadBench
Symbol recognition	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\checkmark$
Concept understanding	$\circ$	$\checkmark$	$\circ$	$\circ$	$\circ$	$\checkmark$	$\circ$
Calculation	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\checkmark$
Spatial perception	$\circ$	$\circ$	$\circ$	$\circ$	$\circ$	$\circ$	$\checkmark$
Formal mathematical language	$\times$	$\times$	$\times$	$\times$	$\times$	$\checkmark$	$\times$
Deductive and inductive reasoning	$\circ$	$\checkmark$	$\circ$	$\circ$	$\circ$	$\checkmark$	$\circ$

Appendix

Table 7: Qualitative coverage of atomic abilities across the competition benchmarks used to construct the composite split. Each cell indicates how frequently the benchmark exercises a given ability:

\checkmark

denotes that the ability is a primary focus of many problems,

\circ

denotes that it appears in a non-trivial but secondary role, and

\times

denotes that it is rarely or never required. Benchmarks: AMO-Bench ( An et al., 2025 ) , IMO-Bench ( Luong et al., 2025 ) , OlymMATH ( Sun et al., 2025 ) , Omni-MATH ( Gao et al., 2024 ) , MathArena ( Balunović et al., 2025 ) , FIMO ( Liu et al., 2023 ) , and OlympiadBench ( He et al., 2024 ) .

Atomic ability	Case
Symbol recognition	Input: Output: $j_{2}=-\left(\dfrac{n}{2}+1\right)-\left(\dfrac{m}{2}+\dfrac{1}{2}\right)b^{-2}$ .
Concept understanding	Input: Irrational Numbers form G-Delta Set in Reals: Let $R\setminus Q$ denote the set of irrational numbers. Let $(R,\tau)$ denote the real number line with the usual (Euclidean) topology. Then $R\setminus Q$ forms a $G_{\delta}$ set in $R$ . Output: 1. Rational Numbers are Countably Infinite: The set $Q$ of rational numbers is countably infinite. 2. Rational Numbers form F-Sigma Set in Reals: Let $Q$ be the set of rational numbers. Let $(R,\tau)$ be the real number line with the usual (Euclidean) topology. Then $Q$ is a $F_{\sigma}$ set in $R$ . 3. De Morgan’s Laws (Set Theory)/Set Difference/General Case/Difference with Union: $S\setminus\bigcup\mathcal{T}=\bigcap_{T^{\prime}\in\mathcal{T}}(S\setminus T^{\prime})$ , where $\bigcup\mathcal{T}:=\{x:\exists T^{\prime}\in\mathcal{T}\text{ such that }x\in T^{\prime}\}$ , that is, the union of $\mathcal{T}$ .
Calculation	Input: Problem: Find the smallest $x$ such that $x\equiv 16\pmod{3}$ , $x\equiv 4\pmod{3}$ , and $x\equiv 7\pmod{18}$ . Output: $7$ .
Spatial perception	Input: Output: Equal(LengthOfLine(AB), x) Equal(LengthOfLine(AC), z) Equal(LengthOfLine(AD), y) Equal(LengthOfLine(BD), 12) Equal(LengthOfLine(CD), 4) PerpendicularBetweenLine(AD, CD) PerpendicularBetweenLine(BD, AD) PerpendicularBetweenLine(CA, BA)
Formal mathematical language	Input: The function $f$ , defined on the set of ordered pairs of positive integers, satisfies the following properties: $f(x,x)=x,\;f(x,y)=f(y,x),{\rm\ and\ }(x+y)f(x,y)=yf(x,x+y).$ Calculate $f(14,52)$ . Show that it is 364. Output: theorem aime_1988_p8 (f : $\mathbb{N}\to\mathbb{N}\to\mathbb{R}$ ) (h_0 : $\forall x,\ 0<x\to f\,x\,x=x$ ) (h_1 : $\forall x\,y,\ 0<x\land 0<y\to f\,x\,y=f\,y\,x$ ) (h_2 : $\forall x\,y,\ 0<x\land 0<y\to(\uparrow x+\uparrow y)f\,x\,y=yf\,x\,(x+y)$ ) : $f\,14\,52=364$ := by sorry

Appendix

Table 8:

Atomic ability	Case
Deductive and Inductive Reasoning	Input: Hermitian Matrix has Real Eigenvalues: Every Hermitian matrix has eigenvalues which are all real numbers. Output: 1. Set up the eigenpair for the Hermitian matrix and record the eigen-equation relating A, v, and $\lambda$ . 2. Form a scalar inner-product relation by left-multiplying the eigen-equation with $v^{}$ , relating $v^{}Av$ to $\lambda\cdot v^{}v$ . 3. Treat these quantities as 1×1 matrices and express the conjugate-transpose of the quadratic form in terms of conjugate-transposes of its factors. 4. Apply the Hermitian property of A and the conjugate-transpose involution to relate the conjugates of $v^{}Av$ and $v^{*}v$ back to themselves, preparing to view them as real scalars. 5. Introduce scalar representatives a and b for the 1×1 forms, note b is nonzero for an eigenvector, and connect their self-adjointness to real-valuedness. 6. Use the scalar equality to express $\lambda$ as the quotient $a/b$ and combine this with the real-valuedness of a and b to relate $\lambda$ to the real numbers.
Mathematics Modeling	Input: 26 children were riding on the bus. At the bus stop 38 more children got on the bus. How many children are on the bus now? Output: Variables: name: initial_children, desc: number of children riding on the bus initially, unit: children name: boarded_children, desc: number of children who got on at the bus stop, unit: children name: total_children, desc: number of children on the bus now, unit: children Constraints: initial_children = 26 boarded_children = 38 total_children = initial_children + boarded_children initial_children $\in Z$ boarded_children $\in Z$ total_children $\in Z$ initial_children $\geq 0$ boarded_children $\geq 0$ total_children $\geq 0$ Objective: solve total_children
Theorem application	Input: Target: Irrational Numbers form G-Delta Set in Reals: Let $R\setminus Q$ denote the set of irrational numbers. Let $(R,\tau)$ denote the real number line with the usual (Euclidean) topology. Then $R\setminus Q$ forms a $G_{\delta}$ set in $R$ . Hints: 1. Rational Numbers are Countably Infinite: The set $Q$ of rational numbers is countably infinite. 2. Rational Numbers form F-Sigma Set in Reals: Let $Q$ be the set of rational numbers. Let $(R,\tau)$ be the real number line with the usual (Euclidean) topology. Then $Q$ is a $F_{\sigma}$ set in $R$ . 3. De Morgan’s Laws (Set Theory)/Set Difference/General Case/Difference with Union: $\displaystyle S\setminus\bigcup\mathcal{T}=\bigcap_{T^{\prime}\in\mathcal{T}}(S\setminus T^{\prime})$ , where $\displaystyle\bigcup\mathcal{T}:=\{x:\exists T^{\prime}\in\mathcal{T}\text{ such that }x\in T^{\prime}\}$ , that is, the union of $\mathcal{T}$ . Output: {{begin-eqn}} $Q=\bigcup_{\alpha\in Q}\{\alpha\}$ (Rational Numbers form F-Sigma Set in Reals) $\leadsto\ R\setminus Q=R\setminus\bigcup_{\alpha\in Q}\{\alpha\}$ $=\bigcap_{\alpha\in Q}(R\setminus\{\alpha\})$ (De Morgan’s Laws: Difference with Union) {{end-eqn}} The result follows from Rational Numbers are Countably Infinite. {{qed}}

Appendix

Table 9:

Agentic Ability	Task	Text-only	Multimodal	Total
Planning	Atomic Ability Selection	190	31	221
	Plan Generation	217	20	237
	Generate the Next Step	561	48	609
Feedback	Verification and Judgement	486	20	506
	Error Location	334	13	347
	Correction and Fix	262	18	280

Appendix

Table 10: Task Volume Distribution of Planning/Feedback Tasks (Text-only vs. Multimodal)

Atomic Ability	Text-only (Count/Percentage)	Multimodal (Count/Percentage)
Symbol Recognition	54 (28.4%)	6 (19.4%)
Concept Understanding	185 (97.4%)	15 (48.4%)
Calculation	147 (77.4%)	26 (83.9%)
Spatial Perception	33 (17.4%)	22 (71.0%)
Formal Math Language	6 (3.2%)	3 (9.7%)
Deductive and Inductive Reasoning	184 (96.8%)	23 (74.2%)

Appendix

Table 11: Atomic Ability Coverage Statistics of Planning Task (Text-only vs. Multimodal)

Trajectory Details
Metric	Text-only	Multimodal
Total Trajectories	243	25
Successfully Written	217	20
Skipped (Insufficient Steps)	26	5
Multimodal Samples	-	20 (100.0%)
Step Count Distribution

Appendix

Table 12: Trajectory & Capability Statistics of Planning Task “Solution Planning” (Text-only vs. Multimodal)

Capability	Acc.	Prec.	Recall	F1
Solution Planning	0.87	0.83	0.91	0.87
Next-step Planning	0.84	0.79	0.90	0.84
Error Localization	0.89	0.85	0.94	0.89
Fix Suggestion	0.81	0.76	0.92	0.83
Overall	0.85	0.81	0.92	0.86

Appendix

Table 13: Agreement between LLM-based evaluation and human annotations across four agentic capability dimensions. Metrics are computed from confusion matrices on a human-annotated subset.

Judge	Acc.	Prec.	Rec.	F1	Agreement
DeepSeek-V3	0.85	0.81	0.92	0.86	1.00
GPT-5-mini	0.86	0.83	0.88	0.85	0.96

Appendix

Table 14: Second-judge robustness check using GPT-5-mini as an alternative judge on the same human-annotated subset. “Agreement” denotes the cross-judge agreement with our default DeepSeek-V3 judge.

Multimodal Tasks
Model	Capability Planning			Solution Planning				Next-step Planning
Model	Pre	Rec	F1	Logic	Sub-goal	Step	Overall	Capability	Subgoal	Overall
General Open-sourced Models
DeepSeek-VL2	47.33	79.14	53.48	54.50	56.50	48.50	53.17	25.00	38.03	31.52
Qwen3-VL-32B-Instruct	52.15	49.73	49.98	67.00	64.00	66.00	65.67	52.08	47.64	49.86
Qwen3-VL-235B-A22B-Instruct	45.62	94.46	60.63	52.00	49.50	50.50	50.67	54.17	55.09	54.63

Appendix

Table 15: Planning performance of multimodal models across capability planning, solution planning, and next-step planning.

Multimodal Tasks
Model	Correctness	Error Localization			Fix Suggestion
Model	Accuracy	Step Judge	Type Classification	Overall	Reason Consistency	Overall
General Models
DeepSeek-VL2	50.00	30.77	61.54	46.15	34.44	15.50
Qwen3-VL-32B	55.56	69.23	61.54	65.38	66.11	29.75
InternVL3.5-38B	55.56	84.62	38.46	61.54	50.00	22.50

Appendix

Table 16: Feedback performance of multimodal models, evaluating correctness judgment, error localization, and fix suggestion.

General Models
Models	Symbol		Spatial
	ExpACC	SymACC	RelationF1
InternVL3_5-241B-A28B	74.5	90.3	60.6
InternVL3_5-38B	72.6	80.2	62.6
Qwen3-VL-235B-A22B	61.8	85.9	76.5
Qwen3-VL-32B	61.1	84.6	75.4

Appendix

Table 17: Action performance of Multimodal Models.

Figure 7: The example of 3 representative failure cases, illustrating typical breakdowns in Planning and Feedback capabilities.

Component	Source	LLM role	Used for
Raw problem pool	150+ reviewed math benchmarks; 27 retained datasets	No free-form generation	Base problem collection
Atomic action tasks	Existing datasets and structured conversions	Mostly schema normalization or controlled rewriting (e.g., GPT-4o for modeling/deductive-inductive reasoning)	Action evaluation
Trajectory synthesis	1,236 text and 136 multimodal candidates	Math-agent trajectory generation (DeepSeek-V3.2, Gemini-3-Pro-Preview) followed by automated and human filtering	Planning and feedback tasks
Feedback labels	Correct/incorrect trajectories	LLM-assisted error labeling, manually validated	Error localization and fix suggestion
Evaluation (judge)	All open-ended planning/feedback/action subtasks	DeepSeek-V3 as default judge; GPT-5-mini as robustness check (Table 14 )	Scoring and human-aligned verification

Appendix

Table 18: Component-level summary of human-curated, LLM-rewritten, LLM-synthesized, and LLM-judged parts of AMB.

Setting	Usual Steps	Token Cost
Gemini-3-Pro (direct)	$>$ 5 steps	$>$ 20,000
Gemini-3-Pro-based agent	2–4 steps	$<$ 4,000 / step

Appendix

Table 19: Qualitative cost observation on difficult AIME25-style problems: direct long-chain reasoning vs. atomized agentic reasoning.

Explore similar work

Jun 2, 2026

PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Models

Despite the pivotal role of numerical reasoning as the cornerstone of mathematical capabilities in large language models (LLMs) across applications, few benchmarks evaluate LLMs by integrating numerical processing and mathematical reasoning, hindering the interpretability of failures in math tasks. We introduce PyraMathBench, a comprehensive hierarchical benchmark with 32,505 questions derived from 7,404 math word problems, spanning 4 key cognitive aspects, 14 subcategories, and 2 modalities. Experiments reveal that LLMs' performance is severely compromised by inadequate numerical computation and weak handling of abstract numerical questions. To address this, we propose the Smart Optimization & Learning-based VErsatile module (SOLVE) and Interactive Relative Policy Optimization (IRPO), which enhance LLMs' numerical-mathematical synergy via efficient tool calls (fuzzy matching and low-quality call rejection). Comparative experiments show Qwen-2.5 achieves a 5.0 score improvement with SOLVE and IRPO training.

Zetian Ouyang, Linlin Wang, Gerard de Melo +1

East China Normal University · Hasso Plattner Institute, University of Potsdam

Numerical Reasoning in Language Models Mathematical Reasoning Benchmarks RL for Language Model Reasoning Mathematical Reasoning LLM Tool Use LLM Mathematical Reasoning

Ask PDF

Apr 2, 2026

LiveMathematicianBench: A Live Benchmark for Research-Level Mathematical Reasoning with Proof Sketches

Mathematical reasoning is a hallmark of human intelligence, and whether large language models (LLMs) can meaningfully perform it remains a central question in artificial intelligence and cognitive science. As LLMs are increasingly integrated into scientific workflows, rigorous evaluation of their mathematical capabilities becomes a practical necessity. Existing benchmarks are limited by synthetic settings and data contamination. We present LiveMathematicianBench, a dynamic multiple-choice benchmark for research-level mathematical reasoning built from recent arXiv papers published after model training cutoffs. By grounding evaluation in newly published theorems, it provides a realistic testbed beyond memorized patterns. The benchmark introduces a thirteen-category logical taxonomy of theorem types (e.g., implication, equivalence, existence, uniqueness), enabling fine-grained evaluation across reasoning forms. It employs a proof-sketch-guided distractor pipeline that uses high-level proof strategies to construct plausible but invalid answer choices reflecting misleading proof directions, increasing sensitivity to genuine understanding over surface-level matching. We also introduce a substitution-resistant mechanism to distinguish answer recognition from substantive reasoning. Evaluation shows the benchmark is far from saturated: Gemini-3.1-pro-preview, the best model, achieves only 43.5%. Under substitution-resistant evaluation, accuracy drops sharply: GPT-5.4 scores highest at 30.6%, while Gemini-3.1-pro-preview falls to 17.6%, below the 20% random baseline. A dual-mode protocol reveals that proof-sketch access yields consistent accuracy gains, suggesting models can leverage high-level proof strategies for reasoning. Overall, LiveMathematicianBench offers a scalable, contamination-resistant testbed for studying research-level mathematical reasoning in LLMs.

Linyang He, Qiyao Yu, Hanze Dong +5

Columbia University · Microsoft Research · University of Amsterdam

Mathematical Reasoning Benchmarks LLM Evaluation LLM Mathematical Reasoning

Ask PDF

Apr 24, 2026

Rethinking Math Reasoning Evaluation: A Robust LLM-as-a-Judge Framework Beyond Symbolic Rigidity

Recent advancements in large language models have led to significant improvements across various tasks, including mathematical reasoning, which is used to assess models' intelligence in logical reasoning and problem-solving. Models are evaluated on mathematical reasoning benchmarks by verifying the correctness of the final answer against a ground truth answer. A common approach for this verification is based on symbolic mathematics comparison, which fails to generalize across diverse mathematical representations and solution formats. In this work, we offer a robust and flexible alternative to rule-based symbolic mathematics comparison. We propose an LLM-based evaluation framework for evaluating model-generated answers, enabling accurate evaluation across diverse mathematical representations and answer formats. We present failure cases of symbolic evaluation in two popular frameworks, Lighteval and SimpleRL, and compare them to our approach, demonstrating clear improvements over commonly used methods. Our framework enables more reliable evaluation and benchmarking, leading to more accurate performance monitoring, which is important for advancing mathematical problem-solving and intelligent systems.

Erez Yosef, Oron Anschel, Shunit Haviv Hakimi +4

Tel-Aviv University · Amazon Prime Video · Ben-Gurion University

Mathematical Reasoning Benchmarks LLM Evaluation LLM-as-a-Judge LLM Answer Verification

Ask PDF