Most existing approaches enhance mathematical reasoning in large language models (LLMs) as a monolithic capability, despite the fact that different mathematical problems require different reasoning skills. Recent work has explored a more fine-grained perspective by decomposing mathematical reasoning into atomic skills and developing specialized modules for individual capabilities. However, how to effectively utilize these atomic abilities remains underexplored. To address this challenge, we propose Atomic MathReasoner (AMR), an instance-level framework for composing atomic reasoning skills. AMR first trains a library of capability-specific LoRA experts and performs gradient-free optimization over held-out validation subsets to discover a diverse set of low-loss composition prototypes. It evaluates the preference of each prototype using the token-level negative log-likelihood of reference solutions, filters ambiguous assignments through a confidence margin, and distills the resulting pseudo-labels into a lightweight router. During inference, the router selects an appropriate composition prototype for each problem, enabling problem-specific utilization of atomic skills without costly online optimization. Experiments on the AtomicMath benchmark across general and mathematics-specialized LLMs demonstrate consistent improvements. On Qwen2.5-Math-Instruct-7B, AMR increases average field accuracy from 64.4% to 68.1% at Level 1 and from 59.3% to 63.7% at Level 2, while improving average logical reasoning performance from 39.4% to 46.8%. These results demonstrate that effectively composing specialized atomic skills provides a promising approach to strengthening mathematical reasoning in LLMs.
Figures & tables
Figure 1: Conventional LLMs treat mathematical reasoning as a monolithic capability, often failing across diverse mathematical problems. In contrast, AMR dynamically composes atomic reasoning experts for each input through instance-level routing, enabling problem-adaptive capability synergy and more accurate mathematical reasoning.
Figure 2: Overview of the proposed Atomic MathReasoner (AMR) framework. Top (Stage 1): Prototype Construction. We first decouple mathematical reasoning into atomic capabilities and train corresponding LoRA modules. A gradient-free optimization algorithm is then applied to the validation set to search for localized capability compositions, constructing a Compositional Prototype Set. Bottom (Stage 2): Route Selection Mechanism. We generate pseudo-labels via token-level NLL and apply a margin-based filtering strategy to obtain a noise-removed training set. Finally, a lightweight router utilizing TF-IDF feature extraction is trained to dynamically predict a suitable combination for unseen problems.
Method
GSM8K
MATH
Baseline
79.6
34.7
DPO
80.4
35.2
Step-DPO
81.3
36.0
Math-Shepherd
81.0
35.8
RSD
81.5
36.5
MathFimer
82.0
37.0
Table 3: Comparison on GSM8K and MATH. Baseline is Llama-3-8B-Instruct.
Method
Overall
Field Avg.
Logic Avg.
Base
50.6
61.9
39.4
Joint LoRA
51.8
62.7
40.9
Best Single LoRA
53.4
63.8
43.0
Static Composition
52.7
63.3
42.2
Dynamic Composition
54.5
64.5
44.5
AMR (Ours)
56.4
65.9
46.8
Table 4: Ablation study on Qwen2.5-Math-7B-Instruct.
Model / Drop Module
Overall
Field Avg.
Logic Avg.
AMR (Ours)
56.4
65.9
46.8
w/o Algebra
55.7
64.8
46.6
w/o Analysis
55.4
64.3
46.5
w/o Backward Rea.
56.1
65.8
46.4
w/o Concept
52.7
63.0
42.4
w/o Forward Rea.
56.2
65.8
46.6
Table 5: Leave-One-Out (LOO) ablation of atomic expert modules.
Hyperparameter
Overall
Field Avg.
Logic Avg.
Prototype Pool Size
Top-4 prototypes
53.8
63.0
44.6
Top-6 prototypes
54.3
63.5
45.1
All ( M=9 ) prototypes
56.4
65.9
46.8
Margin Threshold ( δ )
δ=0
55.9
65.4
46.4
Table 6: Hyperparameter sensitivity of AMR on Qwen2.5-Math-7B-Instruct.
Despite the pivotal role of numerical reasoning as the cornerstone of mathematical capabilities in large language models (LLMs) across applications, few benchmarks evaluate LLMs by integrating numerical processing and mathematical reasoning, hindering the interpretability of failures in math tasks. We introduce PyraMathBench, a comprehensive hierarchical benchmark with 32,505 questions derived from 7,404 math word problems, spanning 4 key cognitive aspects, 14 subcategories, and 2 modalities. Experiments reveal that LLMs' performance is severely compromised by inadequate numerical computation and weak handling of abstract numerical questions. To address this, we propose the Smart Optimization & Learning-based VErsatile module (SOLVE) and Interactive Relative Policy Optimization (IRPO), which enhance LLMs' numerical-mathematical synergy via efficient tool calls (fuzzy matching and low-quality call rejection). Comparative experiments show Qwen-2.5 achieves a 5.0 score improvement with SOLVE and IRPO training.
Zetian Ouyang, Linlin Wang, Gerard de Melo +1
East China Normal University · Hasso Plattner Institute, University of Potsdam
Large language models (LLMs) demonstrate strong performance in math reasoning benchmarks, but their performance varies inconsistently across problems with varying levels of difficulty. This paper describes Adaptive Multi-Expert Reasoning (AMR), a framework that focuses on problem complexity by reasoning with dynamically adapted strategies. An agile routing system that focuses on problem text predicts problems' difficulty and uncertainty and guides a reconfigurable sampling mechanism to manage the breadth of generation. Three specialized experts create candidate responses, which are modified during multiple correction and finalization phases. A neural verifier assesses the correctness of responses, while a clustering-based aggregation technique identifies the final candidate answer based on a combination of consensus and answer quality. When evaluated on the GSM8K dataset, AMR achieved 75.28% accuracy while only using the original training data. This result outperformed the majority of comparable 7B models that were trained on synthetic data. This showcases that models using difficulty-based routing and uncertainty-driven aggregation are efficient and effective in improving math reasoning models' robustness.
Mohamed Ehab, Ali Hamdi
Faculty of Computer Science, October University for Modern Science & Arts, Giza, Egypt
While Large Language Models (LLMs) have demonstrated striking capabilities on frontier mathematical problems, it remains unclear whether they possess the structural mathematical understanding underlying their solutions. In this paper, we take a first step toward systematically studying mathematical understanding in LLMs, from diagnosing its distinct capabilities to leveraging these findings to improve post-training. First, we introduce the notion of Mathematical Primitive to probe structural mathematical understanding and propose \hlei{}, a novel benchmark that evaluates mathematical reasoning along four distinct dimensions: Discovery, Generation, Digestion, and Execution. Second, our systematic diagnosis shows that solution accuracy masks distinct capability profiles, primitives unlock substantial latent execution capacity, and Discovery is the dominant bottleneck in mathematical reasoning. Our post-training analysis further shows that discovery-limited failures are particularly amenable to repair. Finally, building on these findings, we introduce \abs{}, a primitive-privileged self-distillation framework that selectively transfers primitive-guided reasoning into the student model. Extensive experiments demonstrate that \abs{} consistently improves mathematical reasoning over baselines across model scales and challenging benchmarks.
Shuo Xing, Zilin Dai, Chengyuan Qian +7
Texas A&M University · Harvard University · Vanderbilt University +3