Most existing approaches enhance mathematical reasoning in large language models (LLMs) as a monolithic capability, despite the fact that different mathematical problems require different reasoning skills. Recent work has explored a more fine-grained perspective by decomposing mathematical reasoning into atomic skills and developing specialized modules for individual capabilities. However, how to effectively utilize these atomic abilities remains underexplored. To address this challenge, we propose Atomic MathReasoner (AMR), an instance-level framework for composing atomic reasoning skills. AMR first trains a library of capability-specific LoRA experts and performs gradient-free optimization over held-out validation subsets to discover a diverse set of low-loss composition prototypes. It evaluates the preference of each prototype using the token-level negative log-likelihood of reference solutions, filters ambiguous assignments through a confidence margin, and distills the resulting pseudo-labels into a lightweight router. During inference, the router selects an appropriate composition prototype for each problem, enabling problem-specific utilization of atomic skills without costly online optimization. Experiments on the AtomicMath benchmark across general and mathematics-specialized LLMs demonstrate consistent improvements. On Qwen2.5-Math-Instruct-7B, AMR increases average field accuracy from 64.4% to 68.1% at Level 1 and from 59.3% to 63.7% at Level 2, while improving average logical reasoning performance from 39.4% to 46.8%. These results demonstrate that effectively composing specialized atomic skills provides a promising approach to strengthening mathematical reasoning in LLMs.
Figures & tables
Figure 1: Conventional LLMs treat mathematical reasoning as a monolithic capability, often failing across diverse mathematical problems. In contrast, AMR dynamically composes atomic reasoning experts for each input through instance-level routing, enabling problem-adaptive capability synergy and more accurate mathematical reasoning.
Figure 2: Overview of the proposed Atomic MathReasoner (AMR) framework. Top (Stage 1): Prototype Construction. We first decouple mathematical reasoning into atomic capabilities and train corresponding LoRA modules. A gradient-free optimization algorithm is then applied to the validation set to search for localized capability compositions, constructing a Compositional Prototype Set. Bottom (Stage 2): Route Selection Mechanism. We generate pseudo-labels via token-level NLL and apply a margin-based filtering strategy to obtain a noise-removed training set. Finally, a lightweight router utilizing TF-IDF feature extraction is trained to dynamically predict a suitable combination for unseen problems.
Method
GSM8K
MATH
Baseline
79.6
34.7
DPO
80.4
35.2
Step-DPO
81.3
36.0
Math-Shepherd
81.0
35.8
RSD
81.5
36.5
MathFimer
82.0
37.0
Table 3: Comparison on GSM8K and MATH. Baseline is Llama-3-8B-Instruct.
Method
Overall
Field Avg.
Logic Avg.
Base
50.6
61.9
39.4
Joint LoRA
51.8
62.7
40.9
Best Single LoRA
53.4
63.8
43.0
Static Composition
52.7
63.3
42.2
Dynamic Composition
54.5
64.5
44.5
AMR (Ours)
56.4
65.9
46.8
Table 4: Ablation study on Qwen2.5-Math-7B-Instruct.
Model / Drop Module
Overall
Field Avg.
Logic Avg.
AMR (Ours)
56.4
65.9
46.8
w/o Algebra
55.7
64.8
46.6
w/o Analysis
55.4
64.3
46.5
w/o Backward Rea.
56.1
65.8
46.4
w/o Concept
52.7
63.0
42.4
w/o Forward Rea.
56.2
65.8
46.6
Table 5: Leave-One-Out (LOO) ablation of atomic expert modules.
Hyperparameter
Overall
Field Avg.
Logic Avg.
Prototype Pool Size
Top-4 prototypes
53.8
63.0
44.6
Top-6 prototypes
54.3
63.5
45.1
All ( M=9 ) prototypes
56.4
65.9
46.8
Margin Threshold ( δ )
δ=0
55.9
65.4
46.4
Table 6: Hyperparameter sensitivity of AMR on Qwen2.5-Math-7B-Instruct.