Mathematical reasoning has seen rapid progress in large language models (LLMs), yet existing methods optimize predominantly for final-answer correctness, raising the question whether models truly internalize mathematical concepts or merely memorize solution patterns. In human mathematics education, example-based reasoning such as constructing counterexamples to test theorem boundaries reflects deep conceptual understanding, but remains underdeveloped in current LLMs. Enhancing this capability through preference optimization presents two key challenges: (1) the model's limited example-based reasoning ability makes constructing effective preference pairs inherently difficult; and (2) capability acquisition is progressive, as the model must first learn to adopt this strategy before learning to apply it correctly. Therefore we propose INSPIRE, an Internalize-Then-Improve approach combining Reference-Guided Student Internalization (RGSI), which produces high-quality preference candidates under the policy model's own distribution, with a stage-wise rubric preference training strategy that decomposes learning into method-oriented and correctness-oriented stages. Experiments across multiple model scales and families demonstrate consistent improvements, even surpassing larger open-source models, while evaluations on out-of-distribution benchmarks confirm no degradation in general mathematical reasoning ability.
Figures & tables
Figure 1: Overview of Inspire ’s progressive training effect on a mathematical statement requiring example-driven reasoning.
Figure 2: Overview of Inspire . (a) Rubric-based Data Construction : candidate responses are obtained via both self-sampling and Reference-Guided Student Internalization (RGSI) from the base model, then scored by a judge LLM along three dimensions (Method, Reasoning, Correctness). (b) Stage-wise Preference Training : Stage 1 constructs preference pairs prioritizing the method dimension to encourage example-based reasoning; Stage 2 re-samples from the Stage 1 model and constructs pairs prioritizing answer correctness, with Stage 1 replay samples mixed in to prevent forgetting. Note that both responses in Stage 2 employ examples, reflecting the method acquisition from Stage 1; the preference signal now targets correctness.
Dimension
Range
Description
Method ( m )
0–2
Use of example-based analysis
Reasoning ( r )
0–3
Logical quality of reasoning
Correctness ( c )
0–1
Correctness of the final answer
Table 1: Three-dimensional evaluation rubric.
Models
Judgement
Rationale Reasoning
F1 (macro)
Examples (%)
Strict (%)
Loose (%)
Commercial models
KIMI-K2.5 ( Bai et al., 2026 )
51.79
71.63
34.78
45.80
GLM-4.6 ( Zeng et al., 2025 )
58.39
73.19
41.45
45.06
Claude-4.5-Sonnet ( Anthropic, 2025 )
74.24
72.94
39.30
45.64
DeepSeek-R1 ( Guo et al., 2025 )
73.50
80.59
42.52
46.95
Table 2: Main evaluation results on CounterMath. Results are grouped into commercial models, open-source models, and our method with progressive training stages. Bold indicates the best result within each group.
Models
GSM8K
MATH500
AIME 2024
GAOKAO-mQA
MMLU-cMath
Llama-3.1-8B-Instruct
82.94
47.80
6.67
32.48
42.00
+ Stage 1
83.40 (+0.46)
49.00 (+1.20)
6.67 (+0.00)
34.76 (+2.28)
43.00 (+1.00)
+ Stage 1 + Stage 2
83.47 (+0.53)
49.60 (+1.80)
6.67 (+0.00)
35.04 (+2.56)
45.00 (+3.00)
Qwen2.5-Math-1.5B-Instruct
85.22
73.80
10.00
73.22
70.00
+ Stage 1
85.22 (+0.00)
75.20 (+1.40)
10.00 (+0.00)
73.79 (+0.57)
70.00 (+0.00)
+ Stage 1 + Stage 2
85.75 (+0.53)
75.00 (+1.20)
10.00 (+0.00)
74.83 (+1.61)
73.00 (+3.00)
Table 3: Out-of-distribution evaluation results on general mathematical reasoning benchmarks. Values in parentheses indicate the difference relative to the base model.
Strategy
F1
Ex.
Str.
Loo.
Base (+ SFT)
39.06
77.63
14.39
17.02
DPO
39.38
73.19
15.21
17.11
Chosen SFT
41.75
79.52
15.04
18.09
Mixed DPO
42.72
81.17
15.21
17.92
C-DPO
40.27
77.88
15.13
17.43
M-DPO
43.05
81.33
16.53
19.08
Table 4: Ablation on training strategy. The first group compares alternative training strategies; the second group ablates individual stages of our pipeline.
Strategy
F1
Ex.
Str.
Loo.
Base (+ SFT)
39.06
77.63
14.39
17.02
Self-sampling
40.40
78.55
15.54
17.68
Teacher rewriting
39.14
74.01
15.21
17.51
RGSI (Ours)
43.05
81.33
16.53
19.08
Table 5: Ablation on response augmentation strategy in Stage 1. Self-sampling generates candidates without guidance; Teacher rewriting uses DeepSeek-R1 to rewrite references.
Figure 3: DPO training dynamics under different augmentation strategies. RGSI maintains moderate margins and steady gradual accuracy growth, while self-sampling yields insufficient margins and teacher rewriting leads to trivially separable pairs.
Setting
F1
Ex.
Str.
Loo.
Base
38.67
75.25
14.31
16.61
Base + Hint prompt
39.13
77.71
15.95
18.25
Ours (Full)
45.91
84.79
17.30
20.07
Table 6: Comparison between hint prompting and our stage-wise training approach.
Stage
F1
Ex.
Str.
Loo.
Base
38.67
78.21
32.81
37.83
+ SFT
39.06
79.11
33.88
38.89
+ Stage 1
43.05
86.10
37.34
42.19
+ Stage 1 + Stage 2
45.91
87.50
39.56
43.67
Table 7: Cross-judge evaluation on Qwen2.5-Math-7B-Instruct. F1 is judge-independent; Ex., Str., and Loo. are evaluated by GPT-4o.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11Figure 12Figure 13Figure 14Figure 15
Model
Valid responses
m=0
m=1
m=2
π0 (before Stage 1)
20,245
11,455 (56.58%)
2,567 (12.68%)
6,223 (30.74%)
π1 (after Stage 1)
20,171
9,777 (48.47%)
2,466 (12.23%)
7,928 (39.30%)
Appendix
Table 8: Method-score distribution of self-sampled responses before and after Stage 1.
Figure 4: Case study comparing reasoning across training stages on a measure theory problem from CounterMath.
Dimension
Score
Criteria
Method ( m )
2
Concrete mathematical objects (specific values, functions, sets, matrices, etc.) are constructed and play a direct critical role in the reasoning: proving existence, disproving a universal claim, revealing key properties, or advancing a key reasoning step, et al.
1
Concrete objects are present but serve only as incidental illustration or auxiliary explanation, without playing a direct role in the core argument.
0
No concrete objects; the response relies entirely on abstract reasoning or theorem application.
Reasoning ( r )
3
Every step is well-justified; the argument is complete and clearly written.
2
Overall logic is clear, but some details have minor issues.
1
Has a proof framework, but key steps are missing or contain logical gaps.
Appendix
Table 9: Detailed scoring criteria for each dimension and score level of the evaluation rubric.
Stage
F1
Ex.
Str.
Loo.
Base
33.65
65.21
18.83
25.65
+ SFT
33.71
65.79
19.32
26.39
+ Stage 1
35.35
73.03
20.23
27.06
+ Stage 1 + Stage 2
36.36
74.34
21.96
29.03
Appendix
Table 10: Evaluation of INSPIRE on the recent Qwen3-4B-Instruct-2507 backbone. Bold indicates the best result.
Setting
F1
Ex.
Str.
Loo.
Base (+ SFT)
39.06
77.63
14.39
17.02
DPO ( β=0.1 )
38.90
73.27
15.87
17.51
DPO ( β=0.2 )
39.38
73.19
15.21
17.11
DPO ( β=0.5 )
38.94
74.42
13.56
15.71
Ours (Full)
45.91
84.79
17.30
20.07
Appendix
Table 11: Sensitivity of the correctness-only DPO baseline to β on Qwen2.5-Math-7B-Instruct. Bold indicates the best result.
NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China · Pattern Recognition Center, WeChat AI, Tencent Inc, China