Mathematical reasoning has seen rapid progress in large language models (LLMs), yet existing methods optimize predominantly for final-answer correctness, raising the question whether models truly internalize mathematical concepts or merely memorize solution patterns. In human mathematics education, example-based reasoning such as constructing counterexamples to test theorem boundaries reflects deep conceptual understanding, but remains underdeveloped in current LLMs. Enhancing this capability through preference optimization presents two key challenges: (1) the model's limited example-based reasoning ability makes constructing effective preference pairs inherently difficult; and (2) capability acquisition is progressive, as the model must first learn to adopt this strategy before learning to apply it correctly. Therefore we propose INSPIRE, an Internalize-Then-Improve approach combining Reference-Guided Student Internalization (RGSI), which produces high-quality preference candidates under the policy model's own distribution, with a stage-wise rubric preference training strategy that decomposes learning into method-oriented and correctness-oriented stages. Experiments across multiple model scales and families demonstrate consistent improvements, even surpassing larger open-source models, while evaluations on out-of-distribution benchmarks confirm no degradation in general mathematical reasoning ability.
Figures & tables
Figure 1: Overview of Inspire ’s progressive training effect on a mathematical statement requiring example-driven reasoning.
Figure 2: Overview of Inspire . (a) Rubric-based Data Construction : candidate responses are obtained via both self-sampling and Reference-Guided Student Internalization (RGSI) from the base model, then scored by a judge LLM along three dimensions (Method, Reasoning, Correctness). (b) Stage-wise Preference Training : Stage 1 constructs preference pairs prioritizing the method dimension to encourage example-based reasoning; Stage 2 re-samples from the Stage 1 model and constructs pairs prioritizing answer correctness, with Stage 1 replay samples mixed in to prevent forgetting. Note that both responses in Stage 2 employ examples, reflecting the method acquisition from Stage 1; the preference signal now targets correctness.
Dimension
Range
Description
Method ( m )
0–2
Use of example-based analysis
Reasoning ( r )
0–3
Logical quality of reasoning
Correctness ( c )
0–1
Correctness of the final answer
Table 1: Three-dimensional evaluation rubric.
Models
Judgement
Rationale Reasoning
F1 (macro)
Examples (%)
Strict (%)
Loose (%)
Commercial models
KIMI-K2.5 ( Bai et al., 2026 )
51.79
71.63
34.78
45.80
GLM-4.6 ( Zeng et al., 2025 )
58.39
73.19
41.45
45.06
Claude-4.5-Sonnet ( Anthropic, 2025 )
74.24
72.94
39.30
45.64
DeepSeek-R1 ( Guo et al., 2025 )
73.50
80.59
42.52
46.95
Table 2: Main evaluation results on CounterMath. Results are grouped into commercial models, open-source models, and our method with progressive training stages. Bold indicates the best result within each group.
Models
GSM8K
MATH500
AIME 2024
GAOKAO-mQA
MMLU-cMath
Llama-3.1-8B-Instruct
82.94
47.80
6.67
32.48
42.00
+ Stage 1
83.40 (+0.46)
49.00 (+1.20)
6.67 (+0.00)
34.76 (+2.28)
43.00 (+1.00)
+ Stage 1 + Stage 2
83.47 (+0.53)
49.60 (+1.80)
6.67 (+0.00)
35.04 (+2.56)
45.00 (+3.00)
Qwen2.5-Math-1.5B-Instruct
85.22
73.80
10.00
73.22
70.00
+ Stage 1
85.22 (+0.00)
75.20 (+1.40)
10.00 (+0.00)
73.79 (+0.57)
70.00 (+0.00)
+ Stage 1 + Stage 2
85.75 (+0.53)
75.00 (+1.20)
10.00 (+0.00)
74.83 (+1.61)
73.00 (+3.00)
Table 3: Out-of-distribution evaluation results on general mathematical reasoning benchmarks. Values in parentheses indicate the difference relative to the base model.
Strategy
F1
Ex.
Str.
Loo.
Base (+ SFT)
39.06
77.63
14.39
17.02
DPO
39.38
73.19
15.21
17.11
Chosen SFT
41.75
79.52
15.04
18.09
Mixed DPO
42.72
81.17
15.21
17.92
C-DPO
40.27
77.88
15.13
17.43
M-DPO
43.05
81.33
16.53
19.08
Table 4: Ablation on training strategy. The first group compares alternative training strategies; the second group ablates individual stages of our pipeline.
Strategy
F1
Ex.
Str.
Loo.
Base (+ SFT)
39.06
77.63
14.39
17.02
Self-sampling
40.40
78.55
15.54
17.68
Teacher rewriting
39.14
74.01
15.21
17.51
RGSI (Ours)
43.05
81.33
16.53
19.08
Table 5: Ablation on response augmentation strategy in Stage 1. Self-sampling generates candidates without guidance; Teacher rewriting uses DeepSeek-R1 to rewrite references.
Figure 3: DPO training dynamics under different augmentation strategies. RGSI maintains moderate margins and steady gradual accuracy growth, while self-sampling yields insufficient margins and teacher rewriting leads to trivially separable pairs.
Setting
F1
Ex.
Str.
Loo.
Base
38.67
75.25
14.31
16.61
Base + Hint prompt
39.13
77.71
15.95
18.25
Ours (Full)
45.91
84.79
17.30
20.07
Table 6: Comparison between hint prompting and our stage-wise training approach.
Stage
F1
Ex.
Str.
Loo.
Base
38.67
78.21
32.81
37.83
+ SFT
39.06
79.11
33.88
38.89
+ Stage 1
43.05
86.10
37.34
42.19
+ Stage 1 + Stage 2
45.91
87.50
39.56
43.67
Table 7: Cross-judge evaluation on Qwen2.5-Math-7B-Instruct. F1 is judge-independent; Ex., Str., and Loo. are evaluated by GPT-4o.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11Figure 12Figure 13Figure 14Figure 15
Model
Valid responses
m=0
m=1
m=2
π0 (before Stage 1)
20,245
11,455 (56.58%)
2,567 (12.68%)
6,223 (30.74%)
π1 (after Stage 1)
20,171
9,777 (48.47%)
2,466 (12.23%)
7,928 (39.30%)
Appendix
Table 8: Method-score distribution of self-sampled responses before and after Stage 1.
Figure 4: Case study comparing reasoning across training stages on a measure theory problem from CounterMath.
Dimension
Score
Criteria
Method ( m )
2
Concrete mathematical objects (specific values, functions, sets, matrices, etc.) are constructed and play a direct critical role in the reasoning: proving existence, disproving a universal claim, revealing key properties, or advancing a key reasoning step, et al.
1
Concrete objects are present but serve only as incidental illustration or auxiliary explanation, without playing a direct role in the core argument.
0
No concrete objects; the response relies entirely on abstract reasoning or theorem application.
Reasoning ( r )
3
Every step is well-justified; the argument is complete and clearly written.
2
Overall logic is clear, but some details have minor issues.
1
Has a proof framework, but key steps are missing or contain logical gaps.
Appendix
Table 9: Detailed scoring criteria for each dimension and score level of the evaluation rubric.
Stage
F1
Ex.
Str.
Loo.
Base
33.65
65.21
18.83
25.65
+ SFT
33.71
65.79
19.32
26.39
+ Stage 1
35.35
73.03
20.23
27.06
+ Stage 1 + Stage 2
36.36
74.34
21.96
29.03
Appendix
Table 10: Evaluation of INSPIRE on the recent Qwen3-4B-Instruct-2507 backbone. Bold indicates the best result.
Setting
F1
Ex.
Str.
Loo.
Base (+ SFT)
39.06
77.63
14.39
17.02
DPO ( β=0.1 )
38.90
73.27
15.87
17.51
DPO ( β=0.2 )
39.38
73.19
15.21
17.11
DPO ( β=0.5 )
38.94
74.42
13.56
15.71
Ours (Full)
45.91
84.79
17.30
20.07
Appendix
Table 11: Sensitivity of the correctness-only DPO baseline to β on Qwen2.5-Math-7B-Instruct. Bold indicates the best result.
Through encouraging self-exploration, reinforcement learning from verifiable rewards (RLVR) has significantly advanced the mathematical reasoning capabilities of large language models. As the starting point for RLVR, the capacity of supervised fine-tuning (SFT) to memorize new chain-of-thought trajectories provides a crucial initialization that shapes the subsequent exploration landscape. However, existing research primarily focuses on facilitating exploration during RLVR training, leaving exploration-aware SFT under-explored. To bridge this gap, we propose Offline eXploration-Aware (OXA) fine-tuning. Specifically, OXA optimizes two objectives: promoting low-confidence verified teacher-distillation data to internalize previously uncaptured reasoning patterns, and suppressing high-confidence incorrect self-distillation data to redistribute probability mass of incorrect patterns toward potentially correct candidates. Experimental results across 6 benchmarks show that OXA consistently improves mathematical reasoning performance, especially achieving an average gain of +6 Pass@1 and +5 Pass@k points compared to conventional SFT on the Qwen2.5-1.5B-Math. Crucially, OXA elevates initial policy entropy, and performance gains persist throughout extensive RLVR training, demonstrating the long-term value of OXA.
Yongyu Mu, Jiali Zeng, Fandong Meng +2
NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China · Pattern Recognition Center, WeChat AI, Tencent Inc, China
Recent advances in large language model (LLM) reasoning through reinforcement learning rely on annotated datasets for verifiable rewards, which may limit models' ability to surpass human-level performance. While self-play offers a promising alternative, prior methods yield only marginal or even negative gains on post-trained models because they generate problems that cluster around familiar concepts rather than discovering novel ones. We introduce Open-Ended Self-Improving Reasoner (OpenSIR), a self-play framework in which a single LLM alternates teacher and student roles to generate and solve novel problems without external verifiers or annotated data. Starting from a single seed problem, OpenSIR sustains open-ended exploration through diversity rewards that push the model toward unfamiliar concepts and difficulty calibration that keeps problems learnable. Across seven math benchmarks, OpenSIR consistently improves all models, averaging +3.6 points on instruction models and +3.1 on reasoning models, while recent self-play baselines yield marginal or even negative gains; starting from a single trivial seed, it also surpasses GRPO baselines trained on over 7K annotated examples. Despite training only on self-generated math, OpenSIR is the only self-play method that transfers to general reasoning, improving by at least +4.4 points on reasoning models.
Wai-Chung Kwan, Joshua Ong Jun Leang, Pavlos Vougiouklis +3
University of Edinburgh · Imperial College London · Huawei Technologies Research & Development (UK) Limited +1
While Large Language Models (LLMs) have demonstrated striking capabilities on frontier mathematical problems, it remains unclear whether they possess the structural mathematical understanding underlying their solutions. In this paper, we take a first step toward systematically studying mathematical understanding in LLMs, from diagnosing its distinct capabilities to leveraging these findings to improve post-training. First, we introduce the notion of Mathematical Primitive to probe structural mathematical understanding and propose \hlei{}, a novel benchmark that evaluates mathematical reasoning along four distinct dimensions: Discovery, Generation, Digestion, and Execution. Second, our systematic diagnosis shows that solution accuracy masks distinct capability profiles, primitives unlock substantial latent execution capacity, and Discovery is the dominant bottleneck in mathematical reasoning. Our post-training analysis further shows that discovery-limited failures are particularly amenable to repair. Finally, building on these findings, we introduce \abs{}, a primitive-privileged self-distillation framework that selectively transfers primitive-guided reasoning into the student model. Extensive experiments demonstrate that \abs{} consistently improves mathematical reasoning over baselines across model scales and challenging benchmarks.
Shuo Xing, Zilin Dai, Chengyuan Qian +7
Texas A&M University · Harvard University · Vanderbilt University +3