Organizations: NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China · Pattern Recognition Center, WeChat AI, Tencent Inc, China
Through encouraging self-exploration, reinforcement learning from verifiable rewards (RLVR) has significantly advanced the mathematical reasoning capabilities of large language models. As the starting point for RLVR, the capacity of supervised fine-tuning (SFT) to memorize new chain-of-thought trajectories provides a crucial initialization that shapes the subsequent exploration landscape. However, existing research primarily focuses on facilitating exploration during RLVR training, leaving exploration-aware SFT under-explored. To bridge this gap, we propose Offline eXploration-Aware (OXA) fine-tuning. Specifically, OXA optimizes two objectives: promoting low-confidence verified teacher-distillation data to internalize previously uncaptured reasoning patterns, and suppressing high-confidence incorrect self-distillation data to redistribute probability mass of incorrect patterns toward potentially correct candidates. Experimental results across 6 benchmarks show that OXA consistently improves mathematical reasoning performance, especially achieving an average gain of +6 Pass@1 and +5 Pass@k points compared to conventional SFT on the Qwen2.5-1.5B-Math. Crucially, OXA elevates initial policy entropy, and performance gains persist throughout extensive RLVR training, demonstrating the long-term value of OXA.
Figures & tables
Figure 1: Conceptual illustration of entropy collapse versus counteracting entropy collapse.
Model
AIME24
AIME25
BRUMO25
CMIMC25
HMMT25
Minerva
Avg. Perf.
Pass@1
Pass@128
Pass@1
Pass@128
Pass@1
Pass@128
Pass@1
Pass@128
Pass@1
Pass@128
Pass@1
Pass@64
Pass@1
Pass@ k
Qwen2.5-1.5B-Math
Base
9.4
53.3
4.5
36.7
11.5
40.0
2.4
35.0
0.5
26.7
11.9
62.9
6.7
42.4
SFT LP
20.5
76.7
20.5
53.3
29.8
73.3
9.8
50.0
7.1
40.0
20.7
61.4
18.1
59.1
SFT
23.2
80.0
23.8
60.0
29.0
80.0
11.0
52.5
11.6
50.0
22.3
61.4
20.2
64.0
OXA MLE
35.0
83.3
27.1
66.7
36.1
83.3
14.9
57.5
18.5
60.0
25.7
63.2
26.2
69.0
Table 1: Performance comparison of fine-tuning methods and their corresponding RLVR stages. “Base” denotes the pre-trained LLMs, and †indicates models after RLVR training. Pass@1 scores are averaged over 128 samples, except for Minerva which is averaged over 64 samples. Best results within each SFT/RLVR group are bolded.
Figure 2: (a) Sequence length and PPL distributions of the 7B model’s training data. (b) Performance trajectories of the 7B model fine-tuned with OXA MLE and OXA Full on AIME24. (c) Policy entropy dynamics during RLVR training for various fine-tuning methods based on the 7B model. (d) Performance gains of SFT and OXA MLE on the 1.5B model and the Minerva benchmark, grouped by the base model’s pass counts.
Figure 3: Pass@ k curves of 1.5B and 7B models across different fine-tuning methods on CMIMC25 and HMMT25.
Model
SFT LP
SFT
OXA MLE
OXA Full
1.5B
15205
15054
15809
15700
7B
13477
12388
12677
13002
Table 2: Average output lengths of fine-tuned Qwen2.5-1.5B/7B on the AIME24 benchmark.
Figure 4: (a)-(b) Average performance across 6 mathematical benchmarks for LLaMA3.2-3B and Qwen3-1.7B under various fine-tuning methods. (c) Scalability of SFT and OXAMLE as training data increases, compared against ARN-1.1-SFT—a model fine-tuned on millions of samples using the same backbone. (d) Generalization performance of 1.5B variants on out-of-domain benchmarks, including GPQA and MMLU-STEM.
Configuration
AIME24
AIME25
Learning rate (1.5B)
3.0e-4
23.8
23.1
2.5e-4
23.2
23.8
1.0e-4
22.8
23.1
Learning rate (7B)
1.0e-4
45.9
35.4
5.0e-5
47.5
34.2
2.0e-5
45.5
32.7
Table 3: Pass@1 scores of different configurations on AIME24 and AIME25 benchmarks.
Figure 5: (a) Results of OXA Full with Clip-Cov (OXA Full +CC) on AIME25. (b) Comparison across different reasoning data selection strategies on AIME25. “Low PPL”, “Random”, and “High PPL + Long” correspond to SFT LP , SFT, and OXA MLE , respectively.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Impact of data perplexity ranges on reasoning performance. (a)-(b) Performance on AIME24 of 1.5B models fine-tuned on various PPL intervals. (c)-(d) Corresponding results for 7B models across different PPL ranges.
Reinforcement Learning from Verifiable Rewards (RLVR) has become the dominant approach for improving mathematical reasoning in large language models, yet current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states. Investigating this structure, we find that at the anchor token (the position immediately before the answer marker), correct rollouts converge naturally because they must produce the same answer (cosine similarity ~0.84), yet each retains residual variance from its unique reasoning path. Encouraging full alignment at this point pushes the model to extract a unified "correct decision" representation, reducing sensitivity to which reasoning path was taken. Based on this observation, we propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference. On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales, supported by ablations on loss type, anchor position, layer depth, and loss weight.
Ziyue Wang, Aomufei Yuan, Yongfu Zhu +10
1Peking University · 2JINGDONG · 3Shanghai Innovation Institute +2
Reinforcement learning (RL) for mathematical reasoning can suffer from reward sparsity: for challenging problems, LLM fails to sample any correct trajectories, preventing RL from receiving meaningful positive feedback. At the same time, there often exist human-written reference solutions along with the problem (e.g., problems from AoPS), but directly fine-tuning on these solutions offers no benefit because models often cannot imitate human proofs that lie outside their own reasoning distribution. We introduce Reference-Guided Fine-Tuning (ReGFT), a simple and effective method that utilizes human-written reference solutions to synthesize positive trajectories on hard problems and train on them before RL. For each problem, we provide the model with a partial reference solution and let it generate its own reasoning trace, ensuring the resulting trajectories remain in the model's reasoning space while still benefiting from reference guidance. Fine-tuning on these reference-guided trajectories increases the number of solvable problems and produces a checkpoint that receives more positive rewards during RL. Across three benchmarks (AIME24, AIME25, BeyondAIME), ReGFT consistently improves supervised accuracy, accelerates DAPO training, and raises the final performance plateau of RL. Our results show that ReGFT effectively overcomes reward sparsity and unlocks stronger RL-based mathematical reasoning.
Yangzhen Wu, Shanda Li, Zixin Wen +5
UC Berkeley · Carnegie Mellon University · ByteDance Seed
Recent advancements in Reinforcement Learning with Verifiable Rewards (RLVR) have significantly improved Large Language Model (LLM) reasoning, yet models often struggle to explore novel trajectories beyond their initial policy distribution. While offline teacher guidance and entropy-driven strategies have been proposed to address this, they often lack deep integration or are constrained by the model's inherent capacity. In this paper, we propose OGER (Offline-Guided Exploration Reward), a novel framework that unifies offline teacher guidance and online reinforcement learning through a specialized reward modeling lens. OGER employs multi-teacher collaborative training and constructs an auxiliary exploration reward that leverages both offline trajectories and the model's own entropy to incentivize autonomous exploration. Extensive experiments across mathematical and general reasoning benchmarks demonstrate that OGER consistently outperforms competitive baselines, achieving substantial gains in mathematical reasoning while maintaining robust generalization to out-of-domain tasks. We provide a comprehensive analysis of training dynamics and conduct detailed ablation studies to validate the effectiveness of our entropy-aware reward modulation. Our code is available at https://github.com/ecoli-hit/OGER.git.
Xinyu Ma, Mingzhou Xu, Xuebo Liu +4
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China · Hithink RoyalFlush Information Network, Hangzhou, China · Computer and Information Science, University of Macau, Macau, China