Reinforcement learning (RL) for mathematical reasoning can suffer from reward sparsity: for challenging problems, LLM fails to sample any correct trajectories, preventing RL from receiving meaningful positive feedback. At the same time, there often exist human-written reference solutions along with the problem (e.g., problems from AoPS), but directly fine-tuning on these solutions offers no benefit because models often cannot imitate human proofs that lie outside their own reasoning distribution. We introduce Reference-Guided Fine-Tuning (ReGFT), a simple and effective method that utilizes human-written reference solutions to synthesize positive trajectories on hard problems and train on them before RL. For each problem, we provide the model with a partial reference solution and let it generate its own reasoning trace, ensuring the resulting trajectories remain in the model's reasoning space while still benefiting from reference guidance. Fine-tuning on these reference-guided trajectories increases the number of solvable problems and produces a checkpoint that receives more positive rewards during RL. Across three benchmarks (AIME24, AIME25, BeyondAIME), ReGFT consistently improves supervised accuracy, accelerates DAPO training, and raises the final performance plateau of RL. Our results show that ReGFT effectively overcomes reward sparsity and unlocks stronger RL-based mathematical reasoning.
Figures & tables
Figure 1: Comparison of ReFT and ReGFT. Top: ReFT fine-tunes the model using verified correct trajectories obtained from standard sampling. Bottom: ReGFT additionally applies reference-guided sampling to recover hard problems without correct trajectories.
Figure 2: Reinforcement learning performance over training steps on three challenging benchmarks. Models initialized with ReGFT consistently achieve higher accuracy, faster convergence, and a superior final performance compared to the raw checkpoint, demonstrating that reference-guided fine-tuning provides a stronger initialization for RL.
Figure 3: Reinforcement learning performance comparison between ReFT and ReGFT across three benchmarks. While both initializations accelerate early-stage RL, ReGFT consistently achieves higher final accuracy, highlighting the contribution of reference-guided demonstrations beyond self-generated trajectories.
Figure 4: Accuracy comparison between direct SFT on raw human reference solutions and ReGFT . Models trained directly on reference solutions fail to achieve competitive RL performance, underscoring the importance of model-derived reasoning trajectories.
Figure 5: Inference-time scaling performance ( pass@k ) of raw and RL-trained checkpoints across benchmarks. Solid lines denote DAPO-trained models, while dashed line indicate the raw pre-RL checkpoint.
Model
Training Dataset (%)
Train pass@64 (%)
AIME24 (%)
AIME25 (%)
BeyondAIME (%)
Raw
48.4
68.6
59.2
46.7
30.5
ReFT
49.6
70.2
62.1
46.8
31.3
ReGFT
50.1
72.5
60.0
47.8
31.3
Table 1: Performance comparison of models fine-tuned under different settings. We report pass@64 accuracy on the training dataset and evaluation results on AIME24 , AIME25 , and BeyondAIME .
Setting
AIME’24
AIME’25
Beyond-AIME
Raw model
59.2
46.7
30.5
DAPO
Response size 16
63.2
54.3
36.1
Response size 64
67.1
60.5
39.8
ReFT + DAPO
Response size 16
66.2
57.8
35.0
Table 2: Effect of sampling scale per problem under different training settings on AIME 2024, AIME 2025, and Beyond-AIME.
NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China · Pattern Recognition Center, WeChat AI, Tencent Inc, China