Offline Exploration-Aware Fine-Tuning for Long-Chain Mathematical Reasoning

Yongyu Mu; Jiali Zeng; Fandong Meng; JingBo Zhu; Tong Xiao

cs.LGMar 17, 2026

Offline Exploration-Aware Fine-Tuning for Long-Chain Mathematical Reasoning

Authors: Yongyu Mu, Jiali Zeng, Fandong Meng, JingBo Zhu, Tong Xiao

Organizations: NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China · Pattern Recognition Center, WeChat AI, Tencent Inc, China

Abstract

Through encouraging self-exploration, reinforcement learning from verifiable rewards (RLVR) has significantly advanced the mathematical reasoning capabilities of large language models. As the starting point for RLVR, the capacity of supervised fine-tuning (SFT) to memorize new chain-of-thought trajectories provides a crucial initialization that shapes the subsequent exploration landscape. However, existing research primarily focuses on facilitating exploration during RLVR training, leaving exploration-aware SFT under-explored. To bridge this gap, we propose Offline eXploration-Aware (OXA) fine-tuning. Specifically, OXA optimizes two objectives: promoting low-confidence verified teacher-distillation data to internalize previously uncaptured reasoning patterns, and suppressing high-confidence incorrect self-distillation data to redistribute probability mass of incorrect patterns toward potentially correct candidates. Experimental results across 6 benchmarks show that OXA consistently improves mathematical reasoning performance, especially achieving an average gain of $+6$ Pass@1 and $+5$ Pass@ $k$ points compared to conventional SFT on the Qwen2.5-1.5B-Math. Crucially, OXA elevates initial policy entropy, and performance gains persist throughout extensive RLVR training, demonstrating the long-term value of OXA.

Figures & tables

Figure 1: Conceptual illustration of entropy collapse versus counteracting entropy collapse.

Model	AIME24		AIME25		BRUMO25		CMIMC25		HMMT25		Minerva		Avg. Perf.
Model	Pass@1	Pass@128	Pass@1	Pass@128	Pass@1	Pass@128	Pass@1	Pass@128	Pass@1	Pass@128	Pass@1	Pass@64	Pass@1	Pass@ $k$
Qwen2.5-1.5B-Math
Base	9.4	53.3	4.5	36.7	11.5	40.0	2.4	35.0	0.5	26.7	11.9	62.9	6.7	42.4
SFT ${}_{\text{LP}}$	20.5	76.7	20.5	53.3	29.8	73.3	9.8	50.0	7.1	40.0	20.7	61.4	18.1	59.1
SFT	23.2	80.0	23.8	60.0	29.0	80.0	11.0	52.5	11.6	50.0	22.3	61.4	20.2	64.0
OXA ${}_{\text{MLE}}$	35.0	83.3	27.1	66.7	36.1	83.3	14.9	57.5	18.5	60.0	25.7	63.2	26.2	69.0

Table 1: Performance comparison of fine-tuning methods and their corresponding RLVR stages. “Base” denotes the pre-trained LLMs, and †indicates models after RLVR training. Pass@1 scores are averaged over 128 samples, except for Minerva which is averaged over 64 samples. Best results within each SFT/RLVR group are bolded.

Figure 2: (a) Sequence length and PPL distributions of the 7B model’s training data. (b) Performance trajectories of the 7B model fine-tuned with OXA ${}_{\text{MLE}}$ and OXA ${}_{\text{Full}}$ on AIME24. (c) Policy entropy dynamics during RLVR training for various fine-tuning methods based on the 7B model. (d) Performance gains of SFT and OXA ${}_{\text{MLE}}$ on the 1.5B model and the Minerva benchmark, grouped by the base model’s pass counts.

Figure 3: Pass@ $k$ curves of 1.5B and 7B models across different fine-tuning methods on CMIMC25 and HMMT25.

Model	SFT ${}_{\text{LP}}$	SFT	OXA ${}_{\text{MLE}}$	OXA ${}_{\text{Full}}$
1.5B	15205	15054	15809	15700
7B	13477	12388	12677	13002

Table 2: Average output lengths of fine-tuned Qwen2.5-1.5B/7B on the AIME24 benchmark.

Figure 4: (a)-(b) Average performance across 6 mathematical benchmarks for LLaMA3.2-3B and Qwen3-1.7B under various fine-tuning methods. (c) Scalability of SFT and $\text{OXA}_{\text{MLE}}$ as training data increases, compared against ARN-1.1-SFT—a model fine-tuned on millions of samples using the same backbone. (d) Generalization performance of 1.5B variants on out-of-domain benchmarks, including GPQA and MMLU-STEM.

Configuration		AIME24	AIME25
Learning rate (1.5B)	3.0e-4	23.8	23.1
	2.5e-4	23.2	23.8
	1.0e-4	22.8	23.1
Learning rate (7B)	1.0e-4	45.9	35.4
	5.0e-5	47.5	34.2
	2.0e-5	45.5	32.7

Table 3: Pass@1 scores of different configurations on AIME24 and AIME25 benchmarks.

Figure 5: (a) Results of OXA ${}_{\text{Full}}$ with Clip-Cov (OXA ${}_{\text{Full}}$ +CC) on AIME25. (b) Comparison across different reasoning data selection strategies on AIME25. “Low PPL”, “Random”, and “High PPL + Long” correspond to SFT ${}_{\text{LP}}$ , SFT, and OXA ${}_{\text{MLE}}$ , respectively.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Impact of data perplexity ranges on reasoning performance. (a)-(b) Performance on AIME24 of 1.5B models fine-tuned on various PPL intervals. (c)-(d) Corresponding results for 7B models across different PPL ranges.

Explore similar work

Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning
Jun 2, 2026Ziyue Wang, Aomufei Yuan, Yongfu Zhu +10LLM Alignment RL for Language Model Reasoning
1Peking University · 2JINGDONG · 3Shanghai Innovation Institute +2
Learn Hard Problems During RL with Reference Guided Fine-tuning
Mar 1, 2026Yangzhen Wu, Shanda Li, Zixin Wen +5LLM Mathematical Reasoning RL for Language Model Reasoning
UC Berkeley · Carnegie Mellon University · ByteDance Seed
OGER: A Robust Offline-Guided Exploration Reward for Hybrid Reinforcement Learning
Apr 20, 2026Xinyu Ma, Mingzhou Xu, Xuebo Liu +4Reward Shaping RL for Language Model Reasoning
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China · Hithink RoyalFlush Information Network, Hangzhou, China · Computer and Information Science, University of Macau, Macau, China

Jun 2, 2026J/K move · Enter open · S save

Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning

Ziyue Wang, Aomufei Yuan, Yongfu Zhu, Shuai Dong+9

1Peking University · 2JINGDONG · 3Shanghai Innovation Institute+2

LLM Alignment RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards LLM Mathematical Reasoning

Open paper AskPDF arXiv

Reinforcement Learning from Verifiable Rewards (RLVR) has become the dominant approach for improving mathematical reasoning in large language models, yet current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states. Investigating this structure, we find that at the anchor token (the position immediately before the answer marker), correct rollouts converge naturally because they must produce the same answer (cosine similarity ~0.84), yet each retains residual variance from its unique reasoning path. Encouraging full alignment at this point pushes the model to extract a unified "correct decision" representation, reducing sensitivity to which reasoning path was taken. Based on this observation, we propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference. On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales, supported by ablations on loss type, anchor position, layer depth, and loss weight.

Abstract

Explore similar work

Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning

Learn Hard Problems During RL with Reference Guided Fine-tuning

OGER: A Robust Offline-Guided Exploration Reward for Hybrid Reinforcement Learning