SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning

Haonan He; Haodi Lei; Yun Luo; Haoran Zhang; Shunkai Zhang; Yizhuo Li; Shengji Tang; Zhilin Wang

cs.CLAug 14, 2026

SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning

Authors: Haonan He, Haodi Lei, Yun Luo, Haoran Zhang, Shunkai Zhang, Yizhuo Li, Shengji Tang, Zhilin Wang,

Organizations: SU-01 Team, Shanghai Artificial Intelligence Laboratory

Abstract

On-policy distillation (OPD) offers a promising way to transfer reasoning capabilities from stronger teacher models, but applying it to long-context reasoning teachers and short-context students introduces practical challenges, including tokenizer mismatch, teacher-student distribution mismatch, response length explosion, and training instability. In this work, we study this setting by transferring proof-reasoning capabilities from the long-context reasoning model SU-01 to short-context student models. To handle tokenizer differences, we perform OPD in a shared text space and align only tokens that occupy identical text spans under the student and teacher tokenizers. To mitigate the problem of excessive generation length and frequent truncation, we introduce a student reference KL loss and mask the advantages of special termination tokens such as </think> and <|im_end|>. This strategy constrains the student from drifting excessively from its initial policy, thereby mitigating the teacher-student distribution mismatch problem and fostering steady length growth. Experiments on both same-family and different-family student models, including Qwen3, Qwen3.5, Intern-S2, GLM-4.7, Gemma-4, show consistent gains in mathematical reasoning, especially natural-language math proving. Notably, Intern-S2-Preview improves by 21.2 points on ProofBench, reaching 55.2 and surpassing Gemini-2.5-Pro. It also improves on science benchmarks such as HLE and HiPhO, suggesting that OPD transfers reasoning capabilities that generalize beyond the mathematical training domain.

Figures & tables

Figure 1: Evaluation performance on ProofBench using Gemini-2.5-Pro as the judge, under the same evaluation setting as SU-01. With SimpleOPD, Intern-S2-OPD achieves performance comparable to DeepSeek-V3.2-Speciale, demonstrating the effectiveness of our OPD strategy.

Figure 2: Overview of SimpleOPD. The student generates responses under its own tokenizer, while the teacher evaluates the same response under its native tokenizer. We align tokens with identical text spans, use teacher supervision for matched tokens, and stabilize training with KL regularization and termination-token masking.

Figure 3: Training dynamics of Intern-S2-Preview during direct OPD training.

Figure 4: Training dynamics of Qwen3.5-35B-A3B during direct OPD training.

Figure 5: Intern-S2-Preview OPD training dynamics when masking the special tokens such as

{\scriptstyle<}\,\mathrm{/think}{\scriptstyle>}

and

{\scriptstyle<}\,\mathrm{|im\_end|}\,{\scriptstyle>}

. The special-token masking alone helps mitigate length-related instability but cannot resolve the length expansion problem.

Figure 6: Intern-S2-Preview OPD training dynamics when adding student reference KL loss to prevent the student policy from deviating excessively from its initial distribution. The training could be stabilized, and the truncation rate is effectively reduced to nearly zero.

Model	ProofBench@4	AnswerBench@8	AIME25@8
SU-01	45.00	77.50	94.60
Intern-S2-Preview	21.70	76.03	88.33
OPD + Spec Mask	38.10 $(+\Delta\,16.40)$	77.60 $(+\Delta\,1.57)$	95.00 $(+\Delta\,6.67)$
OPD + Ref KL	38.50 $(+\Delta\,16.80)$	79.10 $(+\Delta\,3.07)$	95.80 $(+\Delta\,7.47)$

Table 1: OPD results from SU-01 to Intern-S2-Preview with masking termination tokens and adding student reference KL loss.

Model	ProofBench@4	AnswerBench@8	AIME25@8	AMOBench@8
SU-01	45.00	77.50	94.60	61.75
Same-Tokenizer Models
Qwen3-4B	11.42	47.50	71.25	23.00
Qwen3-4B-OPD	23.72 $(+\Delta\,12.30)$	64.50 $(+\Delta\,17.00)$	90.83 $(+\Delta\,19.58)$	35.00 $(+\Delta\,12.00)$
Qwen3-30B-A3B	13.80	59.13	88.33	36.50
Qwen3-30B-A3B-OPD	36.47 $(+\Delta\,22.67)$	74.46 $(+\Delta\,15.33)$	93.75 $(+\Delta\,5.42)$	52.75 $(+\Delta\,16.25)$

Table 2: Evaluation results of distilling SU-01 to students on ProofBench, AnswerBench, AIME25, and AMOBench. Green values indicate improvements over the corresponding base model.

Figure 7: Cross-family distillation results for GLM-4.7-Flash and Gemma-4-26B-A4B from SU-01.

Figure 8: Comparison of the Intern-S2 base model distilled from SU-01 with EOPD, G-OPD, and SimpleOPD across four reasoning benchmarks. All scores are reported as percentages. Each panel uses an independently scaled vertical axis to make small performance differences visible.

Figure 9: The lexical-overlap curves of Qwen3.5-35B-A3B, Intern-S2-Preview and GLM-4.7-Flash during OPD training from the teacher SU-01.

Model	FrontierScience Olympiad	FrontierScience Research	HLE (text-only)	HiPhO
SU-01	61.5	11.7	20.7	35.0
Intern-S2-Preview	60.6	1.7	19.6	38.6
Intern-S2-OPD	60.9 $(+\Delta\,0.3)$	5.0 $(+\Delta\,3.3)$	20.5 $(+\Delta\,0.9)$	41.1 $(+\Delta\,2.5)$

Table 3: Evaluation results of Intern-S2-OPD distilled from SU-01 on scientific reasoning benchmarks.

Model	Length	ProofBench@4	AnswerBench@8	AIME25@8
SU-01	–	45.00	77.50	94.60
Intern-S2-Preview	–	21.70	76.03	88.33
Qwen3.5-35B-A3B	–	26.78	73.16	94.60
	6k	38.80 $(+\Delta\,17.10)$	77.25 $(+\Delta\,1.22)$	95.00 $(+\Delta\,6.67)$
Intern-S2-OPD	32k	44.50 $(+\Delta\,22.80)$	80.10 $(+\Delta\,4.07)$	95.00 $(+\Delta\,6.67)$
	6k	40.07 $(+\Delta\,13.29)$	77.97 $(+\Delta\,4.81)$	96.25 $(+\Delta\,1.65)$

Table 4: Results of different distillation lengths on Intern-S2-Preview and Qwen3.5-35B-A3B distilled from the teacher SU-01.

Model	ProofBench@4	AnswerBench@8	AIME25@8
DeepSeek-V4-Flash	52.18	77.00	99.17
Intern-S2-Preview	21.70	76.03	88.33
Intern-S2-DS-OPD	39.71 $(+\Delta\,18.01)$	77.94 $(+\Delta\,1.91)$	97.50 $(+\Delta\,9.17)$

Table 5: SimpleOPD results from DeepSeek-V4-Flash to Intern-S2-Preview with masking termination tokens and adding student reference KL loss. The distillation length is 6k for computational efficiency.

Model	Student KL Coefficient	ProofBench@4	AnswerBench@8	AIME25@8
SU-01	–	45.00	77.50	94.60
GLM-4.7-Flash	–	30.75	69.59	92.08
	0.5	37.15 $(+\Delta\,6.40)$	71.28 $(+\Delta\,1.69)$	94.17 $(+\Delta\,2.09)$
	1.0	39.71 $(+\Delta\,8.96)$	71.97 $(+\Delta\,2.38)$	92.91 $(+\Delta\,0.83)$
GLM-4.7-Flash-OPD	1.2	37.96 $(+\Delta\,7.21)$	71.19 $(+\Delta\,1.60)$	94.17 $(+\Delta\,2.09)$

Table 6: Ablation study on the student-reference KL coefficient in the setting from SU-01 to GLM-4.7-Flash. Green values indicate improvements over the GLM-4.7-Flash base model.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Per-token reverse KL divergence in cross-vocabulary distillation. Colored boxes show aligned tokens with KL divergence values, while gold boxes indicate tokens that could not be aligned between teacher and student models.

Long-Context Reasoning in Language Models

See all 42 papers

Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning
May 8, 2026Zhicheng Yang, Zhijiang Guo, Yifan Song +5RL for Language Model Reasoning Language Model Distillation
1The Hong Kong University of Science and Technology (Guangzhou) · 2The Hong Kong University of Science and Technology · 3MBZUAI +2
How Should Teachers Be Prepared? RL on Student-Induced States for On-Policy Distillation
Oct 4, 2026Xiaoyu Ma, Haoyue Liu, Zhichao Wang +2On-Policy Distillation Teacher-Student Learning
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen · Shenzhen Future Network of Intelligence Institute (FNiI-Shenzhen) · Guangdong Provincial Key Laboratory of Future Networks of Intelligence, CUHK(SZ)
Spend Teacher Tokens Where They Matter: Success-Referenced On-Policy Distillation
Oct 2, 2026Xiang Chen, Futao Su, Kong Wang +2Language Model Distillation On-Policy Distillation
Tongji University · Xiaohongshu · Shanghai Jiao Tong University +1

May 8, 2026J/K move · Enter open · S save

Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning

Zhicheng Yang, Zhijiang Guo, Yifan Song, Minrui Xu+4

1The Hong Kong University of Science and Technology (Guangzhou) · 2The Hong Kong University of Science and Technology · 3MBZUAI+2

RL for Language Model Reasoning Language Model Distillation On-Policy Distillation Teacher-Student Learning

Open paper AskPDF arXiv

On-policy distillation (OPD) leverages dense teacher rewards to enhance reasoning models. However, scaling OPD to long-horizon tasks exposes a critical flaw: as the student's generated prefix inevitably diverges from the teacher's thought process, the teacher's dense reward loses local exploitability. Continuing to generate and evaluate tokens on these ``drifted'' trajectories not only degrades reward quality but also incurs massive computational waste. To address this, we introduce Prune-OPD, a framework that dynamically aligns training budgets with supervision quality. By continuously monitoring the local compatibility between student and teacher predictions (e.g., via top-

k

overlap), Prune-OPD detects prefix-drift events in real time. Upon detecting severe drift, it monotonically down-weights subsequent unreliable rewards and triggers dynamic rollout truncation. This allows the training process to halt futile generation and reallocate compute strictly to reliable teacher supervision. Across diverse teacher-student combinations, Prune-OPD consistently aligns computation with supervision reliability. When prefix drift makes dense teacher rewards unreliable, it reduces training time by 37.6%--68.0% while preserving, and often improving, performance on challenging benchmarks (AMC, AIME, HMMT). When student-teacher compatibility remains high, it automatically preserves long-context supervision by expanding the training window. These results suggest that Prune-OPD improves OPD not by blindly shortening rollouts, but by reallocating computation toward locally exploitable teacher rewards.

Abstract

Long-Context Reasoning in Language Models

Explore similar work

Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning

How Should Teachers Be Prepared? RL on Student-Induced States for On-Policy Distillation

Spend Teacher Tokens Where They Matter: Success-Referenced On-Policy Distillation