OPSRD: On-Policy Self-Role Distillation

Weijie Ren; Yanwen Zhang; Hao Li; Zhuolin Qi; Hengyi Zhang; Naibo Wang

cs.CLSep 30, 2026

OPSRD: On-Policy Self-Role Distillation

Authors: Weijie Ren, Yanwen Zhang, Hao Li, Zhuolin Qi, Hengyi Zhang, Naibo Wang

Organizations: Zhejiang University · University of Electronic Science and Technology of China · University of Science and Technology of China

Abstract

Role prompting elicits specialized behavior from large language models through an expert identity, offering a lightweight way to guide reasoning on demanding tasks. However, evaluating or distilling complete role-prompted answers can miss useful next-token preferences when the sampled solution remains incorrect. Transferring these preferences also requires an objective that reaches alternatives the student rarely predicts. We introduce OPSRD, which uses a fixed expert role as privileged teaching context for on-policy self-distillation without reference solutions. A role-free student generates a trajectory, and a frozen instance of the same base model supplies role-conditioned distributions on its exact prefixes, exposing alternatives beyond the sampled continuation. Teacher-weighted forward KL targets alternatives the student underestimates, with clipping to limit individual vocabulary contributions. Supervision is restricted to the highest-entropy half of student positions, concentrating learning where predictions are uncertain. Experiments on three competition-math benchmarks with Qwen3-1.7B, 4B, and 8B show improvements over the base models without role prompts at inference. Forward KL achieves the highest macro-averaged accuracy among the three evaluated divergences at every scale. Code is available at https://github.com/zhansan114514/OPSRD.

Figures & tables

Figure 1: OPSRD overview. The student generates a role-free trajectory; the frozen expert-role teacher evaluates the same prefixes. The highest-entropy half of student positions receive clipped, full-vocabulary forward KL. Only student LoRA parameters are updated, and deployment uses the ordinary prompt at test time.

Figure 2: One OPSRD update. Student and teacher score the same prefix; clipped vocabulary contributions are summed and averaged over the selected high-entropy positions. Only the student adapter is updated before the next rollout. The small distributions are schematic.

Method	AIME24	AIME25	HMMT25	Macro	$\Delta$ Base
Base	46.67	36.11	24.44	35.74	$0.00$
Answer OPSD	51.39	39.17	25.56	38.70	$+2.96$
Base with expert role	46.94	37.22	19.44	34.54	$-1.20$
Base with neutral prompt	45.83	35.83	21.94	34.54	$-1.20$
Role full	55.56	39.72	25.00	40.09	$+4.35$
OPSRD	57.78	41.67	26.67	42.04	$+6.30$

Table 1: Primary Qwen3-1.7B comparison. Avg@12 in percent; gains in percentage points.

Model	Evaluation	Base	Answer OPSD	OPSRD	$\Delta$ Base	$\Delta$ Answer
Qwen3-1.7B	Primary	35.74	38.70	42.04	$+6.30$	$+3.33$
Qwen3-4B	Mean of 4	60.86	61.50	62.52	$+1.67$	$+1.02$
Qwen3-8B	Primary	62.78	64.26	65.83	$+3.06$	$+1.57$

Table 2: Macro Avg@12. The 4B summary averages four evaluation repeats.

Training role	Test role	Macro
Teacher only	No role	42.04
Teacher only	Expert role	39.44
Shared role	No role	38.98
Shared role	Expert role	36.94
Student only	No role	38.24
Student only	Expert role	36.30

Table 3: Role placement and deployment prompt on Qwen3-1.7B.

Figure 3: Divergence ablation. (a) Macro Avg@12 from the primary evaluation at each scale. (b) Forward-minus-reverse gains with paired 95% problem-bootstrap intervals. Each condition uses one training run; JSD uses H100 and the KL runs use A100.

Model	Repeat	Method	Macro
1.7B	Training	Role full	$39.48\pm 1.32$
1.7B	Training	OPSRD	$40.25\pm 1.62$
4B	Evaluation	Base	$60.86\pm 0.67$
4B	Evaluation	Answer OPSD	$61.50\pm 0.77$
4B	Evaluation	OPSRD	$62.52\pm 0.78$

Table 4: Mean macro Avg@12 and sample standard deviation over three training runs at 1.7B and four evaluations of fixed checkpoints at 4B. The fixed 1.7B Base score is 35.74.

Figure 4: Problem-level changes at 1.7B. Left: problems with higher, equal, or lower sample accuracy than Base. Right: observed success coverage within 12 samples. The main gain is more reliable success on partially solved problems.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Setting
Optimizer / steps	AdamW / 100
Learning rate / maximum gradient norm	$5\times 10^{-6}$ / 0.1
Effective batch size / rollouts per problem	36 / 1
LoRA rank / scale	64 / 128
LoRA projections	q, k, v, o, gate, up, down
Training temperature / top- $p$ / top- $k$	1.1 / 0.95 / 20

Appendix

Table 5: Training and evaluation settings.

Figure 5: Selection and role placement. (a) Entropy ranking retains four of eight schematic positions. (b) Teacher-only conditioning keeps the student role-free during training and deployment.

Model	Method	AIME24	AIME25	HMMT25	Macro	$\Delta$ Base
Qwen3-1.7B	Base	46.67	36.11	24.44	35.74	$0.00$
Qwen3-1.7B	Answer OPSD	51.39	39.17	25.56	38.70	$+2.96$
Qwen3-1.7B	OPSRD	57.78	41.67	26.67	42.04	$+6.30$
Qwen3-4B	Base	73.61	66.11	41.11	60.28	$0.00$
Qwen3-4B	Answer OPSD	75.00	68.61	43.89	62.50	$+2.22$
Qwen3-4B	OPSRD	72.22	67.78	44.44	61.48	$+1.20$

Appendix

Table 6: Per-benchmark scores from the primary evaluation at each scale.

Model	Objective	GPU	AIME24	AIME25	HMMT25	Macro
1.7B	Forward KL	A100	57.78	41.67	26.67	42.04
1.7B	Reverse KL	A100	50.56	35.56	25.56	37.22
1.7B	JSD	H100	51.39	38.61	22.22	37.41
4B	Forward KL	A100	72.22	67.78	44.44	61.48
4B	Reverse KL	A100	74.72	67.50	40.56	60.93
4B	JSD	H100	71.94	65.28	43.06	60.09

Appendix

Table 7: Per-benchmark divergence results. JSD uses recorded aggregate scores.

Training role	Test role	AIME24	AIME25	HMMT25	Macro	$\Delta$
Teacher only	No role	57.78	41.67	26.67	42.04	$0.00$
Teacher only	Expert role	51.11	41.94	25.28	39.44	$-2.59$
Shared role	No role	52.22	40.83	23.89	38.98	$-3.06$
Shared role	Expert role	47.50	39.44	23.89	36.94	$-5.09$
Student only	No role	50.28	39.72	24.72	38.24	$-3.80$
Student only	Expert role	47.22	38.33	23.33	36.30	$-5.74$

Appendix

Table 8: Complete placement ablation, relative to teacher-only training with no test role.

Quantity	Mean
Selected valid-token fraction	0.500
Selected-position entropy	0.874
Omitted-position entropy	0.023
Unclipped selected divergence	0.194

Appendix

Table 9: Entropy selection in the primary run.

Fraction	Avg@4	$\Delta$ 50%
25%	43.06	$0.00$
50%	43.06	$0.00$
75%	38.89	$-4.17$

Appendix

Table 10: Selection-fraction pilot with four samples per problem.

Explore similar work

On-Policy Self-Distillation without Any Supervision
Aug 6, 2026Yijiang Li, Bingyang Wang, Yijun Liang +3Unsupervised On-Policy Self-Distillation Self-Distillation Framework
UC San Diego · Georgia Institute of Technology · University of Maryland, College Park +1
dOPSD: On-Policy Self-Distillation for Diffusion Language Models
Jul 5, 2026Phuong Tuan Dat, Qi Li, Xinchao WangUnsupervised On-Policy Self-Distillation Diffusion Language Models
National University of Singapore
RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation
Jun 10, 2026Leyi Pan, Shuchang Tao, Yunpeng Zhai +5Unsupervised On-Policy Self-Distillation Token-Level Supervision
1Tsinghua University · 2Tongyi Lab , Alibaba Group · 3Peking University

Aug 6, 2026, cs.LGJ/K move · Enter open · S save

On-Policy Self-Distillation without Any Supervision

Yijiang Li, Bingyang Wang, Yijun Liang, Yunjie Tian+2

UC San Diego · Georgia Institute of Technology · University of Maryland, College Park+1

Unsupervised On-Policy Self-Distillation Self-Distillation Framework On-Policy Self-Consistency

Open paper AskPDF arXiv

On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine "self"-distillation. In this study, we show that on-policy self-distillation can be achieved using only a model's own generations via internal consistency. We propose unsupervised on-policy self-distillation (U-OPSD). U-OPSD first samples multiple rollouts and constructs a pseudo solution by majority vote under a self-consistency threshold. It then conditions the model's distribution on the pseudo-solution and distills itself on the disagreeing completions, allowing the model to correct itself precisely where it is confidently wrong. Across diverse benchmarks, base models, and training settings, U-OPSD consistently improves over the base models and matches or surpasses supervised methods with ground truth (GT) such as OPSD and GRPO. On five mathematical reasoning benchmarks, i.e., AIME24, AIME25, HMMT25, MATH500, and AMC23, U-OPSD improves over the base model by 8.5% and 10.7% on Qwen3 non-thinking mode at 4B and 8B scales, and outperforms OPSD by 3.2% and 2.3% on average, respectively. In thinking mode, U-OPSD stays on par with OPSD, ahead by 0.9% at 4B and level at 8B and surpassing GRPO by 0.7% and 1.1%, respectively. Code is available at https://github.com/williamium3000/u-opsd.

Abstract

Explore similar work

On-Policy Self-Distillation without Any Supervision

dOPSD: On-Policy Self-Distillation for Diffusion Language Models

RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation