Function-Space Temporal Self-Distillation for Policy Optimization

Aryan Das; Arnab Bhattacharya; Vinay Kumar Verma

cs.LGOct 14, 2026

Function-Space Temporal Self-Distillation for Policy Optimization

Authors: Aryan Das, Arnab Bhattacharya, Vinay Kumar Verma

Organizations: Indian Institute of Technology Kanpur, India

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) improves reasoning in language models, but scalar outcomes provide coarse supervision over response trajectories. Self-distillation offers denser token-level guidance by comparing an unassisted policy with a feedback-informed self-teacher, yet the teacher evolves with the student, creating unstable targets and motivating additional regularization. We introduce Function-Space Policy Optimization (FSPO), a single-policy framework that combines current privileged adaptation with function-space temporal consolidation. FSPO stores sparse historical next-token distributions as fixed temporal targets, providing temporal separation without maintaining a persistent teacher model. It further introduces privileged disagreement correction to explicitly suppress locally overvalued actions, reliability-adaptive grounding that strengthens reference supervision when self-generated evidence repeatedly fails, and quality and coverage-aware temporal memory that retains verified behaviors and improves privileged demonstrations. Across reasoning and tool-use benchmarks, FSPO achieves the highest average accuracy under a 5-hour training budget, improving over the strongest baseline by approximately 4% points on average while reducing peak memory from about 70 GB to 56 GB on Olmo3-7B-Instruct.

Figures & tables

Figure 1: Average Generated Tokens vs. avg@16 Accuracy (Olmo3-7B-Instruct, 5h) across five tasks. Ours reaches the highest accuracy with the fewest tokens.

	Chemistry		Physics		Biology		Materials		Tool use		Avg@5h
	1h	5h	1h	5h	1h	5h	1h	5h	1h	5h
Qwen3-8B	41.2		59.2		30.8		58.9		57.5		49.5
+ GRPO	65.9	74.5	63.8	72.7	35.1	59.9	74.3	77.1	64.9	67.7	70.4
+ GRPO (on-policy)	63.3	63.4	63.6	63.6	49.8	49.8	73.9	74.1	60.2	65.7	63.3
+ SDPO (on-policy)	73.2	80.9	66.6	75.6	50.6	56.8	72.1	78.4	68.0	68.5	72.0
+ FSPO (on-policy)	81.8	83.4	66.7	76.0	71.2	71.2	81.7	81.7	66.6	68.5	76.2

Table 1: Comparison of FSPO with GRPO and SDPO across benchmarks. We report the highest avg@16 achieved within 1-hour and 5-hour wall-clock training budgets for Qwen3-8B and Olmo3-7B-Instruct, with baseline results for GRPO, on-policy GRPO, and SDPO taken from SDPO Hubotter et al. (2026) . The final column reports the average performance across all five benchmarks at the 5-hour budget. FSPO follows the SDPO training and evaluation setting on 4 NVIDIA GH200 GPUs.

Figure 3: Convergence of best-within-window avg@16 accuracy over wall-clock training time for Qwen3-8B (top row) and Olmo3-7B-Instruct (bottom row) across five tasks.

Figure 4: Average time per training step over wall-clock time for Olmo3-7B-Instruct across five tasks.

Figure 5: Catastrophic-forgetting trade-off on Olmo3-7B-Instruct. Each panel corresponds to training on one task, with the horizontal axis showing the best avg@16 achieved on that task within 5 h and the vertical axis showing the mean score on the $5$ prior-capability benchmarks.

Figure 6: Comparison of Memory Footprint. (a) Breakdown for Olmo3-7B-Instruct: policy weights, gradients, AdamW state, and activations plus KV cache are common to all methods. GRPO adds a synced colocated rollout copy, and SDPO adds a persistent EMA teacher copy. FSPO needs no extra model copy, since on-policy rollouts reuse the model being trained. (b) Projecting the same accounting to larger models, the absolute gap grows with parameter count.

Cases	Setting	Accuracy (%) $\uparrow$
Cases	Setting	Biology	Tool Use	Avg.
Case 1	Full objective (ours)	62.3	66.6	64.5
	$\mathcal{L}_{\mathrm{distill}}$ only	54.9	57.2	56.0
	w/o $\mathcal{L}_{\mathrm{neg}}$	61.9	65.8	63.8
	w/o $\mathcal{L}_{\mathrm{temp}}$	59.9	60.3	60.1
	w/o $\mathcal{L}_{\mathrm{CE}}$	59.1	64.3	61.7

Table 2: Ablation study of FSPO on Olmo3-7B-Instruct across Biology and Tool Use. Shaded rows marked (ours) denote the setting used in the full model; unless stated otherwise.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Dataset	HumanEval	MMLU	HellaSwag	Winogrande	TruthfulQA	Avg.
Olmo3-7B-Instruct	–	47.56	58.77	76.04	66.14	57.79	61.26
GRPO	Chemistry	43.90	54.18	71.42	62.51	53.37	57.08
	Physics	44.51	53.95	70.88	63.14	53.92	57.28
	Biology	42.68	54.58	71.95	63.69	53.14	57.21
	Materials	43.29	53.82	70.61	62.75	53.85	56.86
	Tool use	38.41	52.94	69.95	60.77	51.88	54.79

Appendix

Table 3: Prior-capability retention on Olmo3-7B-Instruct. Performance for each method on

5

prior-capability benchmarks after training on each task dataset.

Cases	Setting	Accuracy (%) $\uparrow$
Cases	Setting	Biology	Tool Use	Avg.
Case 7	$G=4$	59.6	63.9	61.8
	$G=8$ (ours)	62.3	66.6	64.5
	$G=16$	61.4	66.7	64.0
Case 8	Forward KL	59.4	64.2	61.8
	JSD, $\alpha=0.25$	60.8	65.3	63.0

Appendix

Table 4: Additional ablations of FSPO on Olmo3-7B-Instruct across Biology and Tool Use. Shaded rows marked (ours) denote the setting used in the full model; unless stated otherwise, all other settings follow the full model.

Parameters	Value
General
Model	Qwen/Qwen3-8B
	allenai/Olmo3-7B-Instruct
Thinking	False
Data
Max. prompt length	2048

Appendix

Table 5: Training configuration of FSPO.

Figure 7: Training dynamics of FSPO averaged across five tasks on Olmo3-7B-Instruct. Metrics are reported as mean

\pm

\sigma

over Biology, Chemistry, Materials, Physics, and Tool Use. When a task’s training completes before the longest run, its final values are held constant so that all five tasks contribute to the average at every step. (a) Weighted contributions of the current, temporal, and reference-grounding objectives under the cyclic

\rho

schedule. (b) Number of guidance-tier assignments per training step. (c) Per-token JSD divergence for the current and temporal components, with cache signature entropy

H_{\mathrm{sig}}

(right axis; science tasks only). (d) Growth of the temporal cache, showing total and ready entries. (e) Mean rollout reward and per-problem solve rate. (f) Mean cache reward and active-token length normalized to its steady-state baseline.

Figure 8: Memory footprint of the temporal cache during training. CPU memory of ready cache entries plotted against cache size, averaged across the same five tasks as in Figure 7 (d) . The black marker denotes the seeded cache at step 1, and the red marker denotes the largest observed cache. Each individual task cache is non-decreasing; completed runs are held at their final value so that the cross-task average grows monotonically.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation
Sep 29, 2026Zhenrui Yue, Huimin Zeng, Yueqi Wang +8RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
University of Illinois · UC San Diego · Rochester Institute of Technology +2
Self-Evolution Reasoner: Continuous Optimization via Policy-Intrinsic Exploration and Exploitation
Oct 25, 2026Wenhang Shi, Yiren Chen, Zhe Zhao +3Language Model Self-Improvement RL for Language Model Reasoning
Renmin University of China · Peking University · Tencent
DASH: Divergence-Adaptive Supervision Horizons for On-Policy Self-Distillation of Reasoning Models
Aug 6, 2026ZhiYan Hou, Xinyu Tang, Hongyan An +9On-Policy Self-Distillation RL for Language Model Reasoning
Institute of Automation, Chinese Academy of Sciences · EverMind · Shanda Group +3

Sep 29, 2026J/K move · Enter open · S save

SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation

Zhenrui Yue, Huimin Zeng, Yueqi Wang, Yaokun Liu+7

University of Illinois · UC San Diego · Rochester Institute of Technology+2

RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards On-Policy Distillation Knowledge Distillation

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for improving large language models (LLMs) on various tasks, yet its sparse outcome rewards lack token-level credit assignment for intermediate steps. To address this, on-policy self-distillation (OPSD) leverages a self-teacher with privileged context to provide additional dense learning signals. However, because the self-teacher is often overconfident and imposes excessive penalties on long reasoning trajectories, OPSD frequently struggles in practice. To mitigate this, we propose self-instructing policy optimization (SIPO) with a contrastive self-teacher to provide dense credit. At each iteration, SIPO samples multiple rollouts per prompt from the current policy, scores them with environment rewards, and constructs two teacher contexts for each rollout by pairing the reference answer with mistakes made within the group. The model then re-evaluates its own responses under both contexts, using the difference between the two teacher log-probabilities as token-level feedback, so that biases shared by both contexts are expected to largely cancel. The resulting objective yields a token-level advantage for every rollout: the reward still sets the main direction of each update while the self-teacher redistributes credit across tokens. Even in groups where every rollout fails and group-relative advantages vanish, SIPO still provides a learning signal. By preserving direct optimization of the task reward while providing dense, token-level feedback, this approach bridges reinforcement learning and on-policy self-distillation. Extensive experiments across multiple reasoning and code-generation benchmarks demonstrate that SIPO outperforms both RLVR and OPSD baselines without an external teacher or additional generation.

Abstract

RL for Language Model Reasoning

Explore similar work

SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation

Self-Evolution Reasoner: Continuous Optimization via Policy-Intrinsic Exploration and Exploitation

DASH: Divergence-Adaptive Supervision Horizons for On-Policy Self-Distillation of Reasoning Models