RISE: Recursive Improvement via Self-Extrapolating Policy Distillation

Yang Li; Semih Yavuz; Shafiq Joty

cs.AISep 4, 2026

RISE: Recursive Improvement via Self-Extrapolating Policy Distillation

Authors: Yang Li, Semih Yavuz, Shafiq Joty

Organizations: Salesforce AI Research

Abstract

On-policy distillation (OPD) provides dense, per-token supervision for language model post-training, but its effectiveness is bottlenecked by teacher quality: external teachers suffer from distribution mismatch, while self-distillation with privileged conditioning is limited by in-context learning capacity. We propose RISE (Recursive Improvement via Self-Extrapolating Policy Distillation), which constructs a synthetic teacher directly from the model's own RLVR training trajectory. By extrapolating the displacement between the current checkpoint and a trailing anchor---in parameter space or output logit space---RISE converts a sparse outcome-induced parameter update into a dense token-level target, without any external model or privileged conditioning. RISE combines RLVR and OPD in a complementary loop: outcome rewards ground the extrapolation toward correct reasoning, while the extrapolated teacher refines token-level decisions. Moreover, since the teacher is refreshed every iteration as the student improves, distillation becomes a recursive improvement mechanism rather than a one-shot compression step. Experiments spanning mathematical reasoning, multi-domain STEM, code generation, and multi-turn agentic tasks show that RISE outperforms RLVR-only training and on-policy self-distillation across all settings.

Figures & tables

Panel of Figure 1 — Figure 1: RISE overview. (a) RLVR updates $\theta_{n}\to\theta_{n\text{+}1}^{\prime}$ (blue); extrapolation amplifies this displacement to construct $\theta_{\text{future}}$ (red); OPD distills $\theta_{\text{future}}$ into the student, yielding $\theta_{n\text{+}1}$ (green). (b) The training loop: RLVR grounds the direction, while OPD refines token-level decisions.

Method	In-Domain Math							OOD
Method	MATH500	AIME24	AIME25	AMC23	Minerva	OlyBench	Avg.	GPQA	IFEval	MMLU	Avg.
Qwen3-8B (DAPOMath)
Base	73.2	27.1	23.1	66.1	21.1	48.6	43.2	50.8	82.4	68.4	67.2
GRPO	83.8	54.4	42.9	89.4	31.5	57.9	60.0	57.0	81.5	73.3	70.6
GRPO+SDPO	83.2	42.3	32.7	84.4	30.6	62.5	55.9	55.7	82.8	69.4	69.3
SDAR	84.5	50.2	38.3	89.4	31.2	64.7	59.7	55.5	82.4	70.1	69.3

Table 1: Mathematical reasoning results. Accuracy (%) across model configurations. We evaluate on in-domain math benchmarks and out-of-distribution (OOD) tasks to assess preservation. Best in bold , second best underlined .

Panel of Figure 2 — Figure 2: RISE reaches higher accuracy in fewer steps across three model scales, demonstrating improved sample efficiency.

Method	Math							STEM
Method	MATH500	AIME24	AIME25	AMC23	Minerva	OlyBench	Avg.	GPQA	SuperGPQA	MMLU	ThrmQA	Avg.
Base	48.2	9.8	10.4	35.0	12.1	27.5	23.8	14.1	8.0	6.6	27.0	13.9
GRPO	69.1	23.8	20.6	59.4	26.6	42.0	40.2	42.4	32.5	59.5	47.8	45.5
GRPO+SDPO	70.5	24.8	20.4	68.1	25.9	43.1	42.1	44.7	31.0	61.7	50.5	47.0
SDAR	69.1	22.9	20.0	65.6	23.7	41.2	40.4	40.6	30.3	58.7	49.1	44.7
RLSD	68.9	23.1	20.0	60.0	24.6	39.9	39.4	33.7	28.8	55.1	46.5	41.0

Table 2: Multi-domain results (Qwen3-4B-Base, mixed math + STEM training). Accuracy (%) on math reasoning and STEM benchmarks. Best in bold , second best underlined .

Method	ALF	WS-S	WS-A
Base	21.9	6.7	0.8
GRPO	75.0	79.8	63.3
RISE (logit)	78.1	78.8	68.0
RISE (weight)	84.4	86.3	74.2

Table 3: Success rate on ALFWorld, Score/Acc on WebShop.

Figure 6: Gain on AIME’24 (avg@16) from extrapolating Qwen3-1.7B GRPO checkpoints at varying

\beta

, relative to

\beta\!=\!1

. Shaded: RISE’s operating range (

\beta_{0}\!=\!1.2

, decaying to 1).

	$\beta_{0}$ (linear decay)			Schedule ( $\beta_{0}\!=\!1.2$ )
	1.2	1.3	1.5	cosine	linear	exp	fixed
Math Avg	62.5	62.3	61.9	63.0	62.5	62.0	61.8

Table 4:

\beta_{0}

and schedule ablation (Qwen3-8B, Math Avg %).

Figure 7: Anchor ablation on Qwen3-1.7B. Lower

\eta

improves Math Avg.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1 : The RLVR weight trajectory is low-dimensional. Principal spectrum of the displacements

\delta_{t}=\theta_{t}-\theta_{0}

from GRPO checkpoints on DAPOMath. Bars show variance explained per component; lines show the cumulative total. A perfectly linear trajectory would place

100\%

on the first component; the observed

68.2\%

reflects curvature along the path, while three components capture

{\sim}87\%

at both scales, confining the trajectory to a low-dimensional subspace.

	Qwen3-1.7B	Qwen3-1.7B-Base	Qwen3-8B	Qwen3-4B-Base	Qwen3-8B-Base	OLMo3-7B-Instruct-SFT
	(DAPOMath)	(DAPOMath)	(DAPOMath)	(multi-domain)	(code)	(OpenR1-Math)
Learning rate	1e-6	1e-6	1e-6	1e-6	1e-6	1e-6
Train batch size	64	64	64	64	64	128
Mini-batch size	32	32	32	32	32	64
Rollouts $n$	8	8	8	8	8	8
Epochs	1	1	1	1	1	1

Appendix

Table B.1: Training hyperparameters.

n

denotes the number of rollouts sampled per prompt. Train batch size is the number of prompts sampled per iteration for rollout generation; mini-batch size is the number of prompts per gradient update within each iteration (the train batch is split into

\text{train batch size}/\text{mini-batch size}

gradient steps). The OPD phase uses the same mini-batch size as the RLVR phase. Qwen2.5-3B agentic runs follow GIGPO’s setup ( Feng et al., 2025 ) ; see their paper for full details.

Method	In-Domain Math							OOD
Method	MATH500	AIME24	AIME25	AMC23	Minerva	OlyBench	Avg.	GPQA	IFEval	MMLU	Avg.
Base	40.0	4.4	2.9	27.5	11.6	19.8	17.7	12.7	14.4	8.0	11.7
GRPO	60.6	9.4	6.9	42.5	19.3	29.2	28.0	26.5	31.1	39.6	32.4
GRPO+SDPO	61.3	8.3	6.7	41.9	20.1	32.3	28.4	28.7	30.5	40.5	33.2
SDAR	55.6	9.6	8.3	43.1	17.7	27.1	26.9	23.2	28.5	42.5	31.4
RLSD	57.2	11.9	7.5	43.1	17.8	28.2	27.6	23.0	29.0	42.4	31.5

Appendix

Table C.1 : Qwen3-1.7B-Base (DAPOMath). Accuracy (%) on math and OOD benchmarks. Best in bold , second best underlined .

Panel of Figure 14 — Figure C.1 : Per-metric convergence curves for all model configurations. Each row shows four representative benchmarks. RISE (both variants) consistently reaches higher accuracy in fewer training steps.

Panel of Figure 15 — Figure C.2 : Code generation convergence curves (Qwen3-8B-Base). Both RISE variants consistently converge faster than GRPO across all three benchmarks under both avg@4 and pass@4 metrics.

		AIME’24		AIME’25
Model	Method	avg@16	pass@16	avg@16	pass@16
Qwen3-8B	GRPO	54.4	78.7	42.9	63.3
	RISE (logit)	56.9 +2.5	79.0 +0.3	46.7 +3.8	62.7 -0.6
	RISE (weight)	58.1 +3.7	81.5 +2.8	45.8 +2.9	67.2 +3.9
Qwen3-1.7B	GRPO	30.0	54.4	26.3	45.8
	RISE (logit)	36.3 +6.3	64.0 +9.6	33.3 +7.0	53.9 +8.1

Appendix

Table C.2 : Average vs. best-of-16 accuracy (%) on AIME benchmarks.

\Delta

denotes improvement over GRPO. RISE improves pass@16 (coverage) alongside avg@16 (mean quality), with the largest coverage gains at 1.7B.

Figure C.3 : Grounding ablation: training reward and policy entropy. Same three Qwen3-8B runs as Fig. 5(a) . Without RLVR (grey), training reward collapses to zero (left) while policy entropy spikes then crashes (right), confirming the phase-transition nature of the collapse.

Model	Method	MATH500	AIME24	AIME25	AMC23	Minerva	OlyBench	Avg.
Qwen3-8B	GRPO	83.8	54.4	42.9	89.4	31.5	57.9	60.0
	RISE (logit)	84.8	56.9	46.7	91.6	32.5	62.4	62.5
	RISE (weight)	84.4	58.1	45.8	91.9	32.6	63.3	62.7
	w/o OPD	84.6	51.7	39.8	90.6	31.2	64.1	60.3
Qwen3-1.7B	GRPO	75.6	30.0	26.3	65.6	23.9	51.1	45.4
	RISE (logit)	75.2	36.3	33.3	78.8	24.9	52.9	50.2

Appendix

Table C.3 : Without-OPD ablation. Accuracy (%) on math benchmarks. The w/o OPD variant adopts

\theta_{\text{future}}

directly without distillation. All runs are configuration-matched to their RISE counterpart.

Qwen3-8B — $\beta_{0}$ sensitivity (linear decay)
Configuration	MATH500	AIME24	AIME25	AMC23	Minerva	OlyBench	Avg.
$\beta_{0}=1.2$	84.8	56.9	46.7	91.6	32.5	62.4	62.5
$\beta_{0}=1.3$	83.3	56.9	47.3	92.5	32.1	61.9	62.3
$\beta_{0}=1.5$	83.5	55.2	47.9	91.2	31.9	61.6	61.9
Qwen3-8B — decay schedule ( $\beta_{0}=1.2$ )
Linear	84.8	56.9	46.7	91.6	32.5	62.4	62.5

Appendix

Table C.4 : Detailed

\beta_{0}

and decay schedule ablation results. Accuracy (%) on individual math benchmarks. The top block varies

\beta_{0}

with linear decay; the middle block varies the schedule with

\beta_{0}\!=\!1.2

. Both blocks use Qwen3-8B. The bottom block varies

\beta_{0}

on Qwen3-1.7B-Base (schedule noted in parentheses).

Model	Variant	MATH500	AIME24	AIME25	AMC23	Minerva	OlyBench	Avg.
Qwen3-8B	Rollout reuse	84.8	56.9	46.7	91.6	32.5	62.4	62.5
Qwen3-8B	Resample	83.7	58.5	47.9	90.6	31.3	63.1	62.5
Qwen3-1.7B -Base	Rollout reuse	59.6	12.1	8.1	45.6	20.2	29.4	29.2
Qwen3-1.7B -Base	Resample	59.6	11.9	7.5	43.8	18.8	29.0	28.4

Appendix

Table C.5 : Resample vs. rollout reuse. Accuracy (%) on individual math benchmarks. “Resample” generates fresh rollouts from

\pi_{\theta_{n+1}^{\prime}}

for OPD; “Rollout reuse” reuses RLVR rollouts.

Qwen3-1.7B — Logit-space
Configuration	MATH500	AIME24	AIME25	AMC23	Minerva	OlyBench	Avg.
$\eta=1.0$	77.4	29.4	26.0	71.3	24.8	48.6	46.2
$\eta=0.95$	75.4	32.7	29.8	73.1	25.0	50.2	47.7
$\eta=0.3$	74.6	33.3	29.4	76.9	25.5	50.6	48.4
$\eta=0.1$	75.2	36.3	33.3	78.8	24.9	52.9	50.2
Qwen3-1.7B — Weight-space

Appendix

Table C.6 : Anchor ablation. Accuracy (%) on individual math benchmarks.

\eta\!=\!1

uses the previous checkpoint as anchor;

\eta\!<\!1

uses an EMA anchor. EMA helps on Qwen3-1.7B but degrades on OLMo3-7B-Instruct-SFT.

Model	Method	MATH500	AIME24	AIME25	AMC23	Minerva	OlyBench	Avg.
Qwen3-8B	GRPO	83.8	54.4	42.9	89.4	31.5	57.9	60.0
	GRPO-2 $\times$	84.5	53.5	42.3	92.5	31.4	58.9	60.5
	RISE (logit)	84.8	56.9	46.7	91.6	32.5	62.4	62.5
	RISE (weight)	84.4	58.1	45.8	91.9	32.6	63.3	62.7
Qwen3-1.7B	GRPO	75.6	30.0	26.3	65.6	23.9	51.1	45.4
	GRPO-2 $\times$	74.5	32.9	30.6	70.6	25.6	49.7	47.3

Appendix

Table C.7 : Compute-matched comparison: RISE vs. GRPO-2

\times

. Accuracy (%) on math benchmarks. GRPO-2

\times

performs a second inner-loop gradient pass on the same rollouts within each iteration, matching RISE’s total gradient budget at equal sampling cost.

Method	In-Domain Math							OOD
Method	MATH500	AIME24	AIME25	AMC23	Minerva	OlyBench	Avg.	GPQA	IFEval	MMLU	Avg.
Qwen3-8B (DAPOMath)
GRPO	84.0 ±0.5	53.1 ±1.2	42.5 ±0.3	88.5 ±2.1	31.4 ±0.4	61.3 ±2.7	60.1 ±0.2	56.3 ±0.5	82.2 ±0.5	71.6 ±1.2	70.0 ±0.4
SDAR	84.7 ±0.4	48.9 ±1.0	39.1 ±1.2	90.0 ±1.3	30.6 ±0.6	64.2 ±0.7	59.6 ±0.6	55.3 ±0.3	83.2 ±0.6	71.4 ±1.5	69.9 ±0.6
RISE (weight)	84.9 ±0.5	55.8 ±1.7	44.1 ±1.2	92.8 ±0.6	32.3 ±0.2	64.4 ±0.9	62.4 ±0.2	58.1 ±0.7	84.1 ±0.5	72.3 ±1.0	71.5 ±0.4
Qwen3-1.7B (DAPOMath)

Appendix

Table C.8 : Multi-seed reproducibility (DAPOMath): mean

\pm

std across 3 random seeds. Standard deviations are shown in gray . RISE’s Math Avg gains over GRPO far exceed seed-to-seed variance at both scales.

Language Model Distillation

Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.

See all 508 papers

Oct 19, 2026

Bridged Self-Distillation for On-Policy Learning

On-policy distillation can transfer the reasoning capabilities of a strong teacher to a smaller student. However, distillation from a stronger teacher model faces two mismatches: its solutions may be difficult for a smaller student to imitate, and its token-level probabilities may be incompatible with, or even unavailable to, the student. Existing methods handle these settings through different forms of supervision. To this end, we propose Bridged Self-Distillation (BSD), a unified framework for distilling from any teacher that provides reasoning traces. The model to be distilled from, termed the external teacher, first provides a reasoning trace as privileged context, which can be generated in an offline fashion. The student generates rollouts without this context, while a copy of the student conditioned on it, serving as the self-teacher, scores the same rollouts in the student's own token space. BSD uses the resulting token-level guidance to adjust updates on unsuccessful rollouts, with a verifier providing the outcome signal. As the external teacher is queried only to produce text, the same training procedure applies to all teacher scenarios: shared-tokenizer, cross-tokenizer, and closed API teachers. Across these scenarios, BSD improves over the base model and outperforms baselines tailored to each scenario, offering a novel and unified pathway for distillation. Code is available: https://github.com/TUMLegalTech/Bridged-Self-Distillation

Runsheng Chen, Yihong Liu, Raoyuan Zhao +3

Technical University of Munich, Germany · LMU Munich, Germany · Munich Center for Machine Learning, Germany

Reasoning Distillation On-Policy Distillation On-Policy Self-Distillation Cross-Tokenizer Knowledge Distillation

Ask PDF

Sep 25, 2026

Recursive Self-Improvement via On-Policy Distillation for Reasoning

On-policy distillation (OPD) trains a student model by having it generate trajectories, then matching its next-token predictions with an external teacher's next-token predictions. This provides dense, token-level supervision to the student. On-policy self-distillation (OPSD) eliminates the need for the external teacher. Specifically, a second frozen copy of the student model, now given the ground truth in its context, serves as the teacher. The student model only receives the problem and learns to mimic the privileged teacher model, while the teacher remains frozen throughout training. Previous work showed that freezing the teacher is useful for training stability, but we argue that this can prevent the teacher from incorporating the improvements learned by the student during training. Our primary contribution is to address this limitation with a recursive framework built around two complementary components. First, we let the privileged teacher co-evolve with the student so that revision learned in one round can guide the next, a process we refer to as Dynamic Co-Evolution (DCE). Second, because stronger revision can also make responses too verbose and self-critical, we additionally train on shorter, verified rewrites of the model's own on-policy responses. We call this complementary objective Self-Refined Concise Learning (SRCL). Overall, our comprehensive evaluations show that DCE+SRCL outperforms OPSD across multiple model scales and four competition-level mathematics benchmarks. Specifically, on Qwen3-8B, DCE+SRCL reaches 65.97% Average@12, outperforming OPSD by 35.62 percentage points while reducing mean output length by 7.80% relative to DCE alone.

Shangjian Yin, Zehao Zhao, Kavosh Asadi +7

Meta AI · University of California, Riverside · Work done at Meta

Language Model Self-Improvement On-Policy Self-Distillation Self-Distillation Mathematical Reasoning

Ask PDF

Aug 19, 2026

Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress

On-policy distillation (OPD) has emerged as an effective framework for post-training language models by pairing student-generated trajectories with dense token-level supervision from a teacher. However, OPD implicitly assumes that teacher-derived rewards are an appropriate proxy for reasoning progress, and therefore treats all teacher feedback equally during policy optimization. While in practice, this assumption does not always hold. We observe that teacher-derived rewards often conflict with genuine reasoning progress, as reasoning steps with clear reasoning advancement may still receive lower distillation rewards, simply due to deviation from teacher's outputs. To address this mismatch, we propose Reasoning-Progress-Aware Reward Filtering for On-Policy Distillation (R2-OPD), which constructs two within-trajectory rankings of reasoning spans, one from teacher-derived rewards and the other from independently estimated progress reward. Distillation rewards are selectively suppressed whenever the two rankings disagree, reducing supervision that conflicts with reasoning progress while preserving effective teacher guidance. Our approach shows consistent improvement over standard OPD especially regarding reasoning performances.

Chen Yang, Haiyuan Wan, Rengrong Xiong +2

The Hong Kong University of Science and Technology (Guangzhou) · Tsinghua University · Zhejiang University +1

Language Model Distillation On-Policy Distillation RL for Language Model Reasoning

Ask PDF

Abstract

Language Model Distillation

Explore similar work

Bridged Self-Distillation for On-Policy Learning

Recursive Self-Improvement via On-Policy Distillation for Reasoning

Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress