RISE: Recursive Improvement via Self-Extrapolating Policy Distillation

Yang Li; Semih Yavuz; Shafiq Joty

cs.AISep 4, 2026

RISE: Recursive Improvement via Self-Extrapolating Policy Distillation

Authors: Yang Li, Semih Yavuz, Shafiq Joty

Organizations: Salesforce AI Research

Abstract

On-policy distillation (OPD) provides dense, per-token supervision for language model post-training, but its effectiveness is bottlenecked by teacher quality: external teachers suffer from distribution mismatch, while self-distillation with privileged conditioning is limited by in-context learning capacity. We propose RISE (Recursive Improvement via Self-Extrapolating Policy Distillation), which constructs a synthetic teacher directly from the model's own RLVR training trajectory. By extrapolating the displacement between the current checkpoint and a trailing anchor---in parameter space or output logit space---RISE converts a sparse outcome-induced parameter update into a dense token-level target, without any external model or privileged conditioning. RISE combines RLVR and OPD in a complementary loop: outcome rewards ground the extrapolation toward correct reasoning, while the extrapolated teacher refines token-level decisions. Moreover, since the teacher is refreshed every iteration as the student improves, distillation becomes a recursive improvement mechanism rather than a one-shot compression step. Experiments spanning mathematical reasoning, multi-domain STEM, code generation, and multi-turn agentic tasks show that RISE outperforms RLVR-only training and on-policy self-distillation across all settings.

Figures & tables

Panel of Figure 1 — Figure 1: RISE overview. (a) RLVR updates $\theta_{n}\to\theta_{n\text{+}1}^{\prime}$ (blue); extrapolation amplifies this displacement to construct $\theta_{\text{future}}$ (red); OPD distills $\theta_{\text{future}}$ into the student, yielding $\theta_{n\text{+}1}$ (green). (b) The training loop: RLVR grounds the direction, while OPD refines token-level decisions.

Method	In-Domain Math							OOD
Method	MATH500	AIME24	AIME25	AMC23	Minerva	OlyBench	Avg.	GPQA	IFEval	MMLU	Avg.
Qwen3-8B (DAPOMath)
Base	73.2	27.1	23.1	66.1	21.1	48.6	43.2	50.8	82.4	68.4	67.2
GRPO	83.8	54.4	42.9	89.4	31.5	57.9	60.0	57.0	81.5	73.3	70.6
GRPO+SDPO	83.2	42.3	32.7	84.4	30.6	62.5	55.9	55.7	82.8	69.4	69.3
SDAR	84.5	50.2	38.3	89.4	31.2	64.7	59.7	55.5	82.4	70.1	69.3

Table 1: Mathematical reasoning results. Accuracy (%) across model configurations. We evaluate on in-domain math benchmarks and out-of-distribution (OOD) tasks to assess preservation. Best in bold , second best underlined .

Panel of Figure 2 — Figure 2: RISE reaches higher accuracy in fewer steps across three model scales, demonstrating improved sample efficiency.

Method	Math							STEM
Method	MATH500	AIME24	AIME25	AMC23	Minerva	OlyBench	Avg.	GPQA	SuperGPQA	MMLU	ThrmQA	Avg.
Base	48.2	9.8	10.4	35.0	12.1	27.5	23.8	14.1	8.0	6.6	27.0	13.9
GRPO	69.1	23.8	20.6	59.4	26.6	42.0	40.2	42.4	32.5	59.5	47.8	45.5
GRPO+SDPO	70.5	24.8	20.4	68.1	25.9	43.1	42.1	44.7	31.0	61.7	50.5	47.0
SDAR	69.1	22.9	20.0	65.6	23.7	41.2	40.4	40.6	30.3	58.7	49.1	44.7
RLSD	68.9	23.1	20.0	60.0	24.6	39.9	39.4	33.7	28.8	55.1	46.5	41.0

Table 2: Multi-domain results (Qwen3-4B-Base, mixed math + STEM training). Accuracy (%) on math reasoning and STEM benchmarks. Best in bold , second best underlined .

Method	ALF	WS-S	WS-A
Base	21.9	6.7	0.8
GRPO	75.0	79.8	63.3
RISE (logit)	78.1	78.8	68.0
RISE (weight)	84.4	86.3	74.2

Table 3: Success rate on ALFWorld, Score/Acc on WebShop.

Figure 6: Gain on AIME’24 (avg@16) from extrapolating Qwen3-1.7B GRPO checkpoints at varying

\beta

, relative to

\beta\!=\!1

. Shaded: RISE’s operating range (

\beta_{0}\!=\!1.2

, decaying to 1).

	$\beta_{0}$ (linear decay)			Schedule ( $\beta_{0}\!=\!1.2$ )
	1.2	1.3	1.5	cosine	linear	exp	fixed
Math Avg	62.5	62.3	61.9	63.0	62.5	62.0	61.8

Table 4:

\beta_{0}

and schedule ablation (Qwen3-8B, Math Avg %).

Figure 7: Anchor ablation on Qwen3-1.7B. Lower

\eta

improves Math Avg.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1 : The RLVR weight trajectory is low-dimensional. Principal spectrum of the displacements

\delta_{t}=\theta_{t}-\theta_{0}

from GRPO checkpoints on DAPOMath. Bars show variance explained per component; lines show the cumulative total. A perfectly linear trajectory would place

100\%

on the first component; the observed

68.2\%

reflects curvature along the path, while three components capture

{\sim}87\%

at both scales, confining the trajectory to a low-dimensional subspace.

	Qwen3-1.7B	Qwen3-1.7B-Base	Qwen3-8B	Qwen3-4B-Base	Qwen3-8B-Base	OLMo3-7B-Instruct-SFT
	(DAPOMath)	(DAPOMath)	(DAPOMath)	(multi-domain)	(code)	(OpenR1-Math)
Learning rate	1e-6	1e-6	1e-6	1e-6	1e-6	1e-6
Train batch size	64	64	64	64	64	128
Mini-batch size	32	32	32	32	32	64
Rollouts $n$	8	8	8	8	8	8
Epochs	1	1	1	1	1	1

Appendix

Table B.1: Training hyperparameters.

n

denotes the number of rollouts sampled per prompt. Train batch size is the number of prompts sampled per iteration for rollout generation; mini-batch size is the number of prompts per gradient update within each iteration (the train batch is split into

\text{train batch size}/\text{mini-batch size}

gradient steps). The OPD phase uses the same mini-batch size as the RLVR phase. Qwen2.5-3B agentic runs follow GIGPO’s setup ( Feng et al., 2025 ) ; see their paper for full details.

Method	In-Domain Math							OOD
Method	MATH500	AIME24	AIME25	AMC23	Minerva	OlyBench	Avg.	GPQA	IFEval	MMLU	Avg.
Base	40.0	4.4	2.9	27.5	11.6	19.8	17.7	12.7	14.4	8.0	11.7
GRPO	60.6	9.4	6.9	42.5	19.3	29.2	28.0	26.5	31.1	39.6	32.4
GRPO+SDPO	61.3	8.3	6.7	41.9	20.1	32.3	28.4	28.7	30.5	40.5	33.2
SDAR	55.6	9.6	8.3	43.1	17.7	27.1	26.9	23.2	28.5	42.5	31.4
RLSD	57.2	11.9	7.5	43.1	17.8	28.2	27.6	23.0	29.0	42.4	31.5

Appendix

Table C.1 : Qwen3-1.7B-Base (DAPOMath). Accuracy (%) on math and OOD benchmarks. Best in bold , second best underlined .

Panel of Figure 14 — Figure C.1 : Per-metric convergence curves for all model configurations. Each row shows four representative benchmarks. RISE (both variants) consistently reaches higher accuracy in fewer training steps.

Panel of Figure 15 — Figure C.2 : Code generation convergence curves (Qwen3-8B-Base). Both RISE variants consistently converge faster than GRPO across all three benchmarks under both avg@4 and pass@4 metrics.

		AIME’24		AIME’25
Model	Method	avg@16	pass@16	avg@16	pass@16
Qwen3-8B	GRPO	54.4	78.7	42.9	63.3
	RISE (logit)	56.9 +2.5	79.0 +0.3	46.7 +3.8	62.7 -0.6
	RISE (weight)	58.1 +3.7	81.5 +2.8	45.8 +2.9	67.2 +3.9
Qwen3-1.7B	GRPO	30.0	54.4	26.3	45.8
	RISE (logit)	36.3 +6.3	64.0 +9.6	33.3 +7.0	53.9 +8.1

Appendix

Table C.2 : Average vs. best-of-16 accuracy (%) on AIME benchmarks.

\Delta

denotes improvement over GRPO. RISE improves pass@16 (coverage) alongside avg@16 (mean quality), with the largest coverage gains at 1.7B.

Figure C.3 : Grounding ablation: training reward and policy entropy. Same three Qwen3-8B runs as Fig. 5(a) . Without RLVR (grey), training reward collapses to zero (left) while policy entropy spikes then crashes (right), confirming the phase-transition nature of the collapse.

Model	Method	MATH500	AIME24	AIME25	AMC23	Minerva	OlyBench	Avg.
Qwen3-8B	GRPO	83.8	54.4	42.9	89.4	31.5	57.9	60.0
	RISE (logit)	84.8	56.9	46.7	91.6	32.5	62.4	62.5
	RISE (weight)	84.4	58.1	45.8	91.9	32.6	63.3	62.7
	w/o OPD	84.6	51.7	39.8	90.6	31.2	64.1	60.3
Qwen3-1.7B	GRPO	75.6	30.0	26.3	65.6	23.9	51.1	45.4
	RISE (logit)	75.2	36.3	33.3	78.8	24.9	52.9	50.2

Appendix

Table C.3 : Without-OPD ablation. Accuracy (%) on math benchmarks. The w/o OPD variant adopts

\theta_{\text{future}}

directly without distillation. All runs are configuration-matched to their RISE counterpart.

Qwen3-8B — $\beta_{0}$ sensitivity (linear decay)
Configuration	MATH500	AIME24	AIME25	AMC23	Minerva	OlyBench	Avg.
$\beta_{0}=1.2$	84.8	56.9	46.7	91.6	32.5	62.4	62.5
$\beta_{0}=1.3$	83.3	56.9	47.3	92.5	32.1	61.9	62.3
$\beta_{0}=1.5$	83.5	55.2	47.9	91.2	31.9	61.6	61.9
Qwen3-8B — decay schedule ( $\beta_{0}=1.2$ )
Linear	84.8	56.9	46.7	91.6	32.5	62.4	62.5

Appendix

Table C.4 : Detailed

\beta_{0}

and decay schedule ablation results. Accuracy (%) on individual math benchmarks. The top block varies

\beta_{0}

with linear decay; the middle block varies the schedule with

\beta_{0}\!=\!1.2

. Both blocks use Qwen3-8B. The bottom block varies

\beta_{0}

on Qwen3-1.7B-Base (schedule noted in parentheses).

Model	Variant	MATH500	AIME24	AIME25	AMC23	Minerva	OlyBench	Avg.
Qwen3-8B	Rollout reuse	84.8	56.9	46.7	91.6	32.5	62.4	62.5
Qwen3-8B	Resample	83.7	58.5	47.9	90.6	31.3	63.1	62.5
Qwen3-1.7B -Base	Rollout reuse	59.6	12.1	8.1	45.6	20.2	29.4	29.2
Qwen3-1.7B -Base	Resample	59.6	11.9	7.5	43.8	18.8	29.0	28.4

Appendix

Table C.5 : Resample vs. rollout reuse. Accuracy (%) on individual math benchmarks. “Resample” generates fresh rollouts from

\pi_{\theta_{n+1}^{\prime}}

for OPD; “Rollout reuse” reuses RLVR rollouts.

Qwen3-1.7B — Logit-space
Configuration	MATH500	AIME24	AIME25	AMC23	Minerva	OlyBench	Avg.
$\eta=1.0$	77.4	29.4	26.0	71.3	24.8	48.6	46.2
$\eta=0.95$	75.4	32.7	29.8	73.1	25.0	50.2	47.7
$\eta=0.3$	74.6	33.3	29.4	76.9	25.5	50.6	48.4
$\eta=0.1$	75.2	36.3	33.3	78.8	24.9	52.9	50.2
Qwen3-1.7B — Weight-space

Appendix

Table C.6 : Anchor ablation. Accuracy (%) on individual math benchmarks.

\eta\!=\!1

uses the previous checkpoint as anchor;

\eta\!<\!1

uses an EMA anchor. EMA helps on Qwen3-1.7B but degrades on OLMo3-7B-Instruct-SFT.

Model	Method	MATH500	AIME24	AIME25	AMC23	Minerva	OlyBench	Avg.
Qwen3-8B	GRPO	83.8	54.4	42.9	89.4	31.5	57.9	60.0
	GRPO-2 $\times$	84.5	53.5	42.3	92.5	31.4	58.9	60.5
	RISE (logit)	84.8	56.9	46.7	91.6	32.5	62.4	62.5
	RISE (weight)	84.4	58.1	45.8	91.9	32.6	63.3	62.7
Qwen3-1.7B	GRPO	75.6	30.0	26.3	65.6	23.9	51.1	45.4
	GRPO-2 $\times$	74.5	32.9	30.6	70.6	25.6	49.7	47.3

Appendix

Table C.7 : Compute-matched comparison: RISE vs. GRPO-2

\times

. Accuracy (%) on math benchmarks. GRPO-2

\times

performs a second inner-loop gradient pass on the same rollouts within each iteration, matching RISE’s total gradient budget at equal sampling cost.

Method	In-Domain Math							OOD
Method	MATH500	AIME24	AIME25	AMC23	Minerva	OlyBench	Avg.	GPQA	IFEval	MMLU	Avg.
Qwen3-8B (DAPOMath)
GRPO	84.0 ±0.5	53.1 ±1.2	42.5 ±0.3	88.5 ±2.1	31.4 ±0.4	61.3 ±2.7	60.1 ±0.2	56.3 ±0.5	82.2 ±0.5	71.6 ±1.2	70.0 ±0.4
SDAR	84.7 ±0.4	48.9 ±1.0	39.1 ±1.2	90.0 ±1.3	30.6 ±0.6	64.2 ±0.7	59.6 ±0.6	55.3 ±0.3	83.2 ±0.6	71.4 ±1.5	69.9 ±0.6
RISE (weight)	84.9 ±0.5	55.8 ±1.7	44.1 ±1.2	92.8 ±0.6	32.3 ±0.2	64.4 ±0.9	62.4 ±0.2	58.1 ±0.7	84.1 ±0.5	72.3 ±1.0	71.5 ±0.4
Qwen3-1.7B (DAPOMath)

Appendix

Table C.8 : Multi-seed reproducibility (DAPOMath): mean

\pm

std across 3 random seeds. Standard deviations are shown in gray . RISE’s Math Avg gains over GRPO far exceed seed-to-seed variance at both scales.

Language Model Distillation

Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.

See all 508 papers

Bridged Self-Distillation for On-Policy Learning
Oct 19, 2026Runsheng Chen, Yihong Liu, Raoyuan Zhao +3Reasoning Distillation On-Policy Distillation
Technical University of Munich, Germany · LMU Munich, Germany · Munich Center for Machine Learning, Germany
Recursive Self-Improvement via On-Policy Distillation for Reasoning
Sep 25, 2026Shangjian Yin, Zehao Zhao, Kavosh Asadi +7Language Model Self-Improvement On-Policy Self-Distillation
Meta AI · University of California, Riverside · Work done at Meta
Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress
Aug 19, 2026Chen Yang, Haiyuan Wan, Rengrong Xiong +2Language Model Distillation On-Policy Distillation
The Hong Kong University of Science and Technology (Guangzhou) · Tsinghua University · Zhejiang University +1

Oct 19, 2026J/K move · Enter open · S save

Bridged Self-Distillation for On-Policy Learning

Runsheng Chen, Yihong Liu, Raoyuan Zhao, Michael A. Hedderich+2

Technical University of Munich, Germany · LMU Munich, Germany · Munich Center for Machine Learning, Germany

Reasoning Distillation On-Policy Distillation On-Policy Self-Distillation Cross-Tokenizer Knowledge Distillation

Open paper AskPDF arXiv

On-policy distillation can transfer the reasoning capabilities of a strong teacher to a smaller student. However, distillation from a stronger teacher model faces two mismatches: its solutions may be difficult for a smaller student to imitate, and its token-level probabilities may be incompatible with, or even unavailable to, the student. Existing methods handle these settings through different forms of supervision. To this end, we propose Bridged Self-Distillation (BSD), a unified framework for distilling from any teacher that provides reasoning traces. The model to be distilled from, termed the external teacher, first provides a reasoning trace as privileged context, which can be generated in an offline fashion. The student generates rollouts without this context, while a copy of the student conditioned on it, serving as the self-teacher, scores the same rollouts in the student's own token space. BSD uses the resulting token-level guidance to adjust updates on unsuccessful rollouts, with a verifier providing the outcome signal. As the external teacher is queried only to produce text, the same training procedure applies to all teacher scenarios: shared-tokenizer, cross-tokenizer, and closed API teachers. Across these scenarios, BSD improves over the base model and outperforms baselines tailored to each scenario, offering a novel and unified pathway for distillation. Code is available: https://github.com/TUMLegalTech/Bridged-Self-Distillation

Abstract

Language Model Distillation

Explore similar work

Bridged Self-Distillation for On-Policy Learning

Recursive Self-Improvement via On-Policy Distillation for Reasoning

Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress