Understanding the Synergy between SFT, RLVR, and OPD in LLM Post-Training

Emre Can Acikgoz; Yang Li; Zeyu Leo Liu; Srijan Bansal; Dilek Hakkani-Tür; Shafiq Joty; Semih Yavuz

cs.LGSep 25, 2026

Understanding the Synergy between SFT, RLVR, and OPD in LLM Post-Training

Authors: Emre Can Acikgoz, Yang Li, Zeyu Leo Liu, Srijan Bansal, Dilek Hakkani-Tür, Shafiq Joty, Semih Yavuz

Organizations: Salesforce AI Research · UIUC

Abstract

Modern LLM post-training composes supervised fine-tuning (SFT), reinforcement learning with verifiable rewards (RLVR), and on-policy distillation (OPD) into multi-stage pipelines, yet these stages are typically designed and evaluated in isolation. We show that this composition is consequential: a stage that improves the current model can make the next stage less effective. Through controlled experiments with Qwen3 models on math and science reasoning, we first characterize OPD across nine student-teacher pairs spanning 2x to 53x parameter ratios and show that OPD effectiveness depends on student-teacher compatibility rather than teacher scale alone. The surrounding stages of OPD reshape this compatibility in three ways: (1) A brief SFT warm-up improves subsequent OPD, while an RLVR-strengthened student regresses under distillation from the same teacher. (2) Adapting the teacher with RLVR raises downstream OPD accuracy in proportion to the capability it adds. Following these two interventions, we find that combining teacher adaptation and student warm-up alone raise average OPD accuracy from 29.2% to 43.8% (50% relative improvement) after the same number of distillation steps, with additional preparatory training. (3) At comparable accuracy, OPD leaves a stronger initialization for downstream RLVR than SFT, with a gap that widens as RL compute scales. Our results suggest that each post-training stage should be chosen not only for the capability it adds, but for the learning interface it creates for the next stage.

Figures & tables

Figure 1: The outcome of each post-training stage depends on the learning state it inherits. (a) An RLVR-warmed student ( green ) starts stronger but regresses under OPD, while a briefly SFT-warmed student ( gold ) improves steadily from the same teacher. (b) An RLVR-adapted teacher transfers substantially more than a SFT-adapted teacher of the same origin. Inset: teacher accuracy along its RLVR training ( gray ) and final OPD accuracy of a student distilled from the teacher at each step ( green ); both plateau together, so student gains stop when the teacher’s do. (c) At comparable starting accuracy, OPD-initialized models ( blue ) reach higher final accuracy under subsequent RLVR than SFT-initialized models, and the gap widens with RL compute.

Figure 2: A larger teacher does not guarantee better transfer. Red lines show the unrecovered gap,

100-\text{DE}

(%). (a) Qwen3-4B-Base gains

10.3

points from Qwen3-14B but leaves

55.5\%

of the teacher’s advantage unrecovered. (b) For the 4B student, DE decreases as teacher size increases from 8B to 32B. (c) With the 14B teacher fixed, smaller students recover a smaller share of the gap.

Figure 3: OPD effectiveness varies across objectives, model states, and training data. (a) Under forward KL, DE decreases with teacher size at every top-

k

support. (b) Qwen2.5-1.5B base students gain more from OPD than instruct students. (c) Post-trained teachers improve peak accuracy in four of five matched-size comparisons. (d) Qwen3-8B gives the highest accuracy on both training datasets for Qwen3-4B-Base.

Figure 4: Reinforceable signal varies across student–teacher pairings. (a) Initial

\rho_{0}^{+}

decreases with teacher size for every student, with point labels showing step-200 accuracy. (b) Selecting maximum

\rho_{0}^{+}

identifies the largest step-200 gain for two of three students. (c) Larger teachers leave a greater share of tokens without positive distillation signal.

Figure 5: Student warm-up: the method and the amount both matter. (a) Brief SFT raises the observed OPD performance and reduces the subsequent drop, whereas the RLVR-prepared student regresses under OPD from the same teacher. (b) Both students contract toward the teacher in reverse KL—the RLVR-warmed student from much farther away—so the regression occurs despite successful teacher matching. (c) As SFT continues, student entropy keeps falling while downstream OPD accuracy saturates after a short warm-up. (d) The SFT validation-loss plateau signals when to stop: SFT beyond it no longer improves downstream OPD.

Figure 6: Teacher adaptation transfers capability, not proximity. (a) OPD from an RLVR-adapted teacher surpasses the off-the-shelf teacher’s own accuracy, whereas a SFT-trained teacher improves slightly over vanilla OPD. (b) The student’s gain tracks the teacher’s gain across RLVR checkpoints and flattens once the teacher plateaus (shaded). (c) Some runs regress while reverse KL continues to decrease, so teacher matching alone does not explain accuracy retention.

Figure 7: An OPD checkpoint is a better RLVR initialization than an SFT checkpoint trained on the same trajectories. (a) RLVR from OPD checkpoints (blue) is still rising at step

260

, while RLVR from SFT checkpoints (gold) stalls by step

100

. Shading marks the range across checkpoints. (b) OPD-initialized runs spend entropy as accuracy rises; SFT-initialized runs keep theirs largely unchanged. (c) Among the tested OPD handoffs, the step-

50

checkpoint has both the highest pre-RLVR accuracy and the highest post-RLVR accuracy, reaching

43.1\%

Figure 8: Final OPD accuracy combined with teacher RLVR adaptation and SFT warm-up.

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Training data	DAPO-Math-17k
Parameter update	Full-parameter training, LoRA rank $0$
Optimizer	AdamW
Learning rate / $(\beta_{1},\beta_{2})$	$10^{-6}$ / $(0.9,0.999)$
Schedule / warm-up	Constant learning rate / 10 steps
Weight decay / gradient norm clipping	$0.01$ / $1.0$

Appendix

Table 1: Shared OPD configuration for the main experiment grid in § 3 .

Setting	Value
Student / teachers	Qwen3-4B-Base / Qwen3-{8B, 14B, 32B}
Training data	DAPO-Math-17k
Loss mode / top- $k$	forward_kl_topk / $k\in\{16,32,64\}$
Update	Direct distillation loss with policy-gradient mode disabled
Optimizer / learning rate	AdamW / $10^{-6}$
Schedule / warm-up	Constant learning rate / 10 steps

Appendix

Table 2: Forward-KL configuration with optimization, sampling, and length settings match Table 1 . The objective uses teacher top-

k

support.

Setting	Value
Primary benchmarks	AIME 2024, AIME 2025, AMC 2023, MATH-500, Minerva Math, OlympiadBench, GPQA-Diamond
Responses per question	4
Temperature / top- $p$ / top- $k$	$0.6$ / $0.95$ / disabled
Prompt / response limit	2,048 / 32,768 tokens
Per-benchmark score	Mean correctness over four responses ( mean@4 )
Configured evaluator	ttrl_math.reward_func in verl

Appendix

Table 3: Evaluation protocol where the primary aggregate assigns equal weight to each of the seven benchmarks.

Training
Dataset	Unique prompts	Average prompt tokens
DAPO-Math-17k	17,917	122
OpenThoughts3-Math	30,000	113
Evaluation
AIME 2024	30	150
AIME 2025	30	145

Appendix

Table 4: Training and evaluation datasets, prompt lengths use the Qwen3 tokenizer with the training chat template.

Student	Teacher	$A_{200}$	$A_{260}$	Gain 260	DE 260 (%)
0.6B	8B	14.29	14.91	+12.65	33.4
0.6B	14B	15.00	13.24	+10.99	27.6
0.6B	32B	13.66	13.24	+10.99	27.1
1.7B	8B	23.00	22.26	+11.47	39.1
1.7B	14B	20.95	20.13	+9.34	29.8
1.7B	32B	20.34	21.36	+10.58	33.0

Appendix

Table 5: Complete Qwen3 base-student grid where accuracy is the seven-benchmark mean in percent. Gain and DE use the pooled initial score for each student.

Figure 9: Complete OPD learning curves across student and teacher sizes. Each panel compares three teachers for one base student. The 0.6B and 1.7B curves show individual runs. For 4B, bold lines show the three-run mean and faint lines show individual runs. The dotted line marks step 200. All curves extend to the shared final evaluation at step 260.

$k$	Teacher	$A$ (%)	Gain	DE (%)	Peak (%)
16	8B	33.66	+14.77	69.7	34.43
16	14B	34.48	+15.59	67.2	34.48
16	32B	31.79	+12.91	54.0	33.48
32	8B	34.22	+15.33	72.3	35.11
32	14B	34.39	+15.50	66.8	35.28
32	32B	32.64	+13.75	57.5	33.12

Appendix

Table 6: Forward-KL results. All cells use Qwen3-4B-Base and one run and peak accuracy is restricted to the same budget. Baseline and teacher references match Table 5 .

Student	State	Initial	Qwen2.5-7B/Qwen3-8B teacher	14B teacher	32B teacher
Qwen3-1.7B	Base	10.79	23.00 (+12.22)	20.95 (+10.16)	20.34 (+9.55)
Qwen3-1.7B	Post-trained	29.51	29.82 (+0.31)	31.17 (+1.65)	29.18 (-0.33)
Qwen3-4B	Base	18.89	32.70 (+13.81)	29.23 (+10.34)	29.25 (+10.36)
Qwen3-4B	Post-trained	39.58	38.03 (-1.55)	39.28 (-0.31)	37.69 (-1.89)
Qwen2.5-1.5B	Base	1.30	19.85 (+18.54)	20.01 (+18.71)	20.44 (+19.14)
Qwen2.5-1.5B	Instruct	18.08	20.69 (+2.61)	20.56 (+2.48)	20.73 (+2.66)

Appendix

Table 7: Student-state and model-family comparisons after 200 OPD steps. Entries report accuracy (%) with gains over the pooled initial student score in parentheses. The smallest teacher is 8B for Qwen3 and 7B for Qwen2.5. Results average three runs per teacher for Qwen3-4B-Base and use one run per configuration otherwise.

Student	Teacher size	Base $A$	Post $A$	Base peak	Post peak
1.7B	8B	19.66	23.00	21.50	23.00
1.7B	14B	22.30	20.95	23.54	23.14
4B	8B	25.05	32.70	31.52	33.38
4B	14B	30.81	29.23	32.35	33.49
8B	14B	31.19	29.34	33.32	35.10

Appendix

Table 8: Comparison of base and post-trained Qwen3 teachers at matched model sizes, using base students throughout. Endpoint accuracy is measured at step 200, and peaks are computed over evaluations through that step. Results with post-trained teachers average three runs for the 4B student, while all other configurations use one run.

	DAPO-Math	OpenThoughts-Math
Teacher	Step 200	Step 200	Step 240
8B	$32.70\pm 0.66$	31.29	31.10
14B	$29.23\pm 0.63$	28.80	29.18
32B	$29.25\pm 0.34$	28.00	30.35

Appendix

Table 9: Qwen3-4B-Base accuracy (%) by distillation dataset. DAPO-Math reports mean

\pm

s.d. over three runs. OpenThoughts-Math is run per teacher.

Benchmark / aggregate	8B teacher	14B teacher	32B teacher
AIME 2024	$13.89\pm 1.73$	$13.61\pm 2.68$	$13.33\pm 0.83$
AIME 2025	$12.78\pm 0.48$	$12.50\pm 0.83$	$11.11\pm 1.27$
AMC 2023	$42.97\pm 1.06$	$39.86\pm 0.92$	$36.45\pm 1.59$
MATH-500	$72.23\pm 0.70$	$61.57\pm 0.70$	$64.13\pm 1.25$
Minerva Math	$21.57\pm 0.70$	$18.29\pm 0.09$	$17.68\pm 0.41$
OlympiadBench	$35.89\pm 0.94$	$30.83\pm 0.97$	$31.00\pm 0.13$

Appendix

Table 10: Qwen3-4B-Base accuracy (%) at step 260, reported as mean

\pm

sample standard deviation over three rollout seeds. Aggregates average the first six, first seven, or all ten benchmarks with equal weight.

Student	0.6B			1.7B			4B
Teacher	8B	14B	32B	8B	14B	32B	8B	14B	32B
Initial rKL	0.740	0.680	0.648	0.447	0.478	0.426	0.370	0.376	0.382
$\rho_{0}^{+}$ (%)	37.32	33.20	29.39	53.34	49.12	34.37	54.79	47.83	33.11

Appendix

Table 11: Initial reverse KL and

\rho_{0}^{+}

for each student–teacher pair.

	Step 200			Step 260
Selector	0.6B	1.7B	4B	0.6B	1.7B	4B
Largest teacher	1.35	2.67	3.44	1.67	0.89	2.92
Smallest teacher	0.71	0.00	0.00	0.00	0.00	0.00
Largest $\rho_{0}^{+}$	0.71	0.00	0.00	0.00	0.00	0.00
Smallest initial rKL	1.35	2.67	0.00	1.67	0.89	0.00

Appendix

Table 12: Teacher-selection regret in accuracy points relative to the best observed teacher at each horizon. Lower is better. 4B comparisons use mean outcomes and diagnostics across three runs.

Figure 10: Task accuracy and teacher matching follow different trajectories. Colored curves show mean benchmark accuracy for Qwen3-4B-Base, with sample-deviation bands across three runs per teacher. Dashed gray curves show mean rKL normalized by each run’s first logged value, using the right axis. Dotted vertical lines mark the peak of each mean accuracy curve through step 200.

Stopping rule	Mean step	Accuracy (%)	Gap (pp)	Updates saved (%)
Fixed step 40	40.0	33.13	0.10	84.6
Fixed step 60	60.0	31.41	1.82	76.9
Fixed step 100	100.0	27.92	5.31	61.5
Fixed step 200	200.0	30.39	2.84	23.1
Fixed step 260	260.0	29.75	3.48	0.0
$\tau=0.4$	66.7	30.13	3.11	74.4

Appendix

Table 13: Stopping comparisons averaged over nine Qwen3-4B-Base runs. Gap is measured from each run’s observed peak through step 260. Update savings use step 260 as the reference.

Figure 11: OPD training dynamics across student and teacher scales.

Setting	Student SFT Warm-Up
Starting student	Qwen3-4B-Base
Solution generator	Qwen3-14B
Prompt source	OpenThoughts-Math
Supervision	Fixed teacher-generated solutions
Update scope	Full-parameter SFT
Training backend / GPUs	FSDP / 8

Appendix

Table 14: Student SFT warm-up hyperparameters.

Setting	Student RLVR Warm-Up
Starting student	Qwen3-4B-Base (Thinking Disabled)
Algorithm	GRPO
Warm-up data	OpenThoughts-Math
Reward	Verifiable outcome (binary)
Update scope	Full-parameter training
Prompt / PPO minibatch size	64 / 64

Appendix

Table 15: Student RLVR warm-up hyperparameters.

Setting	Downstream OPD
Student initialization	Selected SFT checkpoint
Teacher	Qwen3-14B
Training data	DAPO-Math-17K
Objective	Sampled-token reverse KL policy gradient
Task rewards / auxiliary KL loss	Disabled / disabled
Optimizer	AdamW

Appendix

Table 16: Downstream OPD hyperparameters for the SFT warm-up sweep.

Figure 12: Downstream OPD trajectories from different SFT warm-up checkpoints.

SFT Checkpoint $K$	Initial (%)	Peak (%)	Peak $t$	Final (%)	Drop
25	30.68	30.87	40	29.15	1.71
75	31.25	35.06	140	34.65	0.41
150	33.42	35.39	160	35.30	0.09
250	33.81	35.08	180	35.08	0.00
348	33.85	35.18	60	34.48	0.70

Appendix

Table 17: Downstream OPD results across SFT warm-up checkpoints.

SFT Checkpoint $K$	OPD Step $t$	Student Entropy	Reverse KL
25	1	0.2940	0.1417
75	1	0.2559	0.1227
150	1	0.1950	0.1028
250	1	0.1507	0.1153
348	1	0.1580	0.1117

Appendix

Table 18: Student diagnostics at the beginning of downstream OPD.

Setting	SFT Teacher
Starting teacher	Qwen3-14B (Thinking Disabled)
Adaptation data	OpenThoughts3-Math
SFT format	Full-parameter
Supervision	Fixed, verified self -generated solutions
Unique prompts in batch	64
Learning rate	$10^{-6}$

Appendix

Table 19: SFT teacher training hyperparameters.

Setting	RLVR-Adapted Teacher
Starting teacher	Qwen3-14B
Algorithm	GRPO
Adaptation data	OpenThoughts-Math
Reward	Verifiable outcome (binary)
Unique prompts in batch	64
Responses per prompt / temperature	8 / 1.0

Appendix

Table 20: RLVR teacher training hyperparameters.

Setting	SFT/RLVR Teacher
Student	Qwen3-4B-Base
Training data	DAPO-Math-17K
Objective	Sampled-token reverse KL policy gradient
Task rewards / auxiliary KL loss	Disabled / disabled
Optimizer	AdamW
Learning rate / warm-up	$10^{-6}$ / 10 steps

Appendix

Table 21: Downstream OPD training hyperparameters shared across SFT- and RLVR-adapted teachers.

Teacher $K$	rKL 0	$\rho_{0}^{+}$ (%)	$A_{200}$ (%)
50	0.505	47.23	27.41
100	0.476	52.40	28.00
150	0.555	49.08	29.16
200	0.530	52.97	29.77

Appendix

Table 22: Initial diagnostics and step-200 accuracy for Qwen3-1.7B-Base with RLVR-adapted Qwen3-14B teachers. Each configuration uses one run.

Figure 13: Downstream OPD performances across SFT and RLVR adapted teacher checkpoint sweeps.

Setting	Qwen3 family	Qwen2.5 family
Supported base students	0.6B, 1.7B, 4B	1.5B
Teacher	Qwen3-14B	Qwen2.5-14B-Instruct
Training data	OpenThoughts3-Math
Supervision	Fixed, verified teacher-generated solutions
Unique prompts in batch	64	64
Microbatch per GPU	4	4

Appendix

Table 23: SFT initialization settings. The reported comparison on Qwen3-1.7B-Base, Qwen3-4B-Base, and Qwen2.5-1.5B students.

Setting	Qwen3 family	Qwen2.5 family
Supported base students	0.6B, 1.7B, 4B	1.5B, 3B
Teacher	Qwen3-14B	Qwen2.5-14B-Instruct
Training data	OpenThoughts3-Math
Distillation objective	Sampled-token policy gradient reverse-KL
Task rewards	Disabled	Disabled
Prompt / PPO minibatch size	64 / 64	64 / 64

Appendix

Table 24: OPD initialization settings. All students use the same optimization and sampling settings, the teacher and initialization data follow the model family.

Setting	Qwen3 family	Qwen2.5 family
Algorithm	GRPO	GRPO
Training data	DAPO-Math-17K	DAPO-Math-17K
Reward	Verifiable outcome reward ( ttrl_math.reward_func )
Prompt / PPO minibatch size	64 / 64	64 / 64
Responses per prompt	8	8
Rollout temperature	1.0	1.0

Appendix

Table 25: Common Downstream RLVR settings for SFT and OPD-initialized students. The reported horizons reflect the displayed curves, while the launchers specify an epoch-based training schedule.

Figure 14: The OPD initialization advantage reproduces across matched handoff checkpoints, student scales, and model families. Held-out core-7 mean@4 accuracy for Qwen3-1.7B-Base (left), Qwen3-4B-Base (center), and Qwen2.5-1.5B (right), using their corresponding 14B teachers. Within each student family, OPD and SFT use the same downstream RLVR recipe and matched handoff-checkpoint grid. Blue and orange curves denote OPD and SFT initialization. Shading shows the minimum–maximum range across handoff checkpoints within each arm, not variation across random seeds. The reported RLVR horizons are

260

steps for Qwen3 and

200

for Qwen2.5-1.5B.

Figure 15: Recursive composition under different teacher schedules. Four SFT–OPD rounds on Qwen3-1.7B-Base, each using an RLVR-adapted Qwen3-14B checkpoint. Accuracy averages AIME 2024, AIME 2025, AMC 2023, and Minerva Math.

Schedule	Round	rKL	$\rho^{+}$	Acc. (%)
Fixed	1	.42	.48	19.5
Fixed	2	.39	.46	19.0
Fixed	3	.38	.45	18.5
Fixed	4	.38	.44	16.6
Reverse	1	.43	.47	19.8
Progressive	4	.41	.44	21.4

Appendix

Table 26: Diagnostics recorded at the start of OPD and four-benchmark accuracy for every round using teacher

t_{200}

. Reinforceable fractions are shown on a

0

–

1

scale.

Explore similar work

Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR
Sep 3, 2026Boyan Li, Bingsen Chen, Chenghao Yang +3RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
University of Alberta · New York University · NYU Shanghai +3
Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation
Apr 14, 2026Yecheng Wu, Song Han, Han CaiOn-Policy Distillation LLM Post-Training
NVIDIA
How Should Teachers Be Prepared? RL on Student-Induced States for On-Policy Distillation
Oct 4, 2026Xiaoyu Ma, Haoyue Liu, Zhichao Wang +2On-Policy Distillation Teacher-Student Learning
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen · Shenzhen Future Network of Intelligence Institute (FNiI-Shenzhen) · Guangdong Provincial Key Laboratory of Future Networks of Intelligence, CUHK(SZ)

Sep 3, 2026J/K move · Enter open · S save

Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR

Boyan Li, Bingsen Chen, Chenghao Yang, Ping Nie+2

University of Alberta · New York University · NYU Shanghai+3

RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards On-Policy Distillation RL Post-Training

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) and on-policy distillation (OPD) have emerged as two dominant methods for post-training reasoning LLMs. Prior work uses OPD's dense token-level supervision to complement the sparse RL reward, fusing the two signals within a single step: either as a weighted-additive combination or a teacher-modulated rescaling of the RL advantage. In this paper, we show that a simple two-stage scheme, OPD-then-RL, consistently outperforms pure OPD, pure RLVR, and all such joint baselines across logic and math reasoning benchmarks. Beyond the empirical results, we further provide a systematic understanding of this through pass@

k

behavior, learning dynamics, and parameter updates, yielding a consistent explanation: OPD expands the student's coverage of teacher-supported solutions and RL sharpens within that support, while jointly optimizing the two signals causes them to interfere. To provide a practical recipe, we find that the OPD validation score is the key signal for when to switch to RL, and that OPD is a better cold start for RL than SFT. Together, our results establish OPD-then-RL as a simple yet strong way to combine the two methods, turning two entangled signals into complementary stages.

Abstract

Explore similar work

Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR

Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation

How Should Teachers Be Prepared? RL on Student-Induced States for On-Policy Distillation