Organizations: State Key Lab. of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy Sciences · Meituan
The strong generalization performance of on-policy post-training paradigms has motivated studies of their parameter update behaviors. However, these studies treat the observed behaviors only as byproducts in on-policy training, overlooking their potential to serve as optimization principles for improving the generalization of other paradigms such as supervised fine-tuning (SFT). To address this limitation, we investigate whether there exists a specific on-policy update behavior that can achieve such improvements. First, our analyses reveal that SFT updates parameters along consistent directions, while the on-policy paradigm continuously adjusts the direction during training. This difference inspires us to focus on the cumulative update direction of each parameter as a promising behavior. Then, we evaluate its effectiveness for improving generalization by proposing On-Policy direction-constrained Supervised Fine-Tuning (OPSFT), which constrains SFT updates to the direction identified by on-policy paradigms. The strong performance of OPSFT indicates that the generalization advantage of on-policy paradigms can be transferred to SFT through the parameter update direction. Once such a direction is identified, even SFT can generalize with its updates constrained to this direction. This finding offers two practical benefits by combining the strong generalization of on-policy paradigms with the advantages of SFT, including the high training efficiency and ability to leverage high-quality trajectories. For efficiency, we identify update directions that support strong generalization using a few on-policy training steps, and subsequently apply OPSFT to achieve high training efficiency. For leveraging high-quality trajectories, OPSFT can utilize these trajectories to continue improving a post-trained model along its update direction without disrupting the ability learned from on-policy training.
Figures & tables
Figure 1: (a) SFT update parameters along consistent directions during training. (b) In contrast, on-policy paradigms continuously adjust the update direction. (c) To investigate whether the update direction identified by on-policy paradigms can improve the generalization of SFT, we introduce OPSFT that constrains the SFT update of each parameter to the on-policy direction.
Figure 2: Comparisons of update directions between SFT and on-policy (OPD, GRPO) paradigms using Qwen3-1.7B on DeepMath. We present the cosine similarity of interval (left) and cumulative (right) parameter updates among different training stages.
Figure 3: Performance comparison across different models and benchmarks. We compare vanilla SFT, OPSFT with location constraints, OPSFT with direction constraints, and the on-policy paradigm that provides update locations and directions (GRPO).
Model
Method
IFEval
ARC
HaluEval
Hellaswag
Winogrande
PIQA
Mean
Qwen3-4B
Base
87.65
87.71
72.20
52.25
65.82
74.97
73.43
SFT
86.45
87.88
73.40
52.41
66.30
75.46
73.65
GRPO
88.61
87.88
73.70
52.25
66.22
75.03
73.95
OPSFT
88.97
87.97
73.80
52.22
66.46
75.24
74.11
Qwen3-8B
Base
87.05
90.53
75.60
57.20
67.96
76.77
75.85
SFT
87.53
90.61
72.70
57.21
68.19
76.99
75.54
Table 1: Performance on out-of-domain benchmarks. We compare the base model, the final checkpoints of vanilla SFT, GRPO, and OPSFT trained on the DeepMath dataset.
Figure 4: We compare parameter update locations (left) and directions (middle) between datasets from the same (DAPO vs. DeepMath) and different domains (Eurus vs. DeepMath). Then, we train OPSFT on DeepMath using on-policy update directions identified by different datasets (right) . Mean accuracy is computed across the AIME24/25 and HMMT-Feb/Nov benchmarks.
Model
Method
Time
AIME24
AIME25
HMMT25-Feb
HMMT25-Nov
Mean
Qwen3-1.7B
GRPO
4.9h
16.67
17.08
7.92
6.67
12.09
SFT
3.5h
19.48
17.92
8.96
6.25
13.15
DFT
3.9h
18.25
17.72
10.00
6.67
13.16
OPSFT
2.3h
20.00
21.67
9.58
9.17
15.11
Qwen3-4B
GRPO
16.5h
50.83
45.31
26.35
33.33
38.96
SFT
8.1h
46.67
39.79
23.02
27.40
34.22
Table 2: Accuracy and training time of different methods across multiple models and benchmarks. All models are trained on the DeepMath dataset. The time of OPSFT includes both identifying the on-policy update direction and the subsequent OPSFT training.
Model
Method
AIME24
AIME25
HMMT25-Feb
HMMT25-Nov
Mean
Qwen3-1.7B
Base (post-trained)
16.67
17.08
7.92
6.67
12.09
SFT
17.92
19.17
10.83
6.25
13.54
OPSFT
26.25
22.08
10.42
8.33
16.77
Qwen3-4B
Base (post-trained)
50.83
45.31
26.35
33.33
38.96
SFT
50.00
42.08
24.17
28.75
36.25
OPSFT
56.67
44.17
27.92
36.67
41.36
Table 3: Performance comparisons between SFT and OPSFT across multiple benchmarks using different models post-trained by GRPO. All models are trained on the DeepMath dataset.
Table 8
Precision
Method
Updated Param.
AIME24
AIME25
HMMT25-Feb
HMMT25-Nov
Mean
FP32
Base
0.000%
27.08
20.83
9.17
8.33
16.35
GRPO
9.499%
58.13
46.15
26.98
36.77
42.01
BF16
SFT
2.702%
54.16
42.50
27.08
29.58
38.33
OPSFT
0.408%
56.77
45.73
26.15
35.10
40.94
FP32
SFT
89.286%
52.60
44.48
23.85
32.71
38.41
OPSFT
9.454%
56.25
49.17
27.50
38.33
42.81
Table 6: Ablation studies of the parameter precisions. We report the accuracy and the proportion of updated parameters across different methods. Experiments are conducted with Qwen3-8B.
Figure 5: Accuracy evolution of OPSFT using the update direction identified at different GRPO training steps. Experiments are conducted using Qwen3-8B on the DeepMath dataset.
Figure 6: Update behaviors of OPSFT when training Qwen3-8B on the DeepMath dataset. We report the location sparsity, Jaccard overlap of update locations, and matrix-level cosine similarity of cumulative and interval updates across different training steps.
Figure 7: Reasoning behavior comparisons. We report response length, accuracy standard deviation, the number of case-split cues per 1k words, and the number of logical cues per 1k words throughout training with Qwen3-8B on the DeepMath dataset.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Table 13
Method
AIME24
AIME25
HMMT25-Feb
HMMT25-Nov
Mean
Base
13.54
11.35
4.79
4.68
8.59
OPD
24.17
20.93
11.67
10.21
16.75
Vanilla SFT
18.90
18.60
10.40
7.40
13.83
OPSFT (location)
18.90
18.90
9.40
6.70
13.48
OPSFT (direction)
23.02
23.95
10.93
8.95
16.71
Appendix
Table A9: Accuracy of different methods on Qwen3-1.7B using the DeepMath dataset.
Method
Step
AIME24
AIME25
HMMT Feb
HMMT Nov
Mean
Vanilla SFT
100
11.67
12.50
7.50
6.25
9.48
200
18.33
17.92
8.33
7.50
13.02
300
20.83
16.67
7.92
9.17
13.65
400
19.58
21.67
7.08
5.83
13.54
500
19.58
17.92
8.75
6.67
13.23
600
17.50
18.33
10.83
7.08
13.44
Appendix
Table A10: Training trajectories for Qwen3-1.7B. Entries report Acc@8 (%).
Method
Step
AIME24
AIME25
HMMT Feb
HMMT Nov
Mean
Vanilla SFT
100
42.92
34.48
22.08
25.52
31.25
200
43.96
40.52
23.44
28.65
34.14
300
43.75
38.12
23.23
28.44
33.38
400
45.00
39.27
24.06
27.71
34.01
500
46.56
43.12
23.02
29.06
35.44
600
46.46
38.65
22.08
27.71
33.73
Appendix
Table A11: Training trajectories for Qwen3-4B. Entries report Acc@8 (%).
Method
Step
AIME24
AIME25
HMMT Feb
HMMT Nov
Mean
Vanilla SFT
100
49.69
37.29
21.35
28.85
34.30
200
51.35
42.08
22.08
31.98
36.87
300
51.88
40.52
22.40
31.35
36.54
400
50.52
41.67
22.40
31.67
36.56
500
53.02
43.75
24.38
31.15
38.08
600
51.46
43.44
23.23
32.71
37.71
Appendix
Table A12: Training trajectories for Qwen3-8B. Entries report Acc@8 (%).
Location constraint
Direction constraint
AIME24
AIME25
HMMT25-Feb
HMMT25-Nov
Mean
Random
-
41.67
34.17
24.17
23.33
30.84
On-policy
-
46.67
40.00
21.67
27.92
34.07
On-policy
Random
37.50
31.67
18.33
23.33
27.71
On-policy
On-policy
52.50
48.33
31.25
32.50
41.15
Appendix
Table A13: We report the performance of SFT with (i) random update location constraints, (ii) on-policy update location constraints, (iii) on-policy update location constraints with random update sign constraints, and (iv) the complete on-policy update direction constraints used in OPSFT. Experiments are conducted with Qwen3-4B on the DeepMath dataset. GRPO is selected as the on-policy paradigm that provides update location and direction constraints.
Figure A8: We provide the principal values of parameter updates, the rotation angle of parameter matrices’ principal singular vectors, and the stable rank of parameter updates across different layers and methods. Experiments are conducted using Qwen3-4B trained on the DeepMath dataset.
On-policy distillation (OPD) has emerged as an efficient post-training paradigm for large language models. However, existing studies largely attribute this advantage to denser and more stable supervision, while the parameter-level mechanisms underlying OPD's efficiency remain poorly understood. In this work, we argue that OPD's efficiency stems from a form of ``foresight'': it establishes a stable update trajectory toward the final model early in training. This foresight manifests in two aspects. First, at the \textbf{Module-Allocation Level}, OPD identifies regions with low marginal utility and concentrates updates on modules that are more critical to reasoning. Second, at the \textbf{Update-Direction Level}, OPD exhibits stronger low-rank concentration, with its dominant subspaces aligning closely with the final update subspace early in training. Building on these findings, we propose \textbf{EffOPD}, a plug-and-play acceleration method that speeds up OPD by adaptively selecting an extrapolation step size and moving along the current update direction. EffOPD requires no additional trainable modules or complex hyperparameter tuning, and achieves an average training acceleration of 3× while maintaining comparable final performance. Overall, our findings provide a parameter-dynamics perspective for understanding the efficiency of OPD and offer practical insights for designing more efficient post-training methods for large language models.
On-policy distillation (OPD) is increasingly used to improve large language model reasoning, but its training dynamics remain poorly understood. We characterize the trajectory of OPD updates in parameter space and compare it with supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR). A suite of parameter-space diagnostics consistently places OPD in a relaxed off-principal regime: compared with SFT, its updates affect fewer weights and avoid principal directions more strongly, while compared with RLVR, they remain less tightly constrained. Beyond this static localization, OPD exhibits subspace locking: its cumulative updates rapidly enter a narrow low-dimensional channel. Constraining training to the update subspace formed early in training preserves OPD performance but substantially degrades SFT, indicating that the locked subspace is functionally sufficient for OPD. Control experiments further show that sparsifying the update tokens and shifting rollout generation off-policy preserve the rank dynamics, whereas mixing the OPD objective with RLVR changes them. Overall, these results suggest that OPD is not merely an intermediate point between SFT and RLVR, but induces its own update geometry in parameter space.
Zhennan Shen, Yanshu Li, Qingyu Yin +6
1HKUST · 2Brown University · 3Zhejiang University +5
Recent studies have observed that parameter changes during language-model post-training can be concentrated in a small subset of coordinates. This phenomenon has been reported in reinforcement learning, on-policy distillation, and supervised fine-tuning on near-policy data. Its recurrence across different post-training paradigms suggests shared structure in training dynamics. In this paper, we examine this pattern through the diagonal model Fisher, which measures the sensitivity of the model's output distribution to individual parameters and is independent of any particular reward or teacher signal. Theoretically, we show that small diagonal Fisher leads to small expected gradients across a range of training objectives, providing a common explanation for sparse gradient updates. Empirically, we test this connection in RL and OPD. We find that Fisher identifies where gradients are concentrated, and fixed sparse masks selected from the initial Fisher retain a large proportion of the improvement from full training. Finally, we investigate the mechanisms underlying low Fisher in on-policy training. Our results show that high-probability next tokens tend to have similar parameter sensitivities, contributing to low Fisher. Together, these results establish the diagonal model Fisher as a unifying perspective linking update sparsity to on-policy training dynamics in LLM post-training.