Organizations: State Key Lab. of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy Sciences · Meituan
The strong generalization performance of on-policy post-training paradigms has motivated studies of their parameter update behaviors. However, these studies treat the observed behaviors only as byproducts in on-policy training, overlooking their potential to serve as optimization principles for improving the generalization of other paradigms such as supervised fine-tuning (SFT). To address this limitation, we investigate whether there exists a specific on-policy update behavior that can achieve such improvements. First, our analyses reveal that SFT updates parameters along consistent directions, while the on-policy paradigm continuously adjusts the direction during training. This difference inspires us to focus on the cumulative update direction of each parameter as a promising behavior. Then, we evaluate its effectiveness for improving generalization by proposing On-Policy direction-constrained Supervised Fine-Tuning (OPSFT), which constrains SFT updates to the direction identified by on-policy paradigms. The strong performance of OPSFT indicates that the generalization advantage of on-policy paradigms can be transferred to SFT through the parameter update direction. Once such a direction is identified, even SFT can generalize with its updates constrained to this direction. This finding offers two practical benefits by combining the strong generalization of on-policy paradigms with the advantages of SFT, including the high training efficiency and ability to leverage high-quality trajectories. For efficiency, we identify update directions that support strong generalization using a few on-policy training steps, and subsequently apply OPSFT to achieve high training efficiency. For leveraging high-quality trajectories, OPSFT can utilize these trajectories to continue improving a post-trained model along its update direction without disrupting the ability learned from on-policy training.
Figures & tables
Figure 1: (a) SFT update parameters along consistent directions during training. (b) In contrast, on-policy paradigms continuously adjust the update direction. (c) To investigate whether the update direction identified by on-policy paradigms can improve the generalization of SFT, we introduce OPSFT that constrains the SFT update of each parameter to the on-policy direction.
Figure 2: Comparisons of update directions between SFT and on-policy (OPD, GRPO) paradigms using Qwen3-1.7B on DeepMath. We present the cosine similarity of interval (left) and cumulative (right) parameter updates among different training stages.
Figure 3: Performance comparison across different models and benchmarks. We compare vanilla SFT, OPSFT with location constraints, OPSFT with direction constraints, and the on-policy paradigm that provides update locations and directions (GRPO).
Model
Method
IFEval
ARC
HaluEval
Hellaswag
Winogrande
PIQA
Mean
Qwen3-4B
Base
87.65
87.71
72.20
52.25
65.82
74.97
73.43
SFT
86.45
87.88
73.40
52.41
66.30
75.46
73.65
GRPO
88.61
87.88
73.70
52.25
66.22
75.03
73.95
OPSFT
88.97
87.97
73.80
52.22
66.46
75.24
74.11
Qwen3-8B
Base
87.05
90.53
75.60
57.20
67.96
76.77
75.85
SFT
87.53
90.61
72.70
57.21
68.19
76.99
75.54
Table 1: Performance on out-of-domain benchmarks. We compare the base model, the final checkpoints of vanilla SFT, GRPO, and OPSFT trained on the DeepMath dataset.
Figure 4: We compare parameter update locations (left) and directions (middle) between datasets from the same (DAPO vs. DeepMath) and different domains (Eurus vs. DeepMath). Then, we train OPSFT on DeepMath using on-policy update directions identified by different datasets (right) . Mean accuracy is computed across the AIME24/25 and HMMT-Feb/Nov benchmarks.
Model
Method
Time
AIME24
AIME25
HMMT25-Feb
HMMT25-Nov
Mean
Qwen3-1.7B
GRPO
4.9h
16.67
17.08
7.92
6.67
12.09
SFT
3.5h
19.48
17.92
8.96
6.25
13.15
DFT
3.9h
18.25
17.72
10.00
6.67
13.16
OPSFT
2.3h
20.00
21.67
9.58
9.17
15.11
Qwen3-4B
GRPO
16.5h
50.83
45.31
26.35
33.33
38.96
SFT
8.1h
46.67
39.79
23.02
27.40
34.22
Table 2: Accuracy and training time of different methods across multiple models and benchmarks. All models are trained on the DeepMath dataset. The time of OPSFT includes both identifying the on-policy update direction and the subsequent OPSFT training.
Model
Method
AIME24
AIME25
HMMT25-Feb
HMMT25-Nov
Mean
Qwen3-1.7B
Base (post-trained)
16.67
17.08
7.92
6.67
12.09
SFT
17.92
19.17
10.83
6.25
13.54
OPSFT
26.25
22.08
10.42
8.33
16.77
Qwen3-4B
Base (post-trained)
50.83
45.31
26.35
33.33
38.96
SFT
50.00
42.08
24.17
28.75
36.25
OPSFT
56.67
44.17
27.92
36.67
41.36
Table 3: Performance comparisons between SFT and OPSFT across multiple benchmarks using different models post-trained by GRPO. All models are trained on the DeepMath dataset.
Table 8
Precision
Method
Updated Param.
AIME24
AIME25
HMMT25-Feb
HMMT25-Nov
Mean
FP32
Base
0.000%
27.08
20.83
9.17
8.33
16.35
GRPO
9.499%
58.13
46.15
26.98
36.77
42.01
BF16
SFT
2.702%
54.16
42.50
27.08
29.58
38.33
OPSFT
0.408%
56.77
45.73
26.15
35.10
40.94
FP32
SFT
89.286%
52.60
44.48
23.85
32.71
38.41
OPSFT
9.454%
56.25
49.17
27.50
38.33
42.81
Table 6: Ablation studies of the parameter precisions. We report the accuracy and the proportion of updated parameters across different methods. Experiments are conducted with Qwen3-8B.
Figure 5: Accuracy evolution of OPSFT using the update direction identified at different GRPO training steps. Experiments are conducted using Qwen3-8B on the DeepMath dataset.
Figure 6: Update behaviors of OPSFT when training Qwen3-8B on the DeepMath dataset. We report the location sparsity, Jaccard overlap of update locations, and matrix-level cosine similarity of cumulative and interval updates across different training steps.
Figure 7: Reasoning behavior comparisons. We report response length, accuracy standard deviation, the number of case-split cues per 1k words, and the number of logical cues per 1k words throughout training with Qwen3-8B on the DeepMath dataset.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Table 13
Method
AIME24
AIME25
HMMT25-Feb
HMMT25-Nov
Mean
Base
13.54
11.35
4.79
4.68
8.59
OPD
24.17
20.93
11.67
10.21
16.75
Vanilla SFT
18.90
18.60
10.40
7.40
13.83
OPSFT (location)
18.90
18.90
9.40
6.70
13.48
OPSFT (direction)
23.02
23.95
10.93
8.95
16.71
Appendix
Table A9: Accuracy of different methods on Qwen3-1.7B using the DeepMath dataset.
Method
Step
AIME24
AIME25
HMMT Feb
HMMT Nov
Mean
Vanilla SFT
100
11.67
12.50
7.50
6.25
9.48
200
18.33
17.92
8.33
7.50
13.02
300
20.83
16.67
7.92
9.17
13.65
400
19.58
21.67
7.08
5.83
13.54
500
19.58
17.92
8.75
6.67
13.23
600
17.50
18.33
10.83
7.08
13.44
Appendix
Table A10: Training trajectories for Qwen3-1.7B. Entries report Acc@8 (%).
Method
Step
AIME24
AIME25
HMMT Feb
HMMT Nov
Mean
Vanilla SFT
100
42.92
34.48
22.08
25.52
31.25
200
43.96
40.52
23.44
28.65
34.14
300
43.75
38.12
23.23
28.44
33.38
400
45.00
39.27
24.06
27.71
34.01
500
46.56
43.12
23.02
29.06
35.44
600
46.46
38.65
22.08
27.71
33.73
Appendix
Table A11: Training trajectories for Qwen3-4B. Entries report Acc@8 (%).
Method
Step
AIME24
AIME25
HMMT Feb
HMMT Nov
Mean
Vanilla SFT
100
49.69
37.29
21.35
28.85
34.30
200
51.35
42.08
22.08
31.98
36.87
300
51.88
40.52
22.40
31.35
36.54
400
50.52
41.67
22.40
31.67
36.56
500
53.02
43.75
24.38
31.15
38.08
600
51.46
43.44
23.23
32.71
37.71
Appendix
Table A12: Training trajectories for Qwen3-8B. Entries report Acc@8 (%).
Location constraint
Direction constraint
AIME24
AIME25
HMMT25-Feb
HMMT25-Nov
Mean
Random
-
41.67
34.17
24.17
23.33
30.84
On-policy
-
46.67
40.00
21.67
27.92
34.07
On-policy
Random
37.50
31.67
18.33
23.33
27.71
On-policy
On-policy
52.50
48.33
31.25
32.50
41.15
Appendix
Table A13: We report the performance of SFT with (i) random update location constraints, (ii) on-policy update location constraints, (iii) on-policy update location constraints with random update sign constraints, and (iv) the complete on-policy update direction constraints used in OPSFT. Experiments are conducted with Qwen3-4B on the DeepMath dataset. GRPO is selected as the on-policy paradigm that provides update location and direction constraints.
Figure A8: We provide the principal values of parameter updates, the rotation angle of parameter matrices’ principal singular vectors, and the stable rank of parameter updates across different layers and methods. Experiments are conducted using Qwen3-4B trained on the DeepMath dataset.