Organizations: School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen · Shenzhen Future Network of Intelligence Institute (FNii-Shenzhen) · ByteDance · Guangdong Provincial Key Laboratory of Future Networks of Intelligence, CUHK(SZ)
On-policy distillation (OPD) improves a student language model by training on its own generated trajectories with supervision from a stronger teacher. However, when conditioned on student-generated prefixes, the teacher can favor invalid continuations that violate the question's premises, causing OPD to encourage erroneous reasoning steps. As a remedy, we introduce Condition-Contrastive Extrapolated On-Policy Distillation (CCE-OPD), which makes teacher supervision sensitive to question conditions. CCE-OPD incorporates (1) a fixed-prefix comparison in which the same frozen teacher evaluates the complete question and a counterpart with a selected condition masked, and (2) a bounded log-probability contrast that reweights the teacher distribution into an auxiliary distillation target. On 200 diagnostic reasoning states, condition contrast reduces the rate of preferring invalid continuations from 27.0% to 16.0%. Extensive experiments across six mathematical benchmarks show that CCE-OPD improves avg@8 over OPD by an average of 1.60 and 1.95 percentage points when distilling Qwen3-8B into 0.6B and 1.7B students, respectively. It also achieves the strongest results on four transfer benchmarks.
Figures & tables
Figure 1: Overview of CCE-OPD. The same frozen teacher evaluates a shared student prefix under the complete question and a version with a selected condition masked. The clipped condition contrast reweights the full-teacher distribution to form the auxiliary target qE . The student is trained with the resulting auxiliary KL loss together with the original OPD objective.
Source
4 (CVC)
1 (valid)
Other
Student pS
51.8624
30.2846
17.8530
Full teacher qT
59.7318
31.1267
9.1415
Masked teacher qD
89.6143
1.0278
9.3579
Target qE
43.7999
48.9643
7.2358
Table 1: Decision-token probabilities (%) at the same error-before state. Labels refer to the complete assertions with values 400 and 100. The first three rows use full-vocabulary probabilities; qE is normalized on the teacher candidate union with β=0.2 and c=5 . “Other” contains all remaining probability mass.
Scoring source
All 200
Error subset (60)
Student
61 (30.50)
56 (93.33)
Full teacher
54 (27.00)
49 (81.67)
Masked teacher
64 (32.00)
57 (95.00)
Teacher + condition contrast
32 (16.00)
27 (45.00)
Teacher + random contrast
51 (25.50)
46 (76.67)
Table 2: States preferring CVC: count (%). Lower is better. The second column covers all diagnostic states; the third is the subset whose naturally generated next step is erroneous. All sources score the same candidate pairs.
Benchmark
Metric
Qwen3-0.6B-Base (MATH)
Qwen3-1.7B-Base (MATH)
Base
GRPO
OPD
TrOPD
EOPD
CCE-OPD
Base
GRPO
OPD
TrOPD
EOPD
CCE-OPD
MATH500
Avg@8
27.40
49.35
52.08
43.48
52.45
54.00
53.78
63.58
68.45
59.98
68.50
68.95
Pass@8
67.60
77.60
77.00
73.20
76.80
77.40
82.20
83.80
86.40
86.00
85.40
87.60
AIME24
Avg@8
0.42
2.08
3.75
2.50
3.33
4.58
3.33
6.25
9.17
9.17
10.42
12.08
Pass@8
3.33
6.67
6.67
16.67
13.33
10.00
16.67
20.00
20.00
26.67
23.33
23.33
AIME25
Avg@8
0.00
3.33
2.08
0.83
1.67
3.33
1.25
4.17
7.08
6.25
5.83
7.92
Table 3: Mathematical reasoning results (%). Bold and underlined values denote the best and second-best scores within each student scale and metric, including ties. Light blue highlights CCE-OPD. Evaluation protocols are specified in Section 5.1 .
Avg@8 (%)
Benchmark
CCE-OPD
Random
Delete
Other
Symbols
MATH500
68.95
67.38
68.23
67.03
67.58
AIME24
12.08
11.25
9.58
9.17
9.17
AIME25
7.92
7.92
4.58
5.83
5.83
AMC23
40.00
37.81
39.69
38.75
39.69
Minerva
26.01
23.62
25.28
25.14
24.86
Table 4: Second-input ablation for the 1.7B student ( β=0.2 ). CCE-OPD masks the condition; Random masks a length-matched span; Delete removes the question; Other substitutes another question; Symbols matches its token count. Bold marks the best score in each row, including ties.
Figure 2: Sensitivity to extrapolation strength on the six mathematics benchmarks (1.7B student). Orange marks the main setting β=0.2 . Each panel uses its own vertical scale. Full avg@8 and pass@8 scores appear in Table 11 .
Entropy gate
No gate
Benchmark
Avg@8
Pass@8
Avg@8
Pass@8
MATH500
68.95
87.60
68.65
87.20
AIME24
12.08
23.33
10.83
23.33
AIME25
7.92
26.67
6.67
23.33
AMC23
40.00
75.00
39.38
72.50
Minerva
26.01
44.85
23.53
45.59
Table 5: Entropy-gate ablation for the 1.7B student ( β=0.2 ; scores in %). The gate selects positions with H(qT)≥0.8 nats; No gate supervises all valid response positions on eligible questions. Bold marks the best score for each benchmark and metric, including ties.
Method
GPQA D
MMLU Pro
IFEval
CRUX O
Base
24.24
35.90
16.64
34.2250
OPD
29.29
40.23
29.57
39.5125
EOPD
29.29
39.99
30.50
40.3625
CCE-OPD
30.81
40.72
32.53
41.1125
Table 6: Cross-domain results for 1.7B students (%). Bold denotes the best score per task. CRUX-O denotes CRUXEval-O.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Preference change
All states (200)
Error subset (60)
CVC → valid
23
23
Valid → CVC
1
1
Unchanged
176
36
Appendix
Table 7: Paired preference changes after adding condition contrast to the full-teacher score.
Original question
Masked question
Rationale
Numerical information. Margo walks to her friend’s house in 10 minutes. Using the same route, it takes Margo 20 minutes to get back home. If her average walking rate for the entire trip is 4 miles per hour , how many total miles did she walk?
Margo walks to her friend’s house in 10 minutes. Using the same route, it takes Margo 20 minutes to get back home. [MASKED] , how many total miles did she walk?
The average speed connects the total travel time to the requested distance, giving 4×(30/60)=2 miles. Masking it removes the information needed to determine that distance.
Geometric relation. A rectangular prism has dimensions 8 inches by 2 inches by 32 inches. If a cube has the same volume as the prism , what is the surface area of the cube, in square inches?
A rectangular prism has dimensions 8 inches by 2 inches by 32 inches. [MASKED] , what is the surface area of the cube, in square inches?
The equal-volume relation determines the cube’s side length as 38×2×32=8 and its surface area as 6×82=384 . Masking it leaves the cube’s size unspecified.
Complex-variable constraint. Let z be a nonreal complex number such that ∣z∣=1 . Find the real part of 1−z1 .
Let z be a nonreal complex number [MASKED] . Find the real part of 1−z1 .
The unit-modulus constraint fixes the requested real part at 1/2 . Without it, the real part varies with z : it equals 1/2 for z=i and 1/5 for z=2i .
Appendix
Table 8: Masked inputs from the MATH training set. Bold identifies the selected source span; [MASKED] denotes the masking marker. The shared response-format instruction is omitted.
Masking
Source span
Tokens
Condition
If a cube has the same volume as the prism
10
Random span
dimensions 8 inches by 2 inches by
10
Appendix
Table 9: An example of token-length matching between condition masking and random masking. Lengths refer to the replaced source spans.
Hyperparameter
Value
Optimization
Optimizer
AdamW
Learning rate
3×10−6
Learning-rate schedule
Cosine
Weight decay
0.01
Gradient clipping norm
1.0
Appendix
Table 10: Hyperparameter settings for training and evaluation. CCE-OPD uses the same main settings for both student scales; the extrapolation-strength and entropy-gate ablations vary their respective settings.
Avg@8 (%)
Benchmark
0
0.1
0.2
0.3
0.4
0.5
MATH500
68.25
67.70
68.95
68.08
68.35
68.63
AIME24
8.75
10.83
12.08
10.00
10.00
10.00
AIME25
6.67
7.08
7.92
6.67
7.50
5.83
AMC23
39.38
37.19
40.00
39.69
40.94
39.06
Minerva
24.08
25.69
26.01
25.05
25.41
26.33
Appendix
Table 11: Extrapolation strength β for the 1.7B student. The β=0 control retains auxiliary supervision without condition reweighting. Blue marks the main setting; bold marks the best score in each row, including ties.
Benchmark
Questions
Shots
Max. tokens
Responses per question
GPQA-Diamond
198
0
–
–
MMLU-Pro
12,032
5
2,048
1
IFEval
541
0
1,280
1
CRUXEval-O
800
2
8,096
10
Appendix
Table 12: Transfer evaluation settings. GPQA-Diamond scores four options by log likelihood. MMLU-Pro and IFEval use greedy decoding; CRUXEval-O uses temperature 0.2 and top- p=0.95 .
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Science · Ant Group