Quantization-aware training (QAT) aims to recover reasoning capabilities in low-bit large language models (LLMs). Existing recipes typically supervise the quantized student with the original full-precision model on fixed response sequences. However, the resulting low-bit quantized models still struggle on long-horizon reasoning tasks and often fail to terminate within a token budget that suffices for the full-precision model. We identify two challenges in closing this gap. First, fixed-response supervision leaves a mismatch between training prefixes and those encountered during student generation, and this trajectory divergence grows as responses lengthen. Second, on its own prefixes, the student underpredicts some next-token candidates and overpredicts others, and this distributional misalignment becomes more frequent at high-entropy prefixes. We present OnQ, a low-bit recovery pipeline that combines on-policy distillation and candidate-wise Dynamic KL. Starting from block-wise calibration, the quantized student generates responses and receives full-precision teacher supervision on its own prefixes. Dynamic KL weights forward and reverse KL by each candidate's teacher-student probability ratio to balance correction of under- and overprediction. Across Qwen3-1.7B, 4B, and 8B at 2- and 3-bit precision, OnQ achieves the highest nine-benchmark average, with the largest gains on long-horizon reasoning benchmarks. At 2 bits, it outperforms every baseline on every benchmark at every model size, improving the Qwen3-8B average by 10.26 points over the strongest baseline. It also produces shorter responses with fewer truncations and repetitive errors.
Figures & tables
Figure 1: Low-bit fixed-response QAT fails on long-horizon reasoning; OnQ outperforms all QAT baselines (Qwen3-1.7B W2). (a) MATH-500 accuracy and (b) share of responses that reach the token limit, grouped by full-precision response length. (c) Share of prefixes with underpredicted candidates, overpredicted candidates, or both, by student entropy.
Figure 2: Overview of OnQ. Left: On-policy distillation trains the calibrated student on its own generated prefixes. Right: Dynamic KL weights forward and reverse KL by each candidate’s teacher–student probability ratio.
Figure 3: Qwen3-1.7B W2. (a) Teacher and student probabilities at one prefix of the calibrated student. (b) Share of under- and overpredicted candidates at high-entropy prefixes that each objective moves toward the teacher. (c) Share of repeated tokens by response position during training; dashed lines mark the rollout caps, labeled with the training steps that use each cap.
Method
W/A
GSM8K
MATH- 500
AIME 24–26
Human Eval+
MBPP+
LiveCode Bench
MMLU- Redux
GPQA- Diamond
IFEval
Avg.
Qwen3-1.7B
Full precision
16/16
90.55
90.13
41.85
79.07
68.96
53.46
74.36
38.72
71.10
67.58
GPTQ
3/16
53.70
20.13
0.37
4.47
15.43
0.25
10.39
8.75
31.73
16.14
AutoRound
3/16
83.78
75.20
14.44
57.93
55.03
26.92
64.49
29.29
60.63
51.97
EfficientQAT
3/16
74.00
69.60
9.63
57.52
43.03
24.99
60.74
16.50
46.89
44.77
BitDistiller
3/16
80.39
66.33
8.52
56.71
49.47
20.00
61.51
24.75
58.78
47.38
Table 1: Quantization results (%) on Qwen3, averaged over three evaluation seeds (AutoRound: seed 42). QAT and OnQ SDs: Table 9 . Bold: best quantized score per model/bit group. W/A: weight/activation bits.
Figure 4: Accuracy versus mean generated tokens for Qwen3 models at 2 bits (top, a–c) and 3 bits (bottom, d–f), averaged over three evaluation seeds. Dashed lines connect model sizes within each method.
Figure 5: Generation outcomes on MATH-500 (Qwen3-1.7B W2, a) and AIME (Qwen3-4B W3, b). OnQ reduces repetitive errors on both benchmarks.
Method
W/A
GSM8K
MATH- 500
AIME 24–26
Human Eval+
MBPP+
LiveCode Bench
MMLU- Redux
GPQA- Diamond
IFEval
Avg.
Qwen3-1.7B
RTN
3/16
0.68
1.20
1.11
0.00
0.00
0.00
8.72
9.60
12.01
3.70
+ Calibration
3/16
87.11
80.00
17.78
58.54
58.20
30.71
67.16
29.80
61.92
54.58
OnQ
3/16
87.64
84.73
32.59
71.14
62.35
41.77
69.84
30.98
65.87
60.77
RTN
2/16
1.67
2.40
0.00
0.00
0.00
0.00
8.60
11.62
10.91
3.91
+ Calibration
2/16
42.84
25.60
0.00
15.85
19.84
2.84
45.07
21.72
36.41
23.35
Table 2: Scores (%) across recovery stages. OnQ denotes calibration followed by on-policy distillation. OnQ results average evaluation seeds 42, 43, and 44; RTN and calibration results use seed 42. Bold marks the best score within each model/bit group.
Method
MATH- 500
Human Eval+
MMLU- Redux
Fixed-response FKL
59.00
39.63
52.28
Fixed-response DKL
61.00
42.07
52.68
On-policy DKL (OnQ)
69.40
48.78
55.75
Table 3: Fixed-response versus on-policy distillation on Qwen3-1.7B W2 from shared calibration. Scores in %; best scores in bold.
Method
Objective
MATH- 500
Human Eval+
MMLU- Redux
Cold start
–
41.40
3.66
45.12
Vanilla OPD
RKL
63.00
10.98
48.18
OPD + GRPO
RKL + RLVR
66.00
4.27
48.84
GKD
FKL
66.20
11.59
48.82
GRPO
RLVR
31.40
0.61
34.05
OnQ
Dynamic KL
67.20
10.98
49.58
Table 4: On-policy objectives on Qwen3-1.7B W2 (400 steps from a shared cold start). Scores in %; best scores in bold.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Source
Calib.
OPD
Baselines
OpenMathReasoning
1,229
7,680
9,831
OpenCodeReasoning-2
1,229
7,680
9,831
OpenScienceReasoning-2
614
3,840
4,915
Dolci-Think
1,024
–
6,553
Tülu 3
–
5,120
–
IF-multi-constraints
–
1,280
–
Appendix
Table 5: Training data: calibration windows (2,048 tokens), OPD prompts, and question–response pairs shared by the three QAT baselines.
Method
Setting
Steps
Batch
Learning rate
EfficientQAT
W2 / W3
1,024
32
2×10−5 / 10−5
BitDistiller
W2 / W3
768 / 256
128
8×10−6
ReasoningQAT
1.7B W2
1,536
64
5×10−6
4B/8B W2
512
64
10−4
All W3
512
64
10−6
PTQ calibration: 4,096 sequences of 2,048 tokens, seed 42.
Appendix
Table 6: Baseline configurations for Table 1 . Batch size counts sequences per optimizer step. All methods use group size 128 and 2- or 3-bit weights.
Figure 6: Labeling instructions for identifying repetitive errors (excerpt). “Looping” in the instructions denotes the repetitive-error category of this paper.
Method
W/A
GSM8K
MATH- 500
AIME 24–26
Human Eval+
MBPP+
LiveCode Bench
MMLU- Redux
GPQA- Diamond
IFEval
Avg.
Qwen3-1.7B
EfficientQAT
3/16
8.82
11.80
57.78
6.10
9.26
19.12
5.02
54.21
30.44
22.50
BitDistiller
3/16
0.25
7.07
21.48
1.42
3.35
15.45
0.51
6.40
8.81
7.19
ReasoningQAT
3/16
0.03
2.00
13.33
0.61
1.59
3.38
0.12
0.67
5.73
3.05
OnQ
3/16
0.00
0.87
5.56
0.41
1.41
1.23
0.05
0.34
5.85
1.75
EfficientQAT
2/16
16.40
32.53
81.85
25.41
23.90
60.47
8.13
54.88
48.49
39.12
Appendix
Table 7: Truncation rates (%) for Table 1 . QAT baselines and OnQ, averaged over evaluation seeds 42, 43, and 44. Bold marks the lowest rate per model/bit group.
Method
W/A
GSM8K
MATH- 500
AIME 24–26
Human Eval+
MBPP+
LiveCode Bench
MMLU- Redux
GPQA- Diamond
IFEval
Avg.
Qwen3-1.7B
RTN
3/16
94.16
94.00
97.78
98.78
97.09
93.55
99.46
100.00
86.88
95.74
+ Calibration
3/16
0.08
2.60
16.67
0.61
1.06
8.34
0.12
0.51
7.21
4.13
OnQ
3/16
0.00
0.87
5.56
0.41
1.41
1.23
0.05
0.34
5.85
1.75
RTN
2/16
99.85
100.00
100.00
100.00
100.00
99.91
100.00
100.00
100.00
99.97
+ Calibration
2/16
0.68
20.80
52.22
17.68
25.66
58.96
3.67
22.73
36.04
26.49
Appendix
Table 8: Truncation rates (%) for Table 2 ; bold marks the lowest rate per model/bit group.
Method
W/A
GSM8K
MATH- 500
AIME 24–26
Human Eval+
MBPP+
LiveCode Bench
MMLU- Redux
GPQA- Diamond
IFEval
Avg.
Qwen3-1.7B
EfficientQAT
3/16
0.35
1.00
1.70
2.82
1.76
0.61
0.10
4.08
4.44
0.97
BitDistiller
3/16
0.34
1.01
2.31
1.61
0.95
0.43
0.59
2.02
1.61
0.42
ReasoningQAT
3/16
0.30
0.20
2.22
1.86
2.65
0.25
0.08
2.67
0.93
0.19
OnQ
3/16
0.65
1.42
2.31
1.53
0.55
0.63
0.26
0.77
0.21
0.49
EfficientQAT
2/16
1.59
2.75
0.64
1.83
0.85
0.81
0.49
1.82
3.49
0.80
Appendix
Table 9: Sample standard deviations (percentage points) of QAT and OnQ scores in Table 1 over evaluation seeds 42, 43, and 44.
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Zhongguancun Academy +2