Quantization-aware training (QAT) aims to recover reasoning capabilities in low-bit large language models (LLMs). Existing recipes typically supervise the quantized student with the original full-precision model on fixed response sequences. However, the resulting low-bit quantized models still struggle on long-horizon reasoning tasks and often fail to terminate within a token budget that suffices for the full-precision model. We identify two challenges in closing this gap. First, fixed-response supervision leaves a mismatch between training prefixes and those encountered during student generation, and this trajectory divergence grows as responses lengthen. Second, on its own prefixes, the student underpredicts some next-token candidates and overpredicts others, and this distributional misalignment becomes more frequent at high-entropy prefixes. We present OnQ, a low-bit recovery pipeline that combines on-policy distillation and candidate-wise Dynamic KL. Starting from block-wise calibration, the quantized student generates responses and receives full-precision teacher supervision on its own prefixes. Dynamic KL weights forward and reverse KL by each candidate's teacher-student probability ratio to balance correction of under- and overprediction. Across Qwen3-1.7B, 4B, and 8B at 2- and 3-bit precision, OnQ achieves the highest nine-benchmark average, with the largest gains on long-horizon reasoning benchmarks. At 2 bits, it outperforms every baseline on every benchmark at every model size, improving the Qwen3-8B average by 10.26 points over the strongest baseline. It also produces shorter responses with fewer truncations and repetitive errors.
Figures & tables
Figure 1: Low-bit fixed-response QAT fails on long-horizon reasoning; OnQ outperforms all QAT baselines (Qwen3-1.7B W2). (a) MATH-500 accuracy and (b) share of responses that reach the token limit, grouped by full-precision response length. (c) Share of prefixes with underpredicted candidates, overpredicted candidates, or both, by student entropy.
Figure 2: Overview of OnQ. Left: On-policy distillation trains the calibrated student on its own generated prefixes. Right: Dynamic KL weights forward and reverse KL by each candidate’s teacher–student probability ratio.
Figure 3: Qwen3-1.7B W2. (a) Teacher and student probabilities at one prefix of the calibrated student. (b) Share of under- and overpredicted candidates at high-entropy prefixes that each objective moves toward the teacher. (c) Share of repeated tokens by response position during training; dashed lines mark the rollout caps, labeled with the training steps that use each cap.
Method
W/A
GSM8K
MATH- 500
AIME 24–26
Human Eval+
MBPP+
LiveCode Bench
MMLU- Redux
GPQA- Diamond
IFEval
Avg.
Qwen3-1.7B
Full precision
16/16
90.55
90.13
41.85
79.07
68.96
53.46
74.36
38.72
71.10
67.58
GPTQ
3/16
53.70
20.13
0.37
4.47
15.43
0.25
10.39
8.75
31.73
16.14
AutoRound
3/16
83.78
75.20
14.44
57.93
55.03
26.92
64.49
29.29
60.63
51.97
EfficientQAT
3/16
74.00
69.60
9.63
57.52
43.03
24.99
60.74
16.50
46.89
44.77
BitDistiller
3/16
80.39
66.33
8.52
56.71
49.47
20.00
61.51
24.75
58.78
47.38
Table 1: Quantization results (%) on Qwen3, averaged over three evaluation seeds (AutoRound: seed 42). QAT and OnQ SDs: Table 9 . Bold: best quantized score per model/bit group. W/A: weight/activation bits.
Figure 4: Accuracy versus mean generated tokens for Qwen3 models at 2 bits (top, a–c) and 3 bits (bottom, d–f), averaged over three evaluation seeds. Dashed lines connect model sizes within each method.
Figure 5: Generation outcomes on MATH-500 (Qwen3-1.7B W2, a) and AIME (Qwen3-4B W3, b). OnQ reduces repetitive errors on both benchmarks.
Method
W/A
GSM8K
MATH- 500
AIME 24–26
Human Eval+
MBPP+
LiveCode Bench
MMLU- Redux
GPQA- Diamond
IFEval
Avg.
Qwen3-1.7B
RTN
3/16
0.68
1.20
1.11
0.00
0.00
0.00
8.72
9.60
12.01
3.70
+ Calibration
3/16
87.11
80.00
17.78
58.54
58.20
30.71
67.16
29.80
61.92
54.58
OnQ
3/16
87.64
84.73
32.59
71.14
62.35
41.77
69.84
30.98
65.87
60.77
RTN
2/16
1.67
2.40
0.00
0.00
0.00
0.00
8.60
11.62
10.91
3.91
+ Calibration
2/16
42.84
25.60
0.00
15.85
19.84
2.84
45.07
21.72
36.41
23.35
Table 2: Scores (%) across recovery stages. OnQ denotes calibration followed by on-policy distillation. OnQ results average evaluation seeds 42, 43, and 44; RTN and calibration results use seed 42. Bold marks the best score within each model/bit group.
Method
MATH- 500
Human Eval+
MMLU- Redux
Fixed-response FKL
59.00
39.63
52.28
Fixed-response DKL
61.00
42.07
52.68
On-policy DKL (OnQ)
69.40
48.78
55.75
Table 3: Fixed-response versus on-policy distillation on Qwen3-1.7B W2 from shared calibration. Scores in %; best scores in bold.
Method
Objective
MATH- 500
Human Eval+
MMLU- Redux
Cold start
–
41.40
3.66
45.12
Vanilla OPD
RKL
63.00
10.98
48.18
OPD + GRPO
RKL + RLVR
66.00
4.27
48.84
GKD
FKL
66.20
11.59
48.82
GRPO
RLVR
31.40
0.61
34.05
OnQ
Dynamic KL
67.20
10.98
49.58
Table 4: On-policy objectives on Qwen3-1.7B W2 (400 steps from a shared cold start). Scores in %; best scores in bold.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Source
Calib.
OPD
Baselines
OpenMathReasoning
1,229
7,680
9,831
OpenCodeReasoning-2
1,229
7,680
9,831
OpenScienceReasoning-2
614
3,840
4,915
Dolci-Think
1,024
–
6,553
Tülu 3
–
5,120
–
IF-multi-constraints
–
1,280
–
Appendix
Table 5: Training data: calibration windows (2,048 tokens), OPD prompts, and question–response pairs shared by the three QAT baselines.
Method
Setting
Steps
Batch
Learning rate
EfficientQAT
W2 / W3
1,024
32
2×10−5 / 10−5
BitDistiller
W2 / W3
768 / 256
128
8×10−6
ReasoningQAT
1.7B W2
1,536
64
5×10−6
4B/8B W2
512
64
10−4
All W3
512
64
10−6
PTQ calibration: 4,096 sequences of 2,048 tokens, seed 42.
Appendix
Table 6: Baseline configurations for Table 1 . Batch size counts sequences per optimizer step. All methods use group size 128 and 2- or 3-bit weights.
Figure 6: Labeling instructions for identifying repetitive errors (excerpt). “Looping” in the instructions denotes the repetitive-error category of this paper.
Method
W/A
GSM8K
MATH- 500
AIME 24–26
Human Eval+
MBPP+
LiveCode Bench
MMLU- Redux
GPQA- Diamond
IFEval
Avg.
Qwen3-1.7B
EfficientQAT
3/16
8.82
11.80
57.78
6.10
9.26
19.12
5.02
54.21
30.44
22.50
BitDistiller
3/16
0.25
7.07
21.48
1.42
3.35
15.45
0.51
6.40
8.81
7.19
ReasoningQAT
3/16
0.03
2.00
13.33
0.61
1.59
3.38
0.12
0.67
5.73
3.05
OnQ
3/16
0.00
0.87
5.56
0.41
1.41
1.23
0.05
0.34
5.85
1.75
EfficientQAT
2/16
16.40
32.53
81.85
25.41
23.90
60.47
8.13
54.88
48.49
39.12
Appendix
Table 7: Truncation rates (%) for Table 1 . QAT baselines and OnQ, averaged over evaluation seeds 42, 43, and 44. Bold marks the lowest rate per model/bit group.
Method
W/A
GSM8K
MATH- 500
AIME 24–26
Human Eval+
MBPP+
LiveCode Bench
MMLU- Redux
GPQA- Diamond
IFEval
Avg.
Qwen3-1.7B
RTN
3/16
94.16
94.00
97.78
98.78
97.09
93.55
99.46
100.00
86.88
95.74
+ Calibration
3/16
0.08
2.60
16.67
0.61
1.06
8.34
0.12
0.51
7.21
4.13
OnQ
3/16
0.00
0.87
5.56
0.41
1.41
1.23
0.05
0.34
5.85
1.75
RTN
2/16
99.85
100.00
100.00
100.00
100.00
99.91
100.00
100.00
100.00
99.97
+ Calibration
2/16
0.68
20.80
52.22
17.68
25.66
58.96
3.67
22.73
36.04
26.49
Appendix
Table 8: Truncation rates (%) for Table 2 ; bold marks the lowest rate per model/bit group.
Method
W/A
GSM8K
MATH- 500
AIME 24–26
Human Eval+
MBPP+
LiveCode Bench
MMLU- Redux
GPQA- Diamond
IFEval
Avg.
Qwen3-1.7B
EfficientQAT
3/16
0.35
1.00
1.70
2.82
1.76
0.61
0.10
4.08
4.44
0.97
BitDistiller
3/16
0.34
1.01
2.31
1.61
0.95
0.43
0.59
2.02
1.61
0.42
ReasoningQAT
3/16
0.30
0.20
2.22
1.86
2.65
0.25
0.08
2.67
0.93
0.19
OnQ
3/16
0.65
1.42
2.31
1.53
0.55
0.63
0.26
0.77
0.21
0.49
EfficientQAT
2/16
1.59
2.75
0.64
1.83
0.85
0.81
0.49
1.82
3.49
0.80
Appendix
Table 9: Sample standard deviations (percentage points) of QAT and OnQ scores in Table 1 over evaluation seeds 42, 43, and 44.
Quantization-aware training (QAT) can recover much of the accuracy lost when large language models are compressed below four bits. Existing re- covery stages, however, are commonly optimized on fixed completions or teacher-generated answers, whereas the deployed quantized model condi- tions on prefixes generated by itself. Quantization errors can therefore move the model into states that are absent from offline recovery data. We introduce OnlineQAT, a two-stage framework that first obtains a usable low-bit initialization through block-wise QAT and then performs on-policy distillation (OPD) on student-generated responses. At each visited pre- fix, a frozen full-precision teacher provides a sampled reverse-KL training signal. On Qwen3-1.7B, OnlineQAT obtains the best average among the compared quantized methods: 57.28 at W3A16 and 32.52 at W2A16, im- proving over ReasoningQAT by 2.90 and 0.44 points, respectively. The results suggest that student-visited states provide a useful recovery signal beyond fixed-completion training, particularly at three bits.
Wenjun Wang, Heng Li, Yanggan Gu +1
The Hong Kong Polytechnic University · Sun Yat-sen University · PolyU-Daya Bay Technology and Innovation Research Institute
Quantization-aware distillation (QAD) restores much of the short-form question-answering performance lost to sub-3-bit quantization, yet leaves mathematical and code reasoning substantially impaired. Long generations often degenerate into repetitive loops, exhausting the decoding budget without completing a solution. We trace this gap to quantization-amplified exposure bias: QAD trains on fixed corpus prefixes, while quantization-induced deviations compound along the model's own autoregressive trajectories. To address this mismatch, we introduce an on-policy distillation (OPD) stage that places teacher supervision where the quantized model actually goes. Starting from a QAD checkpoint, the student generates through the quantized forward path used at deployment and receives feedback from a frozen full-precision teacher on its own prefixes, combining dense token-level guidance with task-verifier rewards. Across four models at 2.79 and 1.88 effective bits, OPD raises average BF16 performance retention from 35% to 70% on MATH-500 and from 66% to 91% on HumanEval while preserving short-form performance, with reasoning gains substantially exceeding those of continued teacher-forced QAD in matched-budget comparisons. By coupling QAD's stable low-bit initialization with OPD's on-policy reasoning recovery, our framework provides a comprehensive sub-3-bit solution that preserves broad capabilities while restoring long-form reasoning.
Yuanteng Chen, Zhilei Liu, Peisong Wang +7
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Zhongguancun Academy +2
Low-bit quantization reduces the cost of serving reasoning models but can substantially degrade long chains of thought. Each step reads cached keys and values spanning thousands of tokens, allowing quantization errors to accumulate along the trajectory. Standard calibration operates on short segments that omit the earlier context needed by later reasoning steps. Even when this context is retained, cache reconstruction error alone does not capture how quantization errors affect subsequent attention. We present RECAP, a post-training quantization method that calibrates reasoning models on the history they read. Dynamic-prefix calibration separates long historical context from short-block reconstruction. It regenerates prefix keys and values with the current quantizer and rotates the active block to supervise every position along the trajectory. Prefix read geometry further supervises historical cache errors according to their effects on attention outputs, weighting key and value errors asymmetrically. With both components applied only during calibration, RECAP outperforms the baseline across multiple reasoning models and quantization settings. Under W4A4KV4, RECAP improves accuracy over the baseline by 13.3 percentage points on MATH500 with DeepSeek-R1-Distill-Qwen-1.5B and 15.8 points on AIME-120 with Qwen3-4B. These results highlight the importance of preserving historical dependencies during calibration for low-bit reasoning.