Reasoning language models generate thousands of tokens per answer, and generation speed is limited by memory bandwidth, so storing weights in two bits instead of bfloat16 can potentially make inference 8× cheaper. Two-bit methods are usually judged by perplexity and multiple-choice question-answering accuracy, both of which say little about whether a model can still reason through more difficult problems. We present ThinkBit, a strictly scalar two-bit framework built to preserve reasoning, in which every weight is an integer codeword times a bfloat16 scale shared by a group of weights. We first use a sensitivity metric to assign each layer a ternary, 2-bit, or 3-bit tier to allocate bitwidths optimally among layers. This is followed by a layerwise Gumbel-softmax-based scalar quantization algorithm that searches for optimal codewords block by block. Since a layerwise scheme does not capture how quantized layers interact with each other, we recalibrate scales, with the codewords frozen, using a knowledge-distillation-based loss against the teacher model. We then perform a short quantization-aware training (QAT) stage that lets the model mimic the base model's distribution by reassigning codewords. Finally, a paired reasoning test on training problems selects the final checkpoint between the recalibration stage and the QAT stage. On Qwen3-1.7B at matched size, our recalibrated checkpoint scores 46.7 on MATH-500 against 30.0 for EfficientQAT trained on the same data, and 73.0 after QAT on reasoning traces; at 0.52 GB it is smaller than every published 2-bit GGUF of the model yet more accurate on MATH-500. We show that this framework scales and up to 14B, a 3.74 GB model scores 94.8 on MATH-500, against 97.5 for the 29.5 GB bfloat16 model.
Figures & tables
Figure 1: Overview of our 2-bit quantization framework ThinkBit and importance of different stages
Gumbel-softmax ( eqs. 11 and 12 )
ReinMax ( eq. 13 )
Forward weight
soft blend of grid values, c~=∑kπkvk
hard grid value, c=∑kωkvk
Gradient
exact gradient of the soft weight
second-order estimate for the hard weight
Temperature τ
annealed toward 0
kept ≥1
Train–test gap
vanishes only as π becomes one-hot
none: training uses grid weights
Used for
ternary and 3-bit layers
2-bit layers
Table 1: The two relaxations used by codeword search. Gumbel-softmax trains a smooth stand-in for the rounded weight with an exact gradient; ReinMax trains the rounded weight itself with an approximate gradient.
Name
Checkpoint
GSQ
Stages 1–2: tiers allocated and codewords searched; no training
Rec + Stage 4: QAT reassigns codewords (the full framework)
QAT-noRec
GSQ + Stage 4, without recalibration
LW-Rec , LW-QAT
as Rec and QAT , but starting from the layer-wise PTQ solution instead of codeword search
QAT-only
layer-wise start + Stage 4 only: no codeword search, no recalibration
Table 2: Checkpoint names. At 1.7B, Rec-h2 uses a second recalibration pass that quantizes the head and embedding to 2-bits; Rec-h2 (1 pass) uses one.
Stage
Data
Amount
1: KronQ sensitivity
subset of the Stage-2 calibration data
2M tokens
2: Codeword search (GSQ)
own replies, tool conversations, code and FineWeb-Edu
57M tokens
3: Scale recalibration
all sources (i)–(iii) presented in Section 3
268M tokens
4: QAT
all sources (i)–(iii) presented in Section 3
537M tokens
4 ∗ : QAT, reasoning data ( +R )
(i)–(iii) plus complete long reasoning traces ( section 3 )
1.1–2.1B tokens
5: Checkpoint selection
level-3–5 MATH training problems, disjoint from MATH-500
313 problems
Table 3: Data used by each stage. ∗ Longer QAT on the reasoning mixture, used for the selected checkpoints.
Figure 2: Qwen3-1.7B training curves. (a) Scale recalibration (codewords frozen): held-out KL to bf16 falls from 0.641 to 0.282. (b) Long QAT from the recalibrated model: KL increases briefly while codewords start to move, then falls from 0.311 to 0.199 (QAT is scored on the text that follows its own training slice, so its starting KL differs from the end of (a)). (c) Share of integer codewords that differ from the recalibrated model.
Qwen3-1.7B
WikiText-2 ↓
7-task
MMLU
GSM8K
HumanEval
MATH-500
bf16
16.67
58.9
55.5
68.6
41.5
91.2
Stage 2: GSQ-hbf16
46.91
43.8
36.4
27.5
8.5
14.8
Stage 3: Rec-hbf16
19.46
51.5
44.4
51.6
22.6
47.5
Stage 4: QAT-hbf16
16.54
54.6
49.9
50.2
17.7
43.0
Table 4: Recalibration recovers most of what codeword search loses; QAT further improves likelihood but not reasoning. Qwen3-1.7B after each stage ( bf16 head and embedding); Bold: best quantized value.
Model
Checkpoint
Strict acc.
Closed (%)
Acc. on closed
Qwen3-1.7B
bf16
91.2
99.2
91.9
Rec-hbf16
47.5
88.0
53.9
QAT-hbf16
43.0
69.6
61.7
QAT-noRec-hbf16
37.0
57.2
64.5
LW-QAT-hbf16
35.0
68.2
51.3
QAT-only-hbf16
32.1
55.4
58.0
Table 5: Two-bit models fail mainly by not finishing their reasoning; at 14B almost every trace finishes. Sampled MATH-500 split into the share of traces that close </think> and the accuracy on those. Bold: best quantized value per model.
Model
Checkpoint
GB
bits/param
7-task
MMLU
GSM8K
HumanEval
MATH-500
AIME@16
Qwen3-1.7B
bf16
3.44
16
58.9
55.5
68.6
41.5
91.2
41.4
≤ 2.6 bits per parameter
QAT+R-h2 (selected) §
0.52
2.42
53.8
47.0
58.7
25.6
73.0
12.3
Rec+R-h2 §
0.52
2.42
50.6
43.5
51.7
22.0
55.6
3.0
Rec-h2
0.52
2.42
51.1
45.2
51.8
22.6
46.7
1.0
1-bit Bonsai
0.25
1.15
49.7
41.7
43.9
37.2
28.7
0.7
Table 6: At every size, QAT+R-h2 is the most accurate checkpoint on MATH-500 and AIME@16 among files of at most 2.6 bits per parameter. MATH-500: sampled; AIME@16: mean pass@1 over 16 samples. Bonsai [ 55 ] and GGUF [ 75 ] files run in our harness. Rows grouped by bits per parameter; bold: best in the ≤ 2.6-bit group. § Trained on the reasoning mixture. † Official code on our base mixture, same tokens as Rec-h2 , bf16 head and embedding. ‡ Same with a 2-bit head and embedding. ¶ Trained with 4,096-token sequences.
Figure 3: Accuracy against whole-model size for every file of each Qwen3 model scored in our harness: (a–c) sampled, strictly scored MATH-500; (d, e) AIME@16. Filled stars: QAT+R-h2 (selected); hollow stars: Rec-h2 ; diamonds: official EfficientQAT and BitDistiller trained on our base mixture (at 1.7B also with a 2-bit head and embedding); hexagons: PrismML Bonsai (ternary and 1-bit); lines join the model’s published GGUF files (Unsloth [ 75 ] ; UD- prefix omitted) and bf16 . The shaded corner holds every file that is both larger and less accurate than the selected checkpoint, i.e. Pareto-dominated by it; dominated files are drawn hollow. Grey triangles in (a): Qwen3-0.6B at 4 and 8 bits. Dotted lines: 2.6 and 4 bits per parameter (the size groups of the tables).
MATH-500 (think)
Qwen3-1.7B
GB
bits/param
7-task
MMLU
GSM8K
HumanEval
BFCL NL
greedy
sampled
bf16
3.44
16
58.9
55.5
68.6
41.5
81.7
90.2
91.2
≤ 2.6 bits per parameter
PrismML 1-bit Bonsai ∘
0.25
1.15
49.7
41.7
43.9
37.2
74.3
25.4
28.7
PrismML Ternary Bonsai ∘
0.46
2.15
55.0
48.6
60.0
48.2
74.7
42.8
46.4
Ours: QAT+R-h2 (selected) §
0.52
2.42
53.8
47.0
58.7
25.6
73.0
68.6
73.0
Table 7: At 0.52 GB, QAT+R-h2 reasons better than every Qwen3-1.7B file of up to 4 bits per parameter; Ternary Bonsai leads the non-reasoning columns. Same harness for all files; bits/param use the original 1.72B parameters. Rows grouped by bits per parameter; bold: best in the ≤ 2.6-bit group. § QAT on the reasoning mixture. ‡ Second recalibration pass that quantizes the head and embedding. ∗ Not separately re-packed. † Official code on our base mixture, same tokens. ∘ Released files [ 55 ] . ¶ Bits per Qwen3-0.6B parameter. BFCL NL: BFCL non-live.
Figure 4: The framework at a glance. Every weight is an integer codeword times a bf16 scale shared by a group of 128 weights; one checkpoint is kept per model. The rest of this section gives the evidence for each stage.
MATH-500 (thinking)
Qwen3-1.7B variant
WikiText-2 ↓
7-task ↑
MMLU ↑
GSM8K ↑
HumanEval ↑
greedy ↑
sampled ↑
bf16
16.67
58.9
55.5
68.6
41.5
90.2
91.2
GSQ-hbf16
46.91
43.8
36.4
27.5
8.5
15.0
14.8
Scale recalibration (codewords frozen)
RMSNorm gains only
25.29
48.0
38.2
38.8
16.5
27.8
34.0
scales only
19.49
51.4
44.1
51.3
23.2
39.2
48.5
Table 8: Removing any stage keeps the 7-task mean within 1.5 points but costs 6.0–14.6 points of sampled MATH-500. Qwen3-1.7B variants ( bf16 head and embedding); paired tests in table 9 . Shaded: full framework; bold: best quantized value. “Revert” restores the pre-QAT codewords in one layer class. ∘ WikiText-2 from a separate pass ( bf16 : 16.72).
Model
Contrast (A vs. B)
A
B
Δ [95% CI]
p
status
Qwen3-1.7B
codeword search ( QAT vs. LW-QAT )
43.0
35.0
+8.0 [5.6, 10.2]
<10−4
abl
vs. tuned GPTQ start (all tiers, clipped scales)
43.0
32.6
+10.3 [8.0, 12.8]
<10−4
check
vs. GPTQ start + Hessian-weighted rounding
43.0
37.8
+5.2 [3.1, 7.5]
<10−4
check
recalibration before QAT ( QAT vs. QAT-noRec )
43.0
37.0
+6.0 [3.6, 8.3]
<10−4
abl
recalibration on layer-wise-start codewords ( LW-QAT vs. QAT-only )
35.0
32.1
+2.85 [0.85, 4.90]
0.009
abl
both ( QAT vs. QAT-only )
43.0
32.1
+10.8 [8.5, 13.0]
<10−4
abl
Table 9: Codeword search and recalibration each raise sampled MATH-500 significantly, and every selector decision agrees with MATH-500. Paired contrasts on MATH-500 ( n=500 ) and on the 313 selection problems. Bold: significant after Holm correction. Status: abl, stage ablation; check, robustness check; sel, selector decision. § Separate evaluation run ( Rec : 47.2 there, 47.45 in the main run).
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
Algorithm 2
M , Dcal , Dtrain , P
model; calibration and training sequences; selection problems
Nblk , X
number of transformer blocks; calibration activations entering the current block
Format ( section 2.1 )
Wℓ , Wℓ , nℓ
bf16 and quantized weight of linear layer ℓ ( dout×din ); number of weights
greedy, 1 sample; and sampled (temperature 0.6, top- p 0.95, top- k 20, 4 samples, strict scoring); 32k new tokens
Appendix
Table 11: Evaluation settings. GGUF files are dequantized to bf16 ; RACE is excluded from retention means because its passages appear in MMLU’s auxiliary data.
Qwen3-1.7B
ARC-e
ARC-c
Wino
HSwag
PIQA
LAMB
MMLU
Mean
bf16
69.7
43.0
60.9
60.4
72.2
50.5
55.5
58.9
GSQ-hbf16
51.4
29.9
54.3
41.5
64.9
28.4
36.4
43.8
Rec-hbf16
62.4
36.6
57.1
49.2
68.2
42.7
44.4
51.5
QAT-hbf16
66.4
38.2
59.2
52.6
69.0
47.1
49.9
54.6
QAT-h2
65.1
37.5
61.0
51.4
69.2
45.5
48.0
54.0
QAT+R-h2 (selected)
66.3
38.6
58.3
51.2
70.4
44.7
47.0
53.8
Appendix
Table 12: The 7-task means of table 4 task by task (length-normalized accuracy for ARC, HellaSwag and PIQA). Chance-normalized retention after GSQ / recalibration / QAT: 47.6 / 73.1 / 84.2%.
Figure 5: GSQ perplexity relative to the model’s own bf16 as blocks are quantized (before recalibration), Qwen3-1.7B.
Qwen3-1.7B
query/key at ≥ 2 bits ( ThinkBit )
not forced
ternary query / key projections (of 28)
0 / 0
15 / 3
KronQ objective (relative)
1.000
0.997
WikiText-2 after codeword search
46.91
46.35
held-out KL after recalibration
0.282
0.279
WikiText-2 after recalibration
19.46
19.56
sampled MATH-500 after recalibration
47.5
49.3
Appendix
Table 13: Keeping query and key projections at 2 bits or more changes reasoning by less than 2 points (MATH-500 +1.8, p=0.12 ). Otherwise identical settings.
WikiText-2
held-out
MATH-500
after QAT
Qwen3-1.7B start
before rec.
after rec.
KL
after rec.
after QAT
7-task
GSM8K
HumanEval
BFCL NL
GSQ (framework)
46.91
19.46
0.282
47.5
43.0
54.6
50.2
17.7
75.5
GPTQ, all tiers, clipped scales
1,379
22.02
0.402
18.5
32.6
53.8
49.4
17.7
70.0
GPTQ + Hessian-weighted rounding
10,352
21.06
0.367
24.1
37.8
53.8
51.9
17.7
71.9
Appendix
Table 14: Codeword search beats stronger GPTQ starts on reasoning after the same recalibration (rec.) and QAT, while likelihood barely separates them. Bold: best per column.
sampled MATH-500
selector
Model
Contrast (A vs. B)
A
B
Δ [95% CI]
p
Δ [95% CI]
p
Qwen3-1.7B
Rec-h2 : two recalibration passes vs. one
46.7
44.4
+2.3 [0.3, 4.2]
0.027
+0.6
0.71
Rec-h2 (1 pass) vs. QAT-h2
44.4
42.0
–
–
+5.1 [2.4, 8.0]
<10−3
Rec-hbf16 : two passes vs. one pass
46.6
47.5
− 0.9
0.38
− 2.3
0.052
two passes: Rec-h2 vs. Rec-hbf16
46.7
46.6
+0.1
0.96
–
–
QAT-h2 vs. QAT-noRec-h2
42.0
38.2
+3.8 [1.7, 6.0]
<10−3
+3.2 [1.0, 5.6]
0.009
Appendix
Table 15: Quantizing the head and embedding costs no detectable accuracy. Further paired contrasts (pointwise tests, not Holm-corrected).
QAT vs.
7-task Δ [90% CI]
equiv.
MMLU Δ [90% CI]
equiv.
LW-QAT : layer-wise start instead of GSQ
+0.91 [0.37, 1.44]
yes
+1.22 [0.61, 1.84]
no
without the scale recalibration
+0.18 [-0.29, 0.63]
yes
+1.03 [0.47, 1.59]
no
QAT-only (neither)
-0.15 [-0.64, 0.34]
yes
-0.02 [-0.69, 0.65]
yes
uniform 2-bit, GPTQ, recalibration
+0.20 [-0.35, 0.73]
yes
+0.72 [0.08, 1.39]
yes
tuned GPTQ start
+0.79 [0.32, 1.28]
yes
+2.07 [1.47, 2.68]
no
GPTQ + Hessian-weighted rounding
+0.80 [0.31, 1.32]
yes
+1.62 [0.98, 2.29]
no
Appendix
Table 16: Every stage removal is equivalent to the full framework on the 7-task mean (margin ± 1.5); skipping QAT is not. Δ : QAT-hbf16 minus the named checkpoint.
Model
Contrast (A vs. B)
A
B
Δ [95% CI]
p
status
Qwen3-1.7B
QAT-h2 vs. Rec-h2 (0.52 GB)
42.0
46.7
− 4.7 [ − 7.1, − 2.2]
4⋅10−4
post hoc
Rec-h2 vs. GGUF Q2_K [ 62 ] (0.78 GB)
46.7
41.9
+4.8 [2.0, 7.7]
0.001
post hoc
Rec-h2 vs. Rec-hbf16 (0.52 vs. 0.98 GB)
46.7
47.5
− 0.8 [ − 2.7, 1.0]
0.44
post hoc
Qwen3-14B
Rec-hbf16 (6.4 GB) vs. QAT-h2 (3.74 GB)
92.8
90.5
+2.3 [0.9, 3.7]
0.002
post hoc
QAT-h2 vs. Rec-h2 (3.74 GB)
90.5
92.7
− 2.2 [ − 3.8, − 0.6]
0.011
post hoc
codeword search at the recalibrated checkpoint ( Rec-hbf16 vs. LW-Rec-hbf16 ; exploratory)
92.8
89.7
+3.2 [1.8, 4.7]
10−4
post hoc
Appendix
Table 17: At 0.52 GB, Rec-h2 beats QAT-h2 and the larger Q2_K; at 14B every two-bit checkpoint beats the smallest GGUF. Paired MATH-500 contrasts (AIME@16 where noted). Bold: significant after Holm correction together with table 9 . § Separate evaluation run. Status: pre-reg., pre-registered; post hoc, chosen after earlier results; check, robustness check.
Model
A vs. B
A
B
p
kept
Qwen3-1.7B
QAT-hbf16 vs. Rec-hbf16
22.5
28.8
<10−3
Rec-hbf16
Qwen3-1.7B
QAT-h2 vs. Rec-h2
21.9
27.6
<10−3
Rec-h2
Qwen3-1.7B
Rec-hbf16 vs. Rec-h2
28.8
27.6
0.37
Rec-h2
Qwen3-14B
QAT-h2 vs. Rec-hbf16
83.3
87.9
<10−3
Rec-hbf16
Qwen3-14B
QAT-h2 vs. Rec-h2
83.3
87.5
<10−3
Rec-h2
Qwen3-14B
Rec-hbf16 vs. Rec-h2
87.9
87.5
0.73
Rec-h2
Appendix
Table 18: Under the base mixture the selector kept the recalibrated checkpoint in every decision, and MATH-500 agrees. A, B: strict accuracy on the 313 selection problems.
Qwen3-1.7B, codewords frozen
Held-out KL
WikiText-2
GSQ-hbf16
0.641
46.91
+ RMSNorm gains only
0.418
25.29
+ scales only
0.282
19.49
+ scales and RMSNorm gains ( Rec-hbf16 )
0.282
19.46
Appendix
Table 19: The scales carry recalibration: scales alone reach the same held-out KL as scales plus RMSNorm gains. Qwen3-1.7B, same data and schedule.
median
same sign
relative weight error
Tier
∣r∣
as s⋆ (%)
corr.
GSQ s0
recalibrated s
least-squares s⋆
ternary
0.064
51.6
0.08
0.425
0.431
0.325
2-bit
0.054
49.8
0.00
0.198
0.207
0.192
3-bit
0.058
50.8
0.05
0.287
0.296
0.109
all
0.062
51.4
0.07
0.386
0.393
0.277
Appendix
Table 20: Recalibration does not move scales toward the weight-space optimum, and weight error rises in every tier. Per-group scale changes on Qwen3-1.7B; corr.: correlation of r with log(s⋆/s0) .
Qwen3-1.7B
local block error
propagated error
next-token KL
GSQ-hbf16
0.079
0.214
1.300
Rec-hbf16
0.094
0.282
0.571
QAT-hbf16
0.122
0.373
0.371
Appendix
Table 21: Recalibration and QAT make single blocks less faithful but bring the output distribution closer to bf16 . Relative block-output error (local: bf16 input; propagated: fully quantized model) and output KL on WikiText-2.
Figure 6: QAT latent initialization α× learning rate on Qwen3-1.7B (short runs from the recalibrated model). (a) Held-out KL relative to the start. (b) Share of codewords changed. With α=0 fewer than 0.003% of codewords are reassigned. Validation sequences are the chunks that follow each run’s training slice, so runs with different training lengths are scored on different held-out text; this is why the starting KL (0.251) differs from long QAT’s (0.311). KL values are comparable within a run, not across runs.
Figure 7: Where long QAT reassigns codewords on Qwen3-1.7B. (a) Share of each projection type’s codewords that QAT changes; value projections (orange) change most, query and key projections (dark blue) least. (b) Share changed in each transformer block, with the share per tier in the inset text. Codewords at the first and last transformer blocks are reassigned more than other blocks (c) Distribution, before QAT, of the distance ∣W/s−c∣ between each codeword c and its bf16 weight W/s , in grid steps with the recalibrated scales, for all codewords (grey) and for those that QAT reassigns (orange). The dotted line at 0.5 is the rounding boundary: most reassigned codewords lie beyond it, and 99.8% of reassignments move one step toward the bf16 weight.
Thinking budget (tokens)
1k
2k
4k
8k
16k
32k, unforced
bf16
74.2
77.0
82.4
86.0
90.0
90.2
Rec-hbf16
40.4
43.2
44.4
44.6
45.2
40.6
QAT-hbf16
42.2
43.6
44.4
43.4
39.2
39.0
QAT-only-hbf16
33.6
34.6
36.6
31.0
28.8
26.6
0.52 GB checkpoints (2-bit head and embedding) and Q2_K
Rec-h2 , 0.52 GB
40.6
42.0
43.4
42.6
41.0
39.0
Appendix
Table 22: With a forced answer the 0.52 GB checkpoints and Q2_K tie, so the strict lead of Rec-h2 comes from finishing more traces. Greedy thinking-mode MATH-500 on Qwen3-1.7B with the answer forced after the given number of thinking tokens.
Model
Contrast
Δ closing rate
Δ accuracy on closed
problems
Qwen3-1.7B
codeword search
+1.4 [ − 1.0, 3.8]
+10.4 [7.1, 13.6]
414
scale recalibration
+12.3 [9.8, 14.9]
+2.1 [ − 1.4, 5.8]
379
QAT vs. Rec
− 18.4 [ − 21.1, − 15.6]
− 0.5 [ − 3.5, 2.5]
440
Appendix
Table 23: Codeword search mainly raises accuracy on closed traces; recalibration and QAT mainly change how many traces close. Problem-matched sampled MATH-500 differences (95% CI); bold: p<0.01 .
quantized
bits per quantized weight
Model
Checkpoint
weights
measured
computed
file (GB)
Qwen3-1.7B
QAT-hbf16
1.41B
2.011
2.000
0.977
QAT-h2
2.03B
2.046
2.038
0.520
Rec-h2
2.03B
2.046
2.038
0.520
Qwen3-14B
QAT-only-h2
14.77B
2.023
2.013
3.736
Rec-h2
14.77B
2.023
2.013
3.736
Appendix
Table 24: Packed files decode bit-exactly and are within 0.6% of the sizes computed from the grids. Bits per quantized weight include the group scales.
weight memory
decode (tokens/s)
prefill, 1,024
top-1 agreement
Qwen3
(GiB)
batch 1
batch 16
tokens (ms)
(%)
1.7B
3.22 → 1.85
424 → 408
6,232 → 5,998
12.8 → 15.7
98.7
8B
15.27 → 5.68
146 → 212
2,225 → 3,184
27.0 → 28.4
99.1
14B
27.52 → 9.31
86 → 150
1,341 → 2,302
48.0 → 45.0
98.3
Appendix
Table 25: Serving from packed containers cuts weight memory 1.7–3.0 × and speeds up batch-1 decoding at 8B and 14B. vLLM on one H100; memory in GiB.
Figure 8: Qwen3-1.7B: retention on 14 benchmarks after recalibration and after QAT ( Rec-hbf16 , QAT-hbf16 ) (RACE omitted; its test passages appear in MMLU’s auxiliary training data). Numbers above bars are QAT retention.
Figure 9: Qwen3-1.7B: score against whole-model bytes (generation metrics greedy). Grey: published llama.cpp quantizations (Unsloth [ 75 ] ; the UD- prefix is omitted in labels). Red star: ours with a 2-bit head and embedding ( QAT-h2 ); orange diamond: ours with the bf16 tied embedding ( QAT-hbf16 ). Dotted: bf16 . Over 30 individually scored sub-scores (RACE excluded; greedy MATH counted; differences of at most 0.5 points count as ties) the 0.52 GB QAT checkpoint ( QAT-h2 ) wins 23–26 against each 2-bit GGUF.
Model
Checkpoint
GB
strict
lenient
closed (%)
strict if closed
≤ 2.6 bits per parameter
Qwen3-1.7B
QAT+R-h2 (selected)
0.52
73.0
74.6
87.1
83.8
Qwen3-1.7B
Rec-h2
0.52
46.7
47.3
86.0
54.2
Qwen3-1.7B
QAT-h2
0.52
42.0
44.9
70.0
60.0
2.6–4 bits per parameter
Qwen3-1.7B
GGUF UD-IQ2_XXS [ 66 ]
0.61
1.0
2.8
46.8
2.1
Appendix
Table 26: QAT+R-h2 closes more traces than Q2_K and is more accurate on them; the earlier 0.52 GB checkpoints win only by closing more traces. Sampled MATH-500 split into traces that close within 32k tokens and strict accuracy on them. Rows grouped by bits per parameter.
Checkpoint
Size (GB) ↓
7-task ↑
MMLU ↑
GSM8K ↑
HE ↑
MT ↑
M500 ↑
A24/25 ↑
A@16 ↑
Qwen3-14B bf16
29.5
74.3
77.1
88.2
57.3
40.8
97.5
76.7/60.0
73.6
Smaller models at 4–5 bits per parameter (reference)
Qwen3-4B GGUF Q4_K_M [ 70 ]
2.50
66.7
68.3
84.8
70.1
30.0
96.3
73.3/50.0
65.3
Qwen3-8B GGUF Q3_K_M [ 72 ]
4.12
69.2
71.0
86.5
61.0
31.6
95.8
56.7/50.0
64.1
Qwen3-14B, ≤ 2.6 bits per parameter
Ours: Rec-h2
3.74
69.9
69.5
87.7
65.2
32.0
92.7
50.0/33.3
46.5
Appendix
Table 27: At 3.74 GB our checkpoints beat UD-IQ2_XXS (4.47 GB) on reasoning; Q2_K and larger files remain stronger. Qwen3-14B by whole-model bytes. M500: sampled MATH-500; A24/25: greedy AIME; A@16: mean pass@1 over 16 samples; HE: HumanEval; MT: BFCL multi-turn. Rows grouped by bits per parameter. † Not packed; ∗ computed from the grids; § separate evaluation run.
Low-bit quantization reduces the memory footprint of large language models, but preserving accuracy with efficient training remains challenging. Scalar quantization encodes weights independently, while explicit-codebook vector quantization introduces costly search during training. We propose E8-QAT, a 2-bit quantization-aware training framework that uses the E8 lattice as an implicit eight-dimensional codebook. It jointly quantizes weight groups through arithmetic projection without storing or searching an explicit codebook during the training forward pass. A 2-bit core encoding and sparse extension preserve the selected lattice points exactly, adding less than 0.1% to core index storage. On Qwen3.6-35B-A3B, E8-QAT matches the performance of the full-precision model in terms of the average score across the four tasks. E8-QAT improves over the state-of-the-art baseline by 2.03% on IFEval and by 1.51% on LiveCodeBench, under the same data, schedule, and bit width. The 2-bit representation reduces peak memory by 80% and supports a 4× larger batch. At each representation's largest feasible batch, E8-QAT reaches 1.97× the decoding throughput of BF16.
Xinyu Lei, Hong Huang, Jiebin Zhang +6
Peking University · City University of Hong Kong · Tencent
Large Reasoning Models (LRMs) rely on long reasoning traces, making inference expensive. While low-bit quantization reduces per-token decoding cost, we show that aggressive 2-bit inference can fail to deliver end-to-end speedup because instability in the generation process inflates total token count. Instead of merely lowering answer accuracy, 2-bit quantization often produces much longer traces with repetitive loops, budget exhaustion, delayed commitment, and unclosed reasoning segments. We analyze full reasoning traces of Qwen3 reasoning models across mathematical and commonsense benchmarks and show that accuracy degradation is tightly linked to these process-level failures. To address them, we introduce two lightweight controls: FP16 planning, which gives the 2-bit model a short high-precision outline, and loop rescue, which detects repetitive traces and either commits to an earlier answer or falls back to FP16. On MATH-500, loop rescue improves Qwen3-8B accuracy from 17.2% to 74.2%, while planning plus loop rescue improves Qwen3-32B from 65.0% to 87.2%. Overall, our results show that extreme low-bit reasoning becomes practical when its failures are treated as controllable generation pathologies: with lightweight detection and selective FP16 support, 2-bit inference can recover accuracy while preserving real end-to-end speed. Our code is available at: https://github.com/brain-lab-research/quantized-reasoning.
We propose ScaleQ-1.58, a scalable ternary post-training quantization (PTQ) framework for reasoning LLMs. Its core insight stems from an empirical finding: although modern LLMs are typically trained to exhibit chain-of-thought reasoning capabilities, in the PTQ regime, even the latest CAT-Q method based on learning-based differentiable ternarization still leads to performance collapse on challenging mathematics and coding tasks when using conventional calibration schemes that ignore the model's reasoning process. Driven by this finding, we introduce a simple calibration approach, Attend to Your Own Thoughts (AYOT), where reasoning traces and final answers generated by the pre-trained high-precision target LLM on a proper set of calibration samples are used as the context input during the ternarization process, along with the corresponding questions. ScaleQ-1.58 is formed by simply integrating AYOT with CAT-Q, which demonstrates several scaling properties: (1) with only 4M calibration tokens, Qwen3-1.7B ternarized by ScaleQ-1.58 reaches over 90.52% of the performance of the prior best BitNet b1.58 2B4T averaged over 4 mathematics and coding tasks, and our ternary Qwen3-4B shows an absolute gain of 8.97%, while requiring 1,000,000x fewer calibration tokens for quantization; (2) ScaleQ-1.58 generalizes well to both dense and MoE architectures, with performance improving as model scale increases (up to 235B parameters); (3) ScaleQ-1.58 demonstrates strong generalization across tasks of varying difficulty levels, including mathematics, coding and scientific logic reasoning, as well as commonsense reasoning and basic language generation; (4) its performance continues to improve as the number of calibration tokens increases. Notably, AYOT also exhibits strong generalization ability across other quantization bit-widths. Code will be available at https://github.com/IntelChina-AI/BitTern.