ThinkBit: Search Codewords, Recalibrate Scales, Select by Reasoning

Rhea Sanjay Sukthanker; Pashmina Cameron

cs.AIOct 26, 2026

ThinkBit: Search Codewords, Recalibrate Scales, Select by Reasoning

Authors: Rhea Sanjay Sukthanker, Pashmina Cameron

Organizations: Microsoft Applied Sciences Group

Abstract

Reasoning language models generate thousands of tokens per answer, and generation speed is limited by memory bandwidth, so storing weights in two bits instead of bfloat16 can potentially make inference 8 $\times$ cheaper. Two-bit methods are usually judged by perplexity and multiple-choice question-answering accuracy, both of which say little about whether a model can still reason through more difficult problems. We present ThinkBit, a strictly scalar two-bit framework built to preserve reasoning, in which every weight is an integer codeword times a bfloat16 scale shared by a group of weights. We first use a sensitivity metric to assign each layer a ternary, 2-bit, or 3-bit tier to allocate bitwidths optimally among layers. This is followed by a layerwise Gumbel-softmax-based scalar quantization algorithm that searches for optimal codewords block by block. Since a layerwise scheme does not capture how quantized layers interact with each other, we recalibrate scales, with the codewords frozen, using a knowledge-distillation-based loss against the teacher model. We then perform a short quantization-aware training (QAT) stage that lets the model mimic the base model's distribution by reassigning codewords. Finally, a paired reasoning test on training problems selects the final checkpoint between the recalibration stage and the QAT stage. On Qwen3-1.7B at matched size, our recalibrated checkpoint scores 46.7 on MATH-500 against 30.0 for EfficientQAT trained on the same data, and 73.0 after QAT on reasoning traces; at 0.52 GB it is smaller than every published 2-bit GGUF of the model yet more accurate on MATH-500. We show that this framework scales and up to 14B, a 3.74 GB model scores 94.8 on MATH-500, against 97.5 for the 29.5 GB bfloat16 model.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of our 2-bit quantization framework ThinkBit and importance of different stages

	Gumbel-softmax ( eqs. 11 and 12 )	ReinMax ( eq. 13 )
Forward weight	soft blend of grid values, $\tilde{c}=\sum_{k}\pi_{k}v_{k}$	hard grid value, $c=\sum_{k}\omega_{k}v_{k}$
Gradient	exact gradient of the soft weight	second-order estimate for the hard weight
Temperature $\tau$	annealed toward 0	kept $\geq 1$
Train–test gap	vanishes only as $\pi$ becomes one-hot	none: training uses grid weights
Used for	ternary and 3-bit layers	2-bit layers

Table 1: The two relaxations used by codeword search. Gumbel-softmax trains a smooth stand-in for the rounded weight with an exact gradient; ReinMax trains the rounded weight itself with an approximate gradient.

Name	Checkpoint
GSQ	Stages 1–2: tiers allocated and codewords searched; no training
Rec	GSQ + Stage 3: scales and RMSNorm gains recalibrated, codewords frozen
QAT	Rec + Stage 4: QAT reassigns codewords (the full framework)
QAT-noRec	GSQ + Stage 4, without recalibration
LW-Rec , LW-QAT	as Rec and QAT , but starting from the layer-wise PTQ solution instead of codeword search
QAT-only	layer-wise start + Stage 4 only: no codeword search, no recalibration

Table 2: Checkpoint names. At 1.7B, Rec-h2 uses a second recalibration pass that quantizes the head and embedding to 2-bits; Rec-h2 (1 pass) uses one.

Stage	Data	Amount
1: KronQ sensitivity	subset of the Stage-2 calibration data	2M tokens
2: Codeword search (GSQ)	own replies, tool conversations, code and FineWeb-Edu	57M tokens
3: Scale recalibration	all sources (i)–(iii) presented in Section 3	268M tokens
4: QAT	all sources (i)–(iii) presented in Section 3	537M tokens
4 ∗ : QAT, reasoning data ( +R )	(i)–(iii) plus complete long reasoning traces ( section 3 )	1.1–2.1B tokens
5: Checkpoint selection	level-3–5 MATH training problems, disjoint from MATH-500	313 problems

Table 3: Data used by each stage. ∗ Longer QAT on the reasoning mixture, used for the selected checkpoints.

Figure 2: Qwen3-1.7B training curves. (a) Scale recalibration (codewords frozen): held-out KL to bf16 falls from 0.641 to 0.282. (b) Long QAT from the recalibrated model: KL increases briefly while codewords start to move, then falls from 0.311 to 0.199 (QAT is scored on the text that follows its own training slice, so its starting KL differs from the end of (a)). (c) Share of integer codewords that differ from the recalibrated model.

Qwen3-1.7B	WikiText-2 $\downarrow$	7-task	MMLU	GSM8K	HumanEval	MATH-500
bf16	16.67	58.9	55.5	68.6	41.5	91.2
Stage 2: GSQ-hbf16	46.91	43.8	36.4	27.5	8.5	14.8
Stage 3: Rec-hbf16	19.46	51.5	44.4	51.6	22.6	47.5
Stage 4: QAT-hbf16	16.54	54.6	49.9	50.2	17.7	43.0

Table 4: Recalibration recovers most of what codeword search loses; QAT further improves likelihood but not reasoning. Qwen3-1.7B after each stage ( bf16 head and embedding); Bold: best quantized value.

Model	Checkpoint	Strict acc.	Closed (%)	Acc. on closed
Qwen3-1.7B	bf16	91.2	99.2	91.9
	Rec-hbf16	47.5	88.0	53.9
	QAT-hbf16	43.0	69.6	61.7
	QAT-noRec-hbf16	37.0	57.2	64.5
	LW-QAT-hbf16	35.0	68.2	51.3
	QAT-only-hbf16	32.1	55.4	58.0

Table 5: Two-bit models fail mainly by not finishing their reasoning; at 14B almost every trace finishes. Sampled MATH-500 split into the share of traces that close </think> and the accuracy on those. Bold: best quantized value per model.

	$\leq$ 2.6 bits per parameter
Model	Checkpoint	GB	bits/param	7-task	MMLU	GSM8K	HumanEval	MATH-500	AIME@16
Qwen3-1.7B	bf16	3.44	16	58.9	55.5	68.6	41.5	91.2	41.4
	QAT+R-h2 (selected) §	0.52	2.42	53.8	47.0	58.7	25.6	73.0	12.3
	Rec+R-h2 §	0.52	2.42	50.6	43.5	51.7	22.0	55.6	3.0
	Rec-h2	0.52	2.42	51.1	45.2	51.8	22.6	46.7	1.0
	1-bit Bonsai	0.25	1.15	49.7	41.7	43.9	37.2	28.7	0.7

Table 6: At every size, QAT+R-h2 is the most accurate checkpoint on MATH-500 and AIME@16 among files of at most 2.6 bits per parameter. MATH-500: sampled; AIME@16: mean pass@1 over 16 samples. Bonsai [ 55 ] and GGUF [ 75 ] files run in our harness. Rows grouped by bits per parameter; bold: best in the

\leq

2.6-bit group. § Trained on the reasoning mixture. † Official code on our base mixture, same tokens as Rec-h2 , bf16 head and embedding. ‡ Same with a 2-bit head and embedding. ¶ Trained with 4,096-token sequences.

Figure 3: Accuracy against whole-model size for every file of each Qwen3 model scored in our harness: (a–c) sampled, strictly scored MATH-500; (d, e) AIME@16. Filled stars: QAT+R-h2 (selected); hollow stars: Rec-h2 ; diamonds: official EfficientQAT and BitDistiller trained on our base mixture (at 1.7B also with a 2-bit head and embedding); hexagons: PrismML Bonsai (ternary and 1-bit); lines join the model’s published GGUF files (Unsloth [ 75 ] ; UD- prefix omitted) and bf16 . The shaded corner holds every file that is both larger and less accurate than the selected checkpoint, i.e. Pareto-dominated by it; dominated files are drawn hollow. Grey triangles in (a): Qwen3-0.6B at 4 and 8 bits. Dotted lines: 2.6 and 4 bits per parameter (the size groups of the tables).

$\leq$ 2.6 bits per parameter
								MATH-500 (think)
Qwen3-1.7B	GB	bits/param	7-task	MMLU	GSM8K	HumanEval	BFCL NL	greedy	sampled
bf16	3.44	16	58.9	55.5	68.6	41.5	81.7	90.2	91.2
PrismML 1-bit Bonsai ∘	0.25	1.15	49.7	41.7	43.9	37.2	74.3	25.4	28.7
PrismML Ternary Bonsai ∘	0.46	2.15	55.0	48.6	60.0	48.2	74.7	42.8	46.4
Ours: QAT+R-h2 (selected) §	0.52	2.42	53.8	47.0	58.7	25.6	73.0	68.6	73.0

Table 7: At 0.52 GB, QAT+R-h2 reasons better than every Qwen3-1.7B file of up to 4 bits per parameter; Ternary Bonsai leads the non-reasoning columns. Same harness for all files; bits/param use the original 1.72B parameters. Rows grouped by bits per parameter; bold: best in the

\leq

2.6-bit group. § QAT on the reasoning mixture. ‡ Second recalibration pass that quantizes the head and embedding. ∗ Not separately re-packed. † Official code on our base mixture, same tokens. ∘ Released files [ 55 ] . ¶ Bits per Qwen3-0.6B parameter. BFCL NL: BFCL non-live.

Panel of Figure 4 — Figure 4: The framework at a glance. Every weight is an integer codeword times a bf16 scale shared by a group of 128 weights; one checkpoint is kept per model. The rest of this section gives the evidence for each stage.

Scale recalibration (codewords frozen)
						MATH-500 (thinking)
Qwen3-1.7B variant	WikiText-2 $\downarrow$	7-task $\uparrow$	MMLU $\uparrow$	GSM8K $\uparrow$	HumanEval $\uparrow$	greedy $\uparrow$	sampled $\uparrow$
bf16	16.67	58.9	55.5	68.6	41.5	90.2	91.2
GSQ-hbf16	46.91	43.8	36.4	27.5	8.5	15.0	14.8
RMSNorm gains only	25.29	48.0	38.2	38.8	16.5	27.8	34.0
scales only	19.49	51.4	44.1	51.3	23.2	39.2	48.5

Table 8: Removing any stage keeps the 7-task mean within 1.5 points but costs 6.0–14.6 points of sampled MATH-500. Qwen3-1.7B variants ( bf16 head and embedding); paired tests in table 9 . Shaded: full framework; bold: best quantized value. “Revert” restores the pre-QAT codewords in one layer class. ∘ WikiText-2 from a separate pass ( bf16 : 16.72).

Model	Contrast (A vs. B)	A	B	$\Delta$ [95% CI]	$p$	status
Qwen3-1.7B	codeword search ( QAT vs. LW-QAT )	43.0	35.0	+8.0 [5.6, 10.2]	$<10^{-4}$	abl
	vs. tuned GPTQ start (all tiers, clipped scales)	43.0	32.6	+10.3 [8.0, 12.8]	$<10^{-4}$	check
	vs. GPTQ start + Hessian-weighted rounding	43.0	37.8	+5.2 [3.1, 7.5]	$<10^{-4}$	check
	recalibration before QAT ( QAT vs. QAT-noRec )	43.0	37.0	+6.0 [3.6, 8.3]	$<10^{-4}$	abl
	recalibration on layer-wise-start codewords ( LW-QAT vs. QAT-only )	35.0	32.1	+2.85 [0.85, 4.90]	0.009	abl
	both ( QAT vs. QAT-only )	43.0	32.1	+10.8 [8.5, 13.0]	$<10^{-4}$	abl

Table 9: Codeword search and recalibration each raise sampled MATH-500 significantly, and every selector decision agrees with MATH-500. Paired contrasts on MATH-500 (

n{=}500

) and on the 313 selection problems. Bold: significant after Holm correction. Status: abl, stage ablation; check, robustness check; sel, selector decision. § Separate evaluation run ( Rec : 47.2 there, 47.45 in the main run).

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
Algorithm 2
$M$ , $\mathcal{D}_{\mathrm{cal}}$ , $\mathcal{D}_{\mathrm{train}}$ , $\mathcal{P}$	model; calibration and training sequences; selection problems
$N_{\mathrm{blk}}$ , $X$	number of transformer blocks; calibration activations entering the current block
Format ( section 2.1 )
$W_{\ell}$ , $\widehat{W}_{\ell}$ , $n_{\ell}$	bf16 and quantized weight of linear layer $\ell$ ( $d_{\mathrm{out}}\times d_{\mathrm{in}}$ ); number of weights
$G$ , $\gamma(j)$	group size (128); group of input column $j$

Appendix

Table 10: Notation.

Suite	Tasks	Shots / format	Decoding
7-task core	ARC-e/c, WinoGrande, HellaSwag, PIQA, LAMBADA, MMLU	lm-eval defaults (0-shot), log-likelihood	–
Generative	GSM8K (strict match)	5-shot, no chat template	greedy
	HumanEval	0-shot, pass@1	greedy
Tool use	BFCL v4: non-live (syntax-tree match), live, multi-turn, irrelevance	official harness, native tool format	greedy, vLLM server
Broad suite	MMLU-Pro, TriviaQA, TruthfulQA, SciQ, OpenBookQA, CommonsenseQA, BoolQ, RACE, ARC-C, BBH-CoT, MATH-500, GSM8K-CoT, IFEval, MBPP, HumanEval+	task defaults; ARC-C 25-shot; BBH 3-shot CoT; GSM8K-CoT 8-shot	greedy
Thinking mode	AIME24, AIME25 (30 problems each), MATH-500	chat template, thinking on, 0-shot	greedy, 1 sample; and sampled (temperature 0.6, top- $p$ 0.95, top- $k$ 20, 4 samples, strict scoring); 32k new tokens

Appendix

Table 11: Evaluation settings. GGUF files are dequantized to bf16 ; RACE is excluded from retention means because its passages appear in MMLU’s auxiliary data.

Qwen3-1.7B	ARC-e	ARC-c	Wino	HSwag	PIQA	LAMB	MMLU	Mean
bf16	69.7	43.0	60.9	60.4	72.2	50.5	55.5	58.9
GSQ-hbf16	51.4	29.9	54.3	41.5	64.9	28.4	36.4	43.8
Rec-hbf16	62.4	36.6	57.1	49.2	68.2	42.7	44.4	51.5
QAT-hbf16	66.4	38.2	59.2	52.6	69.0	47.1	49.9	54.6
QAT-h2	65.1	37.5	61.0	51.4	69.2	45.5	48.0	54.0
QAT+R-h2 (selected)	66.3	38.6	58.3	51.2	70.4	44.7	47.0	53.8

Appendix

Table 12: The 7-task means of table 4 task by task (length-normalized accuracy for ARC, HellaSwag and PIQA). Chance-normalized retention after GSQ / recalibration / QAT: 47.6 / 73.1 / 84.2%.

Figure 5: GSQ perplexity relative to the model’s own bf16 as blocks are quantized (before recalibration), Qwen3-1.7B.

Qwen3-1.7B	query/key at $\geq$ 2 bits ( ThinkBit )	not forced
ternary query / key projections (of 28)	0 / 0	15 / 3
KronQ objective (relative)	1.000	0.997
WikiText-2 after codeword search	46.91	46.35
held-out KL after recalibration	0.282	0.279
WikiText-2 after recalibration	19.46	19.56
sampled MATH-500 after recalibration	47.5	49.3

Appendix

Table 13: Keeping query and key projections at 2 bits or more changes reasoning by less than 2 points (MATH-500 +1.8,

p{=}0.12

). Otherwise identical settings.

	WikiText-2		held-out	MATH-500		after QAT
Qwen3-1.7B start	before rec.	after rec.	KL	after rec.	after QAT	7-task	GSM8K	HumanEval	BFCL NL
GSQ (framework)	46.91	19.46	0.282	47.5	43.0	54.6	50.2	17.7	75.5
GPTQ, all tiers, clipped scales	1,379	22.02	0.402	18.5	32.6	53.8	49.4	17.7	70.0
GPTQ + Hessian-weighted rounding	10,352	21.06	0.367	24.1	37.8	53.8	51.9	17.7	71.9

Appendix

Table 14: Codeword search beats stronger GPTQ starts on reasoning after the same recalibration (rec.) and QAT, while likelihood barely separates them. Bold: best per column.

		sampled MATH-500				selector
Model	Contrast (A vs. B)	A	B	$\Delta$ [95% CI]	$p$	$\Delta$ [95% CI]	$p$
Qwen3-1.7B	Rec-h2 : two recalibration passes vs. one	46.7	44.4	+2.3 [0.3, 4.2]	0.027	+0.6	0.71
	Rec-h2 (1 pass) vs. QAT-h2	44.4	42.0	–	–	+5.1 [2.4, 8.0]	$<10^{-3}$
	Rec-hbf16 : two passes vs. one pass	46.6	47.5	$-$ 0.9	0.38	$-$ 2.3	0.052
	two passes: Rec-h2 vs. Rec-hbf16	46.7	46.6	+0.1	0.96	–	–
	QAT-h2 vs. QAT-noRec-h2	42.0	38.2	+3.8 [1.7, 6.0]	$<10^{-3}$	+3.2 [1.0, 5.6]	0.009

Appendix

Table 15: Quantizing the head and embedding costs no detectable accuracy. Further paired contrasts (pointwise tests, not Holm-corrected).

QAT vs.	7-task $\Delta$ [90% CI]	equiv.	MMLU $\Delta$ [90% CI]	equiv.
LW-QAT : layer-wise start instead of GSQ	+0.91 [0.37, 1.44]	yes	+1.22 [0.61, 1.84]	no
without the scale recalibration	+0.18 [-0.29, 0.63]	yes	+1.03 [0.47, 1.59]	no
QAT-only (neither)	-0.15 [-0.64, 0.34]	yes	-0.02 [-0.69, 0.65]	yes
uniform 2-bit, GPTQ, recalibration	+0.20 [-0.35, 0.73]	yes	+0.72 [0.08, 1.39]	yes
tuned GPTQ start	+0.79 [0.32, 1.28]	yes	+2.07 [1.47, 2.68]	no
GPTQ + Hessian-weighted rounding	+0.80 [0.31, 1.32]	yes	+1.62 [0.98, 2.29]	no

Appendix

Table 16: Every stage removal is equivalent to the full framework on the 7-task mean (margin

\pm

1.5); skipping QAT is not.

\Delta

: QAT-hbf16 minus the named checkpoint.

Model	Contrast (A vs. B)	A	B	$\Delta$ [95% CI]	$p$	status
Qwen3-1.7B	QAT-h2 vs. Rec-h2 (0.52 GB)	42.0	46.7	$-$ 4.7 [ $-$ 7.1, $-$ 2.2]	$4\cdot 10^{-4}$	post hoc
	Rec-h2 vs. GGUF Q2_K [ 62 ] (0.78 GB)	46.7	41.9	+4.8 [2.0, 7.7]	0.001	post hoc
	Rec-h2 vs. Rec-hbf16 (0.52 vs. 0.98 GB)	46.7	47.5	$-$ 0.8 [ $-$ 2.7, 1.0]	0.44	post hoc
Qwen3-14B	Rec-hbf16 (6.4 GB) vs. QAT-h2 (3.74 GB)	92.8	90.5	+2.3 [0.9, 3.7]	0.002	post hoc
	QAT-h2 vs. Rec-h2 (3.74 GB)	90.5	92.7	$-$ 2.2 [ $-$ 3.8, $-$ 0.6]	0.011	post hoc
	codeword search at the recalibrated checkpoint ( Rec-hbf16 vs. LW-Rec-hbf16 ; exploratory)	92.8	89.7	+3.2 [1.8, 4.7]	$10^{-4}$	post hoc

Appendix

Table 17: At 0.52 GB, Rec-h2 beats QAT-h2 and the larger Q2_K; at 14B every two-bit checkpoint beats the smallest GGUF. Paired MATH-500 contrasts (AIME@16 where noted). Bold: significant after Holm correction together with table 9 . § Separate evaluation run. Status: pre-reg., pre-registered; post hoc, chosen after earlier results; check, robustness check.

Model	A vs. B	A	B	$p$	kept
Qwen3-1.7B	QAT-hbf16 vs. Rec-hbf16	22.5	28.8	$<10^{-3}$	Rec-hbf16
Qwen3-1.7B	QAT-h2 vs. Rec-h2	21.9	27.6	$<10^{-3}$	Rec-h2
Qwen3-1.7B	Rec-hbf16 vs. Rec-h2	28.8	27.6	0.37	Rec-h2
Qwen3-14B	QAT-h2 vs. Rec-hbf16	83.3	87.9	$<10^{-3}$	Rec-hbf16
Qwen3-14B	QAT-h2 vs. Rec-h2	83.3	87.5	$<10^{-3}$	Rec-h2
Qwen3-14B	Rec-hbf16 vs. Rec-h2	87.9	87.5	0.73	Rec-h2

Appendix

Table 18: Under the base mixture the selector kept the recalibrated checkpoint in every decision, and MATH-500 agrees. A, B: strict accuracy on the 313 selection problems.

Qwen3-1.7B, codewords frozen	Held-out KL	WikiText-2
GSQ-hbf16	0.641	46.91
+ RMSNorm gains only	0.418	25.29
+ scales only	0.282	19.49
+ scales and RMSNorm gains ( Rec-hbf16 )	0.282	19.46

Appendix

Table 19: The scales carry recalibration: scales alone reach the same held-out KL as scales plus RMSNorm gains. Qwen3-1.7B, same data and schedule.

	median	same sign		relative weight error
Tier	$\|r\|$	as $s^{\star}$ (%)	corr.	GSQ $s_{0}$	recalibrated $s$	least-squares $s^{\star}$
ternary	0.064	51.6	0.08	0.425	0.431	0.325
2-bit	0.054	49.8	0.00	0.198	0.207	0.192
3-bit	0.058	50.8	0.05	0.287	0.296	0.109
all	0.062	51.4	0.07	0.386	0.393	0.277

Appendix

Table 20: Recalibration does not move scales toward the weight-space optimum, and weight error rises in every tier. Per-group scale changes on Qwen3-1.7B; corr.: correlation of

r

with

\log(s^{\star}/s_{0})

Qwen3-1.7B	local block error	propagated error	next-token KL
GSQ-hbf16	0.079	0.214	1.300
Rec-hbf16	0.094	0.282	0.571
QAT-hbf16	0.122	0.373	0.371

Appendix

Table 21: Recalibration and QAT make single blocks less faithful but bring the output distribution closer to bf16 . Relative block-output error (local: bf16 input; propagated: fully quantized model) and output KL on WikiText-2.

Figure 6: QAT latent initialization

\alpha

\times

learning rate on Qwen3-1.7B (short runs from the recalibrated model). (a) Held-out KL relative to the start. (b) Share of codewords changed. With

\alpha=0

fewer than 0.003% of codewords are reassigned. Validation sequences are the chunks that follow each run’s training slice, so runs with different training lengths are scored on different held-out text; this is why the starting KL (0.251) differs from long QAT’s (0.311). KL values are comparable within a run, not across runs.

Figure 7: Where long QAT reassigns codewords on Qwen3-1.7B. (a) Share of each projection type’s codewords that QAT changes; value projections (orange) change most, query and key projections (dark blue) least. (b) Share changed in each transformer block, with the share per tier in the inset text. Codewords at the first and last transformer blocks are reassigned more than other blocks (c) Distribution, before QAT, of the distance

|W/s-c|

between each codeword

c

and its bf16 weight

W/s

, in grid steps with the recalibrated scales, for all codewords (grey) and for those that QAT reassigns (orange). The dotted line at 0.5 is the rounding boundary: most reassigned codewords lie beyond it, and 99.8% of reassignments move one step toward the bf16 weight.

0.52 GB checkpoints (2-bit head and embedding) and Q2_K
Thinking budget (tokens)	1k	2k	4k	8k	16k	32k, unforced
bf16	74.2	77.0	82.4	86.0	90.0	90.2
Rec-hbf16	40.4	43.2	44.4	44.6	45.2	40.6
QAT-hbf16	42.2	43.6	44.4	43.4	39.2	39.0
QAT-only-hbf16	33.6	34.6	36.6	31.0	28.8	26.6
Rec-h2 , 0.52 GB	40.6	42.0	43.4	42.6	41.0	39.0

Appendix

Table 22: With a forced answer the 0.52 GB checkpoints and Q2_K tie, so the strict lead of Rec-h2 comes from finishing more traces. Greedy thinking-mode MATH-500 on Qwen3-1.7B with the answer forced after the given number of thinking tokens.

Model	Contrast	$\Delta$ closing rate	$\Delta$ accuracy on closed	problems
Qwen3-1.7B	codeword search	+1.4 [ $-$ 1.0, 3.8]	+10.4 [7.1, 13.6]	414
	scale recalibration	+12.3 [9.8, 14.9]	+2.1 [ $-$ 1.4, 5.8]	379
	QAT vs. Rec	$-$ 18.4 [ $-$ 21.1, $-$ 15.6]	$-$ 0.5 [ $-$ 3.5, 2.5]	440

Appendix

Table 23: Codeword search mainly raises accuracy on closed traces; recalibration and QAT mainly change how many traces close. Problem-matched sampled MATH-500 differences (95% CI); bold:

p{<}0.01

		quantized	bits per quantized weight
Model	Checkpoint	weights	measured	computed	file (GB)
Qwen3-1.7B	QAT-hbf16	1.41B	2.011	2.000	0.977
	QAT-h2	2.03B	2.046	2.038	0.520
	Rec-h2	2.03B	2.046	2.038	0.520
Qwen3-14B	QAT-only-h2	14.77B	2.023	2.013	3.736
	Rec-h2	14.77B	2.023	2.013	3.736

Appendix

Table 24: Packed files decode bit-exactly and are within 0.6% of the sizes computed from the grids. Bits per quantized weight include the group scales.

	weight memory	decode (tokens/s)		prefill, 1,024	top-1 agreement
Qwen3	(GiB)	batch 1	batch 16	tokens (ms)	(%)
1.7B	3.22 $\to$ 1.85	424 $\to$ 408	6,232 $\to$ 5,998	12.8 $\to$ 15.7	98.7
8B	15.27 $\to$ 5.68	146 $\to$ 212	2,225 $\to$ 3,184	27.0 $\to$ 28.4	99.1
14B	27.52 $\to$ 9.31	86 $\to$ 150	1,341 $\to$ 2,302	48.0 $\to$ 45.0	98.3

Appendix

Table 25: Serving from packed containers cuts weight memory 1.7–3.0

\times

and speeds up batch-1 decoding at 8B and 14B. vLLM on one H100; memory in GiB.

Figure 8: Qwen3-1.7B: retention on 14 benchmarks after recalibration and after QAT ( Rec-hbf16 , QAT-hbf16 ) (RACE omitted; its test passages appear in MMLU’s auxiliary training data). Numbers above bars are QAT retention.

Figure 9: Qwen3-1.7B: score against whole-model bytes (generation metrics greedy). Grey: published llama.cpp quantizations (Unsloth [ 75 ] ; the UD- prefix is omitted in labels). Red star: ours with a 2-bit head and embedding ( QAT-h2 ); orange diamond: ours with the bf16 tied embedding ( QAT-hbf16 ). Dotted: bf16 . Over 30 individually scored sub-scores (RACE excluded; greedy MATH counted; differences of at most 0.5 points count as ties) the 0.52 GB QAT checkpoint ( QAT-h2 ) wins 23–26 against each 2-bit GGUF.

$\leq$ 2.6 bits per parameter
Model	Checkpoint	GB	strict	lenient	closed (%)	strict if closed
Qwen3-1.7B	QAT+R-h2 (selected)	0.52	73.0	74.6	87.1	83.8
Qwen3-1.7B	Rec-h2	0.52	46.7	47.3	86.0	54.2
Qwen3-1.7B	QAT-h2	0.52	42.0	44.9	70.0	60.0
2.6–4 bits per parameter
Qwen3-1.7B	GGUF UD-IQ2_XXS [ 66 ]	0.61	1.0	2.8	46.8	2.1

Appendix

Table 26: QAT+R-h2 closes more traces than Q2_K and is more accurate on them; the earlier 0.52 GB checkpoints win only by closing more traces. Sampled MATH-500 split into traces that close within 32k tokens and strict accuracy on them. Rows grouped by bits per parameter.

Smaller models at 4–5 bits per parameter (reference)
Checkpoint	Size (GB) $\downarrow$	7-task $\uparrow$	MMLU $\uparrow$	GSM8K $\uparrow$	HE $\uparrow$	MT $\uparrow$	M500 $\uparrow$	A24/25 $\uparrow$	A@16 $\uparrow$
Qwen3-14B bf16	29.5	74.3	77.1	88.2	57.3	40.8	97.5	76.7/60.0	73.6
Qwen3-4B GGUF Q4_K_M [ 70 ]	2.50	66.7	68.3	84.8	70.1	30.0	96.3	73.3/50.0	65.3
Qwen3-8B GGUF Q3_K_M [ 72 ]	4.12	69.2	71.0	86.5	61.0	31.6	95.8	56.7/50.0	64.1
Qwen3-14B, $\leq$ 2.6 bits per parameter
Ours: Rec-h2	3.74	69.9	69.5	87.7	65.2	32.0	92.7	50.0/33.3	46.5

Appendix

Table 27: At 3.74 GB our checkpoints beat UD-IQ2_XXS (4.47 GB) on reasoning; Q2_K and larger files remain stronger. Qwen3-14B by whole-model bytes. M500: sampled MATH-500; A24/25: greedy AIME; A@16: mean pass@1 over 16 samples; HE: HumanEval; MT: BFCL multi-turn. Rows grouped by bits per parameter. † Not packed; ∗ computed from the grids; § separate evaluation run.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 381 papers

E8-QAT: Efficient 2-Bit Quantization-Aware Training with Implicit Lattice Codebooks
Oct 20, 2026Xinyu Lei, Hong Huang, Jiebin Zhang +6LLM Quantization Quantization-Aware Training
Peking University · City University of Hong Kong · Tencent
Extreme Low-Bit Inference in Reasoning Models: Failure Modes and Targeted Recovery
Jun 1, 2026Ekaterina Alimaskina, Darya Rudas, Denis Shveykin +3LLM Quantization Repetition in Language Generation
BRAIn Lab, Moscow, Russia
Attend to Your Own Thoughts: Breaking the Barrier for Post-Training Quantization of Reasoning LLMs through the Lens of 1.58-Bit Quantization
Aug 2, 2026Shigeng Wang, Chao Li, Yangyuxuan Kang +2LLM Quantization Ternary Quantization
Intel Labs China

Oct 20, 2026J/K move · Enter open · S save

E8-QAT: Efficient 2-Bit Quantization-Aware Training with Implicit Lattice Codebooks

Xinyu Lei, Hong Huang, Jiebin Zhang, Hongyu Li+5

Peking University · City University of Hong Kong · Tencent

LLM Quantization Quantization-Aware Training Low-Bit Quantization Vector Quantization

Open paper AskPDF arXiv

Low-bit quantization reduces the memory footprint of large language models, but preserving accuracy with efficient training remains challenging. Scalar quantization encodes weights independently, while explicit-codebook vector quantization introduces costly search during training. We propose E8-QAT, a 2-bit quantization-aware training framework that uses the

E_8

lattice as an implicit eight-dimensional codebook. It jointly quantizes weight groups through arithmetic projection without storing or searching an explicit codebook during the training forward pass. A 2-bit core encoding and sparse extension preserve the selected lattice points exactly, adding less than 0.1% to core index storage. On Qwen3.6-35B-A3B, E8-QAT matches the performance of the full-precision model in terms of the average score across the four tasks. E8-QAT improves over the state-of-the-art baseline by 2.03% on IFEval and by 1.51% on LiveCodeBench, under the same data, schedule, and bit width. The 2-bit representation reduces peak memory by 80% and supports a

4\times

larger batch. At each representation's largest feasible batch, E8-QAT reaches

1.97\times

the decoding throughput of BF16.

Abstract

LLM Quantization

Explore similar work

E8-QAT: Efficient 2-Bit Quantization-Aware Training with Implicit Lattice Codebooks

Extreme Low-Bit Inference in Reasoning Models: Failure Modes and Targeted Recovery

Attend to Your Own Thoughts: Breaking the Barrier for Post-Training Quantization of Reasoning LLMs through the Lens of 1.58-Bit Quantization