Boosting Entropy with Bell Box Quantization

Ningfeng Yang; Tor M. Aamodt

cs.LGMar 2, 2026

Boosting Entropy with Bell Box Quantization

Authors: Ningfeng Yang, Tor M. Aamodt

Organizations: University of British Columbia

Abstract

Quantization-Aware Pre-Training (QAPT) is an effective technique to reduce the compute and memory overhead of Deep Neural Networks while improving their energy efficiency on edge devices. Existing QAPT methods produce models stored in compute-efficient data types (e.g. integers) that are not information theoretically optimal (ITO). On the other hand, existing ITO data types (e.g. Quantile/NormalFloat Quantization) are not compute-efficient. We propose BBQ, the first ITO quantization method that is also compute-efficient. BBQ builds on our key insight that since learning is domain-agnostic, the output of a quantizer does not need to reside in the same domain as its input. BBQ performs ITO quantization in its input domain, and returns its output in a compute-efficient domain where ITO data types are mapped to compute-efficient data types. Without sacrificing compute efficiency, BBQ outperforms prior SOTA QAPT methods by a perplexity reduction of up to 2 points for 4-bit models, up to 4 points for 3-bit models, up to 5 points for 2-bit models, and up to 18 points for 1-bit models. Code is available at https://github.com/1733116199/bbq.

Figures & tables

Signed INT4	-8, $\pm 7$ , $\pm 6$ , $\pm 5$ , $\pm 4$ , $\pm 3$ , $\pm 2$ , $\pm 1$ , 0
MX FP4	$\pm 6$ , $\pm 4$ , $\pm 3$ , $\pm 2$ , $\pm 1.5$ , $\pm 1$ , $\pm 0.5$ , $\pm 0$

Table 1: Possible values of the INT4 and the MX FP4 data type.

Panel of Figure 1 — Figure 1: The three steps of the BBQ quantization formula (Equation 5 ) for $b=4$ . Step 1(a) is the Hadamard Transform (HT) followed by RMS Normalization. Step 1(b) is the probability integral transform (PIT). Step 1(c) is uniform quantization. We name our method Bell Box Quantization because Figure 1(a) looks like a bell and Figure 1(b) looks like a box (rectangle) which is quantized in Figure 1(c) .

Panel of Figure 2 — Figure 2: Comparison of clip (blue) and the standard Gaussian CDF $\Phi$ (orange).

Precision $b$	Zero Point $z$	Possible Values of $q$	INT4	MX FP4
4	0	-8, -7, -6, -5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5, 6, 7	✓	✗
3	0	-4, -3, -2, -1, 0, 1, 2, 3	✓	✓
2	-0.5	-1.5, -0.5, 0.5, 1.5	✗	✓
1	-0.5	-0.5, 0.5	✗	✓

Table 2: The BBQ data type: zero point

z

, possible values of

q

, and whether BBQ can be encoded as INT4 and MX FP4 for each precision

b\in\{1,2,3,4\}

Params	Tokens	Full	Bits	BBQ		QuEST		LSQ
Params	Tokens	Full	Bits	Entropy	Perplexity	Entropy	Perplexity	Entropy	Perplexity
95M	3B	24.75	4	3.93	25.51	3.61	26.37	3.59	27.46
95M	3B	24.75	3	2.96	26.55	2.78	29.04	2.74	30.27
95M	3B	24.75	2	1.97	31.34	1.92	35.58	1.69	36.58
95M	3B	24.75	1	1.00	49.22	1.00	67.78	-	-
125M	5B	21.51	4	3.93	22.15	3.61	22.98	3.60	23.77

Table 3: Entropy and Perplexity of LLaMA models pre-trained on the C4 dataset. The headers are the number of parameters (Params), the number of tokens (Tokens), perplexity without any quantization (Full), activation/weight precision (Bits), and the entropy and perplexity of BBQ, QuEST, and LSQ.

Params	95M	95M	95M	95M	95M	95M	95M	95M	95M	95M	95M	95M
Bits WA	16	4	4	4	4	3	3	3	3	3	1	1
Method	-	BBQ	QuEST	LSQ	NF4	BBQ	QuEST	LSQ	NF3	N2UQ	BBQ	QuEST
Perplex	25	26.7	26.83	27.55	28.5	28.61	29.7	30.85	36.16	29.87	53.8	77.96
Params	95M	95M	95M	95M	95M	95M	125M	125M	125M	125M	125M	125M
Bits WA	2	2	2	2	2	2	3	3	2	2	1	1
Method	BBQ	QuEST	LSQ	NF2	SEQ	N2UQ	BBQ	QuEST	BBQ	QuEST	BBQ	QuEST

Table 4: Perplexity of GPT models pre-trained on the C4 dataset.

Figure 3: Quantized weight entropy vs. training iterations for LLaMA-300M with 2-bit weight and activations, pre-trained on 20 billion C4 tokens (batched into 80 thousand training iterations).

Panel of Figure 4 — Figure 4: Kernel latency on Nvidia RTX 5090 (y-axis) vs. matrix size $N$ (x-axis).

Figure 5: End-to-end LLaMA inference latency of BBQ, FP16 and NF4 on RTX 5090 and A100 GPUs. For each linear layer, BBQ launches an activation quantization kernel (green region), an fp4/int4 matrix multiplication kernel (part of blue regions), and an element-wise scaling kernel (part of orange region).

Row	HT	RMS	PIT	Learn $\gamma$	$\gamma$ Init	Perplexity	Entropy	Notes
1	✓	✓	✓	✓	✓	31.34	1.97	BBQ
2	✗	✓	✓	✓	✓	35.79	1.98
3	✓	✗	✓	✓	✓	35.93	1.98
4	✓	✓	✗	✗	✗	35.58	1.92	QuEST
5	✓	✓	✓	✗	✗	138.3	1.92
6	✓	✓	✓	✗	✓	31.46	1.98

Table 5: Ablation of BBQ features: Hadamard Transform (HT), RMS Normalization (RMS), Probability Integral Transform (PIT), making

\gamma

a learnable parameter (Learn

\gamma

), and

\gamma

initialization (

\gamma

Init). A ✓ means a feature is present, and ✗ means the feature is not included.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Binary Representation	Signed INT4	MX FP4
0b0000	0	0
0b0001	1	0.5
0b0010	2	1
0b0011	3	1.5
0b0100	4	2
0b0101	5	3

Appendix

Table 6: Signed INT4 vs. MX FP4

Precision	Possible Values of $q$
4	-8,-7,-6,-5,-4,-3,-2,-1,0,1,2,3,4,5,6,7
3	-4,-3,-2,-1,0,1,2,3
2	-1.5,-0.5,0.5,1.5
1	-0.5,0.5

Appendix

Table 7: Values that BBQ can represent. As shown, for

b=4

, BBQ can be encoded as INT4. For

b=3

, BBQ can be encoded as INT4 or FP4. For

b\in\{1,2\}

, BBQ can be encoded as FP4.

	4-bit	3-bit	2-bit	1-bit
BBQ	25.51	26.55	31.35	49.80
BBQ-Fast	25.40	26.55	31.43	49.72

Appendix

Table 8: Perplexity of BBQ vs BBQ-Fast, on LLaMA-95M pre-trained with 3 billion C4 tokens.

Model	Dataset	Params n+e	Method	Bits WA	Perplexity
LLaMA	wikitext	95M	None	16	56.11
LLaMA	wikitext	95M	BBQ	4	57.87
LLaMA	wikitext	95M	QuEST	4	58.91
LLaMA	wikitext	95M	LSQ	4	61.62
LLaMA	wikitext	95M	BBQ	3	60.18
LLaMA	wikitext	95M	QuEST	3	64.21

Appendix

Table 9: Zero-shot wikitext perplexity.

Panel of Figure 7 — Figure 7: Histogram of the absolute value of learned $\gamma$ in a language model versus a vision model.

Model	Params	Bits WA	BBQ-Vision			QuEST			LSQ
Model	Params	Bits WA	Ent	TL	VA	Ent	TL	VA	Ent	TL	VA
DEIT-T	5M	4	3.85	2.84	74.68	3.61	2.91	74.3	3.67	2.88	74.88
DEIT-T	5M	3	2.93	2.89	74.24	2.78	2.98	72.2	2.81	3.01	71.36
DEIT-T	5M	2	1.94	3.05	70.16	1.93	3.18	67.24	1.90	3.23	65.32
DEIT-T	5M	1	1.00	3.52	53.54	1.00	3.67	47.55	-	-	-
DEIT-S	20M	4	3.82	2.53	79.46	3.61	2.51	80.00	3.57	2.52	80.04

Appendix

Table 10: Entropy (Ent), Training Cross Entropy Loss (TL), and Validation top-1 Accuracy (VA) of vision models pretrained on Imagenet-100 ( Chun-Hsiao Yeh, 2022 ; Russakovsky et al., 2015 ) .

Figure 8: LLaMA-95M (4-bit) pre-trained on 3 billion C4 tokens (batched over 12 thousand iterations). LSQ is green, QuEST is gray, and BBQ is blue.

Figure 9: LLaMA-95M (3-bit) pre-trained on 3 billion C4 tokens (batched over 12 thousand iterations). LSQ is pink, QuEST is red, and BBQ is green.

Figure 10: LLaMA-95M (2-bit) pre-trained on 3 billion C4 tokens (batched over 12 thousand iterations). LSQ is yellow, QuEST is purple, and BBQ is orange.

Figure 11: LLaMA-95M (1-bit) pre-trained on 3 billion C4 tokens (batched over 12 thousand iterations). QuEST is purple and BBQ is orange.

Figure 12: LLaMA-125M (4-bit) pre-trained on 5 billion C4 tokens (batched over 20 thousand iterations). LSQ is red, QuEST is brown, and BBQ is pink.

Figure 13: LLaMA-125M (3-bit) pre-trained on 5 billion C4 tokens (batched over 20 thousand iterations). LSQ is green, QuEST is gray, and BBQ is blue.

Figure 14: LLaMA-125M (2-bit) pre-trained on 5 billion C4 tokens (batched over 20 thousand iterations). LSQ is purple, QuEST is orange, and BBQ is green.

Figure 15: LLaMA-125M (1-bit) pre-trained on 5 billion C4 tokens (batched over 20 thousand iterations). QuEST is pink and BBQ is orange.

Figure 16: LLaMA-200M (4-bit) pre-trained on 10 billion C4 tokens (batched over 40 thousand iterations). QuEST is green and BBQ is brown.

Figure 17: LLaMA-200M (3-bit) pre-trained on 10 billion C4 tokens (batched over 40 thousand iterations). QuEST is pink and BBQ is gray.

Figure 18: LLaMA-200M (2-bit) pre-trained on 10 billion C4 tokens (batched over 40 thousand iterations). QuEST is orange and BBQ is pink.

Figure 19: LLaMA-200M (1-bit) pre-trained on 10 billion C4 tokens (batched over 40 thousand iterations). QuEST is green and BBQ is brown.

Figure 20: LLaMA-300M (4-bit) pre-trained on 20 billion C4 tokens (batched over 80 thousand iterations). QuEST is green and BBQ is brown.

Figure 21: LLaMA-300M (3-bit) pre-trained on 20 billion C4 tokens (batched over 80 thousand iterations). QuEST is orange and BBQ is green.

Figure 22: LLaMA-300M (2-bit) pre-trained on 20 billion C4 tokens (batched over 80 thousand iterations). QuEST is green and BBQ is red.

Quantization

See all 49 papers

Quantization-Aware Pre-Training with Constrained Empirical Weight Distribution
Sep 26, 2026Ningfeng Yang, Tor M. AamodtQuantization-Aware Training LLM Quantization
University of British Columbia
Efficient Tuning Before Low-Bit Post-Training Quantization for Stochastic Gradient Descent-optimized Models
Jul 13, 2026Peng Xia, Junbiao Pang, Muhammad Ayub SabirActivation Quantization Post-Training Quantization
School of Information Science and Technology, Beijing University of Technology, Beijing 100124, China
ApexQuant: Data-Free Elastic Quantization by Residual Re-Isotropization
Oct 6, 2026Aksel Fristrup, Sumit Pandey, Ankit KariryaaLLM Quantization Rotation-Based Quantization
Henselian (henselian.com) · Department of Computer Science University of Copenhagen Denmark · Towards Deep Learning

Sep 26, 2026J/K move · Enter open · S save

Quantization-Aware Pre-Training with Constrained Empirical Weight Distribution

Ningfeng Yang, Tor M. Aamodt

University of British Columbia

Quantization-Aware Training LLM Quantization Low-Bit Quantization

Open paper AskPDF arXiv

Quantization-Aware Pre-Training (QAPT) can increase the inference efficiency of DNNs, but a problematic behaviour known as rounding boundary weight oscillation can introduce detrimental noise into the training process and significantly reduce convergence speed. While existing methods can reduce this detrimental noise, they either introduce additional hyperparameters or memory overhead, or cannot consistently improve model accuracy. In this work, we propose optimization with

\textbf{C}

onstrained

\textbf{E}

mpirical

\textbf{W}

eight dis

\textbf{T}

ribution (CEWT), the first hyperparameter-free memory-overhead-free oscillation suppression method that consistently improves QAPT performance: an optimizer post-update step that projects weights to the nearest point in weight space whose empirical distribution (histogram) matches a zero-mean Gaussian. Our key insight is many quantizers are designed with the implicit assumption that the to-be-quantized data are permutations of samples from a zero-mean Gaussian, and this assumption is not true during QAPT. By enforcing the zero-mean Gaussian prior as a hard constraint, CEWT can suppress this detrimental noise. Empirical results on various combinations of SOTA quantizers and hypersphere optimizers suggest, that with a geomean increase of 4% in training time, CEWT can consistently reduce the pre-training perplexity (by an average of 2.5 and up to 21 points) of low-precision (down to 1-bit activations and weights and up to 610M parameters) LLaMA/GPT models without introducing any hyperparameters or storage overhead. Code is available at https://github.com/1733116199/cewt

Abstract

Quantization

Explore similar work

Quantization-Aware Pre-Training with Constrained Empirical Weight Distribution

Efficient Tuning Before Low-Bit Post-Training Quantization for Stochastic Gradient Descent-optimized Models

ApexQuant: Data-Free Elastic Quantization by Residual Re-Isotropization