Tetra: Serving Leech-Lattice Quantized LLMs at 2.7 Bits per Parameter

Pier-Jean Malandrino

cs.LGSep 28, 2026

Tetra: Serving Leech-Lattice Quantized LLMs at 2.7 Bits per Parameter

Authors: Pier-Jean Malandrino

Organizations: Scub, Bordeaux, France

Abstract

Leech-lattice quantization gives good quality at two bits per weight, but its codebooks hold more than 10^14 points, too many for a lookup table. Our earlier kernel expanded the codes at load time and read 4.804 bits per weight from GPU memory for 2 bits of code. We present Tetra, a new codebook on the same lattice. A 24-weight block still takes 48 bits, most of which index a 64-state trellis of the Golay code and one shared 16 KiB table. The kernel decodes a block with six table loads and two small lookups inside the matrix-vector product, and reads 2.148 bits per weight. For full models, we retrain one scale per matrix row, store the matrices that lose the most as 4-bit integers, and pay for them with 4-bit embedding tables. Our Qwen3-4B, 8B and 14B files hold 2.73, 2.70 and 2.73 bits per parameter over the whole model. They score 63.37, 69.58 and 75.66 on the full MMLU test set, 4.76, 4.21 and 2.46 points below 4-bit AWQ at 5.3 to 6.0 bits per parameter. They generate 113.8, 95.0 and 57.2 tokens per second in our engine. On GSM8K, through the served kernel, they lose 9.63, 4.62 and 3.26 points to FP16. At 4B our file scores 23.6 points above llama.cpp's IQ2_XXS (2.48 bits per parameter). Every number we measured for a table or figure comes from one NVIDIA L40S GPU. We preregistered the main experiments.

Figures & tables

	Qwen3-4B	Qwen3-8B	Qwen3-14B
projections, Tetra $+$ int4	168 $+$ 84	199 $+$ 53	181 $+$ 99
int4 projections	$v$ , $o$ , down 12–23	$v$ , down 10–26	$v$ , $o$ , down 10–28
4-bit embedding tables	1 (tied head)	2	2
file size (bytes)	1,418,224,685	2,815,098,745	5,087,000,541
bits per parameter, whole model	2.7320	2.6953	2.7305

Table 1: The three sealed files. Layers are numbered from 0: “down 12–23” means down_proj in layers 12 to 23. Bits per parameter are counted as in § 4.1 . The file is larger: it also carries the tokenizer and the config (11.4 MB), and stores the tail in f32 and the row scales in f64. Data: paper2-chain.csv, and the sealing logs for file sizes.

	FP16	AWQ w4g128	Planes14	Tetra	no weights
matrices timed	252	252	252	216	252
median ms	10.973	3.261	4.997	3.424	2.289
GB read per pass	7.27	1.90	2.18	0.95	0.07
b/weight in VRAM	16.000	4.179	4.804	2.148	0.159
GB/s, fastest round	662	583	437	278	32

Table 2: Kernel benchmark on the 4B shapes at tile

T=64

: one process, one L40S, seven rounds with the first two dropped. “No weights” has the same thread layout and reads no block code, only the tail and the row scales. The Tetra arm times the 216 lattice matrices of an early 4B file. Its 36 int4 v_proj run on the other kernel. The lattice arms hold the tail in f32, as the files store it, where the served kernel uses f16. The Tetra row leaves out 4.1 MB of row padding, 0.009 b/weight. The AWQ arm is its authors’ GEMV kernel [ 17 ] ported to our harness, not the Marlin kernel that vLLM serves. Its row includes 10.1 MB of padding in that kernel’s scale and zero buffers, 0.022 b/weight. Without it, w4g128 costs 4.156 b/weight. Ten arms: Appendix A . Data: echelle-formats.csv.

	format and engine	b/param	MMLU	decode tok/s	weights GB
4B	FP16, vLLM	16.00	70.14	83.1	8.04
	f16 dense path, our engine a	16.00		43.0	8.04
	AWQ w4g128, vLLM	5.30	68.14	200.5	2.67
	IQ2_XXS, llama.cpp	2.48	39.78	312.9	1.25
	Tetra , our engine	2.73	63.37	113.8	1.38
	LLVQ, cited b	2 b/w c	62.8

Table 3: Quality, speed and memory at three sizes, on one L40S. MMLU uses all 14,042 test questions, and FP16 and AWQ are scored in our harness. The engine named in each row times the speed, and speeds are never divided across engines. For our files, weights GB are the buffers on the card, row padding included. a The sealed file decoded to f16 and run as an ordinary model, in the same process as the kernel. Its speed is that of any f16 model of this size in our engine, which copies the output head at every token. This row has no MMLU of its own. b Fine-tuned rows of Table 6 of van der Ouderaa et al. [24] , from their harness and MMLU protocol. c Bits per quantized weight, as the source gives them. It gives no rate for the whole model. Data: paper2-results.csv.

	FP16	AWQ	Tetra	FP16 minus Tetra	AWQ minus Tetra
4B	92.12	89.01	82.49	9.63 $[7.69,11.57]$	6.52 $[4.39,8.65]$
8B	93.25	92.95	88.63	4.62 $[3.00,6.25]$	4.32 $[2.83,5.81]$
14B	95.30	95.38	92.04	3.26 $[1.93,4.59]$	3.34 $[2.12,4.55]$

Table 4: GSM8K on all 1,319 test problems (§ 6.1 ). Tetra runs through the served kernel, FP16 and AWQ in vLLM. The gaps are paired, with 95 % intervals. Data: paper2-gsm8k.csv, paper2-gsm8k-gaps.csv.

Tetra	against	weights GB	MMLU	GSM8K
8B	AWQ 4B	2.76 vs 2.67	$+1.44$ $[+0.71,+2.19]$	$-0.38$ $[-2.14,+1.39]$
14B	AWQ 8B	5.04 vs 6.10	$+1.87$ $[+1.20,+2.57]$	$-0.91$ $[-2.35,+0.53]$
8B	FP16 4B	2.76 vs 8.04	$-0.56$ $[-1.27,+0.18]$	$-3.49$ $[-5.18,-1.79]$
14B	FP16 8B	5.04 vs 16.38	$+0.61$ $[-0.06,+1.27]$	$-1.21$ $[-2.64,+0.21]$
14B	FP16 4B	5.04 vs 8.04	$+5.52$ $[+4.79,+6.24]$	$-0.08$ $[-1.54,+1.39]$
AWQ 8B against AWQ 4B		6.10 vs 2.67	$+5.65$ $[+4.96,+6.37]$	$+3.94$ $[+2.43,+5.46]$

Table 5: Our 8B and 14B files against the models one size below, and our 14B against FP16’s 4B: the first model minus the second, in points, paired on 14,042 MMLU questions and 1,319 GSM8K problems, with 95 % intervals. The GSM8K pairs cross the two engines of § 6.3 . The last two rows show what one size is worth to AWQ. Weights as in Table 3 . Data: paper2-sizeup.csv.

	step	b/param	MMLU	gain	95 % interval, McNemar
4B	Tetra $+$ int4 v_proj	2.7475	57.95
	$+$ trained row scales	2.7475	61.11	$+3.15$	$[+2.56,+3.74]$ , $p=1.8\cdot 10^{-25}$
	$+$ int4 $o$ , down; 4-bit tables	2.7320	63.37	$+2.26$	$[+1.66,+2.87]$ , $p=8.2\cdot 10^{-13}$
8B	Tetra $+$ int4 v_proj	3.0683	64.87
	$+$ trained row scales	3.0683	68.16	$+3.29$	$[+2.78,+3.80]$ , $p=7.2\cdot 10^{-38}$
	$+$ int4 down; 4-bit tables	2.6953	69.58	$+1.42$	$[+0.93,+1.92]$ , $p=3.2\cdot 10^{-8}$

Table 6: How each file is built, one step per row, each gain paired against the row above on the 14,042 questions. Bits per parameter count the file as served, with 8-bit embedding tables in the first two rows of each size. Those rows were scored with f16 tables, so the last gain also includes the cost of 4-bit tables. Data: paper2-chain.csv.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

arm	matrices	median ms	GB read	b/weight	GB/s
FP16, our control	252	10.973	7.27	16.000	662
FP16, cuBLAS	252	10.828	7.27	16.000	672
Slot32 (ours, earlier)	252	5.732	2.50	5.510	437
Planes14 (ours, earlier)	252	4.997	2.18	4.804	437
Planes12x (ours, earlier)	252	5.366	1.97	4.342	368
Golay70 (ours, earlier)	252	8.064	1.63	3.589	202

Appendix

Table 7: Each row is one kernel timed on the 4B matrix shapes on one L40S. The first ten rows are the five arms of Table 2 and five more, timed in the same process at tile 64 with the same protocol. The rows marked “earlier” are the layouts of Malandrino [19] , timed again here. ∗ The QTIP measurement of Malandrino [19] , from another process (2026-08-21): the HYB kernel of QTIP’s repository (

L=16

Q=9

V=2

) on pseudo-random codes, in its own launch geometry. It reads no row scales and no tail, and its 2 KiB table is not counted. It runs below the no-weights control. Data: echelle-formats.csv.

quantity	predicted	measured	inside?
Tetra time at tile 64, tile sweep	3.77 ms $[3.5,4.1]$	3.423 ms	no, 0.08 ms under
gain from tile 128 to 64	$+8.3\,\%$ $[0,+15]$	$+16.1\,\%$	no, 1.1 over
row-scale training, 4B	$+3.5$ $[+1.5,+4.5]$	$+3.15$	yes
row-scale training, 8B	$+2.0$ $[+0.5,+3.5]$	$+3.29$	yes
row-scale training, 14B	$+3.2$ $[+1.7,+4.7]$	$+1.67$	no, 0.03 under
sealed composition, 4B	$+1.2$ $[+0.2,+2.2]$	$+2.26$	no, 0.06 over

Appendix

Table 8: Preregistered predictions against the measurements. Sources: the preregistrations in proofs/ and the logs in docs/mesures/ .

Explore similar work

AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization
May 9, 2026Beshr IslamBouli, David JinLarge Language Model Quantization Model Weights
Massachusetts Institute of Technology, Cambridge, MA, USA.
QTEA: Ternary LLMs with Sparse Residual Salient Weight and By-Column Optimization
Aug 31, 2026Yipin Guo, Arun M George, Jie Fu +3Post-Training Quantization
University of Notre Dame
CubicQuant: Parametric Non-Uniform Codebooks for High-Throughput LLM Inference with 1-8-Bit Weights
Aug 7, 2026Xuetian GaoLarge Language Model Quantization Codebooks

May 9, 2026, cs.LGJ/K move · Enter open · S save

AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization

Beshr IslamBouli, David Jin

Massachusetts Institute of Technology, Cambridge, MA, USA.

Large Language Model Quantization Model Weights Codebooks LLM Inference Optimization

Open paper AskPDF arXiv

Post-training weight-only quantization to 4 bits is widely used to reduce the memory and compute costs of large language model inference. Existing PTQ methods, such as AWQ and GPTQ, improve how weights are mapped onto a fixed 4-bit grid through scaling, clipping, or error compensation. To further improve accuracy, methods such as OmniQuant and QuIP# uses gradient-assisted algorithms at the cost of hours of quantization time. In this work, we propose AAAC (Activation-Aware Adaptive Codebooks), a lightweight method for 4-bit LLM weight quantization. AAAC replaces the fixed scalar codebook used in standard quantization with two small learned scalar codebooks (64 bytes) per layer. Each group of weights selects the codebook that minimizes activation-weighted reconstruction error, encoding the choice in the unused sign bit of the group's positive scale and adding zero storage overhead. AAAC completes in 3--30 minutes on a single GPU, and adds no memory beyond the model itself. We evaluate against AWQ, GPTQ, IF4, GPTVQ, OmniQuant, SqueezeLLM, and QuIP# across model families. AAAC outperforms baselines at orders-of-magnitude less quantization time.

Abstract

Explore similar work

AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization

QTEA: Ternary LLMs with Sparse Residual Salient Weight and By-Column Optimization

CubicQuant: Parametric Non-Uniform Codebooks for High-Throughput LLM Inference with 1-8-Bit Weights