BARQ: Balanced Codebook Refinement for Low-Bit LLM Quantization

Chenhang Cui; Xu Xie; Linrui Xu; Xiaohao Liu; Xingyu Zhu; Fei Shen; Tat-Seng Chua

cs.LGOct 3, 2026

BARQ: Balanced Codebook Refinement for Low-Bit LLM Quantization

Authors: Chenhang Cui, Xu Xie, Linrui Xu, Xiaohao Liu, Xingyu Zhu, Fei Shen, Tat-Seng Chua

Organizations: National University of Singapore · Independent Researcher · Central South University

Abstract

As large language models (LLMs) grow in parameter count, model storage and parameter memory traffic have become major bottlenecks to efficient deployment. Codebook-based weight quantization reduces these costs, but imbalanced nearest-codeword assignments during fitting can leave some codewords insufficiently updated, limiting effective codebook utilization. To address this limitation, we propose Balanced Assignment Refinement for Quantization (BARQ), which improves quantization quality through balanced fitting of existing codebooks. Specifically, we first compute joint soft assignments between weight blocks and codewords through entropically regularized optimal transport with uniform marginals and curvature-weighted reconstruction costs, ensuring equal positive fitting mass for every codeword in the exact solution. We then refine the codewords through an assignment-weighted barycentric update, which we prove minimizes the fitting objective for fixed assignments. For finite Sinkhorn iterations, the implemented update retains this optimality provided all codeword masses exceed the denominator floor. Finally, we discard the soft assignments and use the refined codebook for standard hard nearest-codeword encoding, with our analysis establishing sufficient conditions for reducing hard-quantization distortion and evaluation loss. Across multiple LLMs, BARQ achieves lower perplexity and higher mean zero-shot accuracy than the evaluated baselines at comparable bit budgets. The code is available at https://github.com/chenhangcuisg-code/BARQ.

Figures & tables

Figure 1: Comparison of Codebook Fitting. Prior methods concentrate fitting mass on a few codewords, leaving the rest never updated. In contrast, BARQ fits under uniform marginals, giving every codeword positive mass without changing the inference path.

Figure 2: Overview of the BARQ pipeline (VQ: vector quantization; MLP: multilayer perceptron). Curvature-Aware Cost: compute the distances between weight blocks and initial codewords, and set the groupwise entropic temperature. Balanced-Entropic E-step: use log-domain Sinkhorn scaling to obtain soft assignments with uniform codeword and block marginals. Barycentric M-step: update each codeword to the assignment-weighted barycenter of the blocks. Hard Deployment: discard the transport state and encode the weights with the refined codebook using the original nearest-codeword rule. Only the refined codebook and hard indices are stored.

Panel of Table 1 — Table 1: Paired comparison with GPT-VQ across 60 configurations. BARQ lowers PPL in 54/60 configurations and raises Avg. in 53/60; both improve in 47/60. Results cover six models in two panels. PPL is WikiText-2 perplexity; Avg. is the five-task zero-shot mean (%). For each metric, the GPT-VQ baseline and BARQ (Ours) are shown side by side and the better value is bold (superscript: Ours $-$ GPT-VQ). Full per-task accuracies are in Appendix B .

Panel of Table 2 — Table 2: External baseline comparison at the displayed budgets. VPTQ reports 3.000 index bpv, excluding codebooks and metadata. PPL is WikiText-2 perplexity; Avg. is mean five-task accuracy (%). A dash denotes quantization collapse in PPL and an unavailable result in Avg. Bold/underline mark best/second best; $\Delta$ gives BARQ $-$ strongest baseline. Table 6 gives per-task results.

Figure 3: Balanced assignment reduces codeword concentration during fitting. Columns are models and rows are the three settings

(K,N)\in\{(16,128),(16,512),(256,512)\}

. The

x

-axis is the outer refinement step and the

y

-axis is the maximum codeword share, the fraction of blocks held by the most-used codeword. Each curve is the median over the matched shallow and deep attention tensors and the sampled weight groups of that cell, and the band is the interquartile range. Blue dashed: hard assignment; red solid: BARQ.

Figure 4: Flatter codeword concentration and lower curvature-weighted reconstruction error across model scales. Each codebook size has its own panel and each bar is scaled to its paired GPT-VQ value, so the vertical axes read in percent of GPT-VQ and are zoomed to their panel instead of starting at zero; the number on a bar is its absolute value and the red tag the exact reduction. Color marks the model, hatched bars are GPT-VQ and solid bars are BARQ. (a) Mean maximum codeword share

p_{\max}

, lower being flatter at encoding. (b) Curvature-weighted reconstruction error

\mathrm{avg}_{g}\,\lVert W_{g}-\widehat{W}_{g}\rVert_{H}^{2}

, bar labels in units of

10^{-6}

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning	Symbol	Meaning
$N$	number of weight blocks in one group	$P$	soft assignment mass, $P\geq 0$
$d$	block dimension (weights per block)	$q_{k}(P)$	fitting mass of codeword $k$ , $\sum_{n}P_{k,n}$
$x_{n}$	the $n$ -th weight block, $x_{n}\in\mathbb{R}^{d}$	$a_{k},b_{n}$	prescribed row and column marginals of $P$
$K$	nominal codebook size, $K=2^{Bd}$	$\Pi(a,b)$	couplings with marginals $a$ and $b$
$B$	index bits per value	$\varepsilon$	dimensionless entropic regularization
$c_{k}$	the $k$ -th codeword; $C=\{c_{k}\}_{k=1}^{K}$	$s$	groupwise cost scale, fixed per update pair

Appendix

Table 3: Notation used throughout the paper. The left block lists quantities inherited from the second-order vector-quantization setting, and the right block lists the quantities introduced by the fitting-stage refinement.

Panel of Table 4 — Table 4: Per-task breakdown localizes the mean BARQ gains and the individual-task exceptions on the four 1B/8B models. Compact view: Table 1 . Layout follows Table 1 : for each metric the GPT-VQ baseline and BARQ (Ours) are shown side by side and the better value is bold (superscript: Ours $-$ GPT-VQ). PPL is WikiText-2 perplexity; per-task columns are zero-shot accuracy (%); Avg. is their mean.

Qwen3-14B
Setting			PPL $\downarrow$		Zero-shot accuracy (%) $\uparrow$
					ARC-e		ARC-c		HellaSwag		PIQA		WinoGrande		Avg.
$d$	gs	bpv	GPT-VQ	Ours	GPT-VQ	Ours	GPT-VQ	Ours	GPT-VQ	Ours	GPT-VQ	Ours	GPT-VQ	Ours	GPT-VQ	Ours
1	32	2.500	10.80	10.73 –0.07	77.40	73.65 –3.75	54.61	48.04 –6.57	68.91	69.51 +0.60	75.14	75.30 +0.16	71.19	70.48 –0.71	69.45	67.40 –2.05
	64	2.250	11.29	11.14 –0.15	75.51	70.79 –4.72	48.29	47.87 –0.42	65.90	66.74 +0.84	75.14	73.99 –1.15	69.06	70.09 +1.03	66.78	65.90 –0.88
	128	2.125	11.80	11.79 –0.01	70.20	65.99 –4.21	46.33	45.56 –0.77	63.98	64.26 +0.28	73.56	72.14 –1.42	68.67	68.19 –0.48	64.55	63.23 –1.32

Appendix

Table 5: At Qwen3-14B/32B, the strongest gains occur for

K=16

and

K=256

, while the per-task breakdown retains the

K=4

regressions. Compact view: the Qwen3-14B/32B panels of Table 1 . Columns, bolding and superscripts match Table 4 .

Panel of Table 6 — Table 6: Full per-task external-baseline comparison. Compact view: Table 2 . PPL is WikiText-2 perplexity; per-task columns are zero-shot accuracy (%); Avg. is their mean. VPTQ uses 3.000 index bpv, excluding codebooks and metadata. A dash in PPL denotes quantization collapse; other dashes denote unavailable results. Bold marks the best; underlining marks the runner-up. The $\Delta$ row gives BARQ minus the strongest baseline ( green : better; red : worse).

Figure 5: The observed accuracy gain is larger in the larger-codebook strata. (a) Mean gain per model at each codebook size, with the individual configurations overlaid as points. (b) Pooled trend across all models, with the interquartile band. The pooled mean grows with codebook size, while the points retain the regressions at

K=4

Figure 6: Assignment evenness and local reconstruction over the same fitting trajectories. (a) Normalized hard-assignment entropy, where BARQ has the higher step-15 median in all 15 model-by-setting cells. (b) Euclidean reconstruction distortion normalized by each group’s step-zero value, where BARQ has the lower step-15 median in 12 of the 15 cells. In both panels, columns are models, rows are the three

(K,N)

settings, the

x

-axis is the outer refinement step, and the dashed grey curve is the hard-assignment baseline while the solid curve is BARQ.

Figure 7: Hard codeword use is higher at the larger codebook sizes across layers and model scales. Each panel shows effective utilization, the number of used codewords divided by

\min(K,N)

with

N

the number of blocks per group, for three model scales and three codebook sizes. At

K=4

both methods nearly saturate the codebook under nearest-codeword assignment, because every group holds many more blocks than codewords. At

K=16

BARQ is higher at every measured layer. At

K=256

it is higher in 74 of 75 matched layers; one further 8B baseline layer has no BARQ counterpart. The matched weighted aggregate is higher at all three model scales.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 386 papers

Jun 11, 2026

Multi-Bitwidth Quantization for LLMs Using Additive Codebooks

As large language models (LLMs) are increasingly deployed across heterogeneous hardware with varying resource constraints, the ability to adaptively manage the trade-off between performance and efficiency without retraining is critical. We propose Drop-by-Drop, a novel multi-bitwidth post-training quantization framework that enables inference-time precision control over LLM weights from a single trained model. Our method is theoretically grounded in information theory and successive refinement. We establish that LLM weights, which commonly follow a Gaussian distribution, can be optimally reconstructed with increasing fidelity as additional bits are incorporated, under a weighted mean squared error distortion motivated by LLM loss functions. To realize this in practice, Drop-by-Drop incorporates Matryoshka-style supervision into the loss function, exploiting the structure of additive codebooks. Drop-by-Drop produces a single model where ordered subsets of codebooks yield accurate partial reconstructions at each precision level. This approach significantly reduces storage and memory overhead by allowing a single checkpoint to serve multiple bitwidths, while maintaining competitive perplexity and accuracy across major architectures, such as Qwen, LLaMA, Gemma, and Mistral.

Liza Babaoglu, Shuangyi Chen, Ashish Khisti

University of Toronto

LLM Quantization Weight-Only Quantization Iterative Refinement Post-Training Quantization Neural Network Quantization

Ask PDF

Aug 24, 2026

Activation-Weighted Seeded Residual Coding for Low-Bit LLM Weight Repair

Low-bit weight quantization saves storage but leaves errors that degrade LLM quality. We introduce activation-weighted seeded residual coding (AWSRC), a compact repair codec for an existing quantization backbone. Given a reconstructed weight

W_0

, AWSRC encodes the residual

W-W_0

using deterministic seed-generated bases. The sidecar stores seed selectors, low-bit coefficients, and scales rather than an explicit codebook. Two variants combine activation weighting with per-module byte quotas (

\mathrm{AWSRC\text{-}U}

), or blended activation/Fisher weighting with globally ranked progressive prefixes (

\mathrm{AWSRC\text{-}P}_{F}

) that support multiple byte budgets without refitting. On Qwen2.5-3B-Instruct, adding

0.162

scope-bits/weight to an RTN-INT4 baseline closes

88.2\%

78.9\%

, and

71.3\%

of the PPL, KL, and 11-task mean-accuracy gaps to BF16, respectively. AWSRC achieves the highest mean downstream accuracy in byte-matched residual-codec ablations and improves all metrics across model families with up to 32B parameters.

Zehao Liu, Chuangchuang Fang, Yang Ren

Huawei Technologies Research & Development (UK) Limited, Edinburgh, United Kingdom

LLM Quantization Weight-Only Quantization Post-Training Quantization Low-Bit Quantization

Ask PDF

May 9, 2026

AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization

Post-training weight-only quantization to 4 bits is widely used to reduce the memory and compute costs of large language model inference. Existing PTQ methods, such as AWQ and GPTQ, improve how weights are mapped onto a fixed 4-bit grid through scaling, clipping, or error compensation. To further improve accuracy, methods such as OmniQuant and QuIP# uses gradient-assisted algorithms at the cost of hours of quantization time. In this work, we propose AAAC (Activation-Aware Adaptive Codebooks), a lightweight method for 4-bit LLM weight quantization. AAAC replaces the fixed scalar codebook used in standard quantization with two small learned scalar codebooks (64 bytes) per layer. Each group of weights selects the codebook that minimizes activation-weighted reconstruction error, encoding the choice in the unused sign bit of the group's positive scale and adding zero storage overhead. AAAC completes in 3--30 minutes on a single GPU, and adds no memory beyond the model itself. We evaluate against AWQ, GPTQ, IF4, GPTVQ, OmniQuant, SqueezeLLM, and QuIP# across model families. AAAC outperforms baselines at orders-of-magnitude less quantization time.

Beshr IslamBouli, David Jin

Massachusetts Institute of Technology, Cambridge, MA, USA.

LLM Quantization 4-Bit Quantization Weight-Only Quantization Post-Training Quantization Neural Network Quantization

Ask PDF

Abstract

LLM Quantization

Explore similar work

Multi-Bitwidth Quantization for LLMs Using Additive Codebooks

Activation-Weighted Seeded Residual Coding for Low-Bit LLM Weight Repair

AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization