CoS-VQ: Coordinated Shared-Scale Vector Quantization for Large Language Models

Zukang Xu; Zhixiong Zhao; Xing Hu; Zhe Jiang; Xin Si; Dawei Yang

cs.LGOct 27, 2026

CoS-VQ: Coordinated Shared-Scale Vector Quantization for Large Language Models

Authors: Zukang Xu, Zhixiong Zhao, Xing Hu, Zhe Jiang, Xin Si, Dawei Yang

Organizations: Houmo AI · Peking University · Southeast University

Abstract

Large language models (LLMs) require substantial memory, motivating low-bit weight quantization for efficient deployment. Existing vector quantization (VQ) methods often improve accuracy by increasing the effective coding dimension, which can complicate independent lookup and local codeword updates. We propose CoS-VQ (Coordinated Shared-Scale Vector Quantization), which preserves low-dimensional codewords while jointly optimizing hard indices and their shared scale across positions. The key challenge is scale-induced coupling: the optimal scale depends on the entire codeword combination and, in turn, changes candidate rankings across positions, so pointwise updates can stall even after scale re-optimization. CoS-VQ collects candidates at multiple probe scales and constructs a tangent separable upper bound for the residual-aware quadratic objective. By reducing candidate comparisons to lower envelopes of lines over positive scales, it identifies finitely many representative intervals for joint index--scale updates without exhaustively enumerating codeword combinations. On Qwen3.8-27B, CoS-VQ outperforms QTIP and LLVQ on the evaluated language modeling, coding, and reasoning benchmarks. Ablations from the same initialization further show larger gains from joint updates than from scale-only refinement, highlighting joint selection as an alternative to increasing codeword dimensionality. The code is available at https://github.com/xzktx003/Cos_VQ.

Figures & tables

Figure 1: (a) Three shared-scale groups approach a coordinate-wise stall under single-index updates with scale re-optimization. After switching to joint-index updates, the normalized true layer-output error decreases further. Each curve is normalized to 1 at the switching point. (b) Distribution of the number of changed indices among accepted updates, showing that multi-index updates account for the majority.

Figure 2: Overview of Qwen3-4B results. (a) Equal-weight mean of WikiText-2 and C4 perplexity across the evaluated low-bit quantization methods. (b) Comparison of selected methods on five downstream tasks.

Figure 3: Overview of CoS-VQ . (a) Multi-scale probes construct candidate pools. (b) A residual-aware tangent majorizer yields separable quadratic costs and affine comparisons over the shared scale. (c) Lower-envelope breakpoints define intervals for joint index–scale optimization. The lower panels illustrate the envelope geometry and true-error verification. Proofs and boundary cases appear in Appendix A.3.2 .

	Qwen3-4B		Qwen3-8B		Qwen3-14B
Method	PPL.AVG $\downarrow$	ACC.AVG $\uparrow$	PPL.AVG $\downarrow$	ACC.AVG $\uparrow$	PPL.AVG $\downarrow$	ACC.AVG $\uparrow$
BF16	15.77	66.79	12.01	70.37	10.75	73.77
PV-Tuning	17.10	59.57	12.90	66.84	11.36	70.92
QuIP#	21.48	57.03	15.76	65.32	13.37	69.65
QTIP	18.20	60.30	13.91	66.80	12.09	71.50
NoWag	23.46	57.19	15.96	65.73	13.32	69.63

Table 1: Main results on six models. PPL.AVG is the arithmetic mean of WikiText-2 and C4 perplexities. ACC.AVG is the macro-average of ARC-Challenge, ARC-Easy, HellaSwag, LAMBADA OpenAI, PIQA, and WinoGrande. Bold denotes the best result among quantized methods.

Model	Scale	Assign	Wiki2 $\downarrow$	Avg. $\uparrow$
Qwen3-4B	$\times$	$\times$	22.69	58.81
	$\checkmark$	$\times$	21.01	59.32
	$\checkmark$	$\checkmark$	13.68	62.58
Qwen3-8B	$\times$	$\times$	14.41	66.44
	$\checkmark$	$\times$	13.85	67.19
	$\checkmark$	$\checkmark$	9.76	69.43

Table 2: Same-start ablation of shared-scale optimization (Scale) and hard-index assignment (Assign). Avg. denotes the mean accuracy across eight zero-shot tasks.

Method	Wiki2 $\downarrow$	C4 $\downarrow$	AIME25 $\uparrow$	LCBv6 $\uparrow$	GPQA-D $\uparrow$
BF16	7.05	11.45	100.00	85.71	89.90
QTIP	8.29	13.76	93.33	73.65	81.11
LLVQ	8.22	13.85	93.33	72.28	80.48
CoS-VQ	7.84	13.33	93.33	74.92	82.83

Table 3: Qwen3.8-27B with thinking enabled. Bold denotes the best quantized result.

Offline Quantization Time
Model	Q3-4B	Q3-8B	Q3-14B	Q3-32B
Time (h) $\downarrow$	1.3	1.6	2.0	3.5
Inference Throughput
Method	Prefill (tok/s) $\uparrow$		Decode (tok/s) $\uparrow$
BF16	2217.6		21.9
CoS-VQ	2230.2		36.7

Table 4: Offline quantization time on an A100-SXM4-80GB GPU and Llama-3.1-8B-Instruct inference throughput on an RTX 4090 with a 2K-token context.

$d$	bpw	Wiki2 $\downarrow$	ARC-C $\uparrow$	ARC-E $\uparrow$	HellaSwag $\uparrow$	MMLU $\uparrow$
4	2.01	12.96	41.27	68.90	62.11	53.23
5	2.02	12.74	42.50	69.46	62.78	53.89
6	2.04	13.68	42.92	70.12	63.16	55.01
8	2.59	11.87	43.27	70.73	64.28	55.71

Table 5: Effect of base-codeword dimension

d

on Qwen3-4B. The default

d=6

balances accuracy and bitrate under an approximately 2-bpw budget.

Model	Wiki2 $\downarrow$	ARC-C $\uparrow$	HellaS $\uparrow$	WinoG $\uparrow$
Q3-4B	12.18/12.69	42.92/42.63	63.17/62.28	62.90/62.39
Q3-8B	9.76/10.17	54.52/53.87	69.80/69.64	68.67/68.80
Q3-14B	8.51/9.78	53.24/52.76	75.30/75.47	71.98/71.01
Q3-32B	7.74/8.06	60.32/59.78	78.17/78.11	76.09/76.30

Table 6: Ablation of end-to-end scale fine-tuning on Qwen3. Each entry reports fine-tuned / not fine-tuned .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Update	Combination $(a,b)$	Optimal scale	Optimal error
Current state	$(1,6/5)$	$55/61$	$1/61\approx 0.016393$
Change position 1 only	$(3,6/5)$	$35/87$	$9/29\approx 0.310345$
Change position 2 only	$(1,3)$	$2/5$	$2/5=0.4$
Change both jointly	$(3,3)$	$1/3$	$0$

Appendix

Table 7: A checkable coordinated-update instance for Appendix A.2 . Both one-at-a-time updates remain worse even after each refits the scale, whereas a joint modification can eliminate the error.

Method	Wiki2 $\downarrow$	C4 $\downarrow$	ARC-C $\uparrow$	ARC-E $\uparrow$	HellaS $\uparrow$	LAMBADA $\uparrow$	PIQA $\uparrow$	BoolQ $\uparrow$	MMLU $\uparrow$	WinoG $\uparrow$
BF16	13.66	17.88	53.67	78.49	68.43	59.40	74.97	84.98	68.22	65.75
PV-Tuning	14.87	19.33	43.52	69.49	60.58	50.46	72.91	77.55	55.10	60.46
QuIP#	17.98	24.98	40.27	66.58	56.94	47.91	70.08	77.92	52.90	60.38
QTIP	15.78	20.62	43.77	68.69	60.85	51.58	73.12	84.13	58.58	63.77
NoWag	22.69	24.22	42.24	65.87	55.73	46.32	71.00	79.14	53.49	61.96
AQLM	15.12	23.38	40.96	67.89	62.92	55.19	73.07	78.32	56.11	60.46

Appendix

Table 8: Per-task results on Qwen3-4B. WikiText-2 and C4 are perplexities (lower is better); the remaining columns are zero-shot accuracies (%, higher is better). Bold denotes the best result among quantized methods.

Method	Wiki2 $\downarrow$	C4 $\downarrow$	ARC-C $\uparrow$	ARC-E $\uparrow$	HellaS $\uparrow$	LAMBADA $\uparrow$	PIQA $\uparrow$	BoolQ $\uparrow$	MMLU $\uparrow$	WinoG $\uparrow$
BF16	9.72	14.29	56.74	80.68	74.94	64.08	77.69	86.61	72.97	68.11
PV-Tuning	10.51	15.28	51.45	77.36	68.02	61.67	75.90	81.41	63.52	66.61
QuIP#	12.64	18.88	49.83	76.30	64.82	60.99	73.45	81.68	63.70	66.54
QTIP	11.61	16.20	50.77	75.42	68.48	62.68	76.33	84.50	66.03	67.09
NoWag	13.85	18.07	51.79	78.20	63.41	59.38	73.72	83.36	62.38	67.88
AQLM	10.71	17.84	50.17	76.81	70.09	62.95	76.06	81.96	64.25	66.61

Appendix

Table 9: Per-task results on Qwen3-8B. WikiText-2 and C4 are perplexities (lower is better); the remaining columns are zero-shot accuracies (%, higher is better). Bold denotes the best result among quantized methods.

Method	Wiki2 $\downarrow$	C4 $\downarrow$	ARC-C $\uparrow$	ARC-E $\uparrow$	HellaS $\uparrow$	LAMBADA $\uparrow$	PIQA $\uparrow$	BoolQ $\uparrow$	MMLU $\uparrow$	WinoG $\uparrow$
BF16	8.65	12.85	60.24	82.83	78.86	67.88	79.87	89.33	77.20	72.93
PV-Tuning	9.15	13.56	55.46	79.71	73.60	66.54	78.24	83.82	69.30	71.98
QuIP#	10.43	16.31	53.92	78.70	71.16	66.16	76.06	84.10	67.98	71.90
QTIP	9.84	14.33	56.57	78.70	73.86	68.97	78.18	88.38	71.17	72.69
NoWag	11.04	15.60	53.67	79.42	70.23	65.19	77.26	87.19	68.69	71.98
AQLM	9.25	15.53	54.27	79.17	75.17	67.24	78.40	84.40	69.91	71.90

Appendix

Table 10: Per-task results on Qwen3-14B. WikiText-2 and C4 are perplexities (lower is better); the remaining columns are zero-shot accuracies (%, higher is better). Bold denotes the best result among quantized methods.

Method	Wiki2 $\downarrow$	C4 $\downarrow$	ARC-C $\uparrow$	ARC-E $\uparrow$	HellaS $\uparrow$	LAMBADA $\uparrow$	PIQA $\uparrow$	BoolQ $\uparrow$	MMLU $\uparrow$	WinoG $\uparrow$
BF16	7.61	11.51	60.92	83.16	82.58	67.18	82.10	86.36	80.86	72.93
PV-Tuning	8.10	12.26	60.49	82.70	78.47	70.46	79.43	86.36	74.62	75.53
QuIP#	9.73	14.57	60.58	81.44	76.67	69.47	77.75	85.47	72.22	74.59
QTIP	8.53	12.49	60.24	81.69	79.74	69.77	79.60	88.29	77.28	74.19
NoWag	9.19	13.35	61.12	83.08	77.03	70.60	79.00	88.13	74.13	74.90
AQLM	8.15	13.20	61.12	82.53	78.97	70.42	79.49	86.61	74.77	75.22

Appendix

Table 11: Per-task results on Qwen3-32B. WikiText-2 and C4 are perplexities (lower is better); the remaining columns are zero-shot accuracies (%, higher is better). Bold denotes the best result among quantized methods.

Method	Wiki2 $\downarrow$	C4 $\downarrow$	ARC-C $\uparrow$	ARC-E $\uparrow$	HellaS $\uparrow$	LAMBADA $\uparrow$	PIQA $\uparrow$	BoolQ $\uparrow$	MMLU $\uparrow$	WinoG $\uparrow$	IFEval $\uparrow$
BF16	7.22	10.63	55.03	79.67	79.17	73.08	80.96	83.85	67.93	73.40	73.94
PV-Tuning	8.45	13.10	46.16	75.42	54.40	64.10	78.73	76.39	54.80	70.56	60.44
QuIP#	9.75	14.85	43.26	69.65	67.70	61.60	76.82	76.79	52.60	69.22	55.08
QTIP	8.80	13.39	50.77	74.34	75.02	69.71	78.24	80.01	54.39	69.01	64.25
NoWag	10.50	15.28	47.27	74.09	70.14	67.48	77.09	80.52	55.95	69.93	56.75
AQLM	9.50	15.00	30.89	46.25	61.40	60.00	76.50	68.01	42.29	58.25	40.48

Appendix

Table 12: Per-task results on Llama-3.1-8B-Instruct. WikiText-2 and C4 are perplexities (lower is better); the remaining columns are zero-shot accuracies (%, higher is better). Bold denotes the best result among quantized methods.

Method	Wiki2 $\downarrow$	C4 $\downarrow$	ARC-C $\uparrow$	ARC-E $\uparrow$	HellaS $\uparrow$	LAMBADA $\uparrow$	PIQA $\uparrow$	BoolQ $\uparrow$	MMLU $\uparrow$	WinoG $\uparrow$	IFEval $\uparrow$
BF16	3.78	8.00	63.48	83.84	84.63	75.68	83.79	87.68	82.37	78.77	83.55
PV-Tuning	5.70	10.95	52.73	82.20	76.25	72.00	82.59	84.71	72.50	75.45	73.57
QuIP#	5.56	11.21	50.68	79.76	72.74	68.60	81.77	83.03	70.89	75.14	70.61
QTIP	5.08	10.48	54.40	82.60	76.40	72.65	82.50	85.12	73.78	75.51	75.73
NoWag	5.93	11.43	52.65	83.00	73.96	71.53	81.94	84.89	72.56	75.45	71.53
AQLM	5.33	11.04	47.44	72.43	71.63	69.24	81.88	80.58	68.15	71.43	67.10

Appendix

Table 13: Per-task results on Llama-3.1-70B-Instruct. WikiText-2 and C4 are perplexities (lower is better); the remaining columns are zero-shot accuracies (%, higher is better). Bold denotes the best result among quantized methods.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 388 papers

Jun 9, 2026

UniSVQ: 2-bit Unified Scalar-Vector Quantization

Post-training quantization at the 2-bit level enables low-cost deployment and inference acceleration for large language models (LLMs). Scalar quantization (SQ) and vector quantization (VQ) are two primary quantization methods, however, the former suffers from significant performance degradation, and the latter incurs computational and storage overhead. We propose UniSVQ, a unified 2-bit quantization framework that bridges scalar and vector quantization by parameterizing codewords as an affine transform of integer lattices. This structure preserves compatibility with optimized integer kernels while retaining much of VQ's flexibility. We further introduce a data-driven block-wise fine-tuning strategy to directly minimize quantization reconstruction error. Extensive experiments across multiple LLM families and zero-shot benchmarks demonstrate that UniSVQ consistently outperforms state-of-the-art SQ methods and achieves performance comparable to advanced VQ methods, while providing higher inference throughput. Codes are publicly available at https://github.com/AI9Stars/UniSVQ.

Haoyu Wang, Haiyan Zhao, Xingyu Yu +4

Department of Computer Science and Technology, Tsinghua University, Beijing, China · School of Software and Microelectronics, Peking University, Beijing, China · School of Physics, Beihang University, Beijing, China

LLM Quantization Vector Quantization Post-Training Quantization Low-Bit Quantization Neural Network Quantization

Ask PDF

Oct 22, 2026

SQuAT: Spherical Lattice Vector Quantization-Aware Training

Vector quantization (VQ), in particular lattice-based methods, offer strong rate-distortion trade-offs for low-bit large language models (LLMs), yet their behavior under quantization-aware training (VQ-QAT) remains poorly understood. We scale lattice VQ-QAT to billion-parameter LLMs and multi-billion-token training regimes, and introduce SQuAT, a shape-gain VQ-QAT approach trained with a separate straight-through estimator (STE) for the shape and gain. SQuAT decomposes weights into direction and magnitude: normalized directions are quantized using an

E_8

-derived spherical codebook, while gains are quantized independently with low-bit scalar quantizers. We show that, unlike a standard STE applied to the full quantizer, which ignores the shape-gain structure, these separate STEs induce tangent-space updates for directions, yielding an implicit form of spherical (Riemannian) optimization. To enable scalability, we develop a distributed exact nearest-neighbor assignment scheme, in which workers quantize disjoint weight blocks and communicate only discrete indices, reducing VQ-QAT training time by 12.8

\times

and allowing us to scale it to 10B tokens. Across instruction-tuned LLaMA-3.2-1B/3B, Qwen3-4B, and base Qwen3-1.7B, SQuAT consistently outperforms standard VQ-QAT STE under identical quantizers and initialization, and achieves SOTA VQ-QAT accuracy.

Sebastian Loeschcke, Ivan Koryakovskiy, Tycho F. A. van der Ouderaa +4

University of Copenhagen · Qualcomm AI Research

LLM Quantization Vector Quantization Quantization-Aware Training

Ask PDF

Jun 9, 2026

LC-QAT: Data-Efficient 2-Bit QAT for LLMs via Linear-Constrained Vector Quantization

Quantization-aware training (QAT) is essential for extremely low-bit large language models (LLMs). Current QAT methods are mainly based on scalar quantization (SQ), which enables efficient optimization but suffers from severe performance degradation at 2-bit precision. On the other hand, vector quantization (VQ) provides substantially higher representational capacity, but its discrete codebook lookup prevents end-to-end training. We propose LC-QAT, a 2-bit weight-only VQ-QAT framework that represents quantized weights via a learned affine mapping over discrete vectors, which yields a high-quality PTQ initialization and enables fully differentiable end-to-end optimization without explicit codebook lookup in the training forward pass. This strong post-training initialization makes LC-QAT highly data-efficient. Experiments across diverse LLMs demonstrate that LC-QAT consistently outperforms state-of-the-art QAT methods while using only 0.1%--10% of the training data. Our results establish LC-QAT as a practical and scalable solution for extreme low-bit model deployment. Codes are publicly available at https://github.com/AI9Stars/UniSVQ.

Haoyu Wang, Xingyu Yu, Haiyan Zhao +2

Department of Computer Science and Technology, Tsinghua University, Beijing, China · School of Software and Microelectronics, Peking University, Beijing, China · College of Computer Science and Technology, National University of Defense Technology, Hunan, China.

LLM Quantization Quantization-Aware Training Vector Quantization Weight-Only Quantization Neural Network Quantization Data-Efficient Learning

Ask PDF

Abstract

LLM Quantization

Explore similar work

UniSVQ: 2-bit Unified Scalar-Vector Quantization

SQuAT: Spherical Lattice Vector Quantization-Aware Training

LC-QAT: Data-Efficient 2-Bit QAT for LLMs via Linear-Constrained Vector Quantization