CoS-VQ: Coordinated Shared-Scale Vector Quantization for Large Language Models

Zukang Xu; Zhixiong Zhao; Xing Hu; Zhe Jiang; Xin Si; Dawei Yang

cs.LGOct 27, 2026

CoS-VQ: Coordinated Shared-Scale Vector Quantization for Large Language Models

Authors: Zukang Xu, Zhixiong Zhao, Xing Hu, Zhe Jiang, Xin Si, Dawei Yang

Organizations: Houmo AI · Peking University · Southeast University

Abstract

Large language models (LLMs) require substantial memory, motivating low-bit weight quantization for efficient deployment. Existing vector quantization (VQ) methods often improve accuracy by increasing the effective coding dimension, which can complicate independent lookup and local codeword updates. We propose CoS-VQ (Coordinated Shared-Scale Vector Quantization), which preserves low-dimensional codewords while jointly optimizing hard indices and their shared scale across positions. The key challenge is scale-induced coupling: the optimal scale depends on the entire codeword combination and, in turn, changes candidate rankings across positions, so pointwise updates can stall even after scale re-optimization. CoS-VQ collects candidates at multiple probe scales and constructs a tangent separable upper bound for the residual-aware quadratic objective. By reducing candidate comparisons to lower envelopes of lines over positive scales, it identifies finitely many representative intervals for joint index--scale updates without exhaustively enumerating codeword combinations. On Qwen3.8-27B, CoS-VQ outperforms QTIP and LLVQ on the evaluated language modeling, coding, and reasoning benchmarks. Ablations from the same initialization further show larger gains from joint updates than from scale-only refinement, highlighting joint selection as an alternative to increasing codeword dimensionality. The code is available at https://github.com/xzktx003/Cos_VQ.

Figures & tables

Figure 1: (a) Three shared-scale groups approach a coordinate-wise stall under single-index updates with scale re-optimization. After switching to joint-index updates, the normalized true layer-output error decreases further. Each curve is normalized to 1 at the switching point. (b) Distribution of the number of changed indices among accepted updates, showing that multi-index updates account for the majority.

Figure 2: Overview of Qwen3-4B results. (a) Equal-weight mean of WikiText-2 and C4 perplexity across the evaluated low-bit quantization methods. (b) Comparison of selected methods on five downstream tasks.

Figure 3: Overview of CoS-VQ . (a) Multi-scale probes construct candidate pools. (b) A residual-aware tangent majorizer yields separable quadratic costs and affine comparisons over the shared scale. (c) Lower-envelope breakpoints define intervals for joint index–scale optimization. The lower panels illustrate the envelope geometry and true-error verification. Proofs and boundary cases appear in Appendix A.3.2 .

	Qwen3-4B		Qwen3-8B		Qwen3-14B
Method	PPL.AVG $\downarrow$	ACC.AVG $\uparrow$	PPL.AVG $\downarrow$	ACC.AVG $\uparrow$	PPL.AVG $\downarrow$	ACC.AVG $\uparrow$
BF16	15.77	66.79	12.01	70.37	10.75	73.77
PV-Tuning	17.10	59.57	12.90	66.84	11.36	70.92
QuIP#	21.48	57.03	15.76	65.32	13.37	69.65
QTIP	18.20	60.30	13.91	66.80	12.09	71.50
NoWag	23.46	57.19	15.96	65.73	13.32	69.63

Table 1: Main results on six models. PPL.AVG is the arithmetic mean of WikiText-2 and C4 perplexities. ACC.AVG is the macro-average of ARC-Challenge, ARC-Easy, HellaSwag, LAMBADA OpenAI, PIQA, and WinoGrande. Bold denotes the best result among quantized methods.

Model	Scale	Assign	Wiki2 $\downarrow$	Avg. $\uparrow$
Qwen3-4B	$\times$	$\times$	22.69	58.81
	$\checkmark$	$\times$	21.01	59.32
	$\checkmark$	$\checkmark$	13.68	62.58
Qwen3-8B	$\times$	$\times$	14.41	66.44
	$\checkmark$	$\times$	13.85	67.19
	$\checkmark$	$\checkmark$	9.76	69.43

Table 2: Same-start ablation of shared-scale optimization (Scale) and hard-index assignment (Assign). Avg. denotes the mean accuracy across eight zero-shot tasks.

Method	Wiki2 $\downarrow$	C4 $\downarrow$	AIME25 $\uparrow$	LCBv6 $\uparrow$	GPQA-D $\uparrow$
BF16	7.05	11.45	100.00	85.71	89.90
QTIP	8.29	13.76	93.33	73.65	81.11
LLVQ	8.22	13.85	93.33	72.28	80.48
CoS-VQ	7.84	13.33	93.33	74.92	82.83

Table 3: Qwen3.8-27B with thinking enabled. Bold denotes the best quantized result.

Offline Quantization Time
Model	Q3-4B	Q3-8B	Q3-14B	Q3-32B
Time (h) $\downarrow$	1.3	1.6	2.0	3.5
Inference Throughput
Method	Prefill (tok/s) $\uparrow$		Decode (tok/s) $\uparrow$
BF16	2217.6		21.9
CoS-VQ	2230.2		36.7

Table 4: Offline quantization time on an A100-SXM4-80GB GPU and Llama-3.1-8B-Instruct inference throughput on an RTX 4090 with a 2K-token context.

$d$	bpw	Wiki2 $\downarrow$	ARC-C $\uparrow$	ARC-E $\uparrow$	HellaSwag $\uparrow$	MMLU $\uparrow$
4	2.01	12.96	41.27	68.90	62.11	53.23
5	2.02	12.74	42.50	69.46	62.78	53.89
6	2.04	13.68	42.92	70.12	63.16	55.01
8	2.59	11.87	43.27	70.73	64.28	55.71

Table 5: Effect of base-codeword dimension

d

on Qwen3-4B. The default

d=6

balances accuracy and bitrate under an approximately 2-bpw budget.

Model	Wiki2 $\downarrow$	ARC-C $\uparrow$	HellaS $\uparrow$	WinoG $\uparrow$
Q3-4B	12.18/12.69	42.92/42.63	63.17/62.28	62.90/62.39
Q3-8B	9.76/10.17	54.52/53.87	69.80/69.64	68.67/68.80
Q3-14B	8.51/9.78	53.24/52.76	75.30/75.47	71.98/71.01
Q3-32B	7.74/8.06	60.32/59.78	78.17/78.11	76.09/76.30

Table 6: Ablation of end-to-end scale fine-tuning on Qwen3. Each entry reports fine-tuned / not fine-tuned .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Update	Combination $(a,b)$	Optimal scale	Optimal error
Current state	$(1,6/5)$	$55/61$	$1/61\approx 0.016393$
Change position 1 only	$(3,6/5)$	$35/87$	$9/29\approx 0.310345$
Change position 2 only	$(1,3)$	$2/5$	$2/5=0.4$
Change both jointly	$(3,3)$	$1/3$	$0$

Appendix

Table 7: A checkable coordinated-update instance for Appendix A.2 . Both one-at-a-time updates remain worse even after each refits the scale, whereas a joint modification can eliminate the error.

Method	Wiki2 $\downarrow$	C4 $\downarrow$	ARC-C $\uparrow$	ARC-E $\uparrow$	HellaS $\uparrow$	LAMBADA $\uparrow$	PIQA $\uparrow$	BoolQ $\uparrow$	MMLU $\uparrow$	WinoG $\uparrow$
BF16	13.66	17.88	53.67	78.49	68.43	59.40	74.97	84.98	68.22	65.75
PV-Tuning	14.87	19.33	43.52	69.49	60.58	50.46	72.91	77.55	55.10	60.46
QuIP#	17.98	24.98	40.27	66.58	56.94	47.91	70.08	77.92	52.90	60.38
QTIP	15.78	20.62	43.77	68.69	60.85	51.58	73.12	84.13	58.58	63.77
NoWag	22.69	24.22	42.24	65.87	55.73	46.32	71.00	79.14	53.49	61.96
AQLM	15.12	23.38	40.96	67.89	62.92	55.19	73.07	78.32	56.11	60.46

Appendix

Table 8: Per-task results on Qwen3-4B. WikiText-2 and C4 are perplexities (lower is better); the remaining columns are zero-shot accuracies (%, higher is better). Bold denotes the best result among quantized methods.

Method	Wiki2 $\downarrow$	C4 $\downarrow$	ARC-C $\uparrow$	ARC-E $\uparrow$	HellaS $\uparrow$	LAMBADA $\uparrow$	PIQA $\uparrow$	BoolQ $\uparrow$	MMLU $\uparrow$	WinoG $\uparrow$
BF16	9.72	14.29	56.74	80.68	74.94	64.08	77.69	86.61	72.97	68.11
PV-Tuning	10.51	15.28	51.45	77.36	68.02	61.67	75.90	81.41	63.52	66.61
QuIP#	12.64	18.88	49.83	76.30	64.82	60.99	73.45	81.68	63.70	66.54
QTIP	11.61	16.20	50.77	75.42	68.48	62.68	76.33	84.50	66.03	67.09
NoWag	13.85	18.07	51.79	78.20	63.41	59.38	73.72	83.36	62.38	67.88
AQLM	10.71	17.84	50.17	76.81	70.09	62.95	76.06	81.96	64.25	66.61

Appendix

Table 9: Per-task results on Qwen3-8B. WikiText-2 and C4 are perplexities (lower is better); the remaining columns are zero-shot accuracies (%, higher is better). Bold denotes the best result among quantized methods.

Method	Wiki2 $\downarrow$	C4 $\downarrow$	ARC-C $\uparrow$	ARC-E $\uparrow$	HellaS $\uparrow$	LAMBADA $\uparrow$	PIQA $\uparrow$	BoolQ $\uparrow$	MMLU $\uparrow$	WinoG $\uparrow$
BF16	8.65	12.85	60.24	82.83	78.86	67.88	79.87	89.33	77.20	72.93
PV-Tuning	9.15	13.56	55.46	79.71	73.60	66.54	78.24	83.82	69.30	71.98
QuIP#	10.43	16.31	53.92	78.70	71.16	66.16	76.06	84.10	67.98	71.90
QTIP	9.84	14.33	56.57	78.70	73.86	68.97	78.18	88.38	71.17	72.69
NoWag	11.04	15.60	53.67	79.42	70.23	65.19	77.26	87.19	68.69	71.98
AQLM	9.25	15.53	54.27	79.17	75.17	67.24	78.40	84.40	69.91	71.90

Appendix

Table 10: Per-task results on Qwen3-14B. WikiText-2 and C4 are perplexities (lower is better); the remaining columns are zero-shot accuracies (%, higher is better). Bold denotes the best result among quantized methods.

Method	Wiki2 $\downarrow$	C4 $\downarrow$	ARC-C $\uparrow$	ARC-E $\uparrow$	HellaS $\uparrow$	LAMBADA $\uparrow$	PIQA $\uparrow$	BoolQ $\uparrow$	MMLU $\uparrow$	WinoG $\uparrow$
BF16	7.61	11.51	60.92	83.16	82.58	67.18	82.10	86.36	80.86	72.93
PV-Tuning	8.10	12.26	60.49	82.70	78.47	70.46	79.43	86.36	74.62	75.53
QuIP#	9.73	14.57	60.58	81.44	76.67	69.47	77.75	85.47	72.22	74.59
QTIP	8.53	12.49	60.24	81.69	79.74	69.77	79.60	88.29	77.28	74.19
NoWag	9.19	13.35	61.12	83.08	77.03	70.60	79.00	88.13	74.13	74.90
AQLM	8.15	13.20	61.12	82.53	78.97	70.42	79.49	86.61	74.77	75.22

Appendix

Table 11: Per-task results on Qwen3-32B. WikiText-2 and C4 are perplexities (lower is better); the remaining columns are zero-shot accuracies (%, higher is better). Bold denotes the best result among quantized methods.

Method	Wiki2 $\downarrow$	C4 $\downarrow$	ARC-C $\uparrow$	ARC-E $\uparrow$	HellaS $\uparrow$	LAMBADA $\uparrow$	PIQA $\uparrow$	BoolQ $\uparrow$	MMLU $\uparrow$	WinoG $\uparrow$	IFEval $\uparrow$
BF16	7.22	10.63	55.03	79.67	79.17	73.08	80.96	83.85	67.93	73.40	73.94
PV-Tuning	8.45	13.10	46.16	75.42	54.40	64.10	78.73	76.39	54.80	70.56	60.44
QuIP#	9.75	14.85	43.26	69.65	67.70	61.60	76.82	76.79	52.60	69.22	55.08
QTIP	8.80	13.39	50.77	74.34	75.02	69.71	78.24	80.01	54.39	69.01	64.25
NoWag	10.50	15.28	47.27	74.09	70.14	67.48	77.09	80.52	55.95	69.93	56.75
AQLM	9.50	15.00	30.89	46.25	61.40	60.00	76.50	68.01	42.29	58.25	40.48

Appendix

Table 12: Per-task results on Llama-3.1-8B-Instruct. WikiText-2 and C4 are perplexities (lower is better); the remaining columns are zero-shot accuracies (%, higher is better). Bold denotes the best result among quantized methods.

Method	Wiki2 $\downarrow$	C4 $\downarrow$	ARC-C $\uparrow$	ARC-E $\uparrow$	HellaS $\uparrow$	LAMBADA $\uparrow$	PIQA $\uparrow$	BoolQ $\uparrow$	MMLU $\uparrow$	WinoG $\uparrow$	IFEval $\uparrow$
BF16	3.78	8.00	63.48	83.84	84.63	75.68	83.79	87.68	82.37	78.77	83.55
PV-Tuning	5.70	10.95	52.73	82.20	76.25	72.00	82.59	84.71	72.50	75.45	73.57
QuIP#	5.56	11.21	50.68	79.76	72.74	68.60	81.77	83.03	70.89	75.14	70.61
QTIP	5.08	10.48	54.40	82.60	76.40	72.65	82.50	85.12	73.78	75.51	75.73
NoWag	5.93	11.43	52.65	83.00	73.96	71.53	81.94	84.89	72.56	75.45	71.53
AQLM	5.33	11.04	47.44	72.43	71.63	69.24	81.88	80.58	68.15	71.43	67.10

Appendix

Table 13: Per-task results on Llama-3.1-70B-Instruct. WikiText-2 and C4 are perplexities (lower is better); the remaining columns are zero-shot accuracies (%, higher is better). Bold denotes the best result among quantized methods.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 388 papers

UniSVQ: 2-bit Unified Scalar-Vector Quantization
Jun 9, 2026Haoyu Wang, Haiyan Zhao, Xingyu Yu +4LLM Quantization Vector Quantization
Department of Computer Science and Technology, Tsinghua University, Beijing, China · School of Software and Microelectronics, Peking University, Beijing, China · School of Physics, Beihang University, Beijing, China
SQuAT: Spherical Lattice Vector Quantization-Aware Training
Oct 22, 2026Sebastian Loeschcke, Ivan Koryakovskiy, Tycho F. A. van der Ouderaa +4LLM Quantization Vector Quantization
University of Copenhagen · Qualcomm AI Research
LC-QAT: Data-Efficient 2-Bit QAT for LLMs via Linear-Constrained Vector Quantization
Jun 9, 2026Haoyu Wang, Xingyu Yu, Haiyan Zhao +2LLM Quantization Quantization-Aware Training
Department of Computer Science and Technology, Tsinghua University, Beijing, China · School of Software and Microelectronics, Peking University, Beijing, China · College of Computer Science and Technology, National University of Defense Technology, Hunan, China.

Jun 9, 2026J/K move · Enter open · S save

UniSVQ: 2-bit Unified Scalar-Vector Quantization

Haoyu Wang, Haiyan Zhao, Xingyu Yu, Zhangyang Yao+3

Department of Computer Science and Technology, Tsinghua University, Beijing, China · School of Software and Microelectronics, Peking University, Beijing, China · School of Physics, Beihang University, Beijing, China

LLM Quantization Vector Quantization Post-Training Quantization Low-Bit Quantization Neural Network Quantization

Open paper AskPDF arXiv

Post-training quantization at the 2-bit level enables low-cost deployment and inference acceleration for large language models (LLMs). Scalar quantization (SQ) and vector quantization (VQ) are two primary quantization methods, however, the former suffers from significant performance degradation, and the latter incurs computational and storage overhead. We propose UniSVQ, a unified 2-bit quantization framework that bridges scalar and vector quantization by parameterizing codewords as an affine transform of integer lattices. This structure preserves compatibility with optimized integer kernels while retaining much of VQ's flexibility. We further introduce a data-driven block-wise fine-tuning strategy to directly minimize quantization reconstruction error. Extensive experiments across multiple LLM families and zero-shot benchmarks demonstrate that UniSVQ consistently outperforms state-of-the-art SQ methods and achieves performance comparable to advanced VQ methods, while providing higher inference throughput. Codes are publicly available at https://github.com/AI9Stars/UniSVQ.

Abstract

LLM Quantization

Explore similar work

UniSVQ: 2-bit Unified Scalar-Vector Quantization

SQuAT: Spherical Lattice Vector Quantization-Aware Training

LC-QAT: Data-Efficient 2-Bit QAT for LLMs via Linear-Constrained Vector Quantization