SAGE: Pushing 2-Bit Vector-Quantized LLMs toward Real-World Usability

Xianglong Yan; Zenan Wu; Kewei Zhang; Kaicheng Yang; Yulun Zhang

cs.AIOct 10, 2026

SAGE: Pushing 2-Bit Vector-Quantized LLMs toward Real-World Usability

Authors: Xianglong Yan, Zenan Wu, Kewei Zhang, Kaicheng Yang, Yulun Zhang

Organizations: Shanghai Jiao Tong University

Abstract

Extreme low-bit weight-only quantization is an effective approach to reducing the deployment costs of large language models. Below 3 bits, scalar quantization often suffers severe accuracy loss, while vector quantization (VQ) better preserves accuracy through more expressive joint encoding. However, existing VQ methods still exhibit substantial degradation on real-world tasks, and two design challenges remain underexplored: (1) protecting sensitive weight channels within jointly encoded vectors; and (2) allocating a fixed bit budget among representation components, such as direction and radius, to suit different weight matrices. To address these challenges, we propose SAGE: Split Adaptation and Grouping by Energy for weight-only vector quantization. First, Energy-Stratified Grouping (ESG) couples channel grouping with activation-weighted fitting, grouping channels from different energy tiers within each subvector to prioritize sensitive channels. Second, Adaptive Bit Split (ABS) balances direction-codebook capacity and radius precision per matrix, selecting the split with the lowest activation-weighted reconstruction error under a fixed budget. Across benchmarks covering knowledge, mathematics, code generation, instruction following, and tool use, SAGE achieves state-of-the-art performance in the 2-bit quantization setting. On Qwen3-30B-A3B, it retains 95.3% of the BF16 model's average benchmark score. With our custom-designed CUDA kernel, SAGE delivers up to a 2.55-fold non-streaming end-to-end speedup over FP16. The code is available at https://github.com/Zincwolf/sage.

Figures & tables

Figure 1: Accuracy and inference efficiency on Qwen3-4B-Instruct-2507. Left: generative scores, with independent zero-based axes. Right: batch-one non-streaming first-token latency versus output throughput on an RTX 4090; upper left is better. Throughput includes prompt processing. P/D denote input/output token counts. Quantized weights are reconstructed on demand.

Figure 2: Overview of SAGE. Spherical Vector Quantization encodes each weight subvector using signs, a direction index, and a quantized radius. Energy-Stratified Grouping (ESG) uses calibration activations to group channels across energy tiers within subvectors and protect sensitive channels through weighted fitting. Adaptive Bit Split (ABS) compares activation-weighted reconstruction errors to select a direction-index/radius bit split per matrix, using 18 bits per eight weights. Blockwise Optimization updates radius and codebook angles to match full-precision block outputs.

Figure 3: ESG analysis on Qwen3-4B-Instruct-2507: (a) within-subvector energy variation, (b) high-energy channel errors, and (c) total error contributions.

Qwen3-4B-Instruct-2507
Method	BPW	MMLU-R ∗	GSM8K	MATH-500	IFEval	HumanE+ ∗	BFCL-v3	Average
BF16	16.00	81.28	94.01	93.80	84.66	82.32	59.94	82.67
QuIP#	2.00	64.82	70.96	34.40	61.00	32.32	39.74	50.54
QTIP	2.00	72.11	81.73	61.00	74.49	55.49	51.33	66.03
UniSVQ	2.00	42.89	15.62	2.40	24.21	4.27	29.73	19.85
SPHQuant	2.25	66.93	81.35	66.40	80.04	57.32	39.48	65.25

Table 1: Weight-only vector quantization results on six benchmarks (higher is better). Bold indicates the best quantized result for each model and metric. BF16 denotes the full-precision model.

ESG	ABS	MMLU-R ∗	GSM8K	MATH-500	IFEval	HumanE+ ∗	BFCL-v3	Average
✗	✗	66.93	81.35	66.40	80.04	57.32	39.48	65.25
✓	✗	69.05	82.18	68.40	80.04	54.88	45.17	66.62
✗	✓	71.33	84.84	74.40	79.85	58.54	44.02	68.83
✓	✓	73.47	88.70	78.80	82.81	59.76	53.33	72.81

Table 2: ESG and ABS ablation on Qwen3-4B-Instruct-2507. Best results are in bold .

Figure 4: Sensitivity to (a) the energy exponent

\gamma

and (b) calibration set size on Qwen3-4B-Instruct-2507. Scores are averaged over six benchmarks covering diverse generative capabilities.

Figure 5: Inference efficiency. Qwen3-4B-Instruct-2507, RTX 4090 (non-streaming, batch 1). P/D: input/output tokens. Output throughput includes prefill; weights are reconstructed on demand. Packed model sizes include unquantized weights; AWQ uses W4G128 for speed benchmarks.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Kernel data paths and the 64-row GEMM loop. $x_{\mathrm{sub}}$ contains eight reordered activations. In (b), $j\in\{0,1\}$ selects a half-group, and mma adds $AB_{j}^{\top}$ to acc .

Panel of Figure 7 — Figure 7: Inference timing for $D>1$ output tokens. $t_{\mathrm{obs}}$ marks CPU observation of the first token.

Method	First token (ms)	Next token (ms)	Total (s)	Output (tokens/s)	Peak (GiB)
FP16	14.28	9.869	1.268	101.0	7.665
AWQ	13.18	4.115	0.536	238.7	2.611
QuIP#	51.96	5.719	0.779	164.4	1.714
QTIP	48.35	5.638	0.765	167.3	1.898
UniSVQ	21.53	4.159	0.550	232.7	1.714
SPHQ	125.13	3.699	0.595	215.1	1.921

Appendix

Table 3: Short non-streaming workload: 128 input and 128 output tokens, without retained weight copies. First token is GPU TTFT; next token is CPU-observation-based TPOT, each reported as a median over runs. SAGE achieves a

2.63\times

end-to-end speedup over FP16.

Method	Prefill tokens	Prefill relative L2 (%)	Decode maximum relative L2 (%)	Maximum absolute error
Dense	4	0.1192	0.2266	0.01660
SPHQuant	4	0.8494	1.0412	0.16016
QuIP#	4	0.8565	1.2647	0.21484
QTIP	4	1.0206	1.3674	0.19336
UniSVQ	4	5.7379	7.5981	1.02002
SAGE	4	0.6233	0.7746	0.11914

Appendix

Table 4: End-to-end numerical consistency. Lower relative L2 is better.

Prompt 1 Five numbers have an average of 18. If the number 22 is removed, what is the average of the remaining four numbers? Choices: A. 16 B. 17 C. 18 D. 19. Please show your choice in the answer field with only the choice letter, e.g., {"answer": "C"}.

BF16: The user wants me to solve a math problem and provide the answer in a specific format. 1. Understand the Problem: There are 5 numbers. The average (arithmetic mean) of these 5 numbers is 18. One number, which is 22, is removed from the set. I need to find the new average of the remaining 4 numbers. 2. Formulate Equations: Let the five numbers be

x_{1},x_{2},x_{3},x_{4},x_{5}

. The sum of the five numbers is

S_{5}

. Average formula:

\text{Average}=\frac{\text{Sum}}{\text{Count}}

Given:

\frac{S_{5}}{5}=18

. Therefore,

S_{5}=18\times 5

. Calculate

S_{5}

18\times 5=90

. So, the sum of the original 5 numbers is 90. Now, one number is removed. This number is

x_{k}=22

. The sum of the remaining 4 numbers is

S_{4}=S_{5}-x_{k}

. Calculate

S_{4}

S_{4}=90-22=68

. The count of the remaining numbers is 4. The new average is

\text{Average}_{new}=\frac{S_{4}}{4}

. Calculate

\text{Average}_{new}

\frac{68}{4}

68/4=17

. 3. Check the Choices: A. 16 B. 17 C. 18 D. 19 My calculated answer is 17, which corresponds to Choice B. 4. Structure the Output: State the steps clearly. Conclude with the final choice. Format as requested: {"answer": "B"}. Double Check: Original Sum =

5\times 18=90

. Remove 22 (TRUNCATED)

QuIP#: The user wants me to solve a math problem and provide the answer in a specific format. 1. Understand the problem : There are five numbers. The average of these five numbers is 18. One number, specifically 22, is removed. The remaining four numbers must have an average value. I need to determine what that average is from the given choices (A=16, B=17, C=18, D=19). 2. Analyze the data : Let

n

be the number of items in a set. Let

x_{1},x_{2},...,x_{n}

be the elements. The formula for the average is:

\text{average}=\frac{\text{sum}}{n}

. Here, we have 5 numbers whose sum equals

5\times 18=90

. When 22 is removed, there are 4 numbers left. The new sum is 90 - 22 = 68. The new average would be

68/4=17

. 3. Wait, let me check my reasoning. If the original average is 18, with 5 numbers, their total sum is 90. If one number, 22, is removed, the remaining 4 numbers sum to

90-22=68

. Average of the remaining 4 numbers =

68/4=17

. This seems like a straightforward calculation based on averages. However, does this logic work? The problem doesn’t say “one of the numbers was 22”. It says “the number 22 is removed”. Since there were 5 numbers with an average of 18, the sum is indeed 90. Is it possible for the average of the remaining 4 numbers to still be 18? The question asks which choice is correct. If the sum is 90, removing 22 leaves 68. 68 divided by 4 is 17. So the answer should be 17. 4 (TRUNCATED)

QTIP: Thinking Process: 1. Calculate total sum of the initial five numbers: Since the average is 18 and there are 5 numbers, the total sum is

18\times 5=90

. 2. Remove the number 22 from the set. The remaining four numbers have a sum of

90-22=68

. 3. Calculate the new average: Divide the remaining sum by the count of remaining numbers (4). New average =

68/4=17

. 4. Match with choices: Option B corresponds to 17. 5. Format output: Provide the choice letter in the answer field as requested. Step-by-Step Calculation: Sum of original numbers = Average

\times

Count =

18\times 5=90

. After removing 22, Sum =

90-22=68

. Remaining count = 4. New Average =

68/4=17

. Choice B matches 17. Answer should be “B”. {"answer": "B"}

UniSVQ: The user wants me to know the average of the remaining 4 numbers after removing the number 22. The five original numbers were used to get an average of 18. What are the implications? A. It’s about a math problem where I can’t remember what it was when I was at school or work? B. It’s about statistics: What is the mean of a series? C. It’s a question that needs to be answered without knowing the answer. D. It could be related to the average value being something else entirely. E. The question might have nothing to do with statistics, other than the fact that the average is calculated differently in some contexts. (NO FINAL)

SPHQuant: The user wants me to solve a math problem about averages. 1. Analyze the given information: There are five numbers. Their average is 18. One of the numbers (22) is removed. I need to find the average of the remaining four numbers. 2. Formulate the relationship between average and sum: Average = Sum / Count Sum = Average * Count 3. Calculate the total sum of the original five numbers: Count = 5 Average = 18 Sum = 5 * 18 4. Perform the multiplication:

5\times 18=90

5. Determine the sum of the remaining four numbers: Original Sum - Removed Number Sum = 90 - 22

90-22=68

6. Calculate the new average: New Average = Sum of remaining numbers / Count of remaining numbers New Count = 4 New Average = 68 / 4 7. Perform the division:

68/4=17

8. Verify with the choices: A. 16 B. 17 C. 18 D. 19 The calculated result matches choice B. 9. Format the output: The user requested the answer in the answer field with only the choice letter. Format: {"answer": "B"} 10. Final check: Original sum:

5\times 18=90

. Remove 22:

90-22=68

. New count:

5-1=4

. New average:

68/4=17

. Choice is B. Constraint: “Please show your choice in the answer field with only the choice (TRUNCATED)

SAGE: The user wants me to solve a math problem and provide the answer in a specific format. Problem: Five numbers have an average of 18. One number (22) is removed. What is the average of the remaining four numbers? Step 1: Calculate the sum of the original five numbers. Average = Sum / Count

18=\text{Sum}/5

\text{Sum}=18\times 5

\text{Sum}=90

Step 2: Calculate the sum of the remaining four numbers. The number 22 is removed. New Sum = Original Sum - 22 New Sum =

90-22

New Sum = 68 Step 3: Calculate the new average. New Average = New Sum / New Count New Average =

68/4

New Average = 17 Step 4: Format the output. The user requested the choice letter in the answer field. Choices: A. 16, B. 17, C. 18, D. 19. My calculated answer is 17, which corresponds to Choice B. The format requires {"answer": "B"}. Double Check: Original sum = 90. Remove 22 -> Remainder sum = 68. Count = 4. Average = 68 / 4 = 17. Matches Choice B. Final Output Generation: Construct the JSON response. The sum of the five numbers is

18\times 5=90

. After removing 22, the new sum is

90-22=68

. The average of the remaining four numbers is

68/4=17

. This corresponds to Choice B. {"answer": "B"}

Appendix

Table 5: Qwen3.5-4B dialogues with native BF16 or quantized–dequantized (QDQ) weights. Both prompts use seed 42, temperature 1.0, and thinking mode. Generation limits, including reasoning, are 512 tokens for Prompt 1 and 256 for Prompt 2. Gray text denotes reasoning.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 388 papers

Jun 3, 2026

Minimizing the Hidden Cost of Scales: Graph-Guided Ultra-Low-Bit Quantization for Large Language Models

Post-training quantization (PTQ) is critical for the efficient deployment of large language models (LLMs). Recent ultra-low-bit PTQ methods rely on rigid weight-saliency assumptions or position heuristics, introducing substantial hidden scaling overhead. We propose SAGE-PTQ (Saliency-Aware Graph-guided Efficient PTQ), a novel ultra-low-bit quantization framework for LLMs that minimizes hidden scaling cost. SAGE-PTQ separates salient and unsalient weights using distributional statistics, then models subsampled unsalient weights as a sparse graph to estimate the optimal number of groups per layer. SAGE-PTQ applies dual-mode quantization, assigning multi-bit precision to salient weights and binarizing unsalient weights. To reduce scaling overhead, SAGE-PTQ uses one per-channel scale for salient weights and one scalar per unsalient group. Finally, SAGE-PTQ implements adaptive saliency thresholding to select the optimal saliency ratio per matrix. SAGE-PTQ achieves 1.03 weight bits and only 0.004 scaling bits per matrix on average, outperforming state-of-the-art methods such as BiLLM and PB-LLM. On LLaMA-3-8B, SAGE-PTQ achieves 6.74 WikiText2 perplexity, compared to 55.8 for BiLLM, while using less than 50% of BiLLM's GPU memory. On LLaMA-2-70B, SAGE-PTQ provides 1.5x faster decoding on one NVIDIA L40 GPU, demonstrating practical inference efficiency.

Rayyan Abdalla, Amir Hussein, Min Wu +1

LLM Quantization LLM Inference Acceleration Weight-Only Quantization Post-Training Quantization Low-Bit Quantization

Ask PDF

Jun 9, 2026

UniSVQ: 2-bit Unified Scalar-Vector Quantization

Post-training quantization at the 2-bit level enables low-cost deployment and inference acceleration for large language models (LLMs). Scalar quantization (SQ) and vector quantization (VQ) are two primary quantization methods, however, the former suffers from significant performance degradation, and the latter incurs computational and storage overhead. We propose UniSVQ, a unified 2-bit quantization framework that bridges scalar and vector quantization by parameterizing codewords as an affine transform of integer lattices. This structure preserves compatibility with optimized integer kernels while retaining much of VQ's flexibility. We further introduce a data-driven block-wise fine-tuning strategy to directly minimize quantization reconstruction error. Extensive experiments across multiple LLM families and zero-shot benchmarks demonstrate that UniSVQ consistently outperforms state-of-the-art SQ methods and achieves performance comparable to advanced VQ methods, while providing higher inference throughput. Codes are publicly available at https://github.com/AI9Stars/UniSVQ.

Haoyu Wang, Haiyan Zhao, Xingyu Yu +4

Department of Computer Science and Technology, Tsinghua University, Beijing, China · School of Software and Microelectronics, Peking University, Beijing, China · School of Physics, Beihang University, Beijing, China

LLM Quantization Vector Quantization Post-Training Quantization Low-Bit Quantization Neural Network Quantization

Ask PDF

Jun 9, 2026

LC-QAT: Data-Efficient 2-Bit QAT for LLMs via Linear-Constrained Vector Quantization

Quantization-aware training (QAT) is essential for extremely low-bit large language models (LLMs). Current QAT methods are mainly based on scalar quantization (SQ), which enables efficient optimization but suffers from severe performance degradation at 2-bit precision. On the other hand, vector quantization (VQ) provides substantially higher representational capacity, but its discrete codebook lookup prevents end-to-end training. We propose LC-QAT, a 2-bit weight-only VQ-QAT framework that represents quantized weights via a learned affine mapping over discrete vectors, which yields a high-quality PTQ initialization and enables fully differentiable end-to-end optimization without explicit codebook lookup in the training forward pass. This strong post-training initialization makes LC-QAT highly data-efficient. Experiments across diverse LLMs demonstrate that LC-QAT consistently outperforms state-of-the-art QAT methods while using only 0.1%--10% of the training data. Our results establish LC-QAT as a practical and scalable solution for extreme low-bit model deployment. Codes are publicly available at https://github.com/AI9Stars/UniSVQ.

Haoyu Wang, Xingyu Yu, Haiyan Zhao +2

Department of Computer Science and Technology, Tsinghua University, Beijing, China · School of Software and Microelectronics, Peking University, Beijing, China · College of Computer Science and Technology, National University of Defense Technology, Hunan, China.

LLM Quantization Quantization-Aware Training Vector Quantization Weight-Only Quantization Neural Network Quantization Data-Efficient Learning

Ask PDF

Abstract

LLM Quantization

Explore similar work

Minimizing the Hidden Cost of Scales: Graph-Guided Ultra-Low-Bit Quantization for Large Language Models

UniSVQ: 2-bit Unified Scalar-Vector Quantization

LC-QAT: Data-Efficient 2-Bit QAT for LLMs via Linear-Constrained Vector Quantization