TernaryLLM: Ternarized Large Language Model

Tianqi Chen; Zhe Li; Weixiang Xu; Zeyu Zhu; Dong Li; Lu Tian; Emad Barsoum; Peisong Wang

cs.LGJun 11, 2024

TernaryLLM: Ternarized Large Language Model

Authors: Tianqi Chen, Zhe Li, Weixiang Xu, Zeyu Zhu, Dong Li, Lu Tian, Emad Barsoum, Peisong Wang,

Organizations: Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Advanced Micro Devices, Inc., Beijing, China · AIRIA

Abstract

Large language models (LLMs) have achieved remarkable performance on Natural Language Processing (NLP) tasks, but they are hindered by high computational costs and memory requirements. Ternarization, an extreme form of quantization, offers a solution by reducing memory usage and enabling energy-efficient floating-point additions. However, applying ternarization to LLMs faces challenges stemming from outliers in both weights and activations. In this work, observing asymmetric outliers and non-zero means in weights, we introduce Dual Learnable Ternarization (DLT), which enables both scales and shifts to be learnable. We also propose Outlier-Friendly Feature Knowledge Distillation (OFF) to recover the information lost in extremely low-bit quantization. The proposed OFF can incorporate semantic information and is insensitive to outliers. At the core of OFF is maximizing the mutual information between features in ternarized and floating-point models using cosine similarity. Extensive experiments demonstrate that our TernaryLLM surpasses previous low-bit quantization methods on the standard text generation and zero-shot benchmarks for different LLM families. Specifically, for one of the most powerful open-source models, LLaMA-3, our approach (W1.58A16) outperforms the previous state-of-the-art method (W2A16) by 5.8 in terms of perplexity on C4 and by 8.2% in terms of average accuracy on zero-shot tasks.

Figures & tables

Figure 1: An example of the features in the 23rd decoder layer to illustrate the problems incurred by extreme low-bit quantization. The first and second lines correspond to the float-point and quantized models, respectively. Extreme low-bit quantization leads to severe information loss in pretrained LLMs, including a narrowed feature representation range (Figure 1 (a)), loss of prominence in dominant channels (Figure 1 (b)), and disruption of the semantic clustering of related words (Figure 1 (c) and (d)).

Figure 2: The weights in certain groups display noticeable asymmetric outliers and a non-zero mean distribution.

Figure 3: Feature knowledge distillation results for LLaMA-1-7B. Cosine similarity is less sensitive to outliers in features compared to MSE. (a) Ground truth loss of the training. (b) Feature knowledge distillation loss of the training. (c) The reasons for severe oscillations in MSE distillation.

Method	#W	#G	PPL $\downarrow$			Zero-Shot $\uparrow$
Method	#W	#G	WikiText2	C4	PTB	PIQA	ARC-e	ARC-c	HellaSwag	Wino	Avg.
LLaMA-3	16	-	6.1	9.2	10.6	79.9	80.1	50.4	60.2	72.8	68.6
RTN	2	128	1.9E3	2.5E4	1.8E4	53.1	24.8	22.1	26.9	53.1	36.0
RTN	2	-	2.7E6	7.4E6	3.1E6	53.1	24.7	21.9	25.6	51.1	35.3
GPTQ	2	128	2.1E2	4.1E4	9.1E2	53.9	28.8	19.9	27.7	50.5	36.2
GPTQ	2	-	5.7E4	1.0E5	2.7E5	52.8	25.0	20.5	26.6	49.6	34.9

Table 1: Evaluation results of weight-only quantization on the LLaMA-3-8B model. #W indicates weight bits, #G indicates group size, and “-” denotes per-channel quantization.

Method	#W	#G	OPT-125M		OPT-1.3B		OPT-2.7B		OPT-6.7B
Method	#W	#G	WikiText2	C4	WikiText2	C4	WikiText2	C4	WikiText2	C4
FP	16	-	27.65	24.60	14.63	14.72	12.47	13.16	10.86	11.74
RTN	2	64	7.0e3	3.9e3	1.0e4	7.3e3	19.3e4	1.2e5	7.6e3	6.3e3
RTN	2	64	7.0e3	5.0e3	1.0e4	7.7e3	19.3e4	3.8e4	7.6e3	5.2e3
GPTQ	2	64	204.40	133.51	49.58	31.31	29.37	23.23	16.81	16.24
GPTQ	2	128	597.66	597.66	115.16	60.88	61.59	33.83	20.18	18.55

Table 2: C4 and WikiText2 perplexity

\downarrow

of different methods in OPT models. #W indicates weight bits, #G indicates group size, and “-” denotes per-channel quantization.

Method	#W	#G	PPL $\downarrow$			Zero-Shot $\uparrow$
Method	#W	#G	WikiText2	C4	Avg.	PIQA	ARC-e	ARC-c	HellaSwag	Wino	Avg.
LLaMA-2	16	-	5.47	6.97	6.22	76.99	53.58	40.53	72.96	67.25	62.26
GPTQ	2	64	8.97	13.25	11.11	68.39	42.13	31.91	54.64	58.64	51.14
AWQ	2	64	2.06e5	1.54e5	1.8e5	50.00	26.52	26.79	26.14	49.64	35.82
OmniQuant	2	64	9.64	12.73	11.19	68.72	39.77	30.89	53.44	56.12	49.79
PB-LLM	2	64	20.37	44.88	32.63	55.22	29.88	22.01	30.49	50.36	37.59

Table 3: Evaluation results of weight-only quantization on the LLaMA-1-7B and LLaMA-2-7B model. #W indicates weight bits, #G indicates group size, and “-” denotes per-channel quantization.

LLaMA-7B PPL	FP	TWN	DLT
OPT-1.3B	14.63	22.32	20.83
LLaMA-7B	5.68	10.10	9.21

Table 4: Comparison of various ternarization methods on OPT-1.3B and LLaMA-7B model.

Figure 8

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Model	OPT-125M	OPT-1.3B	OPT-2.7B	OPT-6.7B	LLaMA-7B
Learning rate	1e-4	5e-5	1e-4	5e-5	1e-4
Logits KD loss coefficient	0.001	0.001	0.001	0.001	0.001
Feature KD loss coefficient	10	10	10	10	5
Feature KD number layers	6	18	18	18	18

Appendix

Table 8: Training details for different models. We use the same training parameters for LLaMA-1, LLaMA-2 and LLaMA-3.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 362 papers

CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs
Jun 25, 2026Shigeng Wang, Chao Li, Yangyuxuan Kang +2LLM Quantization Ternary Quantization
1Intel Labs China
TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantization
Jun 11, 2026Zhixiong Zhao, Zukang Xu, Zhixuan Chen +3Mixed-Precision Quantization LLM Quantization
Houmo AI, China · 2Southeast University, Nanjing, China
LBLLM: Lightweight Binarization of Large Language Models via Three-Stage Distillation
Apr 21, 2026Siqing Song, Chuang Wang, Yong Lang +2LLM Quantization Activation Quantization
MAIS, Institute of Automation, Chinese Academy of Sciences, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, China · Central Media Technology Institute, Huawei

Jun 25, 2026J/K move · Enter open · S save

CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs

Shigeng Wang, Chao Li, Yangyuxuan Kang, Jiawei Fan+1

1Intel Labs China

LLM Quantization Ternary Quantization LLM Compression LLM Inference Acceleration Post-Training Quantization Neural Network Quantization

Open paper AskPDF arXiv

In this paper, we present CAT-Q, Cost-efficient and Accurate Ternary Quantization, for compressing and accelerating LLMs. Unlike existing state-of-the-art ternary quantization methods that rely on data-intensive and costly quantization-aware training to mitigate severe performance degradation, CAT-Q is a simple yet effective post-training quantization scheme that is readily applicable to LLMs with diverse architectures and model sizes. It has two key components, learnable modulation (LM) and softened ternarization (ST), which are coupled from an optimization perspective. LM leverages a composition of learnable factors to modulate the distribution of pre-trained high-precision weights and the ternary threshold, making them less sensitive to ternarization. ST further introduces a differentiable transition function to guide the ternarization process toward stable convergence. We show that, for pre-trained LLMs with 1.7B to 8B parameters, CAT-Q can efficiently quantize them into ternary models using only 512 calibration samples, while achieving superior performance than the seminal BitNet 1.58-bit v1 and v2 families (with 1.3B to 7B parameters) trained with 100B tokens, yielding about a 100,000X reduction in training tokens. Moreover, we show for the first time that CAT-Q can quantize much larger pre-trained LLMs having 14B to 235B parameters into leading ternary models within just 8 to 60 hours on 8 A100-80GB GPUs. Code is available at https://github.com/IntelChina-AI/BitTern.

Abstract

LLM Quantization

Explore similar work

CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs

TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantization

LBLLM: Lightweight Binarization of Large Language Models via Three-Stage Distillation