Low-Rank Ternary Adaptation for Fine-Tuning Transformers

Alexandru-Dragos Manolache; Yunqiang Li; Jan van Gemert

cs.CVAug 25, 2026

Low-Rank Ternary Adaptation for Fine-Tuning Transformers

Authors: Alexandru-Dragos Manolache, Yunqiang Li, Jan van Gemert

Organizations: Delft University of Technology, The Netherlands · Amazon Development Center

Abstract

Ternary transformers offer extreme memory and compute efficiency, but existing low-bit LoRA-based methods cannot directly fine-tune ternary weights. Current approaches either require dequantization, restoring low-bit base weights to higher precision to merge with adaptation weight, or update only quantization parameters, preventing a merged model that remains ternary. We propose ternary multiplicative adaptation, which represents discrete updates of ternary weights such as sign flips or zeroing through a low-rank Kronecker factorization into two small ternary matrices applied element-wise to ternary weights. This design is parameter-efficient and expressive, preserves the ternary domain, and supports direct merging without dequantization. Experiments on six models across language and vision, including ternarized LLaMA-3 1B and 3B and a ternary ViT-B/16, demonstrate that our method recovers much of the performance lost to quantization and outperforms strong low-bit and ternary baselines. Code is available at https://github.com/alexmanoo/ternary_adaptation.

Figures & tables

Property	QLoRA	QA-LoRA	Ours
Base weights ternary	✓	✗	✓
Dequant-free merge	✗	✓	✓
Ternary merged model	✗	✗	✓
Update type	Additive	Additive	Multiplicative
Applied on	FP weights	Quantizer	Ternary weights

Table 1: Comparison of QLoRA, QA-LoRA, and our ternary multiplicative adapter. Each column indicates whether a method satisfies key properties needed for adapting ternary models. Base weights ternary asks if the backbone can be stored in a ternary format during fine-tuning. Dequant-free merge asks whether the adaptation can be merged directly into the base-weights while keeping the low-bit quantization. Ternary merged model indicates whether the final weights remain ternary after merging. The bottom rows describe how the adaptation is applied: QLoRA adds a low-rank update to dequantized floating-point weights, QA-LoRA adds a low-rank update to quantization parameters, and our method applies a multiplicative ternary adaptation directly to ternary weights. Only our method both operates natively on ternary backbones and produces a merged model that stays ternary while avoiding any dequantization step.

Panel of Figure 1 — Figure 1: Visualization of our ternary multiplicative adapter. Ternary Multiplicative Update (top): a ternary mask $\mathbf{\Delta}_{\text{tern}}\in\{-1,0,1\}^{d_{\text{out}}\times d_{\text{in}}}$ is applied element-wise (Hadamard product) to the pretrained ternary weights $\mathbf{W}_{\text{tern}}$ to obtain merged ternary weights $\mathbf{W}_{\text{merged}}$ , where each entry $\mathbf{\Delta}_{ij}\in\{-1,0,1\}$ respectively keeps, zeros, or flips the sign of $\mathbf{W}_{\text{ij}}$ . Kronecker-structured Ternary Adaptation (bottom): the adaptation $\mathbf{\Delta}_{\text{tern}}$ is constructed as a Kronecker product $\mathbf{A}\otimes\mathbf{B}$ of two small ternary factors; the toy example shows how each entry of $\mathbf{A}$ scales a copy of $\mathbf{B}$ to form a larger ternary matrix. Our adapter uses a compact Kronecker-structured mask to implement discrete keep/zero/flip operations directly on ternary weights, enabling a mergeable fine-tuned model that remains fully ternary.

Method	Base Weight (frozen)	Trainable Update	Merged Weight
LoRA	$\mathbf{W}_{\text{0}}$ (FP)	$\mathbf{\Delta W}=\frac{\alpha}{r}\mathbf{B}\mathbf{A}$	$\begin{aligned} \mathbf{W_{\text{merged}}}&=\mathbf{W}_{\text{0}}+\frac{\alpha}{r}\mathbf{B}\mathbf{A},\end{aligned}$ FP
QLoRA	Per-tensor quantized $\mathbf{W}_{\text{q}}$	$\mathbf{\Delta W}=\frac{\alpha}{r}\mathbf{B}\mathbf{A}$	$\begin{aligned} \mathbf{W_{\text{merged}}}&=\text{Dequant}(\mathbf{W}_{\text{q}})+\frac{\alpha}{r}\mathbf{B}\mathbf{A},\end{aligned}$ FP
QA-LoRA	Group-wise quantized $\mathbf{W}_{\text{0,g}}=s_{g}(\mathbf{Q}_{\text{g}}-\mathbf{Z}_{\text{g}})$	$\begin{aligned} &\text{Low-rank update to zero-point matrix:}\\ &\mathbf{\Delta Z_{g}}=\frac{\alpha}{r}\mathbf{B}\mathbf{A}\end{aligned}$	$\begin{aligned} \mathbf{W_{\text{merged,g}}}&=s_{g}\big(\mathbf{Q}_{\text{g}}-(\mathbf{Z}_{\text{g}}+\mathbf{\Delta Z}_{\text{g}})\big)\\ &=\mathbf{W}_{\text{0,g}}-s_{g}\,\mathbf{\Delta Z}_{\text{g}},\end{aligned}$ INT4
Ours	Ternary $\mathbf{W}_{\text{tern}}\in\{-1,0,1\}^{d_{\text{out}}\times d_{\text{in}}}$	$\mathbf{\Delta W_{\text{tern}}}=\mathbf{A}\otimes\mathbf{B}$	$\begin{aligned} \mathbf{W_{\text{merged}}}&=\mathbf{W}_{\text{tern}}\odot(\mathbf{A}\otimes\mathbf{B}),\end{aligned}$ Ternary

Table 2: LoRA, QLoRA, QA-LoRA and Ours. Each column specifies the frozen base weights, the trainable low-rank adaptation, and the merged weights. LoRA keeps a full-precision base matrix

\mathbf{W}_{\text{0}}

and adds a low-rank update

\mathbf{\Delta W}=\frac{\alpha}{r}\mathbf{B}\mathbf{A}

in floating point. QLoRa stores a per-tensor quantized weight

\mathbf{W}_{\text{q}}

, dequantizes it, then adds the same low-rank update in full precision. QA-LoRA assumes group-wise affine quantization

\mathbf{W}_{\text{0,g}}=s_{g}(\mathbf{Q}_{\text{g}}-\mathbf{Z}_{\text{g}})

and applies the low-rank update to the zero points

\mathbf{Z}_{\text{g}}

; using group size

1

reduces to a per-tensor quantization. Our method starts from ternary base weights

\mathbf{W}_{\text{tern}}\in\{-1,0,1\}^{d_{\text{out}}\times d_{\text{in}}}

, represents the adaptation as a Kronecker product, and merges it multiplicatively, so the final weights remain ternary. The first three methods rely on additive updates in a continuous space, whereas our method keeps the same weight space, such that the merged model stays ternary without needing any dequantization or requantization.

Method	Train. params.	Grads.	Opt.	Acts.
Full-FT (FP32)	$4d^{2}$	$4d^{2}$	$8d^{2}$	$2\,bsd$
LoRA (rank $r$ )	$8dr$	$8dr$	$16dr$	$2\,bsd{+}2\,bsr$
Ours (general)	$4(pq{+}rs)$	$4(pq{+}rs)$	$8(pq{+}rs)$	$2\,bsd$
Ours (square)	$8d$	$8d$	$16d$	$2\,bsd$

Table 3: Per-layer training memory, Bytes (B). Adapters are FP32: 4 B/param, 4 B/grad, 8 B/Adam. Quantized versions are recomputed during each forward pass, gradients are computed using a straight-through estimator. Activations are FP16 at 2 B/element. For a

\mathbf{W}\!\in\!\mathbb{R}^{d\times d}

; General :

\bar{\mathbf{A}}\!\in\!\mathbb{R}^{p\times q}

\bar{\mathbf{B}}\!\in\!\mathbb{R}^{r\times s}

s.t.

p\!\cdot\!r\!=\!q\!\cdot\!s\!=\!d

. Square :

p\!=\!q\!=\!r\!=\!s\!=\!\sqrt{d}

. Our adaptation scales as

O(d)

memory in the square case, versus

O(d^{2})

for full fine-tuning and

O(dr)

for LoRA; matching our expressivity would require LoRA rank

r\!=\!d

Model	Method	Precision	ARC-c	ARC-e	BoolQ	ComQA	HellaSwag	MMLU	OBQA	PiQA	Wino.	Avg.	WikiText-2 PPL $\downarrow$
Llama 3.2-1B	Full Precision	16-bit	31.4	65.2	63.6	47.0	47.7	36.7	26.4	74.6	59.8	50.2	9.7
	PTQ Baselines
	RTN	2-bit	22.8	23.8	62.1	19.2	25.5	22.9	17.2	53.1	48.9	32.8	1.5e6
	GPTQ	2-bit	20.2	31.6	43.1	19.3	26.3	24.1	13.4	55.2	50.0	31.4	1.7e2
	SpinQuant	2-bit	18.3	37.6	62.0	19.5	28.7	22.9	13.4	57.4	53.2	34.7	43.3
	SpinQuant	1.58-bit	20.3	29.7	61.3	20.1	27.0	22.9	13.8	54.5	49.2	33.2	86.5

Table 4: Main results on Llama 3.2-1B and Llama 3.2-3B. Our 1.58-bit method is compared against full-precision (FP) and several post-training quantization (PTQ) baselines. All evaluations are zero-shot. Best results among 1.58-bit methods are in bold . Averages exclude WikiText-2 PPL (

\downarrow

). Our ternary multiplicative adaptation consistently recovers much of the accuracy lost to ternarization and often matches or surpasses 2-bit baselines, while roughly halving the PPL of the ternary SpinQuant backbones.

Shape $\mathbf{W}_{\text{tern}}$ ( $m\times n$ ) $\to$	Adaptation dims $(p\!\times\!q,\ r\!\times\!s)$
$(2048\times 2048)$ $\to$	$(32\!\times\!32,\ 64\!\times\!64)$
$(2048\times 512)$ $\to$	$(32\!\times\!16,\ 64\!\times\!32)$
$(2048\times 8192)$ $\to$	$(32\!\times\!64,\ 64\!\times\!128)$
$(8192\times 2048)$ $\to$	$(64\!\times\!32,\ 128\!\times\!64)$

Table 5: Balanced factorization for self-attention and feed-forward Neural Network weights shapes (excluding the

lm\_head

) for Llama-3.2-1B layers. The factors

(p,q)

and

(r,s)

satisfy

m{=}p\!\cdot\!r

and

n{=}q\!\cdot\!s

, yielding balanced submatrices for all layer shapes. This factorization keeps the number of trainable adapter parameters tiny (about

0.06\%

of the model) while still allowing the multiplicative update to cover every weight in each layer, enabling expressive adaptations at small parameter cost.

Method	Bits (Base/Adapter)	Avg. $\uparrow$	PPL $\downarrow$
QLoRA (requantized)	1.58 / 1.58	37.5	22.9
Ours (merged ternary)	1.58 / 1.58	38.3	22.3

Table 6: Llama-3.2-3B fine-tuning results on Alpaca dataset, compared to QLoRA requantization, showing average accuracy over 9 tasks and WikiText-2 perplexity. QLoRA (requantized) fine-tunes with full-precision adapters but then merges and requantizes the resulting weights back to ternary, incurring a drop in accuracy and PPL, whereas our method works natively in the ternary domain and achieves better performance.

Model	#fewshot	Baseline	Ours	Improvement
BitNet 2B 4T	4	60.1	63.0	+2.9
Falcon Edge 1B	5	52.0	55.1	+3.1
Falcon Edge 3B	5	65.4	66.4	+1.0

Table 7: GSM8K exact-match accuracy (EM, %) for pre-trained ternary models fine-tuned with our adaptation. All models use 1.58-bit weights and are fine-tuned on GSM8K with only the adaptation parameters updated. Overall, our method improves

+1

+3

EM points over the frozen ternary backbones, demonstrating that strongly quantized models can be adapted to downstream math reasoning tasks without increasing inference-time cost.

Method	Bits (Base/Adapter)	Top-1 acc. $\uparrow$
Full-FT	1.58	85.7
QLoRA	1.58 / 16	85.3
QLoRA (requantized)	1.58 / 1.58	78.9
Ours (merged ternary)	1.58 / 1.58	83.0

Table 8: Ternary ViT-B/16 (encoder from TernaryCLIP [ 60 ] ) fine-tuned on ImageNet-100. Full-FT updates all ternary base weights with STE. QLoRA trains full-precision adapters on the ternary backbone, and QLoRA (requantized) additionally merges and requantizes to a single ternary ViT. While QLoRA can match full fine-tuning, merging and requantizing causes a large accuracy drop. Our method avoids this requantization performance drop by fine-tuning and merging natively in the ternary domain.

		Final value (millions of weights)
Init.	From	$-1$	$0$	$+1$
All-ones	$-1$	255.1	4.1	0.0
	$0$	0.0	454.8	0.0
	$+1$	0.0	4.1	255.0
Balanced	$-1$	127.5	4.1	127.5
	$0$	0.0	454.8	0.0

Table 9: Element-wise ternary weight transitions from the SpinQuant Llama-3.2-1B backbone (

\mathbf{W}_{\text{tern}}

) to the merged model (

\mathbf{W}_{\text{merged}}

) after training, for different initialization schemes. Each row shows, in millions of weights, how many parameters with a given initial value ( from ) end up at each final ternary state. The

0

row is identical across all initializations because our multiplicative update cannot reactivate pruned (

0

) weights, so all accuracy gains come from redistributing signs among non-zero weights.

Init.	Unchanged (%)	$-1\leftrightarrow+1$ (%)	NZ $\rightarrow 0$ (%)	Flip in NZ (%)
All-ones	99.16	0.00	0.84	0.00
Balanced	72.95	26.21	0.85	49.21
Normalized	72.76	26.02	1.22	48.85

Table 10: Summary of weight transformations for Llama-3.2-1B. Percentages are computed over all

N{=}973{,}078{,}528

ternary weights. “Unchanged” counts entries whose initial and final values are identical. “

-1\leftrightarrow+1

” counts sign changes between the two non-zero states. “NZ

\rightarrow 0

” counts pruned non-zero weights. “Flip in NZ” reports the fraction of originally non-zero weights that flip sign after adaptation.

Initialization	Avg. $\uparrow$	PPL $\downarrow$
Balanced	35.3	44.6
All-ones	35.1	45.7
Normalized	34.9	52.0

Table 11: Initialization ablation on Llama-3.2-1B.

Figure 2: Training loss on Llama-3.2-1B.

Figure 3: Gradient norm on Llama-3.2-1B.

Parameter-Efficient Fine-Tuning

Parameter-efficient fine-tuning adapts pretrained models by updating small parameter subsets or lightweight modules instead of all weights.

See all 596 papers

TernaryLLM: Ternarized Large Language Model
Jun 11, 2024Tianqi Chen, Zhe Li, Weixiang Xu +6LLM Quantization Ternary Quantization
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Advanced Micro Devices, Inc., Beijing, China +1
RoleQ: Role-Aligned Scales for Converting Language Models to Ternary
Oct 15, 2026Johannes Knittel, Alec Helbling, Hanspeter PfisterLLM Quantization Neural Network Quantization
Harvard University · Georgia Tech
Capability-Stratified Degradation in Ternary Language Models
Aug 28, 2026Anirudh Malik, M Sparsh Mehra, Poojith DevanLLM Quantization Ternary Quantization
OneBit AI

Jun 11, 2024J/K move · Enter open · S save

TernaryLLM: Ternarized Large Language Model

Tianqi Chen, Zhe Li, Weixiang Xu, Zeyu Zhu+5

Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Advanced Micro Devices, Inc., Beijing, China+1

LLM Quantization Ternary Quantization Quantization-Aware Training

Open paper AskPDF arXiv

Large language models (LLMs) have achieved remarkable performance on Natural Language Processing (NLP) tasks, but they are hindered by high computational costs and memory requirements. Ternarization, an extreme form of quantization, offers a solution by reducing memory usage and enabling energy-efficient floating-point additions. However, applying ternarization to LLMs faces challenges stemming from outliers in both weights and activations. In this work, observing asymmetric outliers and non-zero means in weights, we introduce Dual Learnable Ternarization (DLT), which enables both scales and shifts to be learnable. We also propose Outlier-Friendly Feature Knowledge Distillation (OFF) to recover the information lost in extremely low-bit quantization. The proposed OFF can incorporate semantic information and is insensitive to outliers. At the core of OFF is maximizing the mutual information between features in ternarized and floating-point models using cosine similarity. Extensive experiments demonstrate that our TernaryLLM surpasses previous low-bit quantization methods on the standard text generation and zero-shot benchmarks for different LLM families. Specifically, for one of the most powerful open-source models, LLaMA-3, our approach (W1.58A16) outperforms the previous state-of-the-art method (W2A16) by 5.8 in terms of perplexity on C4 and by 8.2% in terms of average accuracy on zero-shot tasks.

Abstract

Parameter-Efficient Fine-Tuning

Explore similar work

TernaryLLM: Ternarized Large Language Model

RoleQ: Role-Aligned Scales for Converting Language Models to Ternary

Capability-Stratified Degradation in Ternary Language Models