RoleQ: Role-Aligned Scales for Converting Language Models to Ternary

Johannes Knittel; Alec Helbling; Hanspeter Pfister

cs.AIOct 15, 2026

RoleQ: Role-Aligned Scales for Converting Language Models to Ternary

Authors: Johannes Knittel, Alec Helbling, Hanspeter Pfister

Organizations: Harvard University · Georgia Tech

Abstract

Weight quantization below four bits shrinks memory and bandwidth at a perplexity that looks acceptable, but often at the cost of factual knowledge and task accuracy. Ternary weights have mostly been obtained by training from scratch or long fine-tuning. We introduce RoleQ, a quantization framework that places scales by the functional roles of Transformer units rather than by the layout of the weight matrix, as conventional formats do. Vectors that read from the residual stream take one scale per output unit; those that write to it, one per input unit. The rule is structural, needs no search, calibration or stored state, and holds on integer and ternary grids. In a rotated basis it matches a per-layer axis search at half the solver work, and at ternary it cuts perplexity by up to half against layout-based placement in two families at every size from 0.6B to 14B, where per-row scales leave MMLU at chance. Our lightweight post-training pipeline converts an existing checkpoint to 1.59 bits per weight in a few GPU-hours, a fraction of the cost of learned ternarization, and leads every post-training ternary method at matched or lower memory-resident bits on perplexity, zero-shot accuracy and MMLU, most widely on knowledge. A sparse second ternary plane placed by a per-unit Fisher saliency keeps the matmul multiplication-free and recovers what ternary loses first: at 8B, a fraction of a bit returns about half the MMLU that conversion costs, well ahead of random or Hessian-based placement at equal bits.

Figures & tables

Figure 1: RoleQ aligns scales with the roles. (A) Readers read from the residual stream through rows, writers write to it through columns. (B) RoleQ places the scales on the unit axis, per row for readers and per column for writers, never on the stream axis. (C) Qwen3-14B, ternary at equal bits: the placement alone lifts MMLU from 25.4 (chance) with per-output-channel scales to 38.5.

Figure 2: The RoleQ-T pipeline. A data-free learned rotation precedes ternary GPTQ with an exact per-vector step, giving codes in

\{-1,0,+1\}

(grey) and one scale per functional unit (blue). The dashed stages are optional: scale distillation refits scales with frozen codes, and Fisher planes add a sparse second plane to salient units (+0.57 bits here). Values below each stage are memory-resident bits per weight for Qwen3-8B-Base.

Qwen3-8B-Base fp16: 6.99 / 70.42 / 76.82
Method	Stage	claimed bpw	resident bpw	WT2 ppl $\,\downarrow$	Zero-shot $\,\uparrow$	MMLU $\,\uparrow$
one ternary plane
Qronos + HIP + MagR	PTQ	1.58	1.591	22.80	46.67	25.93
PT 2 -LLM	PTQ	1.58	1.835	67.95	44.88	26.52
PT 2 -LLM codes + our SD	+SD	1.58	1.835	15.51	56.37	36.74
RoleQ-T (ours)	PTQ	1.585	1.590	22.42 $\pm$ 0.22	50.39 $\pm$ 0.09	33.32 $\pm$ 1.48

Table 1: RoleQ-T leads the tested post-training ternary baselines below 2.3 bpw. WikiText2 perplexity, seven-task zero-shot average and five-shot MMLU; baselines at their best available stage, ours at PTQ and after scale distillation (+SD), mean of two seeds

\pm

half-range. TWLA (higher rate) and CAT-Q (learned) are references. Planes use the text Fisher on Base models and the knowledge Fisher on the post-trained model.

		Qwen3-Base
Configuration	bpw	0.6B	1.7B	8B	14B	Llama-8B
fp16	16.000	12.66	9.39	6.99	6.38	6.30
Ternary: per-OC	1.590	387.2 $\pm$ 34.0	222.3 $\pm$ 8.0	35.47 $\pm$ 0.52	37.89 $\pm$ 0.65	114.4 $\pm$ 8.6
Ternary: per-IC	1.590	1187 $\pm$ 226	305.0 $\pm$ 19.8	33.39 $\pm$ 1.70 4	42.60 $\pm$ 0.48	669.9 $\pm$ 28.1
Ternary: RoleQ (ours)	1.590	351.9 $\pm$ 21.8	136.2 $\pm$ 8.8	31.28 $\pm$ 0.77 4	25.25 $\pm$ 0.34	58.13 $\pm$ 2.29
INT2-g128: per-OC	2.156	79.87 $\pm$ 1.65	37.87 $\pm$ 0.45	13.11 $\pm$ 0.08 4	10.90 $\pm$ 0.05	49.7 $\pm$ 14.0

Table 2: Role-aligned scale placement achieves the lowest perplexity over fixed per-channel placement. WikiText2 perplexity at four Qwen3-Base sizes and Llama-3.1-8B-Base, under matched PTQ settings (random Hadamard

r_{1}

). Mean of two ( 4 four) seeds

\pm

half-range; lower is better.

Configuration / Score	Total bpw	WikiText-2 PPL $\downarrow$	7-Task Zero-Shot Avg. $\uparrow$	MMLU $\uparrow$
One Plane	1.590	$19.59\pm 1.58$	$59.96\pm 0.32$	$48.41\pm 1.46$
Hessian $\times$ Weight	1.988	$17.68\pm 0.14$	$62.32\pm 0.19$	$52.80\pm 0.17$
Random Units	1.991	$15.74\pm 0.18$	$63.07\pm 0.08$	$55.58\pm 0.03$
Knowledge Fisher (Ours)	1.990	$\mathbf{15.30\pm 0.36}$	$\mathbf{64.57\pm 0.28}$	$\mathbf{61.82\pm 0.15}$
Full Precision (FP16)	16.000	9.72	69.36	74.93

Table 3: Our Fisher selector outperforms random and Hessian-based selection at matched bit rates. Results on post-trained Qwen3-8B after scale distillation (two seeds,

\pm

half-range).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Claimed	Resident	Trains?	Calibrates?	Scale axis	Source
RoleQ-T (ours)	1.585	1.590	SD of scales and norm gains	yes, C4	role: readers per row, writers per column	–
Qronos + HIP + MagR	1.58	1.591	no	yes	per-OC, fp scale and zero point	rerun
PT 2 -LLM	1.58	1.835	no	yes	per-OC $\times$ g128, scale and shift	rerun
QTEA	1.70	2.26–2.28	no	yes, WT2	per-OC $\times$ g128 $\alpha,\beta$ + FP8 residual (1:4 sparse) on salient columns	rerun 8B; ckpt 14B
CAT-Q	1.58	1.71	learned ternarization, 60 epochs	yes, C4 ( $512\times 2048$ )	per-row $\times$ g128	ckpt
TWLA	1.58	4.15–4.65	rotation, fit to weights	yes (GPTQ)	per-(row, g128) $\alpha,\mu$ , saliency masks, second plane on 26% (code, not paper)	rerun

Appendix

Table 4: Compared methods. Each method’s claimed and memory-resident rate (derivations in Table 5 ), whether it trains codes or weights, whether it calibrates on data, where its scales sit, and how it enters our tables (rerun: its own code on our checkpoint; ckpt: its released checkpoint decoded and scored on our harness; paper: cited as reported). PTQTP has no public code or checkpoints; its rate and scale axis follow from its own decomposition and storage formula ( Xiao et al., 2025b ) . QTIP, alone and under GuidedQuant’s loss-guided Hessian, is the trellis-coded 2-bit class we do not claim to match; those checkpoints are Llama-3.1-8B-Instruct and CAT-Q’s are post-trained Qwen3-1.7B / 8B; CAT-Q brackets its cost at 1–8 h on eight A100-80GB GPUs from 1.7B to 14B. TWLA releases no checkpoint: its row is the model its released code builds that matches its reported numbers, and its rotation objective is fitted to the weights (its text mentions the calibration set, its code does not use it).

Method	Term	bpw	Resident?	Total
RoleQ-T	ternary codes, $\log_{2}3$	1.585	yes
	one fp16 scale per unit, $16/d$	0.0039	yes
	reintroduced norm gains	0.0007	yes
	$r_{4}$ sign vector per layer	$<0.0001$	yes
	$r_{1}$ / $s_{1}$ / $s_{2}$ , folded into the weights	0	no	1.590
RoleQ-T-g128 / per-OC g128	+ ternary group scales, $g=128$	0.125	yes	1.710

Appendix

Table 5: Bit-accounting derivations. Information bits per weight, counting every term a decoder needs to produce the weight (Qwen3-8B:

d=4096

d_{\mathrm{ff}}=12288

); folded or seed-regenerable rotations cost nothing, and packing is not charged (5 trits per byte would read 1.605). QTEA’s terms use its own bit_width function. PT 2 -LLM’s

\alpha

and

\mu

are counted as fp16 per (row, 128), the layout of its released quantizer; it publishes no per-weight accounting. TWLA’s terms are measured on the model its released code builds, the code that reproduces its reported Qwen3-8B perplexity (TWLA releases no checkpoint; its paper describes one

(\alpha,\mu)

pair per row and a single plane, while the code, of ARB-LLM lineage, adds per-block magnitude masks with separate pairs and a second plane on the salient partition) over all 10 368 blocks: 26.21% of the weights carry a second plane (26.48% on the post-trained model), and the absence of a single exact zero among 1.31 M sampled weights shows that the mask membership is consumed at inference, since

\mu

is added to every element inside a mask regardless of its code. Even without the membership bit, TWLA sits at 3.15–3.65.

Models	Qwen3-0.6B / 1.7B / 8B / 14B-Base; Qwen3-1.7B / 8B post-trained; Llama-3.1-8B-Base; Llama-3.1-8B-Instruct (Table 10 )
Calibration, paper draw	512 C4 sequences $\times$ 512 tokens; standard draw $128\times 2048$ for rows that match a competitor’s protocol and for Llama-3.1-8B-Instruct; post-trained models: 272 chat sequences $\times$ up to 1536 tokens, half UltraChat, half self-generated
Calibration seeds	45 and 46 on every cell of ours; 47 and 48 added where a cell says four seeds
Solver	GPTQ with act-order, damping 0.01 of the mean Hessian diagonal, lazy block 128; exact per-vector ternary step; reader refit in the $\Sigma$ metric
Rotation	recipe: $s_{1}$ (l4) $+s_{2}$ (opt) $+r_{4}$ (block 1024); ablations and placement controls: random Hadamard $r_{1}$
Scale distillation (SD)	400 steps, batch 8, learning rate $3\times 10^{-3}$ , KL to the cached fp teacher, 3,200 RedPajama-1T-Sample windows of 512 tokens; scales and norm gains only, codes frozen; 2 h at 8B on one A100
Perplexity	WikiText2 test at 2048 tokens

Appendix

Table 6: Models, corpora, seeds and evaluation. Full-precision references are measured on our harness.

(a) ternary	0.6B	1.7B	Qwen-8B	14B	Llama-8B
RoleQ	351.9 ±21.8	136.2 ±8.8	31.28 ±0.77 4	25.25 ±0.34	58.13 ±2.29
per-OC	387.2 ±34.0	222.3 ±8.0	35.47 ±0.52	37.89 ±0.65	114.4 ±8.6
per-IC	1187 ±226	305.0 ±19.8	33.39 ±1.70 4	42.60 ±0.48	669.9 ±28.1
writer half	1.10	1.63	1.14	1.50	1.97
reader half	3.37	2.24	1.07	1.69	11.5
best uniform	per-OC	per-OC	per-IC	per-OC	per-OC

Appendix

Table 7: The placement rule: the cells behind Table 2 . (a) Ternary, calibrated,

r_{1}

, PTQ-only: two-seed means with half-ranges (four seeds for RoleQ and per-IC at Qwen-8B), the writer half (per-OC over RoleQ), the reader half (per-IC over RoleQ), the best uniform placement, and RoleQ’s lead over it with the gap as a multiple of the largest seed range. Llama-8B is the one cell in which both halves separate at once, between runs at

9\times

and

18\times

full precision. (b) Two grids omitted from the main table: the per-vector four-level grid (2.01 bpw), whose lead closes and reverses by 8B as it does at the integer widths, and INT8sym, where the axis makes no measurable difference (seed 45). (c) Zero-shot and MMLU scores of the integer grids where they were run and of the 14B ternary triplet (two seeds; lead: RoleQ compared to best uniform placement, ∗ marks notable difference; the 14B per-OC and per-IC MMLU are at chance, ∘ ).

Kind	RoleQ	Picks IC, rotated	Picks IC, unrotated
$W_{\mathrm{in}}$	OC	1 / 16	15 / 16
$W_{\mathrm{gate}}$	OC	0 / 16	14 / 16
$W_{Q}$	OC	0 / 16	11 / 16
$W_{K}$	OC	0 / 16	15 / 16
$W_{V}$	OC	3 / 16	16 / 16
$W_{\mathrm{out}}$	IC	16 / 16	6 / 16

Appendix

Table 8: AdaDim’s per-layer search against the placement rule. Left: at the GPTQ stage on Qwen3-1.7B, in 16 sampled layers, how many matrices of each kind AdaDim’s reconstruction-error argmin sends to the input axis, rotated (

r_{1}

) and unrotated; RoleQ fixes readers to the output axis and writers to the input axis without a search. Right: the cost and the return of the search; AdaDim’s own search quantizes each matrix twice with RTN before one GPTQ pass, and the “half the solver work” of Section 4.2 is against a two-way GPTQ search. We did not run AdaDim end to end: it would differ from RoleQ in 4 of 112 cells at 1.7B and 6 of 112 at 8B, worth 0.003% and 0.023% of

\mathbf{H}

-metric error, three orders of magnitude below the seed noise.

Run	Basis	bpw	Stage	WT2 ppl $\,\downarrow$	Zero-shot $\,\uparrow$	MMLU $\,\uparrow$
ternary body on the role axis	$r_{1}$	1.590	PTQ	30.99 ±0.01	49.13 ±0.31	32.82 ±0.21
	$r_{1}$	1.590	+SD	12.07 ±0.04	59.40 ±0.35	45.88 ±0.87
= RoleQ-T	recipe	1.590	PTQ	22.42 ±0.22	50.39 ±0.09	33.32 ±1.48
	recipe	1.590	+SD	11.33 ±0.06	60.33 ±0.27	47.59 ±2.72
+ ternary group scales g128, per-OC	recipe	1.710	PTQ	19.16 ±0.07	52.37 ±0.28	37.25 ±1.68
	recipe	1.710	+SD	10.87 ±0.06	61.93 ±0.31	51.33 ±0.38

Appendix

Table 9: The 8B recipe ladder in both bases. Every run of ours behind Table 1 : Qwen3-8B-Base, C4

512\times 512

, two seeds, half-range as subscript. The

r_{1}

rows are the ablation that measures what the recipe’s rotation is worth; the recipe rows are the numbers presented in the main text.

Method	Stage	bpw	WT2 ppl ( $\times$ fp)	Zero-shot (%)	MMLU (%)
QTIP 2-bit	PTQ + FT	2.0065	8.83 (1.22)	68.13 (90.6)	59.93 (80.7)
GuidedQuant on QTIP, 2-bit	PTQ	$\sim$ 2.0	10.11 (1.40)	66.29 (85.4)	57.58 (75.2)
RoleQ-T + planes, recipe	PTQ	2.0065	21.58 ±0.86 (2.98)	57.86 ±0.31 (61.9)	40.50 ±0.26 (35.8)
RoleQ-T + planes, recipe	+SD	2.0065	11.06 ±0.06 (1.53)	63.81 ±0.02 (78.5)	51.02 ±0.63 (60.1)
RoleQ-T + planes, $r_{1}$	PTQ	2.0065	26.34 (3.64)	55.75 (56.0)	37.62 (29.1)
RoleQ-T + planes, $r_{1}$	+SD	2.0065	11.50 (1.59)	63.09 (76.5)	47.99 (53.1)

Appendix

Table 10: Llama-3.1-8B-Instruct at 2 bpw against the trellis-coded class. Scores and, in parentheses, the share of the above-chance headroom retained (fp16: 7.23 ppl, 71.50 zero-shot, 68.30 MMLU; chance 35.7 and 25.0). QTIP (whose stage includes its QuIP#-style fine-tune) and GuidedQuant are their released checkpoints re-scored on our harness (QTIP’s WikiText2 reproduces its published value to 0.001, in its own convention on the materialised model). Our plane budget is set so that the rate matches QTIP-2bit to three decimals (1.590 + 0.4165). Calibration on the standard draw

128\times 2048

. The recipe rows are two-seed on every instrument; the

r_{1}

rows (seed 45) are the same method in the ablation basis. This is a negative result: at matched bits the trellis-coded methods lead on every instrument (

-4.3

zero-shot and

-8.9

MMLU against QTIP after distillation,

-2.5

and

-6.6

against GuidedQuant), and the change of basis narrows the gap (from

-5.0

-11.9

r_{1}

) without closing it.

(a) WikiText2 perplexity
Model	Stage	planes	RoleQ-INT2-g128	per-OC INT2-g128	lead	gap $\div$ range
Qwen-8B	PTQ	11.43 ±0.02	11.79 ±0.01	11.88 ±0.03	1.032	$7.5\times$ ∗
Qwen-8B	+SD	9.30 ±0.02	9.44 ±0.00	9.44 ±0.03	1.016	$2.7\times$
Llama-8B	PTQ	13.58 ±0.05	17.42 ±0.15	17.21 ±0.40	1.27	$5.8\times$ ∗
Llama-8B	+SD	9.59 ±0.04	10.25 ±0.04	10.22 ±0.05	1.065	$6.1\times$ ∗
(b) zero-shot average

Appendix

Table 11: Matched bits in the recipe basis. Ternary + text-Fisher planes (2.157 bpw at Qwen-8B, 2.158 at Llama-8B) against the four-level INT2-g128 grid (2.156) with scales per output channel (the standard) and on the role axis (the best four-level control); all three are ours, with the same solver, kernel and calibration; two seeds, half-range as subscript. The lead is the better control’s perplexity over ours (or the point gap against the better control), with the gap as a multiple of the largest seed range among the runs compared; ∗ marks

\geq 3\times

. On Llama the role-axis grid ties per-OC, with the sign against it. The

r_{1}

-basis perplexities at 0.6B and 1.7B are in Table 2 .

Figure 3: The plane budget sweep. Qwen3-1.7B (post-trained), PTQ stage, WikiText2 perplexity, two calibration seeds at every budget; bands are the seed range. (a) Perplexity against the total rate as the second plane is extended to more units (upper axis), for three placements: the per-unit Fisher saliency, Hessian

{}\times{}

weight saliency, and random units; the planes-off run at

1.594

bits is the common origin. Random was run at three budgets. (b) The same cells as a ratio to the Fisher placement: its advantage over random decays from

1.96\times

+0.15

bits to

1.12\times

+0.90

, and Hessian

{}\times{}

weight overtakes it at

+1.20

bits (

0.94\times

). The decay is measured at 1.7B and is not claimed at 8B.

loss-sensitivity scores
Selector	$\rho$	PTQ	after SD
text Fisher (C4 windows)	$-0.27$	9.70 ±0.17	5.45
knowledge Fisher	$-0.47$	10.15 ±0.05	5.58
TaCQ score on the knowledge items	–	10.30 ±0.19	5.59
error-magnitude scores
activation norm	–	13.71	5.73

Appendix

Table 12: Every selector at 1.7B, grouped by the family of the score. Qwen3-1.7B post-trained,

+0.47

bits of planes (2.066 bpw), PTQ WikiText2 perplexity and the same runs after scale distillation (fp 5.64); two seeds with the half-range as subscript, a cell without a subscript is one seed;

\rho

as in Table 3 . The loss-sensitivity scores lie at 9.7–10.3 and the error-magnitude scores at 13.7–17.9, with random placement among the latter; three of the four error-magnitude scores, including the solver’s own measured error, fail to beat random, so a better estimate of the error would not help. The

3.3\times

spread across selectors at PTQ narrows to

1.22\times

after scale distillation.

(a) Rotation	$s_{1}$ objective	slot 2	$r_{4}$	weight-space	PTQ	+SD
none	–	–	–	–	1245.9	67.53
random Hadamard $r_{1}$	–	–	–	$1.00\times$	119.8	23.70
$s_{1}$ , sparsifying (opt)	opt	–	–	$0.65\times$	130.3	23.23
$s_{1}$ , flattening (l4)	l4	–	–	$0.91\times$	89.99	21.39
$s_{1}$ , TWLA’s tri-modal	trigmm	–	–	$1.00\times$	158.4	24.81
$s_{1}+s_{2}+r_{4}$	opt	$s_{2}$	yes	$0.65\times$	92.81	21.00

Appendix

Table 13: Rotation bake-off and solver controls. (a) Qwen3-1.7B-Base, RoleQ format, C4

512\times 512

seed 45, WikiText2 perplexity at PTQ and after SD; the weight-space column is the ternary reconstruction error relative to the random Hadamard. The objective with the lowest weight-space error has the highest perplexity, and the learned rotation paired with per-OC scales collapses. (b) The

s_{2}

and

r_{4}

slots isolated,

s_{1}

= l4 throughout. (c) Solver controls on the writer step: Qwen3-1.7B, minipile

256\times 512

r_{1}

, PTQ seed 42 and 100-step SD on two seeds. (d) The

r_{4}

intervention: the writer-half margin at 1.7B ternary with and without

r_{4}

, two seeds;

r_{4}

is the one slot that mixes a unit axis, and it compresses the half that lives on the writers by two thirds in log terms.

Model, method	standard $128\times 2048$	paper $512\times 512$	ratio
Qwen3-8B-Base, ternary body	33.90 (4.85)	30.32 (4.33)	$1.12\times$
Llama-3.1-8B-Base, ternary body	127.9 (20.3)	56.95 (9.04)	$2.25\times$
Qwen3-1.7B-Base, ternary body	282.6 (30.1)	119.8 (12.8)	$2.36\times$
PT 2 -LLM on Qwen3-8B-Base, WikiText2 draw	28.96 (4.14)	25.73 (3.68)	$1.13\times$
PT 2 -LLM on Qwen3-8B-Base, C4 draw	84.49 (12.1)	67.95 (9.72)	$1.24\times$

Appendix

Table 14: Calibration draw-shape sensitivity. PTQ WikiText2 perplexity of the same configuration calibrated on the standard draw (

128\times 2048

tokens) and on the paper draw (

512\times 512

), seed 45, multiple of fp16 in parentheses. The draw shape moves every method, including PT 2 -LLM on its own corpus, by more than most format effects, and it favours the baselines as much as it favours us. The ternary rows are the RoleQ-T body under

r_{1}

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 360 papers

Aug 28, 2026

Capability-Stratified Degradation in Ternary Language Models

Extreme low-bit inference offers a route toward smaller models and constrained deployment. Ternary language models restrict weights to

\{-1,0,+1\}

, approaching the limit of

\log_2 3 \approx 1.585

bits/weight. The practical question for a pretrained model is not simply whether weights can be quantised but which capabilities survive and whether it remains useful for adaptation. We explore this by converting Qwen3.5-0.8B (752M parameters) to ternary weights using 72.4M tokens of quantisation-aware training (QAT). The resulting model, Cloe, is evaluated across 29 benchmarks, representation diagnostics, and downstream fine-tuning. The evidence shows non-uniform degradation. A linear probe recovers 43.76% of MMLU answers from the full-precision teacher's representations but only 26.19% from Cloe (near chance), indicating specialist factual information is lost. However, Cloe retains measurable performance on ten tasks, averaging 77.1% of teacher performance. Crucially, fine-tuning raises Cloe to 89.8% on SST-2 (95.6% of the matched teacher) and reaches 79.4% teacher retention on XSum. We attribute degradation to a combination of quantisation-induced information loss and incomplete recovery due to the limited QAT budget. We also highlight an evaluation pitfall: standard answer-letter scoring failed (Cloe emitted "A" on 98.6% of MMLU questions), necessitating continuation scoring. Ultimately, ternary conversion is unsuitable as a drop-in general replacement yet remains valuable as a compact substrate for task-specific models.

Anirudh Malik, M Sparsh Mehra, Poojith Devan

OneBit AI

LLM Quantization Ternary Quantization

Ask PDF

Sep 2, 2026

Post-Training Ternarization of Qwen3-4B Capability, Effective Bit Budget, Storage Compression, and Deployment

Ultra-low-bit language models can reduce storage and memory bandwidth, but a nominal "1.58-bit" label does not fully describe the stored representation, retained capability, or runtime behavior. We study an end-to-end post-training conversion of Qwen, an instruction-tuned 4B-parameter model, using KOTMS rotation, E2M-ATQ ternarization, and GPTQ-style error compensation from TWLA. The experiment is weight-only: activations remain at 16-bit precision, so ILA-AMP is omitted. We evaluate effective bit accounting, task capability retention, perplexity, calibration sensitivity, checkpoint composition, and deployment behavior. The final conversion uses 1.641 effective bits per weight for quantized linear weights, with 81.62% of model parameters targeted. Across ten scored capability comparisons, accuracy falls from 64.5% to 54.7%. Degradation is uneven: BoolQ retains 84.6% chance-corrected teacher performance, while ARC-Challenge retains 43.8%. Perplexity rises from 13.639 to 18.748 on WikiText-2, 24.700 to 31.992 on PTB, and 19.831 to 28.966 on C4. A subsequent packing run preserves the ternary planes and scales, reducing reported model size from 8.29 GiB to 3.96 GiB with essentially unchanged perplexity. A separate third-party packing attempt was lossy and is excluded from the primary artifact claim. The packed artifact has not been benchmarked end-to-end for task accuracy or generation throughput. A preliminary Triton GEMV microbenchmark is 4.6x slower than FP16 cuBLAS on one tested shape. We therefore do not claim that compression alone yields faster inference.

Anirudh Malik, M Sparsh Mehra, Poojith Devan

OneBit AI

LLM Quantization Ternary Quantization LLM Compression Weight-Only Quantization Post-Training Quantization

Ask PDF

Jul 15, 2026

ExTernD: Expanded-Rank Ternary Decomposition Ternary LLM PTQ with Accuracy Approaching Any Quantization Level

We introduce ExTernD (Expanded-rank Ternary Decomposition), a post-training factorization of each LLM weight matrix

A \in \mathbb{R}^{m \times n}

into

A \approx B \mathrm{diag}(D) C

with ternary factors

B \in \{-1,0,+1\}^{m \times k}

C \in \{-1,0,+1\}^{k \times n}

and a real scale vector

D \in \mathbb{R}^k

. The inner rank

k = μ\min(m,n)

is deliberately expanded beyond full rank (

μ> 1

), so that components past full rank correct the quantization error of earlier ones. We prove the residual decreases monotonically in

k

and can be driven below any

\varepsilon > 0

: ExTernD approaches bf16 accuracy arbitrarily closely, which no ternary scheme with a fixed plane count can do. Memory and compute scale continuously with

μ

, and factor sparsity continuously with a threshold

τ

, so an accuracy target is hit exactly rather than rounded to the next bit-width. ExTernD matches Q4_K's per-matrix accuracy at 5.2-5.5 effective bpw (5.1-5.5 with importance weighting) on Gemma-4-E2B and Qwen3.5-4B, and a full Qwen3.5-4B conversion at

μ= 3

reaches 10.10 wikitext-2 perplexity against 9.78 for bf16 (+3.2%), placing it near the Q4_K/Q5_K accuracy band at ~5.7 effective bpw.

Chethan Reddy G. P

LLM Quantization Ternary Quantization Weight-Only Quantization Post-Training Quantization Neural Network Quantization

Ask PDF

Abstract

LLM Quantization

Explore similar work

Capability-Stratified Degradation in Ternary Language Models

Post-Training Ternarization of Qwen3-4B Capability, Effective Bit Budget, Storage Compression, and Deployment

ExTernD: Expanded-Rank Ternary Decomposition Ternary LLM PTQ with Accuracy Approaching Any Quantization Level