Capability-Stratified Degradation in Ternary Language Models

Anirudh Malik; M Sparsh Mehra; Poojith Devan

cs.AIAug 28, 2026

Capability-Stratified Degradation in Ternary Language Models

Authors: Anirudh Malik, M Sparsh Mehra, Poojith Devan

Organizations: OneBit AI

Abstract

Extreme low-bit inference offers a route toward smaller models and constrained deployment. Ternary language models restrict weights to $\{-1,0,+1\}$ , approaching the limit of $\log_2 3 \approx 1.585$ bits/weight. The practical question for a pretrained model is not simply whether weights can be quantised but which capabilities survive and whether it remains useful for adaptation. We explore this by converting Qwen3.5-0.8B (752M parameters) to ternary weights using 72.4M tokens of quantisation-aware training (QAT). The resulting model, Cloe, is evaluated across 29 benchmarks, representation diagnostics, and downstream fine-tuning. The evidence shows non-uniform degradation. A linear probe recovers 43.76% of MMLU answers from the full-precision teacher's representations but only 26.19% from Cloe (near chance), indicating specialist factual information is lost. However, Cloe retains measurable performance on ten tasks, averaging 77.1% of teacher performance. Crucially, fine-tuning raises Cloe to 89.8% on SST-2 (95.6% of the matched teacher) and reaches 79.4% teacher retention on XSum. We attribute degradation to a combination of quantisation-induced information loss and incomplete recovery due to the limited QAT budget. We also highlight an evaluation pitfall: standard answer-letter scoring failed (Cloe emitted "A" on 98.6% of MMLU questions), necessitating continuation scoring. Ultimately, ternary conversion is unsuitable as a drop-in general replacement yet remains valuable as a compact substrate for task-specific models.

Figures & tables

Work / direction	Model regime	Core approach	Primary emphasis
BitNet b1.58	Native ternary	Train directly at low precision	Demonstrates viability of ternary training and inference.
CAT-Q	Pretrained conversion	Calibration / post-training conversion	Demonstrates that pretrained models can be brought toward ternary operation.
PTQTP	Pretrained conversion	Dual trit-plane representation	Improves the representation of ternary weights in PTQ settings.
ScaleQ-1.58	Pretrained conversion	Calibration with additional generated signals	Explores stronger post-training ternary conversion.
Ternary Mamba	Pretrained conversion	Grouped quantisation + distillation / QAT	Closest precedent in spirit for converting an existing model and reporting retention.
Falcon-Edge	Native low-bit	Native 1.58-bit pretraining	Shows the alternative strategy of avoiding conversion altogether.

Table 1: Positioning against major directions represented in the project literature.

Property	Value
Base model	Qwen3.5-0.8B
Parameters	$752\text{\,}\mathrm{M}$
Transformer layers	24
Attention structure	18 linear-attention + 6 full-attention
Hidden size	1024
Head dimension	256

Table 2: Core configuration of the Cloe conversion.

Panel of Figure 1 — Figure 1: Overview of the conversion pipeline used to obtain Cloe.

Instrument	What it measures	Why it is included
Continuation scoring	Candidate-answer likelihood	Avoids relying on an answer-letter output channel.
Majority baseline	Trivial class-selection performance	Detects cases where raw accuracy is inflated by label imbalance.
Balanced accuracy	Class-balanced discrimination	Prevents constant predictions from being interpreted as capability.
Linear probe	Recoverable task information in hidden states	Separates inaccessible/removed information from output-head problems.
Effective rank	Diversity of representation directions	Tests whether quantisation causes broad representational collapse.
Matched fine-tuning	Downstream task learnability	Tests whether useful task structure can still be acquired after conversion.

Table 3: Evaluation instruments and the question each instrument answers.

Model	A	B	C	D
Cloe, pre-SFT	13,852	113	14	63
Cloe, post-SFT	6,801	2,505	1,248	3,488
BF16 teacher	301	2,914	2,524	8,303

Table 4: Answer-letter distribution that exposed the degenerate MMLU output channel.

Task	Capability	Cloe	Teacher	Retention	Bar	Over bar
race	Reading	34.0%	38.3%	89%	28.0%	+6.0
piqa	Physical commonsense	60.7%	69.0%	88%	50.7%	+10.0
commonsenseqa	Commonsense	34.0%	39.7%	86%	23.7%	+10.3
qasc	Knowledge-easy	31.3%	37.3%	84%	15.0%	+16.3
sciq	Knowledge-easy	54.7%	66.0%	83%	27.0%	+27.7
hellaswag	Commonsense	37.3%	47.3%	79%	28.0%	+9.3

Table 5: Benchmark performance and capability retention relative to the full-precision teacher. “Bar” denotes the baseline ternary model, while “Over bar” reports the absolute Cloe improvement over that baseline. Tasks shown are those satisfying both the majority-baseline and chance-level balanced-accuracy criteria. Retention is computed relative to the full-precision teacher.

Panel of Figure 2 — Figure 2: Representative retained capabilities. The main visual message is the non-uniform degradation profile: several commonsense and plausibility-oriented tasks remain substantially above chance after ternarisation.

Task	Capability	Cloe	Teacher	Retention	Bar	Over bar
winogrande	Coreference	53.7%	55.3%	97%	50.0%	+3.7
gsm8k	Arithmetic reasoning	2.0%	4.0%	50%	0.0%	+2.0
copa	Causal commonsense	57.0%	64.0%	89%	55.0%	+2.0
rte	Entailment	53.4%	52.7%	101%	52.7%	+0.7
boolq	Reading	62.7%	62.7%	100%	62.7%	+0.0
mrpc	Paraphrase	69.7%	69.7%	100%	69.7%	+0.0

Table 6: Capability boundaries and evaluation-limited tasks. “Bar” denotes the baseline ternary model, while “Over bar” reports the absolute Cloe improvement over that baseline. Retention is reported relative to the full-precision teacher; unusually large retention values are shown without positive highlighting when the teacher score is itself very low.

Representation	Probe accuracy	Own answer accuracy	Effective rank
BF16 teacher	43.76%	44.72%	557.5
Cloe, post-SFT	26.19%	24.31%	602.1
Cloe, pre-SFT	26.68%	22.97%	538.9
Chance	25.00%	25.00%	–

Table 7: Representation-level probe results. Chance for four-way MMLU classification is 25%.

Panel of Figure 3 — Figure 3: MMLU information recoverable from final-layer representations. Cloe approaches the 25% four-way chance level while the teacher retains substantial recoverable signal.

Model	Effective rank	Relative to teacher
BF16 teacher	557.5	100.0%
Cloe, pre-SFT	538.9	96.7%
Cloe, post-SFT	602.1	108.0%

Table 8: Effective-rank comparison.

Task	Cloe before	Cloe after	Teacher retention
SST-2	61.9%	89.8%	95.6%
XSum	11.1 ROUGE-L	19.0 ROUGE-L	79.4%

Table 9: Matched downstream learning results.

Panel of Figure 4 — Figure 4: Downstream learnability expressed relative to identically fine-tuned teacher controls. The absolute task scores remain important and are reported in table 9 .

Property	Direction	Evidence	Interpretation
Specialist factual knowledge	$\downarrow\downarrow\downarrow$	MMLU + linear probe	Much of the teacher’s recoverable specialist information is lost.
Arithmetic reasoning	$\downarrow\downarrow\downarrow$	GSM8K	Not a reliable general arithmetic model after conversion.
Language modelling	$\downarrow\downarrow$	LAMBADA / WikiText-2	Token-level modelling quality degrades substantially.
Commonsense judgement	$\rightarrow$	10-task benchmark subset	Several plausibility-oriented capabilities remain.
Representation diversity	$\rightarrow$	Effective rank	No evidence of broad representational collapse.
Downstream learnability	$\rightarrow$	SST-2 / XSum	Task-specific adaptation remains effective.

Table 10: Capability-stratified outcome profile of ternarisation.

Component	Configuration
CPU	AMD Ryzen 9 7900X (12 cores, 24 threads)
System memory	32 GB RAM
GPU	NVIDIA GeForce RTX 5070
GPU VRAM	12,227 MiB ( 11.94 GiB)
Operating system	Windows 10 Pro (64-bit)
Python	3.10.11

Table 11: Hardware and software environment used for deployment measurements.

Metric	Packed	Latent	Relative change	Interpretation
Disk footprint	634.65 MB	1505.60 MB	2.37 $\times$ smaller	Major storage reduction
Weight memory	1578.36 MiB	1460.59 MiB	+8.1%	Runtime memory overhead
Peak memory	1708.98 MiB	1591.22 MiB	+7.4%	Modest peak-memory increase
Load time	2.858 s	1.570 s	1.82 $\times$ slower	Initialization penalty
Prefill	1806.76 tok/s	1477.03 tok/s	+22.3%	Faster steady-state processing
Decode	7.634 tok/s	6.145 tok/s	+24.2%	Faster token generation

Table 12: Packed versus latent deployment measurements from the current benchmark.

Panel of Figure 5 — Figure 5: Current packed-versus-latent deployment measurements. The strongest observed benefit is storage footprint, while throughput also improves in the measured execution path. Runtime memory and loading remain separate engineering considerations.

Consideration	Current evidence	Engineering implication
Specialist knowledge	MMLU / ARC-C / MedMCQA near chance	Do not treat Cloe as a drop-in replacement for a broad knowledge model.
Arithmetic	GSM8K $\approx$ 2.0%	Applications requiring reliable symbolic arithmetic need dedicated mechanisms.
Truthfulness	TruthfulQA 28.3%; high unanswerable-question confabulation	Safety-sensitive factual deployment requires task-specific safeguards.
Language modelling	LAMBADA perplexity 12.1 $\times$ worse; WikiText-2 3.1 $\times$ worse	General free-form generation quality is materially degraded.
Whole-checkpoint compression	Embeddings remain BF16; overall checkpoint $\approx$ 6.47 bits/parameter	The 1.58-bit figure should be reserved for the ternary weight representation.
Runtime memory	Packed peak $\approx$ 1.67 GiB	Disk compression should not be interpreted as equal runtime-memory compression.

Table 13: Selected limitations and what they imply for deployment.

Panel of Figure 6 — Figure 6: Conceptual deployment thesis suggested by the combined experiments.

Scenario	Why compression matters	Evidence / condition
Edge / constrained inference	Smaller model artifact can reduce storage and transfer burden	Current packed artifact is 634.7 MB.
Private or local assistants	Smaller local model can reduce dependence on remote inference	Requires further end-to-end device validation.
Task-specific classifiers	General knowledge is less important than adaptation	SST-2 reaches 95.6% matched-teacher retention.
Specialised summarisation	Task learnability matters more than broad pretrained knowledge	XSum reaches 79.4% teacher retention after adaptation.
Cost-sensitive inference	Throughput and representation size can affect serving economics	Current packed path shows +22.3% prefill and +24.2% decode versus latent measurement.
General-purpose factual assistant	Broad pretrained knowledge is essential	Not recommended from current evidence.

Table 14: Potential application mapping. These are deployment hypotheses, not claims that every scenario has been experimentally validated in this study.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 362 papers

RoleQ: Role-Aligned Scales for Converting Language Models to Ternary
Oct 15, 2026Johannes Knittel, Alec Helbling, Hanspeter PfisterLLM Quantization Neural Network Quantization
Harvard University · Georgia Tech
Post-Training Ternarization of Qwen3-4B Capability, Effective Bit Budget, Storage Compression, and Deployment
Sep 2, 2026Anirudh Malik, M Sparsh Mehra, Poojith DevanLLM Quantization Ternary Quantization
OneBit AI
Low-Rank Ternary Adaptation for Fine-Tuning Transformers
Aug 25, 2026Alexandru-Dragos Manolache, Yunqiang Li, Jan van GemertParameter-Efficient Fine-Tuning Low-Rank Adaptation
Delft University of Technology, The Netherlands · Amazon Development Center

Oct 15, 2026J/K move · Enter open · S save

RoleQ: Role-Aligned Scales for Converting Language Models to Ternary

Johannes Knittel, Alec Helbling, Hanspeter Pfister

Harvard University · Georgia Tech

LLM Quantization Neural Network Quantization Quantization Ternary Quantization Post-Training Quantization Weight-Only Quantization

Open paper AskPDF arXiv

Weight quantization below four bits shrinks memory and bandwidth at a perplexity that looks acceptable, but often at the cost of factual knowledge and task accuracy. Ternary weights have mostly been obtained by training from scratch or long fine-tuning. We introduce RoleQ, a quantization framework that places scales by the functional roles of Transformer units rather than by the layout of the weight matrix, as conventional formats do. Vectors that read from the residual stream take one scale per output unit; those that write to it, one per input unit. The rule is structural, needs no search, calibration or stored state, and holds on integer and ternary grids. In a rotated basis it matches a per-layer axis search at half the solver work, and at ternary it cuts perplexity by up to half against layout-based placement in two families at every size from 0.6B to 14B, where per-row scales leave MMLU at chance. Our lightweight post-training pipeline converts an existing checkpoint to 1.59 bits per weight in a few GPU-hours, a fraction of the cost of learned ternarization, and leads every post-training ternary method at matched or lower memory-resident bits on perplexity, zero-shot accuracy and MMLU, most widely on knowledge. A sparse second ternary plane placed by a per-unit Fisher saliency keeps the matmul multiplication-free and recovers what ternary loses first: at 8B, a fraction of a bit returns about half the MMLU that conversion costs, well ahead of random or Hessian-based placement at equal bits.

Abstract

LLM Quantization

Explore similar work

RoleQ: Role-Aligned Scales for Converting Language Models to Ternary

Post-Training Ternarization of Qwen3-4B Capability, Effective Bit Budget, Storage Compression, and Deployment

Low-Rank Ternary Adaptation for Fine-Tuning Transformers