Baseline Shape Decides the Verdict: A Controlled Re-Examination of Ternary Language Models at 60K Parameters

Gautam Veldanda

cs.CLSep 24, 2026

Baseline Shape Decides the Verdict: A Controlled Re-Examination of Ternary Language Models at 60K Parameters

Authors: Gautam Veldanda

Organizations: Independent Researcher

Abstract

Ternary (1.58-bit) weights are attractive for microcontroller-class language models, but the sub-1M-parameter regime rests mainly on isolated, single-seed comparisons. One prominent example reports that a routed ternary block (convolution, diagonal SSM and sparse attention mixed by a per-token router) beats a parameter-matched full-precision transformer by 22% at 60K parameters, attributing this to inductive bias. We re-run it under one fixed recipe, three seeds per cell, 98 byte-level runs on one laptop. (i) Baseline shape dominates: at a 16M-byte budget, param-matched transformers span 22.6% in validation loss purely by depth/width choice - far more than any architecture effect we measure there - and the best-shaped transformer ties the routed model, so the published margin is at least partly a baseline-shape effect; the ordering of shapes reverses with budget, so no single fixed shape can be trusted. (ii) At 130M bytes the routed model does win, by 22.2-24.0% over the three transformer shapes we evaluate there - but a plain gated diagonal-SSM block beats it by a further 9.1%, and the routed model's own router puts most of its weight on its recurrent pathway, so the gain does not require routing. (iii) The ternary penalty differs by architecture at the larger budget (+5.3% best transformer vs. +19.5% routed, +28.1% gated SSM), but we cannot attribute that to architecture alone: our transformers keep learned positional embeddings in full precision, 11-22% of their parameters, so they are less quantized than the models they are compared with. (iv) A 90/10 full-precision-then-ternary schedule beats all-ternary training, but only at a stage-2 learning rate about 10x the pretraining peak; at a conventional fine-tuning rate it looks 15.3% worse, reversing the conclusion. The from-scratch baseline was not itself learning-rate tuned, which bounds (iii) and (iv). Code and run logs released.

Figures & tables

	60K comparison	944K comparison
Result (routed ternary vs. FP32 transformer)	6.31 vs. 8.12 ppl (win)	1.0545 vs. 0.9337 nats/byte (loss)
Training tokens	$\approx$ 3M (3,000 steps $\times$ 16 $\times$ 64)	$\approx$ 2B (30,000 $\times$ 256 $\times$ 256)
Tokens per parameter	$\approx$ 50	$\approx$ 2,100
Corpus	0.5 MB TinyStories slice	$\sim$ 1.7 GB TinyStories (reported)
Sequence length	64	256
LR schedule	constant $3\!\times\!10^{-4}$	warmup 1000, cosine $3\!\times\!10^{-4}\!\to\!3\!\times\!10^{-5}$

Table 1: Conditions of the two published Atome-LM comparisons.

Experiment	cells	seeds	runs
Main grid (routed, $3\times 32$ transformer)	8	3	24
Shape sweep (5 shapes @16M, 3 @130M)	12	3	36
Gated SSM	4	3	12
FP-init arms (2 architectures $\times$ 2 stage-2 LRs, incl. stage 1)	6	3	18
Stage-2 LR sweep	10	1	10
Total			98 distinct

Table 2: Run accounting. Two of the tuned FP-init arms’ seed-0 runs are the corresponding sweep runs reused rather than retrained, so 100 run directories correspond to 98 distinct trainings.

System	configuration	params	FP kept	what stays full precision
Routed 3-pathway	$d{=}64$ , 4 layers, $d_{\text{head}}{=}16$ , top- $k{=}4$	60,800	2.3%	SSM scalars, LayerNorms
Gated SSM (ours)	$d{=}44$ , 3 layers, $d_{\text{ff}}{=}2d$	64,196	1.6%	SSM scalars, LayerNorms
Transformer $1\times 52$	$d{=}52$ , 1 layer, $d_{\text{ff}}{=}2d$ , 4 heads	61,880	22.0%	+ positional embedding
Transformer $2\times 36$	$d{=}36$ , 2 layers, $d_{\text{ff}}{=}4d$ , 4 heads	59,112	16.2%	+ positional embedding
Transformer $3\times 32$	$d{=}32$ , 3 layers, $d_{\text{ff}}{=}4d$ , 4 heads	61,888	14.0%	+ positional embedding
Transformer $4\times 28$	$d{=}28$ , 4 layers, $d_{\text{ff}}{=}4d$ , 4 heads	59,640	12.9%	+ positional embedding

Table 3: Exact configurations. Every model ternarizes its token embedding, unembedding and all weight matrices (plus the depthwise conv and router, for the routed block). “FP kept” is the share of parameters that remain full precision: per-channel SSM scalars and LayerNorms everywhere, plus the transformers’ learned positional embeddings, which dominate that column.

System	FP32	ternary	FP32	ternary	penalty @130M
	16M bytes		130M bytes
Gated SSM (ours)	2.704 $\pm$ .041	3.376 $\pm$ .093	1.516 $\pm$ .012	1.942 $\pm$ .018	+28.1%
Routed 3-pathway (Atome)	2.808 $\pm$ .062	3.290 $\pm$ .065	1.668 $\pm$ .024	1.993 $\pm$ .032	+19.5%
Transformer $1\times 52$	2.835 $\pm$ .016	3.426 $\pm$ .034	2.195 $\pm$ .003	2.311 $\pm$ .003	+5.3%
Transformer $2\times 36$	3.080 $\pm$ .031	3.752 $\pm$ .048	2.169 $\pm$ .016	2.320 $\pm$ .002	+7.0%
Transformer $3\times 32$	3.239 $\pm$ .152	3.870 $\pm$ .094	2.143 $\pm$ .028	2.328 $\pm$ .004	+8.6%

Table 4: All systems at 60K parameters. Ternary penalty = (ternary

-

FP)/FP.

stage-2 LR	3e-5	1e-4	3e-4	1e-3	3e-3	arm (3 seeds)	loss	vs. scratch	$2\sigma_{\text{rel}}$
routed	2.758	2.335	2.110	1.944	1.911	routed, 1e-4	$2.299\pm.041$	$+15.3\%$	5.2%
transformer	2.701	2.474	2.348	2.298	2.265	routed, 3e-3	$1.900\pm.010$	$\mathbf{-4.7\%}$	3.4%
						transf., 1e-4	$2.475\pm.014$	$+7.1\%$	1.2%
						transf., 3e-3	$2.262\pm.003$	$\mathbf{-2.1\%}$	0.4%

Table 5: FP-init (90% FP

\to

10% ternary) vs. ternary from scratch, token-matched at 130M. Left: stage-2 LR sweep, seed 0. Right: three-seed arms. The transformer arm is the

1\times 52

shape in both the FP-init runs and the from-scratch reference they are compared against, so the comparison holds shape fixed.

Claim	Evidence	Status
Baseline depth/width choice can decide an architecture verdict at 60K	5-shape sweep, 2 budgets	supported
The published 60K advantage is explained by training budget (H-budget)	16M vs. 130M bytes	not supported
Routing is required for the routed block’s advantage	gated SSM comparison	not supported
A simple gated recurrence suffices to match or beat it	gated SSM, 2 budgets	supported
Recurrence per se is the causal ingredient	not isolated (FFN, mixing differ)	untested
The ternary penalty differs by architecture at 130M	3 families, 2 budgets	supported

Table 6: Claims against the evidence that supports them.

Explore similar work

Sep 14, 2026cs.AI

Breaking the 1.58-bit Barrier for Ternary LLMs

Ternary Large Language Models (LLM) store every weight as one of three symbols

\{-1,0,+1\}

, so the cost of a ternary model is conventionally referenced to the information-theoretic

\log_2 3 \approx 1.585

bits per weight. The prevailing deployment format packs five ternary weights into one byte (five-trit packing), and due to the power-of-two group sizes used in practice this rounds up to

1.625

bits per weight. This effective storage bit-width treats the three symbols

\{-1,0,+1\}

as equiprobable. We measure the actual symbol distribution of 29 ternary LLM models and find that zeros account for up to

51.5\%

of all weights. Motivated by this finding, we introduce BITCOS, a simple distribution-adaptive layout comprised of a dense presence bitmap plus a compacted sign vector, and costs

2 - z

bits per weight element given a zero density

z

in the model's weights. BITCOS stores weights more compactly than the five-trit packing in 26 of the 29 tested models, and reaches

1.485

bits per weight on the sparsest of them. BITCOS is amenable to efficient unpacking on modern processors and GPUs, and we present optimized unpacking sequences for AVX-512, AVX2 and Intel Xe2 GPUs. Measured against production state-of-the-art ternary matrix-vector multiplication kernels, at the zero densities real-world ternary models exhibit, the realized gain with our proposed layout is up to

1.28\times

. Finally, we illustrate end-to-end LLM inference results on 5 different platforms (client and server CPUs, integrated and discrete Xe2 GPUs) where decode throughput improves by up to

1.18\times

on CPUs and

1.27\times

on GPUs.

Evangelos Georganas, Alexander Heinecke, Pradeep Dubey

Intel Corporation

Large Language Model Quantization Model Weights Matrix Multiplication Distributions Barriers

Ask PDF

Jun 25, 2026cs.CL

CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs

In this paper, we present CAT-Q, Cost-efficient and Accurate Ternary Quantization, for compressing and accelerating LLMs. Unlike existing state-of-the-art ternary quantization methods that rely on data-intensive and costly quantization-aware training to mitigate severe performance degradation, CAT-Q is a simple yet effective post-training quantization scheme that is readily applicable to LLMs with diverse architectures and model sizes. It has two key components, learnable modulation (LM) and softened ternarization (ST), which are coupled from an optimization perspective. LM leverages a composition of learnable factors to modulate the distribution of pre-trained high-precision weights and the ternary threshold, making them less sensitive to ternarization. ST further introduces a differentiable transition function to guide the ternarization process toward stable convergence. We show that, for pre-trained LLMs with 1.7B to 8B parameters, CAT-Q can efficiently quantize them into ternary models using only 512 calibration samples, while achieving superior performance than the seminal BitNet 1.58-bit v1 and v2 families (with 1.3B to 7B parameters) trained with 100B tokens, yielding about a 100,000X reduction in training tokens. Moreover, we show for the first time that CAT-Q can quantize much larger pre-trained LLMs having 14B to 235B parameters into leading ternary models within just 8 to 60 hours on 8 A100-80GB GPUs. Code is available at https://github.com/IntelChina-AI/BitTern.

Shigeng Wang, Chao Li, Yangyuxuan Kang +2

1Intel Labs China

Large Language Model Quantization Quantization-Aware Training

Ask PDF

Jun 22, 2026cs.LG

Tapered Language Models

Modern language models, including transformer, recurrent, and memory-based variants, share a common chassis: a stack of identical layers in which parameters are allocated uniformly across depth. This is a default inherited from the original transformer and largely unchanged since, yet a growing body of evidence suggests that layers contribute non-uniformly to the final output, with later layers refining the residual stream rather than transforming it. We ask whether parameter capacity should reflect this asymmetry. Our controlled experiment shows that, under a fixed budget, allocating more capacity to earlier layers and less to later layers improves perplexity over a uniform-width baseline, while the reverse allocation hurts. Building on this result, we introduce Tapered Language Models (TLMs), an architectural principle in which a parameter-bearing component is monotonically tapered across depth under a fixed total budget. MLPs are the natural site for this instantiation: they dominate parameter count across all modern LM families and expose width as a single, clean axis of variation. Across three model scales and four architectures (Transformer, Gated Attention, Hope-attention, and Titans), tapering MLP width via a smooth cosine schedule consistently improves perplexity and downstream benchmark performance over uniform baselines, at no additional parameter or compute cost. These findings establish depth-aware capacity allocation as a simple, architecture-agnostic axis of language model design, a free lever hidden in plain sight.

Reza Bayat, Ali Behrouz, Aaron Courville

1Mila · 2Cornell University · 3Université de Montréal +1

Language Modeling Layer-Wise Transformer Architectures

Ask PDF