Training language models with ternary weights is commonly judged by loss and downstream accuracy, which record only a modest cost relative to full precision. We show that these metrics can conceal a much larger failure. We instead measure knowledge capacity, the factual bits stored per parameter, on synthetic biographies with known information content. We train GPT-2-style models from scratch with 2.5M to 50M parameters at five precisions. Under the standard cosine schedule, ternary models retain as little as 6% of an identically trained fp16 model's capacity. The deficit widens with model size while perplexity rises by only 1.4 to 1.6 times. Measured throughout training, these models first acquire capacity and then lose most of it. We identify this knowledge collapse as a learning-rate dwell instability. Held near 1--2×10−4 with no decay, a pre-collapse model collapses within a few hundred exposures, and returning to a safe rate does not restore capacity. We then locate the collapse in the output head. It happens at a value the model can never predict. The weights there grow unchecked, while every other prediction the model makes is unchanged. We find that making the value predictable removes the collapse, regardless of which attribute carries it. A warmup-stable-decay schedule with a 10% cooldown increases ternary capacity by 2.7 times at 25M and 4.3 times at 50M. Gains are larger at lower precision. Cutting the output head's learning rate prevents failure when training from scratch. The post-training quantization methods we tested recover no measurable capacity below 4 bits. Our findings suggest judging low-precision training by retained capacity during training rather than final loss.
Figures & tables
retention ( ↑ )
2.5M
10M
25M
50M
int8
0.979
0.936
0.926
0.754
int4
0.893
0.686
0.581
0.316 †
int3
0.660
0.468
0.320
0.504
ternary
0.364
0.235
0.090 †
0.058 †
Table 1: Ternary retention falls with model size while the loss penalty stays small. Left : retention Rquant/Rfp16 under the standard cosine protocol; † : depends on the clamp convention (Table 9 ). Right : the perplexity penalty ternary pays relative to fp16 against its capacity gap; PPL ×=eΔloss , capacity gap =1/retention . Appendix A gives the seed count of every cell.
Figure 1: When the knowledge load triples, fp16 stores more and ternary stores less. Learned megabits, R⋅P , at 25M (mean over seeds).
Figure 2: Continuous capacity traces, 25M ternary at full load, one line per run, against the fraction of each run’s schedule. Left : decaying schedules (cosine at 1× and 4× budget, cosine to a 30% floor). Right : constant LR, WSD with a 10% cooldown, and WSD at light load; the dotted line is the light-load reference.
held LR (hold)
final R (s0 / s1, ↑ )
onset (s0 / s1)
outcome
5.0×10−4 (250)
0.348 / 0.360
–
flat (control)
4.0×10−4 (500)
0.355 / 0.368
–
stable
3.0×10−4 (500)
0.362 / 0.107 †
– / 470
s1 late collapse
2.0×10−4 (500)
0.101 † / 0.191 †
220 / 410
both collapse; s1 flickers, then collapses
1.5×10−4 (500)
0.236 / 0.109 †
260 / 170
both collapse; s0 partial recovery at the end
1.2×10−4 (500)
0.105 † / 0.182 †
270 / 290
erosion to 250, then both collapse
Table 2: Held-learning-rate dwell map, 25M ternary full load, branched from the pre-collapse constant-LR checkpoints: final R per seed and, for collapses under the § 3 criterion, onset in exposures of dwell; hold length in parentheses. † : depends on the clamp convention (Table 9 ).
precision
cosine R ( ↑ )
WSD R ( ↑ )
gain
fp16
1.720 ( n=6 )
1.644 ( n=3 )
−4%
int8
1.593 ( n=2 )
1.684 ( n=2 )
+6%
int4
0.999 ( n=2 )
1.229 ( n=2 )
+23%
int3
0.551 ( n=2 )
0.898 ( n=3 )
+63%
ternary
0.155 ( n=6 )
0.412 ( n=5 )
+166%
Table 3: Schedule gain by precision at 25M, full load: capacity under standard cosine vs WSD with a 10% cooldown, and the relative gain.
Figure 3: Left : 25M ternary full-load traces (thin: seeds; bold: mean) under five schedules, each schedule’s learning rate against the hazard window below. Right : final R against exposures spent in the window.
schedule (retention)
10M ( ↑ )
25M ( ↑ )
50M ( ↑ )
standard cosine
0.235 ( n=2 )
0.090 ( n=6 )
0.058 ( n=3 )
per-attribute floor
0.235
0.126
0.109
WSD, 10% cooldown
0.257 ( n=3 )
0.240 ( n=5 )
0.247 ( n=3 )
WSD, 33% cooldown
0.241 ( n=2 )
0.150 † ( n=3 )
0.067 † ( n=1 )
Table 4: Ternary retention by size, standard cosine versus WSD. Indented row: the cosine cells under the per-attribute floor; † : depends on the clamp convention (Table 9 ).
variant
what it changes
final R (s0 / s1, ↑ )
onset
collapse
unmodified
—
0.120 / 0.163
700 / 750
yes
keep date
control for the re-sizing of N
0.127 / 0.138
525 / 550
yes
no birth date
removes the unlearned token
0.372 / 0.355
–
no
no month
removes the unlearned token only
0.365 / 0.387
–
no
month from name
makes the month learnable
0.376 / 0.387
–
no
date last
moves the date to the final sentence
0.108 / 0.136
550 / 675
yes
Table 5: The collapse requires a token the model never learns, and is indifferent to which attribute carries it and where it sits. Corpus variants at 25M ternary under the standard cosine schedule, two seeds each; final R per seed and onset in exposures under the § 3 criterion. N is re-sized so that every variant holds 2.25 bits per parameter. Dashes: no onset; the arm never meets the criterion.
method
int8
int4
int3
ternary
RTN
1.710
0.136
0.000
0.000
AWQ
1.672
0.122
0.000
0.000
GPTQ
1.674
0.142
0.000
0.000
NF4
–
0.174
–
–
QAT, cosine (ours)
1.593
0.999
0.551
0.155
Table 6: Below int4 no post-training method recovers measurable capacity. Capacity R on identical 25M checkpoints and lattice ( ↑ ); 10M in Appendix J .
Table 7: Hyperparameters shared by every run unless a section states otherwise.
size
P
N (full load)
2.5M
1.95M
75k
10M
9.29M
371k
25M
27.3M
1.12M
50M
47.1M
1.96M
Appendix
Table 8: Model sizes. The size is the label used throughout; P is the built model’s parameter count, which R divides by; N is the number of people at full load.
Figure 4: Retention at every cell of Table 1 , under the standard cosine protocol. Dashed: the ternary cells under the per-attribute floor. Open markers: the seeds of the two cells with more than two.
cell
quantity
aggregate clamp
per-attribute floor
Morris
10M ternary cosine
retention
0.235
0.235
0.200
25M ternary cosine
retention
0.090
0.126
0.185 ¶
50M ternary cosine
retention
0.058
0.109
−0.250‡
25M ternary WSD 10%
retention
0.240
0.240
0.224
25M ternary WSD 33%
retention
0.150
0.167
0.171
50M ternary WSD 33%
retention
0.067
0.125
−0.484‡
Appendix
Table 9: Every quantity that depends on the clamp convention, under all three estimators: the † cells of Tables 1 , 2 and 4 , with pre-collapse reference cells for calibration; held-rate rows are the 500-exposure checkpoints. Dashes: not run. \lx@paragraphsign : Morris value over the two grid seeds only. ‡ : one 50M cosine seed scores −1.78 under our Morris implementation, which caps held-out cross-entropy at logV (uncapped, +0.643 ); wherever negative, the column is a signed compression gain (Appendix K ).
Figure 5: Precision curriculum (three seeds) against ternary-from-scratch on the identical learning-rate path (two seeds); stage boundaries marked, the WSD level dashed for reference.
estimator
final R
collapse
cosine, from scratch
clipped (six seeds)
0.118–0.253
6 of 6
identity
0.245 / 0.264
no
stochastic rounding
0.327 / 0.322
no
WSD 10%, from scratch
clipped
0.410 / 0.426
no
identity
0.308 / 0.324
no
stochastic rounding
0.343
no
Appendix
Table 10: The collapse needs a clipped-trained model, not a clipped backward pass. Final R at 25M ternary, full load; the upper block trains from scratch for 1000 exposures, the lower holds a parent at 1×10−4 for 300 exposures with the stated backward pass.
Figure 6: Learning rate at collapse onset for every collapsing run with a logged trace. The shaded band spans the 25M values, which § 6 links to the hazard at each rate and how long cosine spends there.
Figure 7: Hazard map from Table 2 : dwell exposures to sustained collapse at a held learning rate. Color: optimizer; shape: seed; filled: collapse; half-filled: survived but eroded; open: held capacity (censored at the hold length, dotted). Shading marks where the hazard is strongest.
Figure 8: Held-LR runs of Table 2 , branched from the pre-collapse constant-LR checkpoints, one panel per held rate.
Figure 9: Recovery: three collapsed checkpoints returned to the safe constant rate 5×10−4 for 250 exposures, against a fresh model at the same rate.
Lineage
Arm
Final R
Peak
Min
First < 0.25
Onset
Tail R
Tail BD nll
0
baseline
0.117
0.435
0.115
120
–
0.254
10.56
0
head /10
0.322
0.449
0.283
–
–
0.328
9.32
0
embed /10
0.179
0.447
0.114
245
–
0.247
10.47
0
both /10
0.289
0.446
0.280
–
–
0.330
9.30
1
baseline
0.126
0.449
0.129
250
–
0.287
10.24
1
head /10
0.371
0.471
0.359
–
–
0.374
8.86
Appendix
Table 11: Interface-split arms: full-corpus final R , peak and minimum of the subsample trace, the exposure at which the trace first falls below 0.25 and the criterion onset (“–” if never within the horizon), and tail means over the last 50 exposures. Horizons are 300 exposures for lineages 0 and 1 and 500 for lineage 2, whose baseline concatenates the 250-exposure hold and its extension.
Figure 10: Interface split on three parent lineages. Each panel branches one pre-collapse 25M ternary parent (filled marker) to a held body rate of 10−4 , evaluated every five exposures on the subsample; the dashed baseline scales nothing, the coloured arms divide the head, the embedding, or both by ten with weight decay compensated. Dotted lines mark the 0.25 and 0.20 levels of the collapse criterion.
checkpoint
R
month
day
year
year top-1
name
city
max∣z∣
month row
constant LR s0
0.338
2.58
3.20
2.97
0.322
16.81
4.59
356
18.6
constant LR s1
0.353
2.59
3.13
3.06
0.305
16.70
4.45
379
–
WSD 10% s0
0.410
2.70
3.04
2.48
0.395
16.55
4.42
685
19.0
head rate /10
0.371
2.57
2.73
2.86
0.325
16.76
4.63
434
2.5
cosine s1
0.120
7.22
3.18
3.00
0.309
16.89
4.48
10,288
–
cosine (grid s0)
0.122
9.07
3.32
2.61
0.378
16.87
4.42
9,752
39.7
Appendix
Table 12: A collapsed checkpoint has lost nothing except the month. Per-attribute loss in nats under an fp32 forward, on the corpus of each checkpoint’s own seed; max∣z∣ is over the four date-token positions and “month row” is the mean norm of the 12 month rows of the output head. Pre-collapse rows are the constant-LR parents and the two settings that do not collapse; collapsed rows meet the § 3 criterion.
size
precision
R
month
month top-1
month row
all rows
max∣z∣
cosine
25M
fp16
1.722
2.70
0.071
11.2
2.70
1,878
25M
int8
1.670
2.16
0.341
8.0
2.05
1,132
25M
int8
1.515
3.42
0.067
36.8
3.62
6,705
25M
int4
1.016
4.67
0.068
33.9
1.84
4,145
25M
int4
0.982
5.84
0.067
35.6
1.73
5,109
Appendix
Table 13: The month loss sits at or above its uniform value of ln12=2.485 nats in all but two checkpoints, and the output-head rows that predict it grow as precision falls. One final checkpoint per row under an fp32 forward, scored on its own seed’s corpus. “month row” and “all rows” are mean ℓ2 norms; max∣z∣ is over the four date positions.
variant
N
WSD 10% final R
unmodified
1,123,724
0.410 / 0.426
keep date
1,561,465
–
no birth date
1,610,874
0.412 / 0.408
no month
1,204,834
0.411
month from name
1,204,834
0.424
date last
1,123,724
–
Appendix
Table 14: The re-sizing each variant required, and its cooldown arm where one was run. N is chosen so that every corpus holds 2.25 bits per parameter; cosine values are in Table 5 . Dashes: not run.
schedule
placement
final R (s0 / s1)
behavior
cosine
AdamW everywhere
0.253 / 0.120
peak 0.34 at 550–600, collapse
cosine
Muon body, AdamW embed/head
0.158 / 0.204
peak 0.33 at 625–700, collapse
cosine
AdamW, embed/head frozen from 50%
0.182 / 0.276
gradual decline, 600–900
cosine
AdamW, embed/head rate /10
0.265 / 0.126
peak 0.31 at 650–675, collapse
cosine
AdamW, head rate /10
0.371 / 0.375
monotone rise, no collapse
cosine
AdamW, embedding rate /10
0.217 / 0.126
collapse, onset 825 / 700
Appendix
Table 15: Optimizer placement × schedule, 25M ternary full load, two seeds per cell (final R ). The embedding and output head are int4 in every row, and everything not on Muon ( Jordan et al., 2024 ) is on AdamW (Appendix A ). The rate rows scale the learning rate of the named matrices only; “ ηλ held” raises their weight decay by the same factor.
held LR
dwell-map outcome
own R
swap R (both directions)
% of parent
5.0×10−4
flat
0.348
0.281 / 0.303
86
4.0×10−4
stable
0.353
0.286 / 0.297
86
3.0×10−4
mild erosion
0.305
0.019 / 0.014
5
1.0×10−4
collapsed
0.117
0.000 / 0.000
0
Appendix
Table 16: Pre-collapse same-lineage swap control (evaluation only). Each branch starts from the same parent ( R=0.338 ) and is held at a fixed rate for 250 exposures; “swap” pairs the parent’s body with the branch’s embedding and head, and vice versa. Unswapped checkpoints reproduce their recorded R .
Figure 11: Capacity traces for the optimizer-placement runs of Table 15 under cosine (thin: seeds; bold: mean).
Figure 12: Capacity by method and bit-width at 10M and 25M: QAT (green) against the PTQ methods (blue family). PTQ curves quantize seed 0 of each fp16 cell, so their int8 values track that seed; the QAT curve is the cell mean, as in Table 6 .
Figure 13: Attribute-level view of the collapse. Left : per-attribute learned bits per person in one 25M ternary cosine run. Middle : birth date across four cosine seeds and a WSD control. Right : name-prediction loss of the crashed 50M fp16 seed.
Figure 14: Capacity R re-measured while perturbing each trained 25M checkpoint over the same parameter-space range ( α,β∈[−0.04,0.04] , filter-normalized, on every panel). The ternary curve is centered on the two grid_25m seeds, not the pooled six-seed cell.
Figure 15: One-dimensional capacity basin profiles through trained 25M checkpoints at each precision.
Extreme low-bit inference offers a route toward smaller models and constrained deployment. Ternary language models restrict weights to {−1,0,+1}, approaching the limit of log23≈1.585 bits/weight. The practical question for a pretrained model is not simply whether weights can be quantised but which capabilities survive and whether it remains useful for adaptation. We explore this by converting Qwen3.5-0.8B (752M parameters) to ternary weights using 72.4M tokens of quantisation-aware training (QAT). The resulting model, Cloe, is evaluated across 29 benchmarks, representation diagnostics, and downstream fine-tuning. The evidence shows non-uniform degradation. A linear probe recovers 43.76% of MMLU answers from the full-precision teacher's representations but only 26.19% from Cloe (near chance), indicating specialist factual information is lost. However, Cloe retains measurable performance on ten tasks, averaging 77.1% of teacher performance. Crucially, fine-tuning raises Cloe to 89.8% on SST-2 (95.6% of the matched teacher) and reaches 79.4% teacher retention on XSum. We attribute degradation to a combination of quantisation-induced information loss and incomplete recovery due to the limited QAT budget. We also highlight an evaluation pitfall: standard answer-letter scoring failed (Cloe emitted "A" on 98.6% of MMLU questions), necessitating continuation scoring. Ultimately, ternary conversion is unsuitable as a drop-in general replacement yet remains valuable as a compact substrate for task-specific models.
Ultra-low-bit language models can reduce storage and memory bandwidth, but a nominal "1.58-bit" label does not fully describe the stored representation, retained capability, or runtime behavior. We study an end-to-end post-training conversion of Qwen, an instruction-tuned 4B-parameter model, using KOTMS rotation, E2M-ATQ ternarization, and GPTQ-style error compensation from TWLA. The experiment is weight-only: activations remain at 16-bit precision, so ILA-AMP is omitted. We evaluate effective bit accounting, task capability retention, perplexity, calibration sensitivity, checkpoint composition, and deployment behavior. The final conversion uses 1.641 effective bits per weight for quantized linear weights, with 81.62% of model parameters targeted. Across ten scored capability comparisons, accuracy falls from 64.5% to 54.7%. Degradation is uneven: BoolQ retains 84.6% chance-corrected teacher performance, while ARC-Challenge retains 43.8%. Perplexity rises from 13.639 to 18.748 on WikiText-2, 24.700 to 31.992 on PTB, and 19.831 to 28.966 on C4. A subsequent packing run preserves the ternary planes and scales, reducing reported model size from 8.29 GiB to 3.96 GiB with essentially unchanged perplexity. A separate third-party packing attempt was lossy and is excluded from the primary artifact claim. The packed artifact has not been benchmarked end-to-end for task accuracy or generation throughput. A preliminary Triton GEMV microbenchmark is 4.6x slower than FP16 cuBLAS on one tested shape. We therefore do not claim that compression alone yields faster inference.
Weight quantization below four bits shrinks memory and bandwidth at a perplexity that looks acceptable, but often at the cost of factual knowledge and task accuracy. Ternary weights have mostly been obtained by training from scratch or long fine-tuning. We introduce RoleQ, a quantization framework that places scales by the functional roles of Transformer units rather than by the layout of the weight matrix, as conventional formats do. Vectors that read from the residual stream take one scale per output unit; those that write to it, one per input unit. The rule is structural, needs no search, calibration or stored state, and holds on integer and ternary grids. In a rotated basis it matches a per-layer axis search at half the solver work, and at ternary it cuts perplexity by up to half against layout-based placement in two families at every size from 0.6B to 14B, where per-row scales leave MMLU at chance. Our lightweight post-training pipeline converts an existing checkpoint to 1.59 bits per weight in a few GPU-hours, a fraction of the cost of learned ternarization, and leads every post-training ternary method at matched or lower memory-resident bits on perplexity, zero-shot accuracy and MMLU, most widely on knowledge. A sparse second ternary plane placed by a per-unit Fisher saliency keeps the matmul multiplication-free and recovers what ternary loses first: at 8B, a fraction of a bit returns about half the MMLU that conversion costs, well ahead of random or Hessian-based placement at equal bits.
Johannes Knittel, Alec Helbling, Hanspeter Pfister