Error Diffusion Replaces Latent Weights: Training Binary-Weight Language Models on a Few Bits of Dithered State

Pierre-Olivier Bonin

cs.LGOct 25, 2026

Error Diffusion Replaces Latent Weights: Training Binary-Weight Language Models on a Few Bits of Dithered State

Authors: Pierre-Olivier Bonin

Organizations: Independent Researcher

Abstract

Binary-weight language models store one bit per weight, but training them keeps a full-precision latent copy of each weight plus Adam state, about 96 extra bits per weight, so a deployed binary model cannot be adapted on the device that runs it. We replace the latent weight with a 2-8-bit residual buffer quantized by stochastic rounding, which cuts the training state per weight from 97 bits to 3-9, up to 19x less. The resulting error-diffusion (ED) optimizer moves the residual-carrying loop of image halftoning onto the weight-update path. Each buffer accumulates the unrealized part of every update and flips the weight when the accumulated value crosses a threshold, and it works only when its quantization grid has a level exactly at that threshold. Under this threshold-alignment rule, 4-bit buffers match 8-bit buffers on a binary multilayer perceptron benchmark, and the same rule lifts a different latent-free optimizer, Bop, from a 10-point deficit at 4 bits to full-precision accuracy. On character-level transformers with 97% of parameters binary, ED trains within 5-11% of the latent-weight reference at a 2,000-step budget and within 1.7% at convergence. On scaling ladders over enwik8 and wikitext-103, up to 19M parameters on enwik8 (70M with the wikitext-103 embeddings), the latent-free loss stays within -2.3% to +2.9% of the reference at all six sizes once a per-step cap on the buffer update removes a launch instability; without the cap it trails by 10% at the largest tokenized size. A deployed binary model continues training under ED at one tenth of the latent training state, matching latent-weight continuation, and a fused CPU kernel completes the adaptation within a 275 MB training-memory budget, which latent-weight training cannot. A 250-step probe sets the flip learning rate at any new scale.

Figures & tables

Figure 1: MLP-benchmark test accuracy against the training state of the two binary layers ( $2W^{2}\times$ b/wt), widths 64–384, 5 seeds, $C{=}1.0$ . At the accuracy of the latent reference, the 8-bit buffer needs $11\times$ less training state and the 4-bit buffer $19\times$ less.

	ED buffer		dithered Bop EMA
bits	naive $C{=}1.5$	rule $C{=}1.0$	absmax grid	aligned grid
8	$56.4\pm 1.4$	$56.3\pm 1.4$	$55.9\pm 1.5$	$56.1\pm 0.6$
4	$53.2\pm 1.3$	$\mathbf{56.4\pm 1.3}$	$46.5\pm 1.3$	$\mathbf{57.0\pm 1.0}$
2	$33.1\pm 4.9$	$\mathbf{52.3\pm 1.2}$	$38.6\pm 2.1$	$\mathbf{51.0\pm 1.2}$

Table 1: Test accuracy (%, higher is better) at width 192, 5 seeds, for the ED buffer and for dithered Bop’s SR-quantized EMA (FP EMA

56.6\pm 1.5

at 33 b/wt), with and without a grid level at the threshold. Bold marks the better cell of a pair by more than one std. ED cells use flip-lr 1.0 (2-bit rule cell: best of four flip-lrs, 52.3 at 0.5; 51.8 at 1.0); Bop cells take the best of three

\tau

method	$d{=}128$	$d{=}256$	$d{=}512$	$d{=}256$
	2k steps			50k steps
FP32+Adam (96 b/wt)	$\mathbf{1.727\pm 0.045}$	$\mathbf{1.569\pm 0.008}$	$\mathbf{1.570\pm 0.009}$	1.591
latent reference (97 b/wt)	$1.971\pm 0.018$	$1.712\pm 0.008$	$1.576\pm 0.001$	$\mathbf{1.556}$
ED-int8 (9 b/wt)	$2.065\pm 0.021$	$1.851\pm 0.020$	$1.755\pm 0.012$	1.583 (1.586)
ED-int4 (5 b/wt)	$2.076\pm 0.030$	$1.855\pm 0.042$	$1.808\pm 0.047$	1.681

Table 2: Best validation loss (nats per character, lower is better) on character-level Shakespeare. 2k-step columns: FP activations, mean

\pm

std over 3 seeds, one flip-lr per ED cell (100, 60, 80 at

d

128, 256, 512). 50k-step column: binary activations, 1 seed; parentheses: constant flip-lr. Bold marks the lowest loss per column.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Error diffusion in halftoning and in the weight update. Panels (a) and (b) show Floyd–Steinberg halftoning, which renders a grayscale image with binary pixels by carrying each pixel’s rounding residual forward in scan order. Panel (c) shows the ED optimizer applying the same recurrence in time, on a simulated weight with an illustrative drive. A binary weight’s unrealized updates accumulate in a small quantized buffer (2-bit here), and a flip fires when stochastic rounding lands the buffer on the threshold level (§ 5.1 ). Panel (d) compares the cumulative demanded movement with the movement realized by flips. The flips follow the demand with a lag. The gap can exceed $C$ , because the clamp at $\pm C$ discards updates that push a weight further toward its current sign, and stochastic rounding adds zero-mean noise that the buffer does not carry forward.

cell	layers/heads/ $d$	binary / FP params	data, block, batch, steps (eval every)	binact	flip-lr	seeds
Table 3 , 2k	4/4/128, 256, 512	0.79M/52k, 3.1M/103k, 12.6M/207k	Shakespeare char, 256, 64, 2k (250)	no	100, 60, 80 (100, 60, 80)	0, 1, 2
Table 3 , 50k	4/4/256	3.1M/103k	Shakespeare char, 256, 64, 50k (1000)	yes	60 (45)	0
enwik8 ladder	2/4/128, 4/4/256, 6/8/512	0.39M/100k, 3.1M/201k, 18.9M/407k	enwik8 bytes, 256, 64, 15k (1000)	yes	25, 60, 180 (12, 30, 45)	0, 1, 2 (+3, 4 at 512)
wikitext-103 ladder	same three	0.39M/12.9M, 3.1M/25.8M, 18.9M/51.5M	GPT-2 BPE, 128, 64, 15k (30k at 512) (1000)	yes	40, 90, 225	0
deploy-then-adapt	4/4/256	3.1M/201k	Shakespeare bytes, 256, 64, 5k (250)	yes	60 (30)	0, 1, 2
1-bit gradients	4/4/256	3.1M/103k	Shakespeare char, 256, 64, 2k (250)	yes	adaptive drive 0.03	0, 1, 2

Appendix

Table 4: Transformer cells. “Layers/heads/

d

” give depth, head count, and width; “binary” is the number of binary weights and “FP” the FP remnant. Flip-lrs are for the 8-bit buffer with the 4-bit value in parentheses where a 4-bit cell exists. The deploy-then-adapt latent arm was swept over Adam learning rates

\{10^{-4},3{\times}10^{-4},10^{-3}\}

prediction or criterion	record	outcome
Transformer pass criterion: ED-int8 within $+20\%$ of the latent reference after a flip-lr sweep, else report the negative	docs/ADR.md (ADR-010), 2026-07-14	passed, $+4.8/+8.1/+11.4\%$ at $d$ 128/256/512 (3 seeds)
$d{=}512$ flip-lr optimum at 80–90 (drive $1.05\times 10^{-3}$ ), bracket $\{40,80,160\}$ , pass if the interior candidate wins	docs/scaling-plan.md , 2026-07-14	passed, 80 wins (1.782) with 160 tying (1.783)
Ladder gate: ED loss runs parallel to the latent reference across sizes, and the two controls (binary activations in both arms, re-probed flip-lrs) were named before running	docs/scaling-plan.md , docs/RESULTS_s2.md (S2.6), 2026-07-14/15	passed after the controls, $-2.5/+3.3/+3.9\%$
Threshold-aligned regrid rescues dithered Bop’s 4-bit collapse	docs/RESULTS_s2.md (S2.11), 2026-07-14 to 16	passed, 46.5 $\to$ 57.0 at 5 b/wt
Velocity-scaled drive predicts that heavy-ball momentum cells at flip-lr 1.0 destabilize	docs/RESULTS_s2.md (S2.4), 2026-07-15	failed, they did not destabilize (rule does not extend through momentum)
Second-order buffer: if 2+2 bits beats a 4-bit buffer the exchange curve improves, else it is dropped	docs/ADR.md (ADR-013), 2026-07-15	negative, dropped

Appendix

Table 5: Pre-registered predictions and criteria, in the order they were recorded.

Figure 3: Drive at step 250 against the candidate flip-lr for every probe and sweep run behind the transformer cells (three corpora, widths 128–512, 8-bit and 4-bit buffers, from-scratch and warm-started). Filled markers are the flip-lrs selected for the reported cells, and hollow markers are the rejected candidates. The shaded band is the selection rule $[1.0,1.3]\times 10^{-3}$ . Every selected flip-lr lies in the band or within 15% below it, while the selected flip-lrs themselves span 12–225.

$W$	64	96	128	192	256	384
latent reference (97 b/wt)	$40.9\pm 1.2$	$47.3\pm 1.1$	$52.3\pm 1.4$	$57.5\pm 1.0$	$59.5\pm 1.4$	$62.4\pm 0.3$
FP32+Adam (96 b/wt)	$42.8\pm 1.6$	$46.9\pm 1.4$	$50.3\pm 1.5$	$55.8\pm 1.0$	$58.3\pm 0.7$	$61.2\pm 0.6$
ED-int8 (9 b/wt)	$39.8\pm 1.2$	$46.0\pm 2.5$	$50.9\pm 1.2$	$56.3\pm 1.4$	$59.4\pm 0.6$	$62.3\pm 0.3$
dithered Bop int8 (9 b/wt)	$41.3$	$47.8$	$51.7$	$57.2$	$59.6$	$62.0$
ED-int4 (5 b/wt)	$40.3\pm 1.3$	$46.8\pm 1.6$	$50.4\pm 1.2$	$56.4\pm 1.3$	$59.8\pm 0.8$	$62.5\pm 0.5$

Appendix

Table 6: Test accuracy (%) on the MLP benchmark by width

W

, mean

\pm

std over 5 seeds. All ED and Bop series use binary activations and

C{=}1.0

; the FP32+Adam series uses FP activations.

method	9 b/wt	5 b/wt	3 b/wt
ED buffer ( $C{=}1.0$ )	56.3	56.4	52.3
saturating integer latent weight, SR updates, sign readout	56.2	56.1	52.1
magnitude-free sign counter [ Colombo et al., 2024 ]	$58.4\pm 0.8$	52.4	10.1
buffer plus SR-quantized momentum (budget split)	56.3	48.9
second-order $\Delta\Sigma$ cascade	29.1	22.5	–
ECO-style decayed carrier, $\beta=0.9/0.99/1$	45.4 / 55.3 / 56.3 at 8 buffer bits

Appendix

Table 7: Controls at width 192 on the MLP benchmark, test accuracy (%), 5 seeds, binary activations. Budgets are bits of training state per weight.

buffer bits	FP gradients	1-bit gradients	cost (points)
8	$56.3\pm 1.4$	$55.2\pm 1.3$	$-1.0$
4	$56.4\pm 1.3$	$48.6\pm 1.0$	$-7.7$
2	$52.3\pm 1.2$	$38.1\pm 0.9$	$-14.2$

Appendix

Table 8: Cost of 1-bit gradients (SR after a random-Hadamard transform of block 32, per-row absmax, no clipping) by buffer width at width 192 on the MLP benchmark, test accuracy (%), 5 seeds, binary activations,

C{=}1.0

step	val loss	flips/step	buffer $\bar{\|e\|}$	act. scale, blocks 0 / 1 / 2	grad, binary proxies	grad, FP remnant	LN gain mean / max	head norm
1800	2.107	36.7k	0.449	1.26 / 0.87 / 0.73	0.0106	0.81	1.012 / 1.16	19.4
1900	2.053	38.4k	0.455	1.28 / 0.89 / 0.74	0.0097	0.72	1.013 / 1.16	19.6
2000	3.488	79.2k	0.535	3.39 / 2.58 / 1.29	0.0018	0.45	1.014 / 1.17	19.7
2100	3.078	4.0k	0.530	3.22 / 2.59 / 1.31	0.0041	0.55	1.014 / 1.18	19.6
2500	2.685	18.3k	0.520	2.71 / 2.38 / 1.31	0.0063	0.57	1.013 / 1.18	20.4
3000	2.490	23.4k	0.516	2.50 / 2.02 / 1.22	0.0073	0.69	1.012 / 1.17	21.0

Appendix

Table 9: The captured avalanche, initialization 0.5, seed 6: diagnostics at the 100-step evaluations around the event. The FP-remnant columns (LayerNorm gains, head norm, remnant gradient) stay on their prior trend while the binary core’s columns jump.

Figure 4: The same two launches at 100-step resolution: validation loss, flips per step, first-block pre-activation scale, and the gradient norm reaching the binary proxies. The marker is the collapsed launch at step 2000. Everything that defines the event happens inside one evaluation window; recovery runs at a tenth of the normal flip rate.

Artifact	Script(s)	Log(s)
Fig. 2	tools/plot_halftone_concept_sim.py	self-contained, fixed seed
MLP headline, Fig. 1	headline.sh , widthsweep.sh , widthsweep_int4c1.sh , r4_fixes.sh	headline_5seed , widthsweep_5seed , widthsweep_int4c1 , r4_widthsweep_int8c1
Table 1 (ED)	p2clamp.sh , p2boundary.sh	p2clamp , p2boundary_5seed
Table 1 (Bop)	s2bopalign.sh , s2bopalign2.sh	s2bopalign , s2bopalign2
Integer latent, counter, ECO-style carrier	s2intlat.sh , s2baselines.sh	s2intlat , s2baselines
Second-order buffer	s2sigma.sh	s2sigma

Appendix

Table 10: Scripts ( scripts/ , tools/ ) and JSONL logs ( results/ ) behind each artifact. The files of the last row are on the repository branches whose names start with r5-e1 to r5-e5 .

Binary Neural Networks

Binary neural networks use one-bit weights or activations to reduce model storage and computation for efficient learning and inference.

See all 29 papers

Capability-Stratified Degradation in Ternary Language Models
Aug 28, 2026Anirudh Malik, M Sparsh Mehra, Poojith DevanLLM Quantization Ternary Quantization
OneBit AI
E8-QAT: Efficient 2-Bit Quantization-Aware Training with Implicit Lattice Codebooks
Oct 20, 2026Xinyu Lei, Hong Huang, Jiebin Zhang +6LLM Quantization Quantization-Aware Training
Peking University · City University of Hong Kong · Tencent
Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Models
Jul 3, 2026Hamish OgilvyLLM Inference Acceleration Mixed-Precision Quantization
Algolia Research

Aug 28, 2026J/K move · Enter open · S save

Capability-Stratified Degradation in Ternary Language Models

Anirudh Malik, M Sparsh Mehra, Poojith Devan

OneBit AI

LLM Quantization Ternary Quantization

Open paper AskPDF arXiv

Extreme low-bit inference offers a route toward smaller models and constrained deployment. Ternary language models restrict weights to

\{-1,0,+1\}

, approaching the limit of

\log_2 3 \approx 1.585

bits/weight. The practical question for a pretrained model is not simply whether weights can be quantised but which capabilities survive and whether it remains useful for adaptation. We explore this by converting Qwen3.5-0.8B (752M parameters) to ternary weights using 72.4M tokens of quantisation-aware training (QAT). The resulting model, Cloe, is evaluated across 29 benchmarks, representation diagnostics, and downstream fine-tuning. The evidence shows non-uniform degradation. A linear probe recovers 43.76% of MMLU answers from the full-precision teacher's representations but only 26.19% from Cloe (near chance), indicating specialist factual information is lost. However, Cloe retains measurable performance on ten tasks, averaging 77.1% of teacher performance. Crucially, fine-tuning raises Cloe to 89.8% on SST-2 (95.6% of the matched teacher) and reaches 79.4% teacher retention on XSum. We attribute degradation to a combination of quantisation-induced information loss and incomplete recovery due to the limited QAT budget. We also highlight an evaluation pitfall: standard answer-letter scoring failed (Cloe emitted "A" on 98.6% of MMLU questions), necessitating continuation scoring. Ultimately, ternary conversion is unsuitable as a drop-in general replacement yet remains valuable as a compact substrate for task-specific models.

Abstract

Binary Neural Networks

Explore similar work

Capability-Stratified Degradation in Ternary Language Models

E8-QAT: Efficient 2-Bit Quantization-Aware Training with Implicit Lattice Codebooks

Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Models