Organizations: Department of Electrical and Systems Engineering, Washington University in St. Louis, One Brookings Drive, St. Louis, 63130, MO, USA. · Department of Electrical and Computer Engineering, University of California, Santa Cruz, 1156 High Street, Santa Cruz, 95064, CA, USA. · Department of Computer Science and Engineering, Washington University in St. Louis, One Brookings Drive, St. Louis, 63130, MO, USA.
The training loss of binary and ternary neural networks, expanded as a multilinear polynomial over the discrete weights, is structurally equivalent to the energy of a hierarchical high-order Ising model. We exploit this equivalence to apply Ising solvers to discrete-weight network training, while accounting for a fundamental difference in objectives: an Ising solver seeks the ground state of a fixed energy function, whereas training seeks solutions that generalize to unseen data. Our key contribution is ΣΔ-Sampler, an optimizer that reconciles these objectives by embedding a stochastic sampling-and-annealing step as a noisy quantizer inside a first-order ΣΔ noise-shaping loop. The loop pushes noise from weight quantization, high-order frustration, and mini-batch sampling out of the low-frequency adaptation band, so that discrete weight updates follow the smooth descent trajectories embedded in the rugged loss landscape. We validate the framework on a planted quadratic unconstrained binary optimization (QUBO) benchmark with a known minimizer; on CIFAR-10, where a 13M-parameter ternary model comes within 1% of full-precision accuracy; on ImageNet-1k, where a ternary ResNet-18 comes within 0.4% Top-5 accuracy of a ternary baseline; on class-incremental continual learning, where prior-task accuracy is preserved; and on MatMul-free language models up to 2.7B parameters, where it outperforms a straight-through-estimator baseline under matched continued pre-training. Compared with standard quantization-aware training, ΣΔ-Sampler reduces the training memory footprint by over 50%, delivers an order-of-magnitude speed-up, and reduces weight-update memory traffic by four orders of magnitude, making it attractive for on-device training and fine-tuning of large binary and ternary models.
Figures & tables
Figure 1: ΣΔ -Sampler for training discrete neural networks. (a) A feedforward binary neural network with weights constrained to W∈{−1,+1}D . (b) Equivalence between discrete parameter neural networks and a hierarchical high-order Ising machine under stationary data distribution (x,y) . (c) Conceptual loss/energy profiles for increasing Ising order, where low-order expansions give a smooth landscape and higher-order couplings add frustration, many local minima, and high-frequency components in the gradient space. (d) A general sampler architecture for training discrete weight neural networks. (e) The effect of noise-shaping in spectral domain, where the white-noise is pushed out of the low-frequency adaptation band towards high-frequency. (f) Histogram showing noise-shaping of the Ising decomposition, resulting in the smoother optimization contour that benefits generalization. (g) Rugged non-convex optimization landscape due to parameter discretizations, where traditional simulated annealing optimizers are slow and do not perform well. (h) Optimization trajectory of the ΣΔ -Sampler on the smoothened landscape is more stable and computationally scalable. (i) Parameter update rate maps for a ΣΔ -Sampler at different stages of temperature annealing, early( T0 ), mid( Tn ), and late( TN ), while training a single layer, showing adaptive regularization on the model parameters tracks the smooth optimization contour.
Figure 2: Spectral and dynamic analysis of ΣΔ -Sampler on a planted Quadratic Unconstrained Binary Optimization (QUBO) problem. (a) Block diagram of the ΣΔ -Sampler update rule. (b) Optimization plot showing convergence towards the planted global optimum W∗ over 1,000 steps for four different optimizers: SF (single-flip SA), PF and PF-20 (parallel-flip SA at 100% and 20% flip rates respectively), and ΣΔ -Sampler. (c) Principal component visualization of the optimization trajectories, with (d) showing the convergence on a shorter time-scale. (e), (f) Temporally windowed normalized parameter update rate of PF SA and ΣΔ -Sampler showing smoothing of the parameter update trajectories due to noise shaping. (g) Power spectra of the quantization errors (difference in the decision signal vs. update signals) in different optimizers, showing the effect of noise-shaping. (h) Linear extrapolation of average ∣gn∣ and ∣ωn∣ against overall parameter update rate during the optimization process.
Figure 3: ΣΔ -Sampler training dynamics on CIFAR-10 and the spectral signature of ΣΔ noise-shaping. (a) Train (dashed) and test (solid) accuracy of two ternary networks trained end-to-end with the ΣΔ -Sampler pipeline on CIFAR-10: VGG-7 and VGG-8. (b) Block diagram of the used convolution model. (c) Per-epoch probability density function (pdf) of per-parameter inter-flip intervals where color denotes training stage from early (orange) to late (blue) epochs; inset shows the corresponding cumulative distribution function (cdf), where the dashed line marks the median. (d) Normalized empirical power spectral density of the per-parameter ΣΔ integrator state ωjn estimated over a randomly sampled cohort of weights, compared against a 1/f2 reference spectrum (marked in red dashed line) overlaid, with (e) showing the low-frequency zoom of the same PSD. (f) Validation accuracy for ΣΔ -Sampler (solid trajectory) and full-precision counterparts (dashed trajectory) in continual-learning configuration, where the different color codes represent different offline-class replay fractions. (g) Per-phase probability density of per-parameter inter-flip intervals across the offline phase (orange to blue) and the online phase (red). (h) Per-parameter reactivation ratio plotted against the phase-1 update rate.
Method
Weight precision
Accuracy (%)
Time to solution (h)
FPWN (fp baseline)
fp32
92.88
0.38
TWN FP classifier
ternary {−1,0,+1}
91.56*
10.32
TWN discrete
ternary {−1,0,+1}
91.47
10.85
ΣΔ -Sampler FP classifier
ternary {−1,0,+1}
92.16
0.69
ΣΔ -Sampler discrete
ternary {−1,0,+1}
92.20
0.73
ΣΔ -Sampler, ∼ 5M parameters
ternary {−1,0,+1}
91.88
0.67
Table 1: Final test accuracy and time to presented solution on CIFAR-10 for full-precision baseline model, Ternary Weight Networks (TWN) model Liu et al. (2023) and the proposed approach ΣΔ -Sampler on the VGG CNN architecture (13M). *The accuracy reported in the original TWN work is 92.56%
Model
Replay
P1 Off (%)
P2 Off (start → end, %)
P2 On (%)
FP
20%
92.83
72.40→83.85
98.35
FP
50%
92.99
84.94→88.23
97.80
ΣΔ -Sampler
20%
92.15
64.84→77.51
97.60
ΣΔ -Sampler
50%
92.21
78.36→84.36
97.00
Table 2: CIFAR-10 validation accuracy for ΣΔ -Sampler and the full-precision baseline on a 12.98M VGG-8 model with various training configuration and stages.
Method
Top-5 (%)
Top-1 (%)
FP
86.76
65.40
TWN
84.20
61.80
ΣΔ -Sampler
83.81
61.65
Table 3: ImageNet-1000 validation accuracy on a ResNet-18 backbone, comparing ΣΔ -Sampler against a ternary-weight-network (TWN) baseline with a full-precision classification layer and the full-precision (FP) reference.
Figure 4: Scaling up ΣΔ -Sampler training on larger dataset (ImageNet1000) and LLMs (a) Training and test accuracy of ResNet18 on Imagenet1000 with ΣΔ -Sampler as optimizer. (b) Training loss of various sized MatMul-Free LLMs on FineWeb-Edu with 532.5K tokens per training step. (c) Continued pre-train on the same dataset from the same quantized checkpoint with STE and ΣΔ -Sampler optimizers respectively. (d) Per-step DRAM transactions due to model weight updates for different models, where transactions become sparser for ΣΔ -Sampler as training progresses.
Model
Discrete params
Mean loss
Tail-100 loss
MMFree-370M
308.3 M
6.8312
6.4544
MMFree-1.3B
1.233 B
6.3630
5.9684
MMFree-2.7B
2.538 B
6.4871
5.9846
Table 4: Numerical results of MatMul-Free language-model pre-training on FineWeb-Edu for all sizes. Tail-100 is the mean training loss over the final 100 steps.
(a) Component
FP / STE
ΣΔ -Sampler discrete
ΣΔ -Sampler continuous
Weight or ternary code
4 / 1(4)
2
4
Gradient, or decision signal ωn
4
4
4
AdamW first moment m1
4
0
4
AdamW second moment v
4
0
4
Stochastic draw buffer
—
transient
—
Total (B/param)
16/17
6
16
Table 5: Persistent training-state accounting. (a) Per-parameter cost of each stored component. (b) Analytical aggregated memory allocation across models. Activation memory is excluded throughout, since it is identical across optimizers under matched architecture and batch size.
Model
H
Layers
Heads
Intermediate
Discrete
Fraction
MMFree-370M
1024
24
16
2816
308.3 M
82.4%
MMFree-1.3B
2048
24
32
5632
1.23 B
90.3%
MMFree-2.7B
2560
32
40
6912
2.54 B
93.9%
Table 6: MatMul-Free LLM configurations
Symbol
Identifier
Default
Role
T0
temp_init
2.0
Initial temperature
τ
temp_decay
5.0
Cooling time-constant
βema
ema_beta_log
0.95
EMA decay for log-gradient statistic
μmin
thld_min
0.0
Threshold floor
γ
thld_gain
1.0
Threshold gain on Gˉn
τthld
thld_tau
2000
AGC ramp time-constant (steps)
Table S1: Default hyperparameters of the ΣΔ -Sampler per-layer dynamics. Identifiers in monospace are the field names of the Model_HParams dataclass.
Figure S1: Computational graph of a single MatMul-free language model layer.
Model
H
Layers
Heads
Intermediate
Discrete params
ΣΔ layers
MMFree-370M
1024
24
16
2816
308,281,344
144
MMFree-1.3B
2048
24
32
5632
1,233,125,376
144
MMFree-2.7B
2560
32
40
6912
2,537,553,920
192
12L768 (S5.3)
768
12
12
2048
109,510,656
72
Table S2: Language-model configurations. Vocabulary is 32,000 for all sizes, embedding and language-model head are untied, and the SwiGLU intermediate width is 256⌈(8H/3)/256⌉ for hidden width H . Discrete params counts the ternary parameters under ΣΔ -Sampler and layers counts the number of ΣΔ -Sampler layers.
Setting
Value
Meaning
mode
direct ternary
W=Won−Woff , components in {0,1}
T0
2.0
initial temperature
τ
0.5
Tn=T0/(T0+n/τ)
ΣΔ feedback
enabled
second-order temperature scaling, (Tn)2
single accumulator
enabled
difference mode (ωon−ωoff)/2
pure-integration warmup
20 steps
gradient integrated before feedback engages
Table S3: ΣΔ -Sampler configuration for the language-model experiments, identical at all three pre-training scales.
Model
Cap
Mean loss
Tail-100
MMFree-370M
uncapped
7.2055
6.7011
MMFree-370M
10−1
6.7278
6.1271
MMFree-370M
10−2
6.6890
6.1819
MMFree-370M
10−3
6.8312
6.4544
MMFree-1.3B
uncapped
7.0237
6.5612
MMFree-1.3B
10−1
6.7439
6.1921
Table S4: Update-rate cap sweep for from-scratch pre-training. Tail-100 is the mean training loss over the final 100 steps. Best tail-100 per size in bold.
Update rule
Eval loss at step 0
Final eval loss
Δ vs STE
BitNet STE
3.1721
3.1644
—
ΣΔ -Sampler, cap 10−5
3.1951
3.0800
−0.084
ΣΔ -Sampler, cap 10−4
3.1950
3.0241
−0.140
ΣΔ -Sampler, cap 10−3
3.1951
3.0913
−0.073
ΣΔ -Sampler, cap 10−1
3.1950
7.1105
+3.946
Table S5: Continued pre-training from a common quantized checkpoint, 100 steps, held-out evaluation loss. All arms share the same data, schedule, and continuous-parameter optimizer, and differ only in the ternary weight update rule. The initial loss difference is attributed to initialization of the scaling factors for both arms.
Cap
Mean transactions/step
Reduction
Final loss
uncapped
186,493
103×
6.6555
10−1
86,727
222×
6.0654
10−2
27,487
701×
6.1407
10−3
4,209
4,578×
6.4093
Table S6: Weight-update traffic and final loss against the update-rate cap for MMFree-370M. The full-precision reference is 19,267,584 transactions per step.
Varied parameter
Best val
Final val
Last-10 mean
thld_tau =5000
92.20
91.54
91.76
sd_accum_epochs =5
92.03
91.74
91.52
temp_decay =20
92.03
91.86
91.76
thld_target_pruned_init =0.3
92.01
91.68
91.64
wd =0
91.99
91.78
91.70
ema_beta_log =0.9
91.94
91.31
91.35
Table S7: 200 -epoch reruns of the ten configurations promoted from the screen, ordered by best validation accuracy. Each row differs from the baseline only in the parameter shown. Last-10 is the mean over the final ten epochs.
Group
Setting
Best val
Final val
Accumulation window
sd_accum_epochs =5∗
91.24
90.57
sd_accum_epochs =10
91.18
90.91
sd_accum_epochs =20∗
91.36
90.99
Threshold gain
0.5 , target 0.5∗
91.32
91.14
× AGC target
0.5 , target 0.7
91.07
91.05
0.5 , target 0.9∗
91.44
91.03
Table S8: 100 -epoch screen over all twenty-four configurations, grouped by the parameter varied. The row marked † is the baseline configuration. Rows marked ∗ were promoted to the 200 -epoch reruns of Table S7 .
Figure S2: GPU power benchmarking of three optimizers on VGG-7 / CIFAR-10. (a) Instantaneous board power against wall-clock time. Full precision and ΣΔ -Sampler draw comparable power at comparable device utilization, whereas TWN sustains only 3.2% of the above-idle power and 2.6% of the step throughput of the full-precision baseline. (b) Energy per epoch. TWN is an order of magnitude above the other two, while ΣΔ -Sampler incurs a 1.8× overhead relative to full precision that we attribute to unfused kernels rather than to the update rule.
Optimizer
Mean power
Above idle
Util.
Steps / 600 s
s/epoch
J/epoch
J/step
Full precision
166.6 W
116.8 W
42.3%
30,133
9.96
1,659
3.32
TWN
53.5 W
3.7 W
34.3%
797
376.5
20,145
40.29
ΣΔ -Sampler
137.9 W
88.1 W
52.7%
13,776
21.78
3,004
6.01
Table S9: Measured power, throughput, and derived energy for the three optimizers. Above-idle power subtracts the 49.8 W idle draw of the device. Energy figures use total board power.
Figure S3: Sensitivity of optimizers to residual connectivity on CIFAR-10. (a) Final training accuracy of STE and ΣΔ -Sampler on CIFAR-style ResNets of increasing depth, with and without residual connections. (b) Peak validation accuracy of ResNet-56 against the number of blocks retaining a skip, counted from the input. (c) Test accuracy of the same architecture under higher-order residual connections.
In this paper, we present CAT-Q, Cost-efficient and Accurate Ternary Quantization, for compressing and accelerating LLMs. Unlike existing state-of-the-art ternary quantization methods that rely on data-intensive and costly quantization-aware training to mitigate severe performance degradation, CAT-Q is a simple yet effective post-training quantization scheme that is readily applicable to LLMs with diverse architectures and model sizes. It has two key components, learnable modulation (LM) and softened ternarization (ST), which are coupled from an optimization perspective. LM leverages a composition of learnable factors to modulate the distribution of pre-trained high-precision weights and the ternary threshold, making them less sensitive to ternarization. ST further introduces a differentiable transition function to guide the ternarization process toward stable convergence. We show that, for pre-trained LLMs with 1.7B to 8B parameters, CAT-Q can efficiently quantize them into ternary models using only 512 calibration samples, while achieving superior performance than the seminal BitNet 1.58-bit v1 and v2 families (with 1.3B to 7B parameters) trained with 100B tokens, yielding about a 100,000X reduction in training tokens. Moreover, we show for the first time that CAT-Q can quantize much larger pre-trained LLMs having 14B to 235B parameters into leading ternary models within just 8 to 60 hours on 8 A100-80GB GPUs. Code is available at https://github.com/IntelChina-AI/BitTern.
Ternary transformers offer extreme memory and compute efficiency, but existing low-bit LoRA-based methods cannot directly fine-tune ternary weights. Current approaches either require dequantization, restoring low-bit base weights to higher precision to merge with adaptation weight, or update only quantization parameters, preventing a merged model that remains ternary. We propose ternary multiplicative adaptation, which represents discrete updates of ternary weights such as sign flips or zeroing through a low-rank Kronecker factorization into two small ternary matrices applied element-wise to ternary weights. This design is parameter-efficient and expressive, preserves the ternary domain, and supports direct merging without dequantization. Experiments on six models across language and vision, including ternarized LLaMA-3 1B and 3B and a ternary ViT-B/16, demonstrate that our method recovers much of the performance lost to quantization and outperforms strong low-bit and ternary baselines. Code is available at https://github.com/alexmanoo/ternary_adaptation.
Alexandru-Dragos Manolache, Yunqiang Li, Jan van Gemert
Delft University of Technology, The Netherlands · Amazon Development Center
Weight quantization below four bits shrinks memory and bandwidth at a perplexity that looks acceptable, but often at the cost of factual knowledge and task accuracy. Ternary weights have mostly been obtained by training from scratch or long fine-tuning. We introduce RoleQ, a quantization framework that places scales by the functional roles of Transformer units rather than by the layout of the weight matrix, as conventional formats do. Vectors that read from the residual stream take one scale per output unit; those that write to it, one per input unit. The rule is structural, needs no search, calibration or stored state, and holds on integer and ternary grids. In a rotated basis it matches a per-layer axis search at half the solver work, and at ternary it cuts perplexity by up to half against layout-based placement in two families at every size from 0.6B to 14B, where per-row scales leave MMLU at chance. Our lightweight post-training pipeline converts an existing checkpoint to 1.59 bits per weight in a few GPU-hours, a fraction of the cost of learned ternarization, and leads every post-training ternary method at matched or lower memory-resident bits on perplexity, zero-shot accuracy and MMLU, most widely on knowledge. A sparse second ternary plane placed by a per-unit Fisher saliency keeps the matmul multiplication-free and recovers what ternary loses first: at 8B, a fraction of a bit returns about half the MMLU that conversion costs, well ahead of random or Hessian-based placement at equal bits.
Johannes Knittel, Alec Helbling, Hanspeter Pfister