The weights of a trained network do not have uniform sensitivity to model accuracy: the 1% most sensitive weights carry about a fifth of the damage a bit flip can do. This distribution decides the bit-flip tolerance of a model, the fraction of weights a code must cover and the bits each weight needs; existing schemes work with the distribution training leaves. We show that we can systematically modulate or alter the weight sensitivity distribution of a trained model, thereby shifting weights from one sensitivity bin to another. The sensitivity of a stored weight is the product of its gradient and the step a flip takes in the storage format, and LEAN adds a composite regularizer on both factors, a range term that bounds the flip step and a tail term that lowers the gradient of the most sensitive weights, with one coefficient λ setting how far the distribution moves. Ten epochs with the term, as a fine-tune or as the last epochs of training, shrink the sensitive tail 5 to 13 times on nine vision models (ResNet-18, ResNet-50, VGG-16 on CIFAR-10, CIFAR-100, ImageNet) within 0.5 points of accuracy; real flips confirm that the reshaped weights do less damage, a plain gradient-norm penalty does not reproduce the effect, and a preliminary extension to three small language models shows the same tail shaping. With a BCH code over 1% of the weights the model then survives 300 progressive bit-search flips in every training seed and 500 in two of three, where three of four published defenses with the same code fail by 300 and WCAT falls to 75% at 500. The reshaped distribution also cuts the parity for a fixed guarantee 3.7 times and makes a sensitivity-ordered 8/6/4/3-bit format affordable, which stores ImageNet ResNet-50 at a 3.29-bit weight payload, 2.4 times smaller than INT8, above the accuracy of LSQ at 3 bits.
Figures & tables
Fig. 1: Effect of LEAN on ResNet-18, CIFAR-10. (a) The raw weight-sensitivity distribution: number of weights per sensitivity bin (40 log-spaced bins of the sign-flip damage di , as a percentage of the baseline maximum); the bulk of the weights sits between 0.01 and 1% and the HIGH band is a thin tail, which LEAN moves to the left. (b) The cumulative distribution, counted from the tail: the number of weights whose damage is at least a threshold s ; one ten-epoch fine-tune leaves 9 times fewer weights above the baseline HIGH threshold at the same accuracy, while Hessian-aware training leaves the tail where normal training left it (1.01% against 1.00% above the threshold). (c) Accuracy under progressive bit search when the same 1% BCH code protects each model: the normally trained and the Hessian-aware model fall to chance by 300 flips, the LEAN model ( λ=40 ) keeps 94% after 500 in two of three training seeds and after 300 in all (Table V ).
Fig. 2: Overview. Step 1: a normally trained network has a heavy-tailed weight-sensitivity distribution; the top 1% of weights (HIGH) carry about 20% of the first-order damage of sign-bit flips. Step 2: ten epochs with the extra term λPLEAN , as a fine-tune or as the last epochs of training, move weights out of HIGH and MID into LOW; λ sets how far, and accuracy changes by at most 0.5 points. Step 3: the deployed model needs 3.7 times less parity for the same guarantee, survives 300 to 500 progressive bit-search flips with a 1% BCH code, and stores at 3.3 bits per weight in a sensitivity-ordered format.
Fig. 3: The LEAN procedure. The trained network is quantized and its sensitivity distribution is measured once, which fixes the HIGH and MID thresholds; ten epochs with the LEAN term follow, as a fine-tune or as the last epochs of training, with the tail T re-selected and the tail term applied, through double backpropagation, every fourth step; the model is then re-measured against the fixed thresholds and packed. λ is the only knob; the low-bit format uses a separate quantization-aware fine-tune (last paragraph of this section).
Fig. 4: One LEAN fine-tune, ResNet-18 CIFAR-10, λ=40 : (a) the regularizer falls 28 times while the cross-entropy stays flat; (b) test accuracy dips after the first epoch and recovers to within 0.2 points of the baseline.
data
model
base
LEAN
×
top-1% share
acc.
C10
ResNet-18
112k
13k (0.11%)
9
19.5 → 10.8
95.1 → 94.9
C10
ResNet-50
235k
18k (0.08%)
13
16.0 → 10.9
95.0 → 94.8
C10
VGG-16
147k
16k (0.11%)
9
28.3 → 17.9
92.9 → 92.8
C100
ResNet-18
112k
11k (0.10%)
10
9.1 → 5.1
77.2 → 77.1
C100
ResNet-50
237k
31k (0.13%)
8
10.2 → 7.1
75.9 → 75.4
C100
VGG-16
148k
13k (0.09%)
11
18.3 → 8.6
72.3 → 72.3
TABLE I: HIGH-bin weights before and after LEAN , the share of total first-order damage carried by the top 1% of weights (%, scale-free, measured on 20 held-out batches; the ablation of Table IV measures the same quantity on its own probe batches, 20.4 → 11.7%) and clean accuracy (%), each as base → LEAN ; λ=40 unless noted.
Fig. 5: Raw weight-sensitivity distribution of the nine models: number of weights per sensitivity bin (40 log-spaced bins of di as a percentage of the baseline maximum; the first bin collects every weight below 10−3 ), baseline solid, LEAN dashed, with the baseline LOW, MID and HIGH bands. Figure 6 shows the same data as a cumulative distribution. The bulk of the distribution is unchanged; the HIGH tail moves left into the MID and LOW bands.
Fig. 6: Cumulative weight-sensitivity distribution of the nine models, counted from the tail (number of weights whose sensitivity is at least s ), on log-log axes; baseline solid, LEAN dashed. The shaded bands are the baseline LOW, MID and HIGH bins. The LEAN curve separates from the baseline at the MID boundary and stays about one decade lower across the HIGH band. The box in each panel gives the count above the HIGH threshold before and after.
Fig. 7: The knob. HIGH-bin share (bars, log scale) and clean accuracy (line) against λ for six models, at the values trained for the sweep. The HIGH bin falls monotonically and saturates near 0.1%; accuracy stays within 0.5 points up to λ=40 and drops at λ=100 .
Fig. 8: The limit of the reshaping. (a) HIGH share against λ (solid, left axis) and change in clean accuracy (dotted, right axis) for ResNet-18 CIFAR-10 and ResNet-50 ImageNet; the share flattens near 0.1% while accuracy begins to fall. (b) HIGH weights per layer before and after LEAN , ResNet-18 CIFAR-10, λ=40 : the deep layers empty, the first convolution and first stage keep most of what remains. (c) HIGH reduction factor for the twelve models, largest to smallest: CIFAR, then ImageNet, then the language models.
ResNet-18
VGG-16
recipe
ep.
rate
λ
acc
HIGH
MID
acc
HIGH
MID
control
10
0.0005 → 0
0
95.3
1.00
8.0
93.6
1.00
8.0
last epochs
10
0.0005 → 0
40
95.2
0.40
6.4
93.7
0.25
6.9
longer
20
0.0006 → 0
40
95.3
0.20
3.9
93.5
0.06
6.2
stronger
10
0.0005 → 0
100
95.0
0.19
3.9
93.2
0.05
5.3
rate held
10
0.002
40
94.9
0.14
0.9
93.2
0.04
2.9
TABLE II: LEAN in the last epochs of a 200-epoch run, CIFAR-10; HIGH and MID against the thresholds of the matching control of the same run. The fine-tune rows are the paper’s models, measured against their own baseline.
model
recipe
acc
HIGH
MID
model
recipe
acc
HIGH
MID
ResNet-18
control
78.5
1.00
8.0
ResNet-50
control
78.9
1.00
8.0
C-100
last 10
78.4
0.30
4.9
C-100
last 10
78.6
0.50
5.6
20 ep.
78.4
0.12
2.4
20 ep.
78.6
0.17
3.1
λ =100
78.3
0.13
2.6
λ =100
78.6
0.20
3.3
rate held
78.1
0.09
0.7
rate held
78.2
0.04
1.2
VGG-16
control
74.8
1.00
8.0
ResNet-50
control
95.3
1.00
8.0
TABLE III: The same recipes on four more pairs (C-10, C-100: CIFAR), λ=40 unless stated, ten epochs unless stated. HIGH and MID in % of weights.
Fig. 9: LEAN inside training, λ=40 ; ResNet-18 CIFAR-10 in the top row, VGG-16 CIFAR-10 below. (a, d) Test accuracy for epochs 180 to 200 of a 200-epoch run; at epoch 190 the run is forked into a plain control and a branch with the LEAN term on. (b, e) The penalty over the ten LEAN epochs inside training (learning rate 0.0005 to 0) and in the fine-tune (0.002, cosine). (c, f) HIGH share and accuracy of the control, the last-epochs model, the last-epochs model with the rate held at 0.002, and the fine-tune. (g) HIGH share per recipe on four more pairs, accuracy inside the bars.
model
acc
HIGH
top-1%
worst 100
base top
Δ ratio
baseline
95.1
1.00%
20.4%
0.031
0.0069
1.00
Ptail only
94.5
0.87%
19.3%
0.030
0.0061
0.99
Ptail only, λ =400
94.1
0.35%
16.3%
0.031
0.0062
1.07
Pclip only
95.1
0.13%
12.8%
0.012
0.0019
0.51
gradient norm, λ =0.01
94.5
0.64%
19.5%
0.026
0.0058
1.02
LEAN (both)
94.9
0.11%
11.7%
0.009
0.0014
0.51
TABLE IV: Ablation and real-flip check, ResNet-18 CIFAR-10, λ=40 unless noted. Columns are defined in the text.
no code
with 1% BCH code
method
HIGH
10
25
50
100
100
200
300
500
normal
1.00
41
11
10
10
93
26
17
10
PC [ 8 ]
0.89
44
11
10
10
95
33
22
14
Hessian [ 11 ]
1.01
44
11
10
10
95
31
22
14
FaR [ 10 ]
1.05
16
10
10
10
95
95
45
12
WCAT [ 9 ]
0.17
80
48
18
11
95
95
94
75
TABLE V: ResNet-18 CIFAR-10, INT8, accuracy (%) after n progressive bit-search flips, without and with the same 1% BCH code; HIGH in %. Clean accuracy 95. Bottom block: the models trained with LEAN in the last epochs.
no code
with 1% BCH code
model
normal
LEAN
normal
LEAN
C10 ResNet-18
95.4
44.6
92.7
3.9
C10 ResNet-50
95.8
25.3
86.1
5.3
C10 VGG-16
56.2
7.3
30.2
5.4
C100 ResNet-18
96.0
75.3
96.7
70.7
C100 ResNet-50
96.7
82.7
85.3
55.3
TABLE VI: T-BFA, 30-flip budget, mean attack success rate (%) over three class pairs, without and with the 1% code; LEAN at the λ of Table I . Data and caveats in the text.
model
cov.
parity
BER prot.
BER unprot.
PBS-25 drop
normal
20.0%
18.0
95.0
88.0
0.00
normal
19.2%
17.6
94.9
–
0.48
normal
16.8%
15.4
95.0
–
0.48
normal
15.2%
13.9
95.0
–
0.52
normal
12.5%
11.5
95.0
88.3
0.52
LEAN λ =10
11.9%
10.9
95.0
90.6
0.00
TABLE VII: Parity for a fixed guarantee, ResNet-18 CIFAR-10, INT8 with the selective code; coverage in % of weights, parity in % of weight bits, accuracy (%) at 6% BER on the covered and on the uncovered weights, and accuracy drop (points) at 25 flips. The 6% BER figure on the uncovered weights is not reported for the coverage sweep, since at one seed it depends on which layers the random errors land in.
Fig. 10: Cost of protection, storage and weight fetch before and after LEAN . (a) Parity for the same protection guarantee (Table VII ), ResNet-18 CIFAR-10, measured; coverage in brackets. (b) Weight memory of ImageNet ResNet-50 at INT8, where the 1% code’s parity is embedded in LOW weights, and in the tiered 3.29-bit format without and with the 1% BCH and t=19 sign-plane codes (3.63 bits per weight); the 4.09-bit ImageNet hardening mode of Table X is not shown. (c) DRAM weight-fetch energy per inference with the 45 nm access energies of Han et al. [ 20 ] ; on-chip SRAM scales identically (32 to 13 μ J).
format
bits/w
top-1
INT8
8.00
80.7
LSQ uniform [ 14 ]
4.00
78.4
KURE uniform [ 15 ]
4.00
75.9
tiered, λ=0
4.00
78.9
tiered, LEAN λ=40
4.00
79.6
LSQ uniform
3.00
74.4
TABLE VIII: ImageNet ResNet-50 top-1 (%) at matched (4.00) and comparable (3.00 vs 3.29) bits per weight; tiered rows are mean of three seeds.
BCH only
+ sign-plane code
model
25
50
100
100
200
300
500
C10 ResNet-18
91
68
14
94
94
94
90
C10 ResNet-34
93
89
73
94
94
94
93
C10 ResNet-50
82
63
28
94
93
92
90
C10 VGG-16
83
65
27
93
92
91
86
C100 ResNet-18
67
56
11
75
72
71
67
TABLE IX: CIFAR models at 3.29 bits per weight under progressive bit search, accuracy (%): BCH on the 1% tier only (3.45 b/w) and with the interleaved sign-plane code (3.63 b/w).
1% tier, t =19
1% tier, t =79
3% tier, t =79
model
clean
100
200
500
100
200
500
100
200
500
ResNet-18
67.7
54
27
1
63
56
25
67
64
54
ResNet-34
71.1
66
58
22
70
69
64
71
71
70
ResNet-50
71.8
8
1
0
48
24
2
69
68
59
VGG-16
72.1
65
59
39
71
69
62
73
73
73
TABLE X: ImageNet models at 3.29 bits per weight under progressive bit search, accuracy (%) after 100 / 200 / 500 flips, with the sign-plane code at t=19 (3.63 b/w) or t=79 (3.68), and with the 3% tier and t=79 (4.09). No retraining.
INT8 (+code)
3.63 b/w
4.09 b/w
model
Mw †
MB
mJ
MB
mJ
MB
mJ
CIFAR ResNet-18
11.2
11.2
1.79
5.1
0.81
5.7
0.92
CIFAR ResNet-34
21.3
21.3
3.41
9.7
1.55
10.9
1.74
CIFAR ResNet-50
23.6
23.6
3.78
10.7
1.71
12.1
1.93
CIFAR VGG-16
14.7
14.7
2.35
6.7
1.07
7.5
1.20
ImageNet ResNet-18
11.7
11.7
1.87
5.3
0.85
6.0
0.96
TABLE XI: Weight memory (MB) and estimated DRAM weight-fetch energy per inference (mJ) for the deployed formats; formats and assumptions in the text. † Millions of weights.
model
λ
HIGH
MID
ppl (fp32)
ppl (INT4)
Qwen2-0.5B
0 (control)
1.00%
8.0%
13.86
21.1
Qwen2-0.5B
10
0.58%
4.9%
14.51
19.3
Llama-3.2-1B
0 (control)
1.00%
8.0%
11.47
21.4
Llama-3.2-1B
3
0.43%
4.1%
12.09
18.9
Llama-3.2-1B
10
0.19%
2.1%
13.96
19.3
Qwen2-1.5B
0 (control)
1.00%
8.0%
10.78
13.3
TABLE XII: LEAN on language models: HIGH and MID shares against the control’s thresholds, and WikiText-2 perplexity at fp32 and after per-channel INT4 (lower is better).
model
code
clean
10
20
50
100
Llama-3.2-1B control
none
12.2
dead
dead
dead
dead
Llama-3.2-1B LEAN λ =3
none
12.6
dead
dead
dead
dead
Llama-3.2-1B control
1%
12.2
12.5
14.4
122
210
Llama-3.2-1B LEAN λ =3
1%
12.6
20.8
28.5
41.9
61.4
Qwen2-0.5B control
none
14.3
dead
dead
dead
dead
Qwen2-0.5B LEAN λ =10
none
14.7
dead
dead
dead
dead
TABLE XIII: Preliminary: progressive sign-bit attack on INT8 language models, perplexity after n flips, without and with a 1% code by each model’s own ranking; “dead” is above 106 .
Fig. 11: LEAN on three language models (Table XII in bars): (a) HIGH and (b) MID shares against the control’s thresholds, with the reduction factor; (c) WikiText-2 perplexity of the control and the LEAN model, with the cost. λ=10 for Qwen2-0.5B, 3 for the other two.
Neural network robustness studies how models withstand adversarial inputs, data shifts, noise, and hardware faults while maintaining reliable predictions.
We characterize per-bit-position fault sensitivity in ML inference across 16 workloads -- spanning transformer-based models and attention-free CNNs -- and across three floating-point formats. Our central empirical finding is a sharp bit-sensitivity transition: flipping any of the least-significant fraction bits up to a data-type-specific threshold, Xsafe, degrades task metrics by less than 1% under deterministic single-bit stress tests. Sensitivity rises through the upper fraction bits and spikes at the exponent-mantissa boundary, where a single-bit flip causes catastrophic collapse. Because low-order bits are largely inconsequential while high-order and exponent bits are critical, uniform SECDED protection -- which guards every bit equally at 12.5% storage overhead -- is unnecessarily conservative. We derive per-data-type Xsafe floors (FP16: 6, BF16: 4, FP32: 15) and workload-aware tiers that widen the unprotected region for resilient model classes, raising ECC savings to 37.5-62.5% without retraining. Text-conditioned diffusion models dictate the conservative floor; vision encoders, NLU models, and resilient LLMs tolerate wider bypass regions. These floors and tiers drive an Unequal Error Protection (UEP) codec with per-cacheline data-type tags and a dual-partition SRAM architecture for ML accelerators. Validation across 870+ fault-injection runs confirms selective protection holds under contiguous 2- and 3-bit upsets. The codec reduces ECC area by 27.8% relative to uniform SECDED; dual-voltage operation of the non-critical partition lowers gross BF16 read energy by about 17%, with a roughly 4% dual-partition macro-area overhead.
Muhammad Husnain Mubarik, Karthik Mohan Kumar, Pedro Antonio Pena +2
Binary-weight language models store one bit per weight, but training them keeps a full-precision latent copy of each weight plus Adam state, about 96 extra bits per weight, so a deployed binary model cannot be adapted on the device that runs it. We replace the latent weight with a 2-8-bit residual buffer quantized by stochastic rounding, which cuts the training state per weight from 97 bits to 3-9, up to 19x less. The resulting error-diffusion (ED) optimizer moves the residual-carrying loop of image halftoning onto the weight-update path. Each buffer accumulates the unrealized part of every update and flips the weight when the accumulated value crosses a threshold, and it works only when its quantization grid has a level exactly at that threshold. Under this threshold-alignment rule, 4-bit buffers match 8-bit buffers on a binary multilayer perceptron benchmark, and the same rule lifts a different latent-free optimizer, Bop, from a 10-point deficit at 4 bits to full-precision accuracy. On character-level transformers with 97% of parameters binary, ED trains within 5-11% of the latent-weight reference at a 2,000-step budget and within 1.7% at convergence. On scaling ladders over enwik8 and wikitext-103, up to 19M parameters on enwik8 (70M with the wikitext-103 embeddings), the latent-free loss stays within -2.3% to +2.9% of the reference at all six sizes once a per-step cap on the buffer update removes a launch instability; without the cap it trails by 10% at the largest tokenized size. A deployed binary model continues training under ED at one tenth of the latent training state, matching latent-weight continuation, and a fused CPU kernel completes the adaptation within a 275 MB training-memory budget, which latent-weight training cannot. A 250-step probe sets the flip learning rate at any new scale.
We show that neural network weights can be explicilty fintuned to admit a smaller grammar. Weight Pair Encoding (WeightPE) does so by placing a lossy Re-Pair compressor inside a straight-through estimator. The int8 weights of the network are flattened into one string, and near-matching Re-Pair patterns are made exactly equal within a global L2 budget. The network computes with the rewritten weights and trains through them with a straight-through estimator. Unlike a flat codebook of fixed-size entries, a grammar offers variable-length patterns and reuses them hierarchically inside larger ones. On the MLP weights of ViT-B/16 and ViT-L/16 finetuned on CIFAR-10, WeightPE produces a Re-Pair grammar 0.43x and 0.38x the size of the one produced by an equivalent int8 QAT run, at a cost of 1.9 and 1.1 accuracy points. The trend extends to different grammar compressors (LZ78, SEQUITUR), over which the networks has not be finetuned against. To our knowledge, this is the first time grammar size has been used as an explicit training objective for network weights.
Irene Tallini, Daniele Solombrino, Alberto Cazzaniga +1
Sapienza University of Rome Rome, Italy · Sapienza University of Rome and Paradigma Rome, Italy