Robustness to bit errors is a key requirement for the reliable use of neural networks (NNs) on emerging approximate computing platforms and error-prone memory technologies. A common approach to achieve bit error tolerance in NNs is injecting bit flips during training according to a predefined error model. While effective in certain scenarios, training-time bit flip injection introduces substantial computational overhead, often degrades inference accuracy at high error rates, and scales poorly for larger NN architectures. These limitations make error injection an increasingly impractical solution for ensuring robustness on future approximate computing platforms and error-prone memory technologies. In this work, we investigate the mechanisms that enable NNs to tolerate bit errors without relying on error-aware training. We establish a direct connection between bit error tolerance and classification margins at the output layer. Building on this insight, we propose a novel loss function, the Margin Cross-Entropy Loss (MCEL), which explicitly promotes logit-level margin separation while preserving the favorable optimization properties of the standard cross-entropy loss. Furthermore, MCEL introduces an interpretable margin parameter that allows robustness to be tuned in a principled manner. Extensive experimental evaluations across multiple datasets of varying complexity, diverse NN architectures, and a range of quantization schemes demonstrate that MCEL substantially improves bit error tolerance, up to 15 % in accuracy for an error rate of 1 %. Our proposed MCEL method is simple to implement, efficient, and can be integrated as a drop-in replacement for standard CEL. It provides a scalable and principled alternative to training-time bit flip injection, offering new insights into the origins of NN robustness and enabling more efficient deployment on approximate computing and memory systems.
Figures & tables
Fig. 1 : Illustration of output-layer logits with a margin m . Five classes are shown as an example. Class 0 is the prediction and the extent of the margin to the second highest logit determines the error tolerance of the NN.
Fig. 2 : Intuition behind the tanh-clipped Margin Cross-Entropy (MCEL) Loss. Raw logits are first passed through a scaled tanh function, bounding class scores to a finite interval [−L,L] and preventing unbounded growth of logit magnitudes. Dotted lines: Saturation limits imposed by tanh. Solid curve: tanh function. Horizontal violet line: Competing (non-target) class score. The enforced margin m requires the target class score to exceed competing class scores by a fixed fraction of the available dynamic range, i.e. 2Lm .
Name
# Train
# Test
# Dim
# classes
FashionMNIST (VGG3)
60000
10000
(1,28,28)
10
SVHN (VGG7)
73257
26032
(3,32,32)
10
CIFAR10 (MobileNetV2)
50000
10000
(3,32,32)
10
Imagenette (ResNet18)
9470
3925
(3,64,64)
10
TABLE I : Datasets used for experiments.
Name
Architecture
EP
BS
LR
SS
Gamma
VGG3
In → C64 → MP2 → C64 → MP2 → FC2048 → FC10
100
256
10−3
10
0.5
VGG7
In → C64 → C64 → MP2 → C128 → C128 → MP2
50 (100)
256
10−3
5 (10)
0.25 (0.5)
In → C256 → C256 → MP2 → FC512 → FC10
MobileNetV2
In → C32 → IRB(t1,c32)
100
128
10−3
10
0.5
In → IRB(t6,c64,s2) → IRB(t6,c96)
In → C256 → AAP → FC10
TABLE II : QNNs with fully connected (FC), convolutional (C), and maxpool (MP) layers. SCB: Skip-connection block. IRB: Inverted residual block. AAP: Adaptive average pool. Convolutional layers are followed by batch normalization layers, except output layers. EP: epochs, BS: batch size, LR: learning rate, SS: step size, Gamma: decay factor (multiplied with LR every SS epochs). In brackets are the hyperparameters for the BNN models.
Fig. 3 : Accuracy over bit error rate for 2-, 4-, and 8-bit QNNs, as well as BNNs. The error rate is specified under a given bit flip injection rate on the x-axes. CEL is the state of the art (SOTA), MCEL is our proposed method. We do not perform any bit flip injection during training, we only inject errors during inference to assess error tolerance.
Fig. 4 : Evolution of the margins between the highest and the second highest logit the NN returns (i.e. MLM for Mean Logit Margin) during training. Left column: CEL. Right column: MCEL. QNNs with 4-bit quantization and BNNs shown as examples. Y-axis: Average of top2 margins (see Equation 1 for one margin and Figure 1 for an illustration) over all per-training-sample top2 margins of one epoch.
Neural network robustness studies how models withstand adversarial inputs, data shifts, noise, and hardware faults while maintaining reliable predictions.
Deep learning (DL) models are increasingly being deployed into radiation-challenging environments in spacecraft, particle accelerators, and, soon, space-based data centers. Instantiated as a learned orchestration of numbers stored in persistent memory, models are susceptible to continual degradation as high-energy particles corrupt data. At inference time, new data passed through the network fundamentally break the off-line assumptions and guarantees of a static model determined during training. To preserve the quality and efficacy of the models, fault tolerance against bit flips is a clear objective. In this work we developed a runtime method for repairing the quantized weights of neural networks (NNs) after bit flips on commodity, commercial off-the-shelf (COTS) hardware, without storing extra replicas. Our work extends the well-established Fault-Aware Training (FAT) paradigm by adding an alignment penalty to concentrate undamaged activations and weights into a low-rank subspace, and we then reconstruct flagged corrupted coordinates from the uncorrupted survivors. We call this method {\it Projection Healing} and apply it to six regression benchmarks on quantized NNs. Holding a FAT checkpoint fixed and toggling only the runtime heal gives a further ∼23% reduction in the native-unit excess integral. Across those datasets the method extends the operational scrub interval by a median of 2.04× at a +5% RMSE degradation and 1.96× at +50%, compared with 1.62× and 1.07× for classical Single Error Correction, Double Error Detection (SECDED). This introduces a new path towards designing resilient data-driven models, complementing established hardware redundancy, ECC, and FAT-based techniques.
Peter Xiangyuan Ma, Joshua S. Bloom
Department of Astronomy, University of California, Berkeley, CA 94720, USA
We characterize per-bit-position fault sensitivity in ML inference across 16 workloads -- spanning transformer-based models and attention-free CNNs -- and across three floating-point formats. Our central empirical finding is a sharp bit-sensitivity transition: flipping any of the least-significant fraction bits up to a data-type-specific threshold, Xsafe, degrades task metrics by less than 1% under deterministic single-bit stress tests. Sensitivity rises through the upper fraction bits and spikes at the exponent-mantissa boundary, where a single-bit flip causes catastrophic collapse. Because low-order bits are largely inconsequential while high-order and exponent bits are critical, uniform SECDED protection -- which guards every bit equally at 12.5% storage overhead -- is unnecessarily conservative. We derive per-data-type Xsafe floors (FP16: 6, BF16: 4, FP32: 15) and workload-aware tiers that widen the unprotected region for resilient model classes, raising ECC savings to 37.5-62.5% without retraining. Text-conditioned diffusion models dictate the conservative floor; vision encoders, NLU models, and resilient LLMs tolerate wider bypass regions. These floors and tiers drive an Unequal Error Protection (UEP) codec with per-cacheline data-type tags and a dual-partition SRAM architecture for ML accelerators. Validation across 870+ fault-injection runs confirms selective protection holds under contiguous 2- and 3-bit upsets. The codec reduces ECC area by 27.8% relative to uniform SECDED; dual-voltage operation of the non-critical partition lowers gross BF16 read energy by about 17%, with a roughly 4% dual-partition macro-area overhead.
Muhammad Husnain Mubarik, Karthik Mohan Kumar, Pedro Antonio Pena +2
The deployment of deep neural networks (DNNs) in safety-critical edge environments necessitates robustness against hardware-induced bit-flip errors. While empirical studies indicate that reducing numerical precision can improve fault tolerance, the theoretical basis of this phenomenon remains underexplored. In this work, we study resilience as a structural property of neural architectures rather than solely as a property of a dataset-specific trained solution. By deriving the expected squared error (MSE) under independent parameter bit flips across multiple numerical formats and layer primitives, we show that lower precision, higher sparsity, bounded activations, and shallow depth are consistently favored under this corruption model. We then argue that logic and lookup-based neural networks realize the joint limit of these design trends. Through ablation studies on the MLPerf Tiny benchmark suite, we show that the observed empirical trends are consistent with the theoretical predictions, and that LUT-based models remain highly stable in corruption regimes where standard floating-point models fail sharply. Furthermore, we identify a novel even-layer recovery effect unique to logic-based architectures and analyze the structural conditions under which it emerges. Overall, our results suggest that shifting from continuous arithmetic weights to discrete Boolean lookups can provide a favorable accuracy-resilience trade-off for hardware fault tolerance.
Alan T. L. Bacellar, Sathvik Chemudupati, Shashank Nag +4
The University of Texas at Austin · Federal University of Rio de Janeiro · Instituto de Telecomunicações, Porto, Portugal