As Large Language Models (LLMs) scale in size and complexity, the consequences of failures during training become increasingly severe. A major challenge arises from Silent Data Corruption (SDC): hardware-induced faults that bypass system-level detection mechanisms. SDC may behave like benign numerical noise, but can also cause harmful gradient corruption that leads to loss spikes, divergence, or stalled progress. This work provides a controlled study of how intermittent SDC affects LLM pretraining. Using targeted fault injection at the level of GPU matrix-multiply instructions, we characterize the sensitivity of different bit positions, kernel functions, and execution stages. Our analysis shows that locally originating faults can produce impactful corruption, including NaN propagation, short-lived spikes in loss, gradient norm, and attention logits, as well as persistent parameter divergence. Building on the observed corruption signatures, we propose a lightweight detection method that identifies potentially harmful parameter updates. Experiments on LLaMA models with 60M, 350M, and 1.3B parameters demonstrate that recomputing the most recent training step upon detection can effectively mitigate the impact of these events.
Figures & tables
Fig. 1: Evaluation loss, parameter difference, gradient norm before clipping, and maximum attention logits for different bit positions and kernels ( FPi and BPi , referring to the i-th GEMM kernel in the forward or backward pass, respectively). Evaluation loss and parameter difference represent the value observed after the training span. The gradient norm and the maximum attention logits represent the maximum values observed over the training span.
Fig. 2: Training loss for fault-injection runs with and without gradient norm clipping, and for the corresponding baseline run (blue). With gradient norm clipping enabled (orange), injected faults create repeated loss spikes but training continues, illustrating that gradient norm clipping mitigates most harmful spikes. Without gradient norm clipping (green), a single corrupted gradient causes the optimizer’s second moment to become infinity, freezing parameter updates and stalling training.
Fig. 3: Effect of a single fault injection with a fault length of three on the training loss and the parameter update magnitude Rt . The injected fault triggers a sharp spike in Rt , after which the training loss exhibits a bump before gradually stabilizing at a slightly elevated level. This example illustrates how gradient corruption propagates through the optimizer and leads to harmful parameter updates.
Fig. 4: Effect of the detector sensitivity parameter α on recomputation behavior over 2,000 training steps for the 60M-parameter model. Results are shown per bit position and averaged across all bit positions (in black). Left: Recompute precision, indicating how often recomputation corresponds to true corruption. Increasing α raises the false alarm rate, leading to unnecessary recomputation. Center: Detection rate, showing increased sensitivity as α increases. Right: Evaluation loss, which exhibits an increase for smaller α relative to the baseline variation ( 3.881±0.002 , after 2,000 steps).
Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.
LLM training is increasingly vulnerable to silent data corruption (SDC), yet existing protection methods largely treat Transformer computations uniformly because their vulnerability remains poorly understood. We present the first systematic characterization of SDC vulnerability across major computation interfaces in both the forward and backward passes of Transformer training. Our analysis reveals two distinct error propagation mechanisms: forward-pass vulnerability is highly location dependent, with faults on the Q/K path producing persistent training deviations, whereas backward-pass vulnerability is largely governed by gradient exponent distributions rather than computation locations. Motivated by these observations, we propose TrainSDC, a characterization-guided protection framework consisting of Q/K-path recomputation, residual-gain monitoring, and exponent-aware gradient scaling. Experiments on Llama 3.2-1B and Qwen3-0.6B show that TrainSDC maintains training behavior close to fault-free execution under both sparse and dense fault injection while introducing only 1.65%-6.76% runtime overhead.
Frontier large language model training consumes massive accelerator fleets over long wall-clock times, making stability failures extremely costly. However, global symptoms such as loss divergences often emerge only after the underlying fault has already affected the training dynamics, further increasing the sunk compute. In which internal representation can a training fault become persistently and interpretably observable before it appears in global training metrics? We propose a mechanism-aligned principle for training observability: identify the internal state into which a fault is persistently written, derive the structure induced by its mechanism, and construct monitors in representations aligned with the computation of the affected module. For FlashAttention, we show that biased backward errors accumulate into a persistent low-dimensional structure in weight updates, producing spectral concentration. For MoE routers, we characterize routing dynamics through per-token routing behavior rather than parameter similarity, which may change without altering the routing function. Experiments show that the two monitors respond selectively to changes in their corresponding modules, before degradation becomes visible in global training metrics.
Large language models (LLMs) are increasingly integrated into high-performance computing (HPC) workflows, accelerating scientific discovery through diverse perspectives such as code generation and domain-specific decision-making. Yet, how soft errors propagate and affect LLM inference remains largely unexplored. To bridge this gap, we present a comprehensive study on error propagation in LLM inference, enabled by our proposed LLMFI, a configurable and deterministic fault-injection framework. Using LLMFI, we systematically inject faults across three open-weighted LLMs and thirteen representative tasks, covering reasoning, multilingual, mathematical, and coding domains. In addition, we conduct fine-grained case studies that reveal critical vulnerability patterns. Overall, our study yields 17 takeaways that advance the understanding of error propagation in LLM inference and introduces four low-overhead directions to improve reliability through software-only modification, offering practical guidance for future error detection and mitigation.
Yafan Huang, Sheng Di, Guanpeng Li
University of Iowa · Argonne National Laboratory · University of Florida