Analysis of LLM Vulnerability to GPU Soft Errors: An Instruction-Level Fault Injection Study
Authors: Duo Chai, Zizhen Liu, Shuhuai Wang, Songwei Pei, Cheng Liu, Huawei Li, Shangguang Wang
Organizations: School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications, Beijing 100876, China. · State Key Lab of Processors, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China
Large language models (LLMs) are highly compute- and memory-intensive, posing significant demands on high-performance GPUs. At the same time, advances in GPU technology driven by shrinking transistor sizes and lower operating voltages have made these devices increasingly susceptible to soft errors, particularly in radiation-prone environments such as satellite computing scenarios. While prior work has examined GPU soft errors, most studies have focused on general-purpose applications or conventional neural networks primarily used for vision tasks such as classification and detection. However, vulnerability analysis of modern LLMs remains limited, despite their rapid adoption in diverse application scenarios. Given the unique characteristics of LLMs, their resilience to soft errors may differ substantially from earlier models. To bridge this gap, we conduct the first instruction-level fault injection study of LLM inference to analyze the factors affecting LLM vulnerability. Our approach reveals reliability characteristics from multiple perspectives, highlighting the effects of model architecture, parameter scale, and task complexity. These findings provide new insights into LLM reliability and inform the design of more effective fault tolerance mechanisms.
Figures & tables
Fig. 1: Typical fault injection approaches at different abstraction levels. (a) Algorithm-level fault injection, which injects bit-flip errors into weights and activations at neural network layers. (b) Instruction-level fault injection, which injects bit-flip errors into GPU instructions. (c) Microarchitecture-level fault injection, which injects bit-flip errors at major registers and components of the GPU. Essentially, microarchitecture-level fault injection depicts how hardware errors propagate to instructions and algorithms eventually. It is accurate but can be extremely slow. In contrast, algorithm-level fault injection simplifies the error modeling to network operations and ignores low-level details for higher fault simulation speed. Instruction-level fault injection stays between microarchitecture level and algorithm-level fault injection for balanced fault simulation accuracy and speed.
Ability
NVBitFI
Ours
Faults per run
1
k
Bit flip mode
random
random & fixed
Instruction selection
random
random & fixed
LLM-based evaluation
unsupported
supported
TABLE I: Comparison between our framework and NVBitFI ( k denotes the number of faults per run).
Fig. 2: The proposed LLM reliability evaluation framework based on NVBitFI.
Model
P
L
HS
AH
VS
CL
GPT2
124M
12
768
12
50257
1k
GPT2-large
774M
36
1280
20
50257
1k
Llama3.2-1B
1.23B
16
2048
32
128256
128k
Llama3.2-3B
3.21B
28
3072
24
128256
128k
Qwen3-0.6B
0.59B
28
1024
16
151936
40k
Qwen3-1.7B
1.72B
28
2048
16
151936
40k
TABLE II: Detailed parameter description of the selected models.
Experiment
Fault Mode
Fault Count
Run Count
Abnormal behavior
Random
1–8
4,000 per model
DUE causes
Random
1
3,000 per model
SDC causes
Random
1
3,000 per model
Instruction types
Fixed type
1
500 per type
Bit positions
Fixed bit
1
500 per bit
Component types
Random
1
5,000 per model
TABLE III: Summary of Fault Injection Experiments.
Fig. 3: GPU behavior distribution of the LLM inference system under multi-fault scenarios.
Fig. 4: E1–E6 represent the following types of faults. E1: Out Of Range Address, E2: MMU Fault, E3: Out Of Range Register, E4: Misaligned Address, E5: Illegal Instruction Operands, E6: Others.
Fig. 5: Case 1 illustrates an SDC caused by numerical errors. Case 2 illustrates an SDC caused by address errors. Green indicates registers containing correct data, while red indicates registers containing corrupted data, and the registers marked with red boxes represent registers where bit flip occurs.
Fig. 6: Model vulnerability factors of different LLMs. The error bars represent the standard error of the MVF over multiple experiments.
Fig. 7: Instruction vulnerability factor (IVF) of different instructions. Solid color represents the portion contributed by DUE and the diagonal line represents the portion contributed by SDC.
Fig. 8: Approximate model vulnerability factor of different models. The error bars represent the standard error of the MVF over multiple experiments.
Fig. 9: Comparison of MVF obtained from experimental measurements and approximate computations. The numbers on top of the bars indicate the percentage difference.
Fig. 10: Instruction distribution of different LLMs. Instructions that are fault-injectable but have a low proportion (typically less than 0.01) are grouped into the Others category.
Fig. 11: The impact of bit-flip errors at different bit positions on system reliability in terms of DUE and SDC.
Fig. 12: Component vulnerability across LLM architectures.
Fig. 13: Comparison of error rates in different layers. The horizontal axis represents different layers, and the vertical axis represents vulnerability.
Fig. 14: The model’s functional probability decreases as the number of faults injected in a single run increases, where functional probability is defined as the proportion of correct inferences among all experiments.
Model
Lambada(ACC)
PIQA(ACC)
HellaSwag(ACC)
WikiText-2(PPL)
XSum(ROUGE-1)
GSM8K(ACC)
GPT2
32.56
62.89
28.92
37.37
11.66
1.14
GPT2-large
47.66
70.35
36.40
22.61
11.80
1.52
Llama3.2-1B
62.22
74.32
47.68
11.56
12.74
6.60
Llama3.2-3B
70.04
76.71
55.21
9.26
12.86
25.85
Qwen3-0.6B
40.07
67.30
37.47
26.14
11.85
41.70
Qwen3-1.7B
50.30
72.09
46.14
21.04
13.06
69.14
TABLE IV: The scores of the models on different datasets.
Model
Lambada
PIQA
HellaSwag
WikiText-2
XSum
GSM8K
GPT2
↓
↓
↓
↑
↑
↓
Llama3.2
↑
↑
↓
↑
↓
↓
Qwen3
↓
↓
↑
↓
↑
↓
TABLE V: The reliability changes of models with the same architecture after increasing the number of parameters. ( ↑ represents more reliability, ↓ represents less reliability).
Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.