Numerical Instability and Chaos: Quantifying the Unpredictability of Large Language Models
Authors: Chashi Mahiul Islam, Alan Villarreal, Mao Nishino, Shaeke Salman, Xiuwen Liu
Organizations: Department of Computer Science, Florida State University, Tallahassee, USA · Department of Mathematics, Florida State University, Tallahassee, USA
As Large Language Models (LLMs) are increasingly integrated into agentic workflows, their unpredictability stemming from numerical instability has emerged as a critical reliability issue. While recent studies have demonstrated the significant downstream effects of these instabilities, the root causes and underlying mechanisms remain poorly understood. In this paper, we present a rigorous analysis of how unpredictability is rooted in the finite numerical precision of floating-point representations, tracking how rounding errors propagate, amplify, or dissipate through Transformer computation layers. Specifically, we identify a chaotic "avalanche effect" in the early layers, where minor perturbations trigger binary outcomes: either rapid amplification or complete attenuation. Beyond specific error instances, we demonstrate that LLMs exhibit universal, scale-dependent chaotic behaviors characterized by three distinct regimes: 1) a stable regime, where perturbations fall below an input-dependent threshold and vanish, resulting in constant outputs; 2) a chaotic regime, where rounding errors dominate and drive output divergence; and 3) a signal-dominated regime, where true input variations override numerical noise. We validate these findings extensively across multiple datasets and model architectures.
Figures & tables
Fig. 1 : Overview of effective directional number D(ϵ,v) at layer index 32 (Float32) for v1,v50,v500,v2000,v4096 .
Fig. 2 : Layer-wise propagation profile at ϵ=0.1 . Directional structure is visible: higher- σ directions yield larger amplification than low- σ directions, consistent with a signal-dominated regime.
Fig. 3 : Layer-wise propagation profile at ϵ=10−10 . Directional structure collapses: multiple singular directions, coordinate directions, and a random direction exhibit similar growth. This supports an instability-dominated regime where effective sensitivity is governed by scale rather than singular value.
Model
Dataset
Mean Inst.
Median Inst.
Max Drift
Mean Margin
Min Margin
GPT-OSS
AdvBench
6.76 ×105
0.0
2.92 ×10−4
0.6196
0.6196
GPT-OSS
TruthfulQA
2.63 ×105
0.0
3.43 ×10−4
0.9767
0.9767
Llama-3.1
AdvBench
3.78 ×106
0.0
1.02 ×10−4
0.6175
0.6175
Llama-3.1
TruthfulQA
8.84 ×106
0.0
1.46 ×10−2
0.5351
0.5035
TABLE I : AGGREGATE STABILITY METRICS. Llama-3.1-8B RESULTS USE 100 PROMPTS PER DATASET (GPU). GPT-OSS-20B RESULTS USE 10 PROMPTS PER DATASET (CPU). MEDIAN INSTABILITY OF 0.0 INDICATES PREVALENT CONSTANT-LIKE PLATEAUS, WHILE LARGE MEAN INSTABILITY INDICATES RARE BUT LARGE FINITE-DIFFERENCE SPIKES.
Fig. 4 : Micro-continuity analysis: consecutive differences exhibit long constant-like stalls with rare discrete jumps, producing a staircase-like cumulative change in the output representation.
Metric
v1 - v2
v1 - v10
v1 - v4096
Flip Frequency (%)
16.14
16.29
15.42
Fragmentation
861
866
779
Crossing Density
51.0
52.5
49.0
TABLE II : DECISION BOUNDARY IRREGULARITY METRICS ACROSS SINGULAR VECTOR PLANES NEAR A NEAR-TIE POINT x0 .
Fig. 5 : Binary decision maps near a constructed near-tie point. White: L1≥L2 (Token 1 wins). Black: L1<L2 (Token 2 wins). The fragmented regions indicate that near-tie decisions can flip under microscopic perturbations, including along low- σ directions.
Fig. 6 : Directional stability in the (v1,v2) plane. (a) Maximum stable perturbation smax(θ) varies by 4× across angles. (b) Cartesian projection reveals asymmetric polygonal boundary. Geometric distortion demonstrates that stability is determined by input-dependent rounding dynamics, not by singular values.
Fig. 7 : Maximum stable perturbation magnitude along all 4096 singular vectors. Despite singular values spanning σ1=615.3 to σ4096≈0 , the stability boundary remains nearly constant ( smax∼10−10 ) across the entire spectrum. This universality proves instability pervades the entire embedding manifold.
Fig. 8 : Directional number vs. ϵ at layer index 32 under different floating-point precisions. Precision shifts the ϵ ranges where plateau behavior and rapid growth appear, while preserving the overall scale dependence.
Fig. 9 : Absolute directional condition number convergence via noise averaging. The dashed red line shows the unaveraged baseline, while the blue curve shows the averaged estimate stabilizing at ∼ 600 by n=100 samples, converging to the theoretical singular value (dotted line, 615.31). Averaging eliminates rounding-error artifacts and recovers true model sensitivity.
Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.
Transformers are the state-of-the-art architecture for large language models, and a key to their scalability is the strategic usage of low-precision arithmetic. We develop a mixed-precision analysis of transformer inference, deriving bounds for the condition numbers and forward error of the architecture's constituent parts. Notably, we compare the numerical stability of LayerNorm and RMSNorm in the massive-outlier regime, tighten the error bound of softmax in the presence of attention sinks, and quantify the impact of its shifted evaluation on the sensitivity to perturbations. Furthermore, we derive novel sequence-length-independent bounds on the local Lipschitz constant of self-attention. Our worst-case error bound for transformer inference suggests that its numerical stability is determined by the interplay between weight magnitude and the growth of the residual stream. Crucially, and as validated by experiments with GPT-2, our analysis establishes that the scaling of residual-projection weights preserves the propagation of the relative rounding error unless it forces a qualitative transition in the dynamics of the residual stream.
Stanislav Budzinskiy, Wenyi Fang, Longbin Zeng +1
Faculty of Mathematics, University of Vienna, Kolingasse 14-16, 1090, Vienna, Austria · Huawei Technologies
Large language models (LLMs) are increasingly deployed under diverse numerical precision configurations, including standard floating-point formats (e.g., bfloat16 and float16) and quantized integer formats (e.g., int16 and int8), to meet efficiency and resource constraints. However, minor inconsistencies between LLMs of different precisions are difficult to detect and are often overlooked by existing evaluation methods. In this paper, we present PrecisionDiff, an automated differential testing framework for systematically detecting precision-induced behavioral disagreements in LLMs. PrecisionDiff generates precision-sensitive test inputs and performs cross-precision comparative analysis to uncover subtle divergences that remain hidden under conventional testing strategies. To demonstrate its practical significance, we instantiate PrecisionDiff on the alignment verification task, where precision-induced disagreements manifest as jailbreak divergence-inputs that are rejected under one precision may produce harmful responses under another. Experimental results show that such behavioral disagreements are widespread across multiple open-source aligned LLMs and precision settings, and that PrecisionDiff significantly outperforms vanilla testing methods in detecting these issues. Our work enables automated precision-sensitive test generation, facilitating effective pre-deployment evaluation and improving precision robustness during training.
Yifei Wang, Tianlin Li, Xiaohan Zhang +4
Shanghai Jiao Tong University, China · Beihang University, China · Nanyang Technological University, Singapore +1
Quantization is a powerful strategy to build capable and resource-efficient large language models (LLMs) by reducing the bitwidth of the parameters. While quantized LLMs achieve state-of-the-art performance on unperturbed inputs using standard predictive metrics, their performance on perturbed inputs, measured using reliability metrics, remains underexplored, despite its importance for reliable deployment. To address this gap, we first conduct a comprehensive reliability evaluation of quantized LLMs consisting of three key components: (1) Uncertainty: We assess the trustworthiness of LLMs quantized to 2, 3, 4, and 8 bits using six different quantization methods, employing established uncertainty metrics. (2) Calibration: We assess how well-calibrated the uncertainty estimates of quantized models are across model scales and bit precisions. (3) Robustness: We design character-level and word-level input perturbations to evaluate the reliability of quantized models under semantically-preserving variations in the inputs that arise in real-world applications. Second, we characterize how reliability scales with the total number of model bits. Our study reveals that while the performance scales monotonically with the total number of bits, the reliability scalings are nonlinear. A reliability peak occurs for 4-bit quantized models, indicating that quantizing moderately sized models offers the best reliability-efficiency trade-off. Additionally, our empirical findings reveal that quantization enhances the robustness of LLMs to natural input perturbations.
Sirine Ayadi, Sándor Daróczi, Stephan Günnemann +1
School of Computation, Information and Technology, Technical University of Munich Munich Data Science Institute · Pruna AI