Hidden Reliability Risks in Large Language Models: Systematic Identification of Precision-Induced Output Disagreements

Yifei Wang; Tianlin Li; Xiaohan Zhang; Xiaoyu Zhang; Wei Ma; Mingfei Cheng; Li Pan

cs.AIApr 2, 2026

Hidden Reliability Risks in Large Language Models: Systematic Identification of Precision-Induced Output Disagreements

Authors: Yifei Wang, Tianlin Li, Xiaohan Zhang, Xiaoyu Zhang, Wei Ma, Mingfei Cheng, Li Pan

Organizations: Shanghai Jiao Tong University, China · Beihang University, China · Nanyang Technological University, Singapore · Singapore Management University, Singapore

Abstract

Large language models (LLMs) are increasingly deployed under diverse numerical precision configurations, including standard floating-point formats (e.g., bfloat16 and float16) and quantized integer formats (e.g., int16 and int8), to meet efficiency and resource constraints. However, minor inconsistencies between LLMs of different precisions are difficult to detect and are often overlooked by existing evaluation methods. In this paper, we present PrecisionDiff, an automated differential testing framework for systematically detecting precision-induced behavioral disagreements in LLMs. PrecisionDiff generates precision-sensitive test inputs and performs cross-precision comparative analysis to uncover subtle divergences that remain hidden under conventional testing strategies. To demonstrate its practical significance, we instantiate PrecisionDiff on the alignment verification task, where precision-induced disagreements manifest as jailbreak divergence-inputs that are rejected under one precision may produce harmful responses under another. Experimental results show that such behavioral disagreements are widespread across multiple open-source aligned LLMs and precision settings, and that PrecisionDiff significantly outperforms vanilla testing methods in detecting these issues. Our work enables automated precision-sensitive test generation, facilitating effective pre-deployment evaluation and improving precision robustness during training.

Figures & tables

Figure 1. The same adversarial prompt may trigger inconsistent behaviors across LLM implementations running under different numerical precision settings, motivating a systematic study of precision-induced behavioral disagreements. The suffix shown is generated by PrecisionDiff.

Format	Sign	Exponent	Fraction	Machine Epsilon ( $\epsilon$ )
FP16	1 bit	5 bits	10 bits	$\approx 9.77\times 10^{-4}$
BF16	1 bit	8 bits	7 bits	$\approx 7.81\times 10^{-3}$
Int16	1 bit	0 bits	15 bits	1 (Fixed)
Int8	1 bit	0 bits	7 bits	1 (Fixed)

Table 1. Comparison of Numerical Precision Formats Comparison of FP16, BF16, Int16, and Int8 formats showing their bit allocations for sign, exponent, and fraction, along with their respective machine epsilon values.

Figure 2. An overview of PrecisionDiff. It consists of two stages, namely precision-induced disagreements detection via differential testing and root-cause analysis via layer-wise divergence localization. A two-stage framework. Stage 1 performs differential testing via dual-precision momentum-guided GCG to expose inconsistent behavioral outputs across precision configurations. Stage 2 performs root-cause analysis via layer-wise activation monitoring to localize divergence sources.

Model	BF16 vs FP16		BF16 vs INT16		INT16 vs INT8
Model	Success	Iter	Success	Iter	Success	Iter
Guanaco 7B	78.0%	83.8	84.0%	78.2	100.0%	23.2
Llama-2 7B	68.0%	44.7	64.0%	24.0	98.0%	9.3
Mistral-7B	96.0%	75.2	94.0%	79.1	100.0%	39.5
Vicuna-7B	94.0%	48.9	91.5%	14.8	100.0%	10.8
Average	84.0%	63.2	83.4%	49.0	99.5%	20.7

Table 2. Success rates under differential precision testing across multiple precision transitions.

Category	Count	SR	Avg. Iter
Cybercrime	99	92.9%	41.4
Malware	33	90.9%	60.6
Fraud	99	89.9%	32.6
Privacy Violation	33	84.8%	51.5
Violence	143	82.5%	54.8
Hate Speech	33	75.8%	41.9

Table 3. Per-category success rates and average iterations over representative models and precision settings.

Figure 3. The change of Loss during the Adversarial Prompt Optimizer process.

	Vicuna-7B		Llama2-7B
Method	Success	Iter	Success	Iter
Random Search	2.0%	19.0	0.0%	N/A
Fuzzing (AFL++)	8.0%	84.0	0.0%	N/A
Genetic Algorithm	8.0%	12.3	0.0%	N/A
Standard GCG ( FP16 )	50.0%	59.0	8.0%	94.3
PrecisionDiff (Ours)	72.0%	45.0	68.0%	19.4

Table 4. Comparison of PrecisionDiff with baseline methods (50 test cases, max_step=200).

	Llama-3 8B		Llama-2 7B
Precision Transition	Success	Avg. Iter	Success	Avg. Iter
BF16 vs FP16	100.0%	87.7	90.0%	11.3
BF16 vs INT16	98.0%	109.2	95.8%	15.4
INT16 vs INT8	96.0%	71.5	90.0%	13.8

Table 5. Success rates under differential precision testing across multiple precision transitions when do_sample=True.

	BF16 vs FP16		FP16 vs BF16
Model	Success	Avg. Iter	Success	Avg. Iter
Guanaco 7B	62.0%	73.0	52.0%	81.2
Llama-2 7B	68.0%	19.4	72.0%	24.2
Mistral-7B	92.0%	55.7	72.0%	76.2
Vicuna-7B	72.0%	45.0	62.0%	48.7

Table 6. Success rates under differential precision testing across multiple precision transitions when max_step=200.

Figure 4. Jailbreak decision boundaries of large language models under different precision configurations.

Figure 5. The layer-wise source localization of disagreements on four open-source models between different precisions.

LLM Reliability

Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.

See all 789 papers

Greedy Decoding Is Not Precision-Invariant: Cross-Precision Output Divergence in LLM Inference
Sep 22, 2026Gaoyuan Du, Anam Nawaz Khan, Rex Zhou +4Language Model Decoding LLM Inference
University of Tennessee, Knoxville · University of Chicago · Amazon +1
Demystifying Numerical Instability in LLM Inference: Achieving Reproducible Inference for Mission-Critical Tasks with HEAL
Jun 19, 2026Zhenting Zhu, Lucas Thai, Shan Yu +5Mixed-Precision Quantization LLM Inference
1UCLA · 2UC Berkeley · University of Chinese Academy of Sciences
Accuracy is Not Enough: A Divergence-Based Approach to Evaluate Fidelity Loss in Quantized LLMs
Sep 7, 2026Shahzeb Qamar, Lorenz Sparrenberg, Christian Bauckhage +5LLM Quantization LLM Evaluation
Fraunhofer IAIS, Germany · PhenoRob Cluster of Excellence, University of Bonn, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Germany +4

Sep 22, 2026J/K move · Enter open · S save

Greedy Decoding Is Not Precision-Invariant: Cross-Precision Output Divergence in LLM Inference

Gaoyuan Du, Anam Nawaz Khan, Rex Zhou, Xiaoyang Liu+3

University of Tennessee, Knoxville · University of Chicago · Amazon+1

Language Model Decoding LLM Inference LLM Reliability

Open paper AskPDF arXiv

Greedy decoding from large language models is commonly treated as deterministic. We show it is not precision-invariant: the same model, prompt, and decoding algorithm produce different outputs in BF16 versus FP16 on identical hardware. Across our evaluations of six models (1.1B-7B parameters, four families; divergence additionally characterised at 12B) and three benchmarks, 49-100% of prompts diverge; a single token flip often cascades into trajectory-level divergence. We develop an empirical error-propagation analysis and find that 22 layers of accumulated body error do not distinguish flipping from non-flipping steps; the outcome depends primarily on the top-two logit margin at the LM head relative to the directional perturbation between the top-two candidates. The analysis makes five testable predictions about intervention outcomes, including that applying more FP32 compute (broader scope) makes agreement worse. The experiments match all five predictions. The best-performing low-overhead intervention we evaluate, selective FP32 LM head recomputation, triggered only when the margin falls below a threshold, delivers +22-36 pp exact agreement on A10G (+12-21 pp on L4 and A100) at less than 4% latency overhead in low-batch (batch size <=4) single-stream inference. We map the applicability boundary across six models and four batch sizes, and hypothesise that training-time precision stability is a determining factor. The method is a partial mitigation rather than a universal determinism guarantee: its benefit vanishes when body-originated error dominates, including at batch size >=8 and under end-to-end FP8 in our tests.

Abstract

LLM Reliability

Explore similar work

Greedy Decoding Is Not Precision-Invariant: Cross-Precision Output Divergence in LLM Inference

Demystifying Numerical Instability in LLM Inference: Achieving Reproducible Inference for Mission-Critical Tasks with HEAL

Accuracy is Not Enough: A Divergence-Based Approach to Evaluate Fidelity Loss in Quantized LLMs