Deterministic LLM Inference Across GPU Kernels: Power-of-Two INT8 Quantization Scales and the Limits of Tolerance-Based Conformance

Teng-Ruei Chen

cs.LGAug 25, 2026

Deterministic LLM Inference Across GPU Kernels: Power-of-Two INT8 Quantization Scales and the Limits of Tolerance-Based Conformance

Authors: Teng-Ruei Chen

Organizations: Krixvon, Taipei 100, Taiwan

Abstract

Conformance suites for quantized GEMM kernels ask whether two implementations agree within a tolerance. We measure what such a suite can detect. Injecting nine faults into a reference INT8 pipeline over 8,232 layer--fault--regime cells of Qwen3-1.7B, we find that every one of five epilogue faults -- scale precision, double rounding, multiplication order, output truncation, fused ordering -- moves the output by at most a single bfloat16 spacing, and by exactly one whenever it moves it at all, across 5,880 cells. A tolerance of one spacing is therefore blind to the entire class by construction: four of the five faults are detected by no check in the suite, and the fifth only under power-of-two scales. Faults that violate the accumulator's exactness preconditions, or that break operand sharing, are detected without exception, and a null fault never fires. What a tolerance-based suite of this shape establishes is therefore narrower than interchangeability: that the preconditions hold, that operands are shared, and that differences stay within one spacing. The power-of-two constraint that exposes the one detected fault is also deployable. Requantizing every weight scale to its nearest power of two makes CUTLASS and Triton agree bitwise at every linear layer (196/196 and 252/252, against 8/196 and 10/252 under the checkpoints' own scales) and yields byte-identical generated token sequences at 1.7B, 8B and 14B (8/8 prompts, against 0/8 at all three). Observed perplexity point estimates are +0.32%, -0.28% and +0.48%; the 90% intervals cover zero at the two smaller sizes but not at 14B, reaching +0.71% and +0.76%. A previously reported +157% perplexity for this intervention was an artifact of a probe that rewrote scales without requantizing the weights; separating the effects attributes 99.8% of it to the resulting weight--scale mismatch rather than to the power-of-two constraint itself.

Figures & tables

Fig. 1: Overview. Top: a W8A8 linear layer’s integer accumulation is exact and order-independent under a verified bound, so cross-kernel differences can arise only in the epilogue; nine fault families are injected with ground truth known by construction and scored against a prediction matrix (63 cells pre-data; corrected and re-pinned after a disclosed smoke run). Bottom left: the suite is reliable about its own preconditions and about operand provenance, and structurally blind to epilogue defects that stay within one bfloat16 spacing—which is where single-rounding and precision faults live. Bottom centre: power-of-two scales, built by requantization rather than scale rewriting, give bitwise cross-kernel agreement at observed perplexity costs between

-0.28\%

and

+0.48\%

	bitwise	ground truth	deploy. cost	pre-reg.
CRADLE [ 12 ]	–	–	–	–
Predoo [ 13 ]	–	–	–	–
Eq. checking [ 14 ]	$\bullet$	–	–	–
SMT tensor cores [ 15 ]	$\bullet$	–	–	–
Dev. frameworks [ 8 ]	–	–	–	–
Nondet. LLM [ 9 ]	–	–	–	–

TABLE I: Where this paper sits. “Bitwise” = compares outputs at bit granularity; “ground truth” = evaluates a test suite against faults whose presence is known by construction; “deployable cost” = measures the accuracy price of a determinism mechanism on served checkpoints; “pre-reg.” = predictions or protocol pinned before measurement.

epilogue
fault	what it does
F1 scale precision	scales cast to bfloat16 before multiplying
F2 double rounding	extra rounding to bfloat16 between the two scale multiplies
F3 scale order	$\mathrm{acc}\cdot(s_{a}s_{w})$ in place of $(\mathrm{acc}\cdot s_{a})\cdot s_{w}$
F4 output truncation	output cast truncates toward zero instead of round-to-nearest-even
F5 fused order	per-column scale folded into the reduction, so the sum becomes a float sum

TABLE II: The fault catalogue. Epilogue faults are single, locally plausible mistakes a conventionally correct kernel could contain; precondition and operand faults violate what the exactness argument assumes; the null fault changes nothing, making silence informative.

fault	(1)	(2)	(3)	(4)	(5)	(6)	(7)
F1 precision	$\circ$	$\circ$	$\circ$	$\circ$	$\circ$	$\circ$ m	—
F2 dbl. round	$\circ$	$\circ$	$\circ$	$\circ$	$\circ$	$\circ$ m	—
F3 order	$\circ$	$\circ$	$\circ$	$\circ$	$\circ$	$\circ$	—
F4 truncate	$\circ$	$\circ$	$\circ$	$\circ$	$\bullet$ ∗	$\circ$ m	—
F5 fused	$\circ$	$\circ$	$\circ$	$\circ$	$\circ$	$\circ$ m	—
F6 overflow	$\circ$	$\bullet$	$\bullet$	$\bullet$	$\circ$ m	$\circ$	—

TABLE III: The 77-cell prediction matrix as re-pinned before the full run (the pre-data original had 63 cells; F9 was one row).

\bullet

= should fire,

\circ

= should stay silent, — = not applicable; superscript

m

= predicted miss (five pre-data, the sixth introduced by the F9 split); superscript

*

= corrected post-data. Checks: (1) shared operands, (2) INT32 no-overflow, (3) lossless float32 entry, (4) exact accumulator, (5) power-of-two identity, (6) real-scale tolerance, (7) token-level risk.

fault	cells	obs.	det.	detected by
epilogue
F1 scale precision	1,176	439	0	—
F2 double rounding	1,176	448	0	—
F3 scale order	1,176	205	0	—
F4 truncation	1,176	972	494	pow2 identity
real scales	588	478	0	—

TABLE IV: Detection by fault. “Obs.” counts cells where the injected fault changed at least one output element; unobservable cells are excluded from the detection denominator (Section III-D ). “Det.” counts cells where at least one check fired.

Fig. 2: Where each fault lands, as the real-scale tolerance check sees it. Above the dashed rule are the five epilogue faults: every one produces a maximum ULP distance of exactly

0

1

—never

2

—across all 5,880 cells, so the entire class sits inside the one-spacing interval that legal implementations are allowed to occupy. Below it, the operand fault F9 reaches distances from 68 to 35,571 in

67\%

of its cells. A tolerance partitions the fault space at the wrong boundary.

	per-layer		end to end
model	own	pow2	own	pow2
Qwen3-1.7B	8/196	196/196	0/8	8/8
Qwen3-8B	10/252	252/252	0/8	8/8
Qwen3-14B	—	—	0/8	8/8

TABLE V: Cross-kernel agreement at two granularities. Per-layer: identical captured operands through both kernels, over all linear layers. End to end: greedy generation on 8 pinned prompts of 64 tokens, “identical” meaning the token sequence matches byte for byte. At 14B the capture path dequantizes to bfloat16 and exceeds the 24 GB card, so no per-layer figure exists.

Fig. 3: Accuracy cost of the nearest-power-of-two constraint: relative perplexity change with

90\%

paired cluster-bootstrap intervals over 256 pinned WikiText windows, 10,000 draws. The interval covers zero at 1.7B and 8B and excludes it at 14B (drawn in red); point estimates span

-0.28\%

+0.48\%

, and the interval upper ends reach

+0.71\%

(1.7B) and

+0.76\%

(14B). Throughput at 1.7B moves between

-9.6\%

and

+1.4\%

(prefill) and

-1.6\%

and

+0.9\%

(decode) across six batch–length settings on a shared machine; 8B and 14B throughput was not measured.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 382 papers

Aug 13, 2026

The Integer Alibi: Localizing Cross-Kernel Divergence in INT8-Quantized LLM Inference

Two GPU kernels implementing the same scaled INT8 GEMM interface are usually treated as interchangeable. We test that assumption: holding the checkpoint, prompts, hardware, inference engine, decoding, and quantization configuration fixed, we swap only the INT8 linear kernel (CUTLASS versus Triton) inside vLLM. At 1.7B each arm reproduces itself bit-for-bit across cold restarts, yet the arms agree on no sequence in any end-to-end comparison we ran (0/8, 0/16, and 0/64). What makes this more than a benchmark discrepancy is an integer alibi: for shared INT8 operands under a verified no-overflow bound, the INT32 dot product is exact and order-independent, so the accumulator cannot be the source of any difference. Feeding both kernels identical operands from every linear layer of Qwen3-1.7B and 8B (196 and 252 layers), we find bit-identical outputs under power-of-two scales, confirming a pinned prediction list 196/196 and 252/252 (pre-registered at 1.7B, pinned but not blind at 8B), and observed differences of at most one bfloat16 spacing under the checkpoints' real scales. This localizes the divergence to scale application and output rounding after the exact accumulator. Applied as a probe checkpoint, the same intervention restores end-to-end bitwise agreement (8/8 and 16/16 sequences). Cross-implementation FP8 GEMM shows a different signature: both the prevalence and the magnitude of differences grow with reduction depth, while the INT8 fraction stays at parts per million and within one spacing over a 64x range of K. Teacher-forced replay ties layers to tokens: flips concentrate at small logit margins, which predict flip risk with ROC-AUC 0.94 on 16,384 positions. We will release the pre-registration, per-layer predictions, manifests with kernel-selection evidence, and a conformance procedure that turns these controls into a concrete check for kernel interchangeability.

Teng-Ruei Chen

Krixvon, Taipei 100, Taiwan

GPU Kernel Correctness Verification LLM Inference LLM Quantization ML Reproducibility

Ask PDF

May 20, 2026

Runtime-Certified Bounded-Error Quantized Attention

KV cache quantization reduces the memory cost of long-context LLM inference, but introduces approximation error that is typically validated only empirically. Existing systems rely on average-case robustness, with no mechanism to detect or recover from failures at runtime. We present a tiered KV cache architecture that enables runtime-certified attention: INT8 keys and INT4 values are stored in GPU memory, while FP16 originals are retained in system RAM for deterministic fallback. A two-term error decomposition yields per-head, per-step bounds on (i) attention distribution distortion from key quantization and (ii) value reconstruction error. These bounds are computed online and used to drive adaptive precision selection and a multi-stage fallback ladder, which guarantees recovery to the exact dense attention output when required. Across PG-19, NIAH, and RULER benchmarks on LLaMA~3.1-8B with contexts up to 128K, the system matches dense FP16 KV quality within noise for language modelling and retrieval tasks, while recovering catastrophic failures observed in naive INT8/INT4 baselines. Value-sensitive tasks at short context expose a controlled trade-off between compression and fidelity, which can be eliminated via tighter value tolerances or FP16-value fallback. The certification is local (per-head, per-step) and does not guarantee end-to-end model correctness, but ensures that each attention computation is either bounded relative to an FP16 reference or exactly recovered via fallback. This reframes KV cache quantization as a runtime-verified computation rather than a fixed approximation. The goal is not raw speedups, but enabling safe deployment of aggressive KV compression under strict quality constraints.

Dean Calver

Independent Researcher

LLM Inference KV Caching Attention Quantization KV-Cache Quantization

Ask PDF

Aug 19, 2026

Compress and Forget: bitsandbytes Quantization Amplifies Proactive Interference in LLMs

Proactive interference (PI) is a documented failure mode in large language models in which retrieval of a repeatedly overwritten value degrades as prior overwrites accumulate, mirroring a classical phenomenon in human working memory. Post-training quantization (PTQ) is now the default deployment path for open-weight models, yet its effect on this failure mode has not been tested. We evaluate three precision levels (FP16, INT8, INT4/NF4, via bitsandbytes) across three architecturally distinct instruction-tuned models (Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, Phi-3.5-mini-instruct), holding the retrieval task fixed. INT4 quantization significantly reduces accuracy under high interference in every model (e.g., from 81.0% to 68.3% for Qwen), confirmed by paired McNemar's tests (

p \le 2.6 \times 10^{-6}

) and a mixed-effects regression spanning all interference levels; INT8, often assumed safe, also carries a smaller but real penalty in two of three models. The effect is specific to semantically similar (word-type) distractors and reverses sign under a numeric control condition, and is mechanistically linked to a rise in same-key intrusion errors under INT4 (from 21.5% to 24.6% of trials,

p = 4.8 \times 10^{-7}

). A follow-up ablation shows the effect originates in the quantized transformer backbone rather than the output projection layer. These results suggest that bitsandbytes 4-bit quantization can impose an additional cost on applications relying on long, updatable, semantically dense contexts, even when aggregate benchmark accuracy appears largely unaffected. We release our code and tokenizer-verified vocabulary construction method at https://github.com/ShayanShahrabi/compress-and-forget

Shayan Shahrabi-Farahani, Dara Rahmati

Shahid Beheshti University · Shahid Beheshti University, Tehran, Iran

LLM Quantization Post-Training Quantization Low-Bit Quantization Knowledge Conflicts in Language Models

Ask PDF

Abstract

LLM Quantization

Explore similar work

The Integer Alibi: Localizing Cross-Kernel Divergence in INT8-Quantized LLM Inference

Runtime-Certified Bounded-Error Quantized Attention

Compress and Forget: bitsandbytes Quantization Amplifies Proactive Interference in LLMs