The Integer Alibi: Localizing Cross-Kernel Divergence in INT8-Quantized LLM Inference

Teng-Ruei Chen

cs.LGAug 13, 2026

The Integer Alibi: Localizing Cross-Kernel Divergence in INT8-Quantized LLM Inference

Authors: Teng-Ruei Chen

Organizations: Krixvon, Taipei 100, Taiwan

Abstract

Two GPU kernels implementing the same scaled INT8 GEMM interface are usually treated as interchangeable. We test that assumption: holding the checkpoint, prompts, hardware, inference engine, decoding, and quantization configuration fixed, we swap only the INT8 linear kernel (CUTLASS versus Triton) inside vLLM. At 1.7B each arm reproduces itself bit-for-bit across cold restarts, yet the arms agree on no sequence in any end-to-end comparison we ran (0/8, 0/16, and 0/64). What makes this more than a benchmark discrepancy is an integer alibi: for shared INT8 operands under a verified no-overflow bound, the INT32 dot product is exact and order-independent, so the accumulator cannot be the source of any difference. Feeding both kernels identical operands from every linear layer of Qwen3-1.7B and 8B (196 and 252 layers), we find bit-identical outputs under power-of-two scales, confirming a pinned prediction list 196/196 and 252/252 (pre-registered at 1.7B, pinned but not blind at 8B), and observed differences of at most one bfloat16 spacing under the checkpoints' real scales. This localizes the divergence to scale application and output rounding after the exact accumulator. Applied as a probe checkpoint, the same intervention restores end-to-end bitwise agreement (8/8 and 16/16 sequences). Cross-implementation FP8 GEMM shows a different signature: both the prevalence and the magnitude of differences grow with reduction depth, while the INT8 fraction stays at parts per million and within one spacing over a 64x range of K. Teacher-forced replay ties layers to tokens: flips concentrate at small logit margins, which predict flip risk with ROC-AUC 0.94 on 16,384 positions. We will release the pre-registration, per-layer predictions, manifests with kernel-selection evidence, and a conformance procedure that turns these controls into a concrete check for kernel interchangeability.

Figures & tables

Work	Treatment variable	Numeric path	Observation scale	Relation to this paper
Schlögl et al. [ 9 ]	Platform, SIMD, convolution algorithm	General NN inference	Layer, model output	Stacks are not numerically neutral; we fix the platform and isolate one kernel
Yuan et al. [ 10 ]	Batch size, GPU count/type, precision	BF16, FP32	Token, accuracy, length	Greedy LLM outputs are configuration-sensitive; we fix configuration and add an exact integer control
He [ 11 ]	Batch size (serving shape)	Floating-point kernels	Kernel to token	Buys batch invariance with throughput; we study fixed-shape interchangeability
TBIK [ 12 ]	Tensor-parallel size	Floating-point reductions	Bitwise model output	Aligns reduction trees; the integer path needs only an epilogue contract
Pape et al. [ 13 ]	Whole inference backend	Mostly floating-point serving	Benchmark score	Backend is a hidden hyperparameter; we isolate one kernel within it
HEAL [ 14 ]	Heterogeneous GPUs, 16-bit path	FP16/BF16 boundaries	Activation, token, task	Attributes error to kernel boundaries, uses teacher forcing; we add an exact-accumulator control and a scale intervention

TABLE I: Prior work organized by treatment variable. The rightmost column states what this paper adds rather than what the prior work lacks; several of these studies isolate mechanisms we rely on.

	Qwen3-1.7B	Qwen3-8B
int8 linear layers	196	252
predicted bitwise-safe	196	252
min headroom to $2^{24}$ (bits)	2.82	3.36
capture tile $M$ (tokens/layer)	512	512
uniform pow2 scales: bit-identical layers	196/196	252/252
real scales: bit-identical layers	8/196	10/252

TABLE II: Per-layer verification. “Safe” means every observed accumulator magnitude stayed below

2^{24}

on the pinned prompts, so bitwise identity is required under power-of-two scales, and observed. The 1.7B prediction list was pre-registered before any kernel comparison; the 8B list was hash-pinned before the authoritative rerun but after an earlier result on the same data was known (Section IV ). Under real scales, every observed difference stays within one bf16 spacing.

Panel of Figure 3 — Fig. 1: Observed accumulator headroom per layer (distance of the maximum $|{\rm acc}|$ below the $2^{24}$ float32 representability threshold, in bits; larger is safer). Every layer of both models clears the threshold, so the prediction lists mark all 448 layers bitwise-safe: their exact accumulators enter the epilogue without information loss. The 1.7B list was pre-registered before any kernel comparison; the 8B list was pinned before the authoritative rerun but is not blind (Section IV ). The claim is conditional on the pinned evaluation prompts; worst-case bounds do not clear the threshold.

Panel of Figure 4 — Fig. 2: Cross-implementation divergence versus reduction depth $K$ ; both series use $M{=}256$ , $N{=}2048$ and the same $K$ grid, but their designs otherwise differ and are not matched. FP8 (CUTLASS vs. torch._scaled_mm ): 20 seeds, nested-prefix inputs so that only $K$ varies within a seed, fixed per-tensor scalar scales, Gaussian operands cast to fp8; the fraction grows from 8% to 53% of elements and is best fit, among four pre-listed candidates under a common objective with held-out folds, by generalized saturation with exponent $\alpha=0.518$ (seed-bootstrap CI90 $[0.5181,0.5187]$ ; jackknife and leave-one- $K$ -out ranges $[0.516,0.529]$ ), a shape compatible with square-root-driven boundary crossing under saturation. INT8 (CUTLASS vs. Triton): a single unreplicated sweep in which uniform int8 operands and random per-token and per-channel scales are redrawn at each $K$ ; 18 differing elements in total ( $1.9$ – $7.6$ ppm), showing no growth detectable at this sample size. The exact accumulator leaves only per-element epilogue rounding, which does not accumulate with depth; the INT8 series tests that expectation weakly, and the per-layer results of Section V carry the stronger evidence.

comparison	1.7B	8B
same arm, two cold runs (each arm)	8/8	—
CUTLASS vs. Triton, real scales	0/8	0/16
same, 64-prompt rail run	0/64	—
CUTLASS vs. Triton, pow2 probe	8/8	16/16
SGLang vs. vLLM arms, real scales	0/8	—
SGLang vs. vLLM, pow2 probe	1/8	—

TABLE III: End-to-end greedy agreement; “identical” means the full token sequence matches bitwise. Continuations are 64 tokens except the 64-prompt rail run, which generates 256. At 1.7B each arm is self-consistent across two cold restarts; the 8B and 64-prompt runs were not repeated. Arms disagree with each other under real scales and agree bitwise under the pow2 probe checkpoint.

Panel of Figure 6 — Fig. 3: Cross-kernel token flips against the logit margin, teacher-forced on identical contexts (64 prompts $\times$ 256 positions, identity-verified artifacts). Flip rate falls monotonically from 34% in the lowest-margin decile to zero above a margin of ${\sim}9$ ; overall 769 of 16,384 positions flip (4.7%). The margin predicts flips with ROC-AUC 0.942 (prompt-cluster CI90 $[0.935,0.949]$ ); a margin-binned calibration achieves held-out Brier 0.0352 against a 0.0440 constant-prevalence baseline.

Check	Requirement	A violation suggests, or first check
Shared operands	Both arms consume the same int8 tensors and scales, verified from artifacts	The comparison is not kernel-only; no localization is licensed
INT32 no-overflow	$16256K<2^{31}$ for the layer’s $K$	Accumulation may wrap, so the integer alibi does not apply at all and no localization is licensed
Lossless FP32 entry	Observed $\|{\rm acc}\|\leq 2^{24}$ per layer	Only the universal no-loss guarantee at the epilogue boundary is lost; individual values may still be exact, so judge per value or widen the tolerance rather than discard the layer
Exact accumulator	Bit-identical INT32 accumulation across arms	An integer-path defect: the arithmetic admits no legal difference
Pow2-scale identity	Bit-identical outputs under power-of-two scales, excluding subnormal and overflow results	First check the listed preconditions and each arm’s intermediate semantics; if those hold, an epilogue defect, since rounding order cannot matter under ( 2 )
Real-scale tolerance	Cross-arm bf16 ulp distance $\leq 1$ , assuming both arms round once from fp32 intermediates; near-zero and non-finite elements reported separately	Suggests reduced intermediate precision, fusion, or another departure from single-rounding semantics rather than mere double rounding

TABLE IV: The conformance procedure implied by the controls in this paper. The scripts of this study instantiate these checks for one scaled-INT8 kernel pair; another pair would require adapting them rather than running them unchanged. Each row names what a violation points to first; the exclusions are the conditions under which the underlying argument does not apply.

GPU Kernel Correctness Verification

See all 14 papers

Deterministic LLM Inference Across GPU Kernels: Power-of-Two INT8 Quantization Scales and the Limits of Tolerance-Based Conformance
Aug 25, 2026Teng-Ruei ChenLLM Quantization Neural Network Quantization
Krixvon, Taipei 100, Taiwan
Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures
Sep 22, 2026Liam Cooper, Shinnung Jeong, Hyeran Jeon +2GPU Kernel Optimization LLM Inference Acceleration
Georgia Institute of Technology · University of California, Merced
Realizing Native INT8 Compute for Diffusion Transformers on Consumer GPUs: A Fused INT8 GEMM Kernel for Ideogram 4.0
Jun 12, 2026Ali Asaria, Tony Salomone, Deep GandhiGPU Kernel Optimization Diffusion Transformer
Transformer Lab

Aug 25, 2026J/K move · Enter open · S save

Deterministic LLM Inference Across GPU Kernels: Power-of-Two INT8 Quantization Scales and the Limits of Tolerance-Based Conformance

Teng-Ruei Chen

Krixvon, Taipei 100, Taiwan

LLM Quantization Neural Network Quantization GPU Kernel Correctness Verification ML Reproducibility

Open paper AskPDF arXiv

Conformance suites for quantized GEMM kernels ask whether two implementations agree within a tolerance. We measure what such a suite can detect. Injecting nine faults into a reference INT8 pipeline over 8,232 layer--fault--regime cells of Qwen3-1.7B, we find that every one of five epilogue faults -- scale precision, double rounding, multiplication order, output truncation, fused ordering -- moves the output by at most a single bfloat16 spacing, and by exactly one whenever it moves it at all, across 5,880 cells. A tolerance of one spacing is therefore blind to the entire class by construction: four of the five faults are detected by no check in the suite, and the fifth only under power-of-two scales. Faults that violate the accumulator's exactness preconditions, or that break operand sharing, are detected without exception, and a null fault never fires. What a tolerance-based suite of this shape establishes is therefore narrower than interchangeability: that the preconditions hold, that operands are shared, and that differences stay within one spacing. The power-of-two constraint that exposes the one detected fault is also deployable. Requantizing every weight scale to its nearest power of two makes CUTLASS and Triton agree bitwise at every linear layer (196/196 and 252/252, against 8/196 and 10/252 under the checkpoints' own scales) and yields byte-identical generated token sequences at 1.7B, 8B and 14B (8/8 prompts, against 0/8 at all three). Observed perplexity point estimates are +0.32%, -0.28% and +0.48%; the 90% intervals cover zero at the two smaller sizes but not at 14B, reaching +0.71% and +0.76%. A previously reported +157% perplexity for this intervention was an artifact of a probe that rewrote scales without requantizing the weights; separating the effects attributes 99.8% of it to the resulting weight--scale mismatch rather than to the power-of-two constraint itself.

Abstract

GPU Kernel Correctness Verification

Explore similar work

Deterministic LLM Inference Across GPU Kernels: Power-of-Two INT8 Quantization Scales and the Limits of Tolerance-Based Conformance

Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures

Realizing Native INT8 Compute for Diffusion Transformers on Consumer GPUs: A Fused INT8 GEMM Kernel for Ideogram 4.0