Reproducibility is important for LoRA-adapted language models used in repeated evaluation, auditing, and decision-making. However, Split-K kernels used to introduce parallelism in low-rank LoRA multiplications can have the unintended consequence of numerical nondeterminism when partial results are combined using floating-point atomic additions. We investigate this non-determinism in vLLM across GSM8K mathematical reasoning, MBPP code generation, and Jev-style structured toxicity decisions on Civil Comments and introduce a deterministic Triton Split-K implementation. Under identical-input replay, Atomic Split-K=8 produces different FP32 shrink outputs in 97.61% of comparisons, while our Deterministic Split-K=8 shows no observed variation. Repeated atomic inference changes generated outputs in 17.30% of GSM8K and 14.96% of MBPP comparisons, sometimes altering task correctness. Structured-decision probabilities vary in 81.47% of atomic comparisons, with decision flips. Controlled tracing demonstrates how shrink-output differences propagate into the KV cache and how accumulated cache differences can change token decisions and final answers. Our deterministic Triton Split-K=8 implementation preserves parallelism, achieving 16.8-21.1% latency-based speedup over unsplit reduction with only 1.02-2.62% additional latency relative to Atomic Split-K=8.
Figures & tables
Figure 1: Thread-block parallelism in LoRA shrink. Split-K=1 assigns one CTA per output tile, whereas Split-K=8 assigns eight CTAs per tile across independent K partitions. The highlighted row illustrates one output tile.
Figure 2: Deterministic Split-K=8 LoRA shrink. Eight CTAs compute FP32 partials for one output tile. The last-finishing CTA combines them in a fixed order and writes the scaled result.
Figure 3: Run-to-run numerical variability in the LoRA shrink operation during prefill and first decode. Atomic Split-K=8 exhibits numerical variation under identical inputs, while Split-K=1 and Deterministic Split-K=8 show no observed differences.
Figure 4: Numerical differences between two Atomic Split-K=8 executions during prefill and decode at generated token 36. Colors indicate maximum absolute tensor differences on a logarithmic scale; gray denotes exact agreement. The first propagating difference appears in layer 2, with cached K/V differences from layer 3 onward. Subsequent differences reflect both accumulated state variation and numerical variation during later operations.
Figure 5: Single-coordinate causal intervention for GSM8K prompt 813. An observed FP32 difference crosses a BF16 rounding boundary and changes the token decision and final answer under controlled Split-K=1 continuation. FP32 positions are schematic.
Dataset
Reduction
Output changes
Correctness flips
GSM8K
Atomic SK8
467 (17.30%)
84
SK1
0
0
Det. SK8
0
0
MBPP
Atomic SK8
404 (14.96%)
53
SK1
0
0
Det. SK8
0
0
Table 1: Output and correctness changes under repeated greedy inference. Each reduction method has 2,700 comparisons per dataset, using repetitions 1-9 against repetition 0.
Metric
Atomic SK8
SK1
Det. SK8
Probability changes (%)
81.47
0
0
Mean absolute drift
0.00143
0
0
Maximum absolute drift
0.05672
0
0
Decision flips
7
0
0
Affected comments
2
0
0
Accuracy (%)
94.25
94.20
94.30
Table 2: Probability and decision stability on Civil Comments. Each reduction method has 9,000 comparisons across 1,000 comments.
Reduction
Fixed batch
Cross-batch
Atomic SK8
1748/4608 (37.93%)
168/384 (43.75%)
SK1
0/4608 (0%)
152/384 (39.58%)
Det. SK8
0/4608 (0%)
150/384 (39.06%)
Table 3: Output differences under fixed and changing batch sizes, aggregated across LoRA ranks 16 and 32.
Model
Rank
Speedup vs. SK1 (%)
Latency overhead vs. Atomic (%)
Llama
16
21.0
1.62
32
19.5
1.50
Gemma
16
20.6
2.49
32
18.9
2.17
Nemotron
16
18.3
1.13
32
16.8
1.20
Table 4: End-to-end latency comparison of Deterministic Split-K=8 against SK1 and Atomic Split-K=8, averaged across GSM8K and MBPP.
Figure 6: Output-throughput improvement of Atomic Split-K=8 and Deterministic Split-K=8 relative to Split-K=1 across batch sizes 1, 16, 32, and 64 using Llama on GSM8K.
Large language model (LLM) outputs are expected to be reproducible under greedy decoding, yet in practice the same model, prompt, and software stack produce different outputs on different GPUs. The root cause is floating-point non-associativity combined with hardware-dependent kernel selection. Inference frameworks select different matrix-multiplication kernels on each architecture, with different parallel reduction orders and unspecified tensor-core arithmetic, and the resulting rounding differences can flip output tokens. Existing solutions have imperfect cross-architecture reproducibility and incur a significant performance penalty. We present a solution employing a set of fixed-configuration fused-upcast GEMM kernels that load 16-bit weights from memory, upcast them to FP32 in registers, and accumulate with IEEE-754 arithmetic in a reduction order that is a pure function of the problem shape and is therefore independent of the device, its SM count, or kernel scheduling. By fixing the floating-point reduction order as a function of problem shape alone, every GPU runs the same operation sequence, so cross-architecture reproducibility of the linear layers reduces to correct IEEE-754 arithmetic rather than to rounding differences staying below a tie-flip threshold. We confirm our solution's linear-layer outputs are bitwise identical across NVIDIA Ampere, Ada, and Hopper GPUs, while running 1.17 to 3.1× faster end-to-end than the state-of-the-art solution and cutting weight-memory traffic in half.
Liam Cooper, Shinnung Jeong, Hyeran Jeon +2
Georgia Institute of Technology · University of California, Merced
Optimized floating-point inference is not reproducible across environments: the same model on different hardware, compilers or thread counts can produce different bits. We present tritify, an open-source runtime that executes the forward pass of real models -- decoder-only language models, Mixture-of-Experts decoders, and the Segment Anything 2.1 image/video stack -- with the entire run-time path in exact integer arithmetic; floating point is confined to offline quantization and the float reference used for verification. We make three separate claims and verify each. (1) Cross-hardware reproducibility: the integer path produces bit-identical output run to run, across CPU instruction-set architectures, and across CPU/GPU, carried by a reproducing SHA-256 of the output. (2) Decision fidelity: the integer path reproduces the float reference's decision on the models and inputs tested, with the numeric exceptions measured; per-layer fidelity through sixteen layers of OLMoE-1B-7B and of a 106-billion-parameter MoE shows no fidelity cliff. (3) Location independence: a model's computation can be split across instruction sets, CPU and GPU, and networked machines with zero effect on the output bits. GLM-4.5-Air (106,852,251,264 parameters, 427 GB of integer codes) generated its answer across three machines, none of which can hold its integer codes in RAM, bit-identically to a single-host reference. The consequence is verifiable distributed inference: an untrusted server or a fleet of devices runs the model while an open scalar oracle re-derives the exact bytes; a guarantee against a dishonest server is future work. We do not claim a per-device speed-up, bit-equality of floating-valued intermediates, or confidentiality. Code: https://github.com/pelloran-inc/tritify
Reproducible evaluation, verifiable inference and on-policy reinforcement learning (RL) need an LLM engine to give a request the same output bits whichever requests share its batch. Batch-invariant kernels exist, but engines still select each kernel and its parameters from the batch. Two kernels that compute the same sum can round it differently. The engines' deterministic modes remove the choice by keeping one conservative kernel per operator and pay for it in latency. Many fast kernels are batch-invariant by themselves, so it is enough to fix the few batch-dependent values that kernel selection reads. Detent expresses kernel selection as a program that the engine executes. It finds the fewest values in this program that must be fixed and tunes them for the workload before deployment. On multi-GPU deployments of Llama-3.1-70B and Qwen3-30B-A3B in vLLM and SGLang, Detent is batch-invariant. Tuned per load, it serves at 1.04-1.17x the latency per output token of Stock (each engine with its standard settings), against 1.67-10.3x for the deterministic modes. In SGLang it matches the GSM8K accuracy of SGLang's deterministic mode. Detent covers multiple deployment combinations, including multiple engines, GPU generations, tensor-parallel degrees, and dense and MoE models. Seeded RL rollouts of models up to Llama-3.1-70B replay bit for bit under other batches and cache splits. A trainer on Detent's kernels and values matches every rollout logprob bit for bit at 1.03x Stock's rollout time per token. BF16 GRPO then stays stable where Stock's run collapses.
Peixu Xin, Runchuan Feng, Jiazhen Guo +2
Independent Researcher, Boston University · Google · RadixArk +1