Reproducibility is important for LoRA-adapted language models used in repeated evaluation, auditing, and decision-making. However, Split-K kernels used to introduce parallelism in low-rank LoRA multiplications can have the unintended consequence of numerical nondeterminism when partial results are combined using floating-point atomic additions. We investigate this non-determinism in vLLM across GSM8K mathematical reasoning, MBPP code generation, and Jev-style structured toxicity decisions on Civil Comments and introduce a deterministic Triton Split-K implementation. Under identical-input replay, Atomic Split-K=8 produces different FP32 shrink outputs in 97.61% of comparisons, while our Deterministic Split-K=8 shows no observed variation. Repeated atomic inference changes generated outputs in 17.30% of GSM8K and 14.96% of MBPP comparisons, sometimes altering task correctness. Structured-decision probabilities vary in 81.47% of atomic comparisons, with decision flips. Controlled tracing demonstrates how shrink-output differences propagate into the KV cache and how accumulated cache differences can change token decisions and final answers. Our deterministic Triton Split-K=8 implementation preserves parallelism, achieving 16.8-21.1% latency-based speedup over unsplit reduction with only 1.02-2.62% additional latency relative to Atomic Split-K=8.
Figures & tables
Figure 1: Thread-block parallelism in LoRA shrink. Split-K=1 assigns one CTA per output tile, whereas Split-K=8 assigns eight CTAs per tile across independent K partitions. The highlighted row illustrates one output tile.
Figure 2: Deterministic Split-K=8 LoRA shrink. Eight CTAs compute FP32 partials for one output tile. The last-finishing CTA combines them in a fixed order and writes the scaled result.
Figure 3: Run-to-run numerical variability in the LoRA shrink operation during prefill and first decode. Atomic Split-K=8 exhibits numerical variation under identical inputs, while Split-K=1 and Deterministic Split-K=8 show no observed differences.
Figure 4: Numerical differences between two Atomic Split-K=8 executions during prefill and decode at generated token 36. Colors indicate maximum absolute tensor differences on a logarithmic scale; gray denotes exact agreement. The first propagating difference appears in layer 2, with cached K/V differences from layer 3 onward. Subsequent differences reflect both accumulated state variation and numerical variation during later operations.
Figure 5: Single-coordinate causal intervention for GSM8K prompt 813. An observed FP32 difference crosses a BF16 rounding boundary and changes the token decision and final answer under controlled Split-K=1 continuation. FP32 positions are schematic.
Dataset
Reduction
Output changes
Correctness flips
GSM8K
Atomic SK8
467 (17.30%)
84
SK1
0
0
Det. SK8
0
0
MBPP
Atomic SK8
404 (14.96%)
53
SK1
0
0
Det. SK8
0
0
Table 1: Output and correctness changes under repeated greedy inference. Each reduction method has 2,700 comparisons per dataset, using repetitions 1-9 against repetition 0.
Metric
Atomic SK8
SK1
Det. SK8
Probability changes (%)
81.47
0
0
Mean absolute drift
0.00143
0
0
Maximum absolute drift
0.05672
0
0
Decision flips
7
0
0
Affected comments
2
0
0
Accuracy (%)
94.25
94.20
94.30
Table 2: Probability and decision stability on Civil Comments. Each reduction method has 9,000 comparisons across 1,000 comments.
Reduction
Fixed batch
Cross-batch
Atomic SK8
1748/4608 (37.93%)
168/384 (43.75%)
SK1
0/4608 (0%)
152/384 (39.58%)
Det. SK8
0/4608 (0%)
150/384 (39.06%)
Table 3: Output differences under fixed and changing batch sizes, aggregated across LoRA ranks 16 and 32.
Model
Rank
Speedup vs. SK1 (%)
Latency overhead vs. Atomic (%)
Llama
16
21.0
1.62
32
19.5
1.50
Gemma
16
20.6
2.49
32
18.9
2.17
Nemotron
16
18.3
1.13
32
16.8
1.20
Table 4: End-to-end latency comparison of Deterministic Split-K=8 against SK1 and Atomic Split-K=8, averaged across GSM8K and MBPP.
Figure 6: Output-throughput improvement of Atomic Split-K=8 and Deterministic Split-K=8 relative to Split-K=1 across batch sizes 1, 16, 32, and 64 using Llama on GSM8K.