BlockLock: Reproducible LLM Inference with Fixed Batch Geometry
Authors: Xiuyuan Qi, Pedro Reviriego, Javier Conde, Shanshan Liu
Organizations: University of Electronic Science and Technology of China, Chengdu, Sichuan, China · Universidad Polit´ecnica de Madrid, Madrid, Community of Madrid, Spain.
Reproducible LLM inference requires that a request produces the same output regardless of the other requests in the batch. Existing solutions pay for this in different ways: batch-invariant kernels constrain every operator, and isolating prefill with fixed-shape decode ({our CoRun-style baseline}) gives up shared prefill compute and pads decode. We present BlockLock, which keeps shared batched prefill and pins only the numerical path of the protected request: it fixes the geometry that the request sees, with {two modes: BlockLock-Rows uses fixed-width rows and retains batched decode of all requests; BlockLock-Pack uses terminal-anchored packing with an isolated dummy segment, then extracts the protected request's KV state for target-only decode. Under a stated per-deployment contract, both modes preserve the target's output bitwise when its companions change.} Across five {dense} models (Qwen3-8B/14B, Llama-3.1-8B-Instruct, and Gemma-3-4B/12B-IT) and three single-GPU deployments (NVIDIA A6000, A100, and H100), we evaluate our approach on both tinyBenchmarks and BFCL{; we also include initial tensor-parallel experiments on two NVIDIA A6000 GPUs. Both modes preserve target sequences and Top20-logit bits in the reported tests.} At equal delivered work, {BlockLock-Rows} reduces latency by 53.5--55.0% and improves throughput by 115.2--122.2% relative to vLLM Batch-Invariant; compared with batch-invariant kernels, it needs no custom kernel implementations and constrains only the protected requests.
Figures & tables
Figure 1: Batch-induced nondeterminism. Processing the exact same target prompt x (blue) alongside different companion requests ( f1,f2 vs. f3 ) alters tensor padding and reduction orders, flipping near-tie logits to produce divergent outputs.
Figure 2: Limitations of the CoRun-style baseline. (1) Prefill executes sequentially ( B=1 ) to avoid cross-request interference, forfeiting batching efficiency. (2) Decode enforces a fixed 256-slot batch shape, wasting compute on masked padding slots when concurrency is low. The 256-slot envelope depicts CoRun’s evaluated configuration.
Figure 3: Execution geometry comparison. (a) Padded batching pads all rows to the longest request in the batch, causing x ’s padding offset to shift with its companions. (b) BlockLock-Rows executes a single [B,W] prefill where every row is aligned to a fixed class width W (a multiple of block size L ), preserving x ’s exact geometry and batched decode across all B requests. (c) BlockLock-Pack concatenates requests into a [1,BW] block with x anchored at the terminal interval [P−Lx,P) and an isolated dummy segment absorbing length variations, followed by target-only KV extraction and decode.
Figure 4: Reproducibility and phase efficiency across GPUs and models. (a,b) Generated-token and Top-20 logit-trace agreement within each fixed batch size. (c) Per-target token agreement. (d–f) Token agreement by dataset. (g) Prefill and decode time normalized to STD. (h) Per-dataset decode latency (evaluated at 500 output tokens per request). (i) Useful-input prefill throughput relative to STD.
Model
Method
Prefill (ms)
Decode (ms)
Total (ms)
Speedup
Output tokens
Tokens/s
Qwen3-8B
Padded batching
66.99
1213.59
1280.58 ± 4.17
3.362 ×
256
199.91
CoRun
91.55
4214.20
4305.75 ± 18.27
1.000 ×
256
59.46
BlockLock-Rows
76.58
1236.13
1312.71 ± 3.83
3.280 ×
256
195.02
BlockLock-Pack
83.86
1110.04
1193.91 ± 10.37
3.606 ×
64
53.61
Qwen3-14B
Padded batching
106.16
1756.99
1863.16 ± 2.32
3.211 ×
256
137.40
CoRun
137.02
5846.13
5983.14 ± 0.43
1.000 ×
256
42.79
Table 1: Phase latency and throughput comparison across eight datasets on NVIDIA A6000. At B=4 , all schemes decode four requests.
vLLM
Torch
Dataset
STD
BI
Rows
Pack
Padded
Rows
Pack
Qwen3-8B
MMLU
871.32
2032.84 (+133.3%)
902.68 (+3.6%)
874.84 (N/A)
1194.34
1201.19 (+0.6%)
1101.97 (N/A)
AI2 ARC
820.51
2006.00 (+144.5%)
884.39 (+7.8%)
857.39 (N/A)
1134.29
1133.47 (-0.1%)
1068.36 (N/A)
HellaSwag
901.02
2049.74 (+127.5%)
952.99 (+5.8%)
925.91 (N/A)
1342.03
1353.24 (+0.8%)
1158.08 (N/A)
GSM8K
845.56
2000.99 (+136.6%)
901.43 (+6.6%)
872.76 (N/A)
1179.14
1194.50 (+1.3%)
1084.28 (N/A)
Table 2: Means of Prefill+decode time (ms). B=4 ; full-batch methods generate 256 output tokens per group. Relative latency changes below the times use vLLM STD or Torch Padded as the within-engine baseline. Positive values mean higher latency: 100(t/tbase−1)% . N/A denotes a decrease.
Figure 5: Mean instrumented prefill and decode latency across the seven evaluated methods, averaged over eight datasets ( B=4 , 64 output tokens per decoded request). Dark and light bars show prefill and decode. Asterisks mark target-only Pack (one output request); other methods decode four.
Width W / output T
Padded
CoRun
Rows
Pack
1024 / 128
3.542
8.472 (+139.2%)
3.761 (+6.2%)
2.654 (N/A)
1024 / 2048
72.988
134.345 (+84.1%)
75.812 (+3.9%)
36.678 (N/A)
4096 / 128
8.187
9.016 (+10.1%)
9.176 (+12.1%)
6.132 (N/A)
4096 / 2048
129.529
135.103 (+4.3%)
141.141 (+9.0%)
39.669 (N/A)
Blazedit / 64
2.133
4.421 (+107.3%)
3.061 (+43.5%)
2.339 (N/A)
BurstGPT / 64
1.173
4.175 (+255.9%)
1.198 (+2.1%)
1.134 (N/A)
Table 3: Fixed-target workload time (s). B=4 . Padded, CoRun, and Rows decode all four requests; Pack decodes the protected target only. Each trace workload is one representative four-request group. Relative latency changes below the times use Padded as the baseline. Positive values mean higher latency: 100(t/tbase−1)% . N/A denotes a decrease.
Approach
Bitwise-reproducible across companions
Needs custom kernels
Batched prefill efficiency
Decode efficiency
Scope / limits
Padded batching
No (52.59% overall)
No
Yes (pads to longest)
Shared batched
No determinism guarantee
Batch-invariant kernels
Yes (across batch shapes)
Yes (per op/GPU)
Shared, constrained
Shared, constrained
Constrains all requests
LLM-42
Yes (via verification)
No
Fast path + verify
Rollback on mismatch
Verification/rollback cost
CoRun
Yes (never mixed)
No
No ( B=1 prefill)
Fixed-shape padded slots
Underutilizes prefill
BlockLock-Rows
Yes (per deployment)
No
Shared [B,W]
Batched ( B requests)
Per deployment
BlockLock-Pack
Yes (per deployment)
No
Shared [1,BW]
Target only ( 1 request)
Per deployment
Table 4: Qualitative comparison of reproducible LLM inference approaches. BlockLock guarantees bitwise-identical outputs per deployment with repeat-stable kernels without requiring custom operator implementations.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Platform
Model
Padded Seq. Agree. (%)
BlockLock Agree. (%)
Evaluated Conditions
A100
Qwen3-14B
57.4
100.0
135
A100
Qwen3-8B
56.6
100.0
159
A6000
Qwen3-14B
55.8
100.0
105
A6000
Qwen3-8B
62.9
100.0
160
A6000
Gemma-3-4B-IT
40.4
100.0
130
A6000
Llama-3.1-8B
44.9
100.0
160
Appendix
Table 1: Pair-weighted generated-sequence agreement (%) and Top-20 raw-logit agreement (%) across the evaluated single-GPU and TP=2 configurations with T = 500 generated output tokens per request ( B∈{1,2,4,8,16} , 10 repetitions per condition). The single-GPU subtotal and the combined total are distinct populations. CoRun , BlockLock-Rows , and BlockLock-Pack achieve 100.0% across all 1,336 conditions.
Figure 1: Qwen3-8B on NVIDIA RTX A6000 ( BF16 , prompt , T=500 ): (a) generated-sequence agreement; (b) prefill and decode cuda time. Method labels follow the correspondence defined at the start of Section E .
Figure 2: Qwen3-14B on NVIDIA RTX A6000 ( BF16 , prompt , target question Q1, T=500 ): (a) generated-sequence agreement; (b) top-20 raw-logit agreement; (c) prefill and decode cuda time. Method labels follow the correspondence defined at the start of Section E .
Figure 3: Llama-3.1-8B-Instruct on NVIDIA RTX A6000 ( BF16 , prompt , T=500 ): (a) generated-sequence agreement; (b) prefill and decode cuda time. Method labels follow the correspondence defined at the start of Section E .
Figure 4: Gemma-3-4B-IT on NVIDIA RTX A6000 ( BF16 , prompt , target question Q1, T=500 ): (a) generated-sequence agreement; (b) top-20 raw-logit agreement; (c) prefill and decode cuda time. Method labels follow the correspondence defined at the start of Section E .
Figure 5: Gemma-3-12B-IT on NVIDIA RTX A6000 ( BF16 , prompt , target question Q0, T=500 ): (a) generated-sequence agreement; (b) top-20 raw-logit agreement; (c) prefill and decode cuda time. Method labels follow the correspondence defined at the start of Section E .
Figure 6: Qwen3-8B on NVIDIA A100 ( BF16 , prompt , target question Q2, T=500 ): (a) generated-sequence agreement; (b) top-20 raw-logit agreement; (c) prefill and decode cuda time. Method labels follow the correspondence defined at the start of Section E .
Figure 7: Llama-3.1-8B-Instruct on NVIDIA A100 ( BF16 , prompt , target question Q0, T=500 ): (a) generated-sequence agreement; (b) top-20 raw-logit agreement; (c) prefill and decode cuda time. Method labels follow the correspondence defined at the start of Section E .
Figure 8: Gemma-3-4B-IT on NVIDIA A100 ( BF16 , prompt , target question Q0, T=500 ): (a) generated-sequence agreement; (b) top-20 raw-logit agreement; (c) prefill and decode cuda time. Method labels follow the correspondence defined at the start of Section E .
Figure 9: Gemma-3-12B-IT on NVIDIA A100 ( BF16 , prompt , target question Q0, T=500 ): (a) generated-sequence agreement; (b) top-20 raw-logit agreement; (c) prefill and decode cuda time. Method labels follow the correspondence defined at the start of Section E .
Figure 10: Qwen3-8B on NVIDIA H100 ( BF16 , native , target question Q1, T=500 ): (a) generated-sequence agreement; (b) prefill and decode cuda time. Method labels follow the correspondence defined at the start of Section E .
Figure 11: Qwen3-14B on NVIDIA H100 ( BF16 , prompt , target question Q1, T=500 ): (a) generated-sequence agreement; (b) prefill and decode cuda time. Method labels follow the correspondence defined at the start of Section E .
Figure 12: Llama-3.1-8B-Instruct on NVIDIA H100 ( BF16 , prompt , T=500 ): (a) generated-sequence agreement; (b) prefill and decode cuda time. Method labels follow the correspondence defined at the start of Section E .
Figure 13: Gemma-3-4B-IT on NVIDIA H100 ( BF16 , prompt , T=500 ): (a) generated-sequence agreement; (b) prefill and decode cuda time. Method labels follow the correspondence defined at the start of Section E .
Reproducible evaluation, verifiable inference and on-policy reinforcement learning (RL) need an LLM engine to give a request the same output bits whichever requests share its batch. Batch-invariant kernels exist, but engines still select each kernel and its parameters from the batch. Two kernels that compute the same sum can round it differently. The engines' deterministic modes remove the choice by keeping one conservative kernel per operator and pay for it in latency. Many fast kernels are batch-invariant by themselves, so it is enough to fix the few batch-dependent values that kernel selection reads. Detent expresses kernel selection as a program that the engine executes. It finds the fewest values in this program that must be fixed and tunes them for the workload before deployment. On multi-GPU deployments of Llama-3.1-70B and Qwen3-30B-A3B in vLLM and SGLang, Detent is batch-invariant. Tuned per load, it serves at 1.04-1.17x the latency per output token of Stock (each engine with its standard settings), against 1.67-10.3x for the deterministic modes. In SGLang it matches the GSM8K accuracy of SGLang's deterministic mode. Detent covers multiple deployment combinations, including multiple engines, GPU generations, tensor-parallel degrees, and dense and MoE models. Seeded RL rollouts of models up to Llama-3.1-70B replay bit for bit under other batches and cache splits. A trainer on Detent's kernels and values matches every rollout logprob bit for bit at 1.03x Stock's rollout time per token. BF16 GRPO then stays stable where Stock's run collapses.
Peixu Xin, Runchuan Feng, Jiazhen Guo +2
Independent Researcher, Boston University · Google · RadixArk +1
Diffusion language models (dLLMs) generate text by iteratively denoising multiple token positions in parallel, offering an attractive alternative to strictly autoregressive decoding. In practice, however, block-wise dLLM inference exposes a difficult granularity trade-off: small blocks preserve local conditioning but require many denoising steps, whereas large blocks expose more parallelism but can make premature commitments and accumulate cache error. Existing acceleration methods typically choose a single block size per request, leaving the complementarity among block sizes unused. We show that block size itself is a useful branching dimension. Different block sizes induce related but non-identical KV-cache trajectories: branches often share an initial prefix, bifurcate at semantically decisive positions, and later agree on syntactically lightweight tokens. Motivated by this structure, we propose BlockBatch, a training-free online inference framework that executes multiple block-size branches for the same request inside a batched forward pass. BlockBatch coordinates these branches through confidence-gated token merging, leader-based synchronization, and periodic full-sequence refreshes that re-anchor local block updates to a globally consistent KV state. Across 3 representative dLLMs and 4 datasets, BlockBatch reduces denoising NFEs by 26.6% on average and achieves a 1.33× average end-to-end speedup over Fast-dLLM while preserving accuracy. These results identify block-size diversity as a practical and previously underexplored axis for branch-parallel dLLM inference.
Rowhammer on GPU DRAM has enabled adversarial bit flips in model weights; shared KV-cache blocks in LLM serving systems present an analogous but previously unexamined target. In vLLM's Prefix Caching, these blocks exist as a single physical copy without integrity protection. Using software fault injection under ideal bit targeting, we characterize worst-case severity and identify three properties: (1) Silent divergence - 13 of 16 BF16 bit positions produce coherent but altered outputs, indistinguishable from legitimate responses without a clean baseline. (2) Selective propagation - only requests sharing the targeted prefix are affected. (3) Persistent accumulation - no temporal decay occurs, so cumulative damage grows linearly with subsequent requests. Together, these constitute a threat profile distinct from weight corruption: silent divergence and selective propagation enable detection evasion; persistent accumulation then proceeds unchecked, yielding damage amplification bounded only by how long the block remains cached. A checksum-based countermeasure detects any single-bit corruption at scheduling time, bounding cumulative damage to one batch independent of the block's cache lifetime, with negligible overhead. These results argue for integrity protection of prefix blocks before end-to-end exploitation is demonstrated.