BlockLock: Reproducible LLM Inference with Fixed Batch Geometry
Authors: Xiuyuan Qi, Pedro Reviriego, Javier Conde, Shanshan Liu
Organizations: University of Electronic Science and Technology of China, Chengdu, Sichuan, China · Universidad Polit´ecnica de Madrid, Madrid, Community of Madrid, Spain.
Reproducible LLM inference requires that a request produces the same output regardless of the other requests in the batch. Existing solutions pay for this in different ways: batch-invariant kernels constrain every operator, and isolating prefill with fixed-shape decode ({our CoRun-style baseline}) gives up shared prefill compute and pads decode. We present BlockLock, which keeps shared batched prefill and pins only the numerical path of the protected request: it fixes the geometry that the request sees, with {two modes: BlockLock-Rows uses fixed-width rows and retains batched decode of all requests; BlockLock-Pack uses terminal-anchored packing with an isolated dummy segment, then extracts the protected request's KV state for target-only decode. Under a stated per-deployment contract, both modes preserve the target's output bitwise when its companions change.} Across five {dense} models (Qwen3-8B/14B, Llama-3.1-8B-Instruct, and Gemma-3-4B/12B-IT) and three single-GPU deployments (NVIDIA A6000, A100, and H100), we evaluate our approach on both tinyBenchmarks and BFCL{; we also include initial tensor-parallel experiments on two NVIDIA A6000 GPUs. Both modes preserve target sequences and Top20-logit bits in the reported tests.} At equal delivered work, {BlockLock-Rows} reduces latency by 53.5--55.0% and improves throughput by 115.2--122.2% relative to vLLM Batch-Invariant; compared with batch-invariant kernels, it needs no custom kernel implementations and constrains only the protected requests.
Figures & tables
Figure 1: Batch-induced nondeterminism. Processing the exact same target prompt x (blue) alongside different companion requests ( f1,f2 vs. f3 ) alters tensor padding and reduction orders, flipping near-tie logits to produce divergent outputs.
Figure 2: Limitations of the CoRun-style baseline. (1) Prefill executes sequentially ( B=1 ) to avoid cross-request interference, forfeiting batching efficiency. (2) Decode enforces a fixed 256-slot batch shape, wasting compute on masked padding slots when concurrency is low. The 256-slot envelope depicts CoRun’s evaluated configuration.
Figure 3: Execution geometry comparison. (a) Padded batching pads all rows to the longest request in the batch, causing x ’s padding offset to shift with its companions. (b) BlockLock-Rows executes a single [B,W] prefill where every row is aligned to a fixed class width W (a multiple of block size L ), preserving x ’s exact geometry and batched decode across all B requests. (c) BlockLock-Pack concatenates requests into a [1,BW] block with x anchored at the terminal interval [P−Lx,P) and an isolated dummy segment absorbing length variations, followed by target-only KV extraction and decode.
Figure 4: Reproducibility and phase efficiency across GPUs and models. (a,b) Generated-token and Top-20 logit-trace agreement within each fixed batch size. (c) Per-target token agreement. (d–f) Token agreement by dataset. (g) Prefill and decode time normalized to STD. (h) Per-dataset decode latency (evaluated at 500 output tokens per request). (i) Useful-input prefill throughput relative to STD.
Model
Method
Prefill (ms)
Decode (ms)
Total (ms)
Speedup
Output tokens
Tokens/s
Qwen3-8B
Padded batching
66.99
1213.59
1280.58 ± 4.17
3.362 ×
256
199.91
CoRun
91.55
4214.20
4305.75 ± 18.27
1.000 ×
256
59.46
BlockLock-Rows
76.58
1236.13
1312.71 ± 3.83
3.280 ×
256
195.02
BlockLock-Pack
83.86
1110.04
1193.91 ± 10.37
3.606 ×
64
53.61
Qwen3-14B
Padded batching
106.16
1756.99
1863.16 ± 2.32
3.211 ×
256
137.40
CoRun
137.02
5846.13
5983.14 ± 0.43
1.000 ×
256
42.79
Table 1: Phase latency and throughput comparison across eight datasets on NVIDIA A6000. At B=4 , all schemes decode four requests.
vLLM
Torch
Dataset
STD
BI
Rows
Pack
Padded
Rows
Pack
Qwen3-8B
MMLU
871.32
2032.84 (+133.3%)
902.68 (+3.6%)
874.84 (N/A)
1194.34
1201.19 (+0.6%)
1101.97 (N/A)
AI2 ARC
820.51
2006.00 (+144.5%)
884.39 (+7.8%)
857.39 (N/A)
1134.29
1133.47 (-0.1%)
1068.36 (N/A)
HellaSwag
901.02
2049.74 (+127.5%)
952.99 (+5.8%)
925.91 (N/A)
1342.03
1353.24 (+0.8%)
1158.08 (N/A)
GSM8K
845.56
2000.99 (+136.6%)
901.43 (+6.6%)
872.76 (N/A)
1179.14
1194.50 (+1.3%)
1084.28 (N/A)
Table 2: Means of Prefill+decode time (ms). B=4 ; full-batch methods generate 256 output tokens per group. Relative latency changes below the times use vLLM STD or Torch Padded as the within-engine baseline. Positive values mean higher latency: 100(t/tbase−1)% . N/A denotes a decrease.
Figure 5: Mean instrumented prefill and decode latency across the seven evaluated methods, averaged over eight datasets ( B=4 , 64 output tokens per decoded request). Dark and light bars show prefill and decode. Asterisks mark target-only Pack (one output request); other methods decode four.
Width W / output T
Padded
CoRun
Rows
Pack
1024 / 128
3.542
8.472 (+139.2%)
3.761 (+6.2%)
2.654 (N/A)
1024 / 2048
72.988
134.345 (+84.1%)
75.812 (+3.9%)
36.678 (N/A)
4096 / 128
8.187
9.016 (+10.1%)
9.176 (+12.1%)
6.132 (N/A)
4096 / 2048
129.529
135.103 (+4.3%)
141.141 (+9.0%)
39.669 (N/A)
Blazedit / 64
2.133
4.421 (+107.3%)
3.061 (+43.5%)
2.339 (N/A)
BurstGPT / 64
1.173
4.175 (+255.9%)
1.198 (+2.1%)
1.134 (N/A)
Table 3: Fixed-target workload time (s). B=4 . Padded, CoRun, and Rows decode all four requests; Pack decodes the protected target only. Each trace workload is one representative four-request group. Relative latency changes below the times use Padded as the baseline. Positive values mean higher latency: 100(t/tbase−1)% . N/A denotes a decrease.
Approach
Bitwise-reproducible across companions
Needs custom kernels
Batched prefill efficiency
Decode efficiency
Scope / limits
Padded batching
No (52.59% overall)
No
Yes (pads to longest)
Shared batched
No determinism guarantee
Batch-invariant kernels
Yes (across batch shapes)
Yes (per op/GPU)
Shared, constrained
Shared, constrained
Constrains all requests
LLM-42
Yes (via verification)
No
Fast path + verify
Rollback on mismatch
Verification/rollback cost
CoRun
Yes (never mixed)
No
No ( B=1 prefill)
Fixed-shape padded slots
Underutilizes prefill
BlockLock-Rows
Yes (per deployment)
No
Shared [B,W]
Batched ( B requests)
Per deployment
BlockLock-Pack
Yes (per deployment)
No
Shared [1,BW]
Target only ( 1 request)
Per deployment
Table 4: Qualitative comparison of reproducible LLM inference approaches. BlockLock guarantees bitwise-identical outputs per deployment with repeat-stable kernels without requiring custom operator implementations.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Platform
Model
Padded Seq. Agree. (%)
BlockLock Agree. (%)
Evaluated Conditions
A100
Qwen3-14B
57.4
100.0
135
A100
Qwen3-8B
56.6
100.0
159
A6000
Qwen3-14B
55.8
100.0
105
A6000
Qwen3-8B
62.9
100.0
160
A6000
Gemma-3-4B-IT
40.4
100.0
130
A6000
Llama-3.1-8B
44.9
100.0
160
Appendix
Table 1: Pair-weighted generated-sequence agreement (%) and Top-20 raw-logit agreement (%) across the evaluated single-GPU and TP=2 configurations with T = 500 generated output tokens per request ( B∈{1,2,4,8,16} , 10 repetitions per condition). The single-GPU subtotal and the combined total are distinct populations. CoRun , BlockLock-Rows , and BlockLock-Pack achieve 100.0% across all 1,336 conditions.
Figure 1: Qwen3-8B on NVIDIA RTX A6000 ( BF16 , prompt , T=500 ): (a) generated-sequence agreement; (b) prefill and decode cuda time. Method labels follow the correspondence defined at the start of Section E .
Figure 2: Qwen3-14B on NVIDIA RTX A6000 ( BF16 , prompt , target question Q1, T=500 ): (a) generated-sequence agreement; (b) top-20 raw-logit agreement; (c) prefill and decode cuda time. Method labels follow the correspondence defined at the start of Section E .
Figure 3: Llama-3.1-8B-Instruct on NVIDIA RTX A6000 ( BF16 , prompt , T=500 ): (a) generated-sequence agreement; (b) prefill and decode cuda time. Method labels follow the correspondence defined at the start of Section E .
Figure 4: Gemma-3-4B-IT on NVIDIA RTX A6000 ( BF16 , prompt , target question Q1, T=500 ): (a) generated-sequence agreement; (b) top-20 raw-logit agreement; (c) prefill and decode cuda time. Method labels follow the correspondence defined at the start of Section E .
Figure 5: Gemma-3-12B-IT on NVIDIA RTX A6000 ( BF16 , prompt , target question Q0, T=500 ): (a) generated-sequence agreement; (b) top-20 raw-logit agreement; (c) prefill and decode cuda time. Method labels follow the correspondence defined at the start of Section E .
Figure 6: Qwen3-8B on NVIDIA A100 ( BF16 , prompt , target question Q2, T=500 ): (a) generated-sequence agreement; (b) top-20 raw-logit agreement; (c) prefill and decode cuda time. Method labels follow the correspondence defined at the start of Section E .
Figure 7: Llama-3.1-8B-Instruct on NVIDIA A100 ( BF16 , prompt , target question Q0, T=500 ): (a) generated-sequence agreement; (b) top-20 raw-logit agreement; (c) prefill and decode cuda time. Method labels follow the correspondence defined at the start of Section E .
Figure 8: Gemma-3-4B-IT on NVIDIA A100 ( BF16 , prompt , target question Q0, T=500 ): (a) generated-sequence agreement; (b) top-20 raw-logit agreement; (c) prefill and decode cuda time. Method labels follow the correspondence defined at the start of Section E .
Figure 9: Gemma-3-12B-IT on NVIDIA A100 ( BF16 , prompt , target question Q0, T=500 ): (a) generated-sequence agreement; (b) top-20 raw-logit agreement; (c) prefill and decode cuda time. Method labels follow the correspondence defined at the start of Section E .
Figure 10: Qwen3-8B on NVIDIA H100 ( BF16 , native , target question Q1, T=500 ): (a) generated-sequence agreement; (b) prefill and decode cuda time. Method labels follow the correspondence defined at the start of Section E .
Figure 11: Qwen3-14B on NVIDIA H100 ( BF16 , prompt , target question Q1, T=500 ): (a) generated-sequence agreement; (b) prefill and decode cuda time. Method labels follow the correspondence defined at the start of Section E .
Figure 12: Llama-3.1-8B-Instruct on NVIDIA H100 ( BF16 , prompt , T=500 ): (a) generated-sequence agreement; (b) prefill and decode cuda time. Method labels follow the correspondence defined at the start of Section E .
Figure 13: Gemma-3-4B-IT on NVIDIA H100 ( BF16 , prompt , T=500 ): (a) generated-sequence agreement; (b) prefill and decode cuda time. Method labels follow the correspondence defined at the start of Section E .