Reproducible evaluation, verifiable inference and on-policy reinforcement learning (RL) need an LLM engine to give a request the same output bits whichever requests share its batch. Batch-invariant kernels exist, but engines still select each kernel and its parameters from the batch. Two kernels that compute the same sum can round it differently. The engines' deterministic modes remove the choice by keeping one conservative kernel per operator and pay for it in latency. Many fast kernels are batch-invariant by themselves, so it is enough to fix the few batch-dependent values that kernel selection reads. Detent expresses kernel selection as a program that the engine executes. It finds the fewest values in this program that must be fixed and tunes them for the workload before deployment. On multi-GPU deployments of Llama-3.1-70B and Qwen3-30B-A3B in vLLM and SGLang, Detent is batch-invariant. Tuned per load, it serves at 1.04-1.17x the latency per output token of Stock (each engine with its standard settings), against 1.67-10.3x for the deterministic modes. In SGLang it matches the GSM8K accuracy of SGLang's deterministic mode. Detent covers multiple deployment combinations, including multiple engines, GPU generations, tensor-parallel degrees, and dense and MoE models. Seeded RL rollouts of models up to Llama-3.1-70B replay bit for bit under other batches and cache splits. A trainer on Detent's kernels and values matches every rollout logprob bit for bit at 1.03x Stock's rollout time per token. BF16 GRPO then stays stable where Stock's run collapses.
Figures & tables
Figure 1: ShareGPT replays multi-turn chats. Their prompts are short and mostly hit the prefix cache, so most of the work is decode. arXiv sends 8k-token documents with Poisson arrivals and long prefills. Both run at loads up to Stock’s peak. Closed loop sends the arXiv documents one or eight at a time, without queueing (§ 8.1 ). (a) Qwen3-30B-A3B; (b) Llama-3.1-70B. Bars: geometric mean over a workload’s loads, with Detent tuned per load (§ 6 ).
Figure 2: The batch regroups a request’s additions, so its bits change. (a) Where prefill ends. (b) Kernel selection sets the split count; each split sums its KV blocks, and a combine adds the sums in order. (c) Detent fixes selection; the user supplies kernels.
Figure 3: Kernel selection reads the batch, and only measurement shows which kernels agree. (a) Where four choices switch (log axes; H100; tile on Llama-3.2-1B, MoE row on Qwen3-30B-A3B, splits and all-reduce on Llama-3.1-70B under CUDA graphs). (b) Agreement is measured. (c) Too many FA3 configurations on H100 to measure (log scale). (d) Fixing choices is slow.
Figure 4: Detent isolates kernel selection from the batch.
Figure 5: The solver separates kernel selection from the batch with a few pins. Left, the whole selection of FA3 on Llama-3.1-70B; right, the MoE’s on Qwen3-30B-A3B. In FA3, red inputs are Runtime or computed at launch, purple ones other Unpinnable inputs, each value is tinted by the inputs it derives from, and gray values derive only from deployment facts and a knob.
Figure 6: The tuner searches one kernel family at a time. A case on Qwen3-30B-A3B: candidates multiply a family’s pin and knob values, VRA keeps the legal ones, and the tuner measured 76 of the 3,020 legal assignments.
Arm
BI
Kernel selection
Matrix mult.
Attention split
All-reduce
MoE row key
Stock
×
stock heuristics
cuBLAS
by the batch
custom, by the batch’s bytes
nearest the batch
SGLang-Det vLLM-Det
✓
one kernel per operator
DeepGEMM (Triton in vLLM)
none
NCCL tree
fixed
Detent
✓
SelIR, pinned
DeepGEMM (tuned pad)
fixed stride or none
ours, pinned
pinned
Table 1: Arms (§ 8.1 ). BI: batch-invariant. Stock: each engine as shipped, deterministic mode off. SGLang-Det, vLLM-Det: the engines’ deterministic modes (Appendix A ). MoE: Qwen3-30B-A3B only. No arm uses FlashInfer’s all-reduce fusion; Appendix C ablates it and the matrix-multiplication and all-reduce kernels.
Figure 7: Latency per output token over Stock’s in SGLang, median of three runs. (a, c) Qwen3-30B-A3B, (b, d) Llama-3.1-70B. A1–A4 (W-A), B1–B3 (W-B): loads up to Stock’s peak. C1, C2 (W-B-c): one and eight in flight. (a, b) p50, SGLang-Det also p99, each workload on its own scale. (c, d) Tuned Detent at p50, p90 and p99: 1.01–1.22 × at p50, 1.04–1.34 × at p90 and p99.
Figure 8: Rows: the workload tuples of W-A, W-B and W-B-c. Columns: the loads. Cells: the row’s p50 over the p50 of the column’s tuned tuple. (a) Qwen3-30B-A3B, (b) Llama-3.1-70B. Outlined: a tuple at its own workload’s loads. On another workload’s loads a tuple can take up to 4.17 × (Qwen3-30B-A3B) and 1.22 × (Llama-3.1-70B) as long.
Figure 9: Latency per output token over Stock’s at the loads vLLM serves, p50 and p99 (rows). SGLang (left), vLLM (right). (a, b) Qwen3-30B-A3B, (c, d) Llama-3.1-70B.
Figure 10: Detent keeps GSM8K accuracy. (a) Accuracy with 95% Wilson intervals. Right: Detent minus SGLang-Det with its 95% interval. (b) Answers that change across G1–G3 ( Detent : G2 vs. G3, as its G1 tuple splits the KV).
Point
Stock
SGLang-Det
Detent
Δ [95%]
p
Qwen3-30B-A3B
G1
93.71
93.63
93.48
− 0.15 [ − 0.92, +0.66]
0.85
G2, G3
93.93, 94.01
93.63
94.16
+0.53 [ − 0.15, +1.02]
0.14
Llama-3.1-70B
G1
94.69
94.69
94.77
+0.08 [ − 0.39, +0.50]
1.00
G2, G3
94.92, 94.62
94.69
94.84
+0.15 [ − 0.36, +0.57]
0.75
Table 2: GSM8K accuracy (%) on the 1,319 test problems, four comparisons. At G2 and G3, SGLang-Det’s and Detent ’s outputs are identical, so the two load points form one comparison and only Stock has two accuracies (G2, G3). Δ is Detent minus SGLang-Det in percentage points with its 95% interval, and p the p -value of the exact McNemar test. Wilson 95% intervals on the accuracies are about ± 1.3 percentage points.
Figure 11: Detent makes every request deterministic at 3.5–5.1% above Stock’s median latency. LLM-42 costs more even when only 2% of requests are deterministic. Llama-3.1-8B on one H100, ShareGPT online, median of three servers per point. (a) Online, median end-to-end latency against request rate. LLM-42 at 100% (4.6–6.9 s) is off this scale. (b) Online at 18 requests per second, LLM-42 as its deterministic share grows. (c, d) Offline output throughput on ShareGPT and arXiv, one run per configuration. LLM-42 runs on its port to our SGLang base. Detent ’s tuner judged a tie on this workload and committed the default tuple.
Deterministic
12 QPS
14 QPS
16 QPS
18 QPS
2%
1.155
1.161
1.185
1.198
5%
1.187
1.191
1.212
1.236
10%
1.212
1.221
1.242
1.289
20%
1.265
1.302
1.379
1.530
50%
2.023
2.408
2.842
3.249
100%
3.946
4.322
4.756
5.147
Table 3: Online: ported LLM-42’s median end-to-end latency over Detent ’s on Llama-3.1-8B, at four request rates and six deterministic shares.
Deterministic
ShareGPT
arXiv
2%
1.098
0.975
5%
1.224
0.982
10%
1.357
0.987
20%
1.471
0.996
50%
1.727
1.063
100%
2.147
1.217
Table 4: Offline: Detent ’s output throughput over ported LLM-42’s on Llama-3.1-8B, at six deterministic shares.
Figure 12: Detent ’s trainer matches the rollout’s bits at near-stock cost. Its GRPO run does not collapse. (a) Bit-identical tokens and (b) time per token over Stock’s, TBIK’s protocol on Llama-3.1-8B. Sanity test on DeepSeek-R1-Distill-Qwen-1.5B, (c) training reward (20-step mean), (d) AIME 2024.
Figure 13: On eight more deployments and a B200, solved from specs alone, Detent is batch-invariant. Bars: closed-loop median at W-A and W-B-c. Pins: attention + all-reduce + MoE. Stock fails: battery cells where Stock’s bits change.
Model, H100s
Detent
SGLang-Det
BI
Replay, rescoring
/ Stock
/ SGLang-Det
BI, mixed
Llama-3.1-8B, 1
yes
512/512
0.917 ×
1.15 ×
no
Qwen3-30B-A3B, 4
yes
512/512
0.837 ×
1.27 ×
no
Llama-3.1-70B, 4
yes
512/512
0.872 ×
1.35 ×
no
Table 5: Seeded rollouts. 512/512: bit-identical replays, rescorings. Ratios: Detent ’s throughput over Stock’s and SGLang-Det’s.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Operator
Stock
SGLang-Det
vLLM-Det
FA3
split count from a batch-wide heuristic
one split
one split; ahead-of-time scheduler and cascade attention off
All-reduce (TP > 1)
custom, symmetric-memory or NCCL kernels by message size
NCCL tree with fixed channels
NCCL tree, one channel; custom and symmetric-memory all-reduce off
Fused MoE
tuned table, nearest token count
one hard-coded configuration
one hard-coded configuration; kernels without batch-invariant support refused
GEMM
cuBLAS, kernel picked by shape
batch-invariant mm , addmm on DeepGEMM
Triton persistent matmul for linear layers; cuBLAS without split-K
Other row-local operators
native kernels
batch-invariant log_softmax , mean , rms_norm
batch-invariant log_softmax , softmax , mean , rms_norm
Sampling
default backend
PyTorch backend
unchanged
Appendix
Table 6: What the engines’ deterministic modes fix.
Figure 14: What Detent ’s GEMM and all-reduce kernels add to Stock’s GPU time per output token at each workload, split by family. Left bar: the GEMM read unpadded. Right bar: with the padding that workload’s tuple fixes. The padding leaves the all-reduce untouched. A1 and A4 in (a) and A1 and C1 in (b) are served unpadded, so their bars are identical. (a) Qwen3-30B-A3B, +0.6% and +0.6% on average; (b) Llama-3.1-70B, +2.8% and +2.1%. Diamonds: the sums, with twice the measurement noise of the unpadded reading.
Measure
Qwen3-30B-A3B
Llama-3.1-70B
Kernel swap: Detent ’s kernel over Stock’s
GEMM: DeepGEMM / cuBLAS
0.99 (0.93–1.05)
1.02 (0.99–1.15)
with the tuple’s padding
1.00 (0.93–1.06)
1.01 (0.99–1.06)
All-reduce: ours / Stock’s pick
1.10 (0.97–1.29)
1.16 (1.04–1.37)
Added to Stock’s GPU time
+0.6% ( − 0.3 to +1.6)
+2.8% (+0.1 to +10.3)
with the tuple’s padding
+0.6% ( − 0.3 to +1.3)
+2.1% (+0.5 to +5.5)
Appendix
Table 7: Ablation of Stock’s kernels. Top: Detent ’s kernels over Stock’s in GPU time (geometric mean and range over the workloads), and the GPU time they add to Stock’s GPU time per output token (mean and range). Middle: the all-reduce row broken down by the algorithm the tuple fixes and by the algorithm Stock picks for each message (in parentheses: share of Detent ’s extra all-reduce time). Bottom: Detent ’s decode step over Stock’s when Stock runs FlashInfer’s all-reduce fusion.