Detent: Batch-Invariant LLM Serving and RL Rollouts at Near-Stock Speed
Organizations: Independent Researcher, Boston University · Google · RadixArk · Princeton University and Georgia Institute of Technology
Abstract
Reproducible evaluation, verifiable inference and on-policy reinforcement learning (RL) need an LLM engine to give a request the same output bits whichever requests share its batch. Batch-invariant kernels exist, but engines still select each kernel and its parameters from the batch. Two kernels that compute the same sum can round it differently. The engines' deterministic modes remove the choice by keeping one conservative kernel per operator and pay for it in latency. Many fast kernels are batch-invariant by themselves, so it is enough to fix the few batch-dependent values that kernel selection reads. Detent expresses kernel selection as a program that the engine executes. It finds the fewest values in this program that must be fixed and tunes them for the workload before deployment. On multi-GPU deployments of Llama-3.1-70B and Qwen3-30B-A3B in vLLM and SGLang, Detent is batch-invariant. Tuned per load, it serves at 1.04-1.17x the latency per output token of Stock (each engine with its standard settings), against 1.67-10.3x for the deterministic modes. In SGLang it matches the GSM8K accuracy of SGLang's deterministic mode. Detent covers multiple deployment combinations, including multiple engines, GPU generations, tensor-parallel degrees, and dense and MoE models. Seeded RL rollouts of models up to Llama-3.1-70B replay bit for bit under other batches and cache splits. A trainer on Detent's kernels and values matches every rollout logprob bit for bit at 1.03x Stock's rollout time per token. BF16 GRPO then stays stable where Stock's run collapses.
Figures & tables
| Arm | BI | Kernel selection | Matrix mult. | Attention split | All-reduce | MoE row key |
|---|---|---|---|---|---|---|
| Stock | stock heuristics | cuBLAS | by the batch | custom, by the batch’s bytes | nearest the batch | |
| SGLang-Det vLLM-Det | one kernel per operator | DeepGEMM (Triton in vLLM) | none | NCCL tree | fixed | |
| Detent | SelIR, pinned | DeepGEMM (tuned pad) | fixed stride or none | ours, pinned | pinned |
| Point | Stock | SGLang-Det | Detent | [95%] | |
| Qwen3-30B-A3B | |||||
| G1 | 93.71 | 93.63 | 93.48 | 0.15 [ 0.92, +0.66] | 0.85 |
| G2, G3 | 93.93, 94.01 | 93.63 | 94.16 | +0.53 [ 0.15, +1.02] | 0.14 |
| Llama-3.1-70B | |||||
| G1 | 94.69 | 94.69 | 94.77 | +0.08 [ 0.39, +0.50] | 1.00 |
| G2, G3 | 94.92, 94.62 | 94.69 | 94.84 | +0.15 [ 0.36, +0.57] | 0.75 |
| Deterministic | 12 QPS | 14 QPS | 16 QPS | 18 QPS |
|---|---|---|---|---|
| 2% | 1.155 | 1.161 | 1.185 | 1.198 |
| 5% | 1.187 | 1.191 | 1.212 | 1.236 |
| 10% | 1.212 | 1.221 | 1.242 | 1.289 |
| 20% | 1.265 | 1.302 | 1.379 | 1.530 |
| 50% | 2.023 | 2.408 | 2.842 | 3.249 |
| 100% | 3.946 | 4.322 | 4.756 | 5.147 |
| Deterministic | ShareGPT | arXiv |
|---|---|---|
| 2% | 1.098 | 0.975 |
| 5% | 1.224 | 0.982 |
| 10% | 1.357 | 0.987 |
| 20% | 1.471 | 0.996 |
| 50% | 1.727 | 1.063 |
| 100% | 2.147 | 1.217 |
| Model, H100s | Detent | SGLang-Det | |||
|---|---|---|---|---|---|
| BI | Replay, rescoring | / Stock | / SGLang-Det | BI, mixed | |
| Llama-3.1-8B, 1 | yes | 512/512 | 0.917 | 1.15 | no |
| Qwen3-30B-A3B, 4 | yes | 512/512 | 0.837 | 1.27 | no |
| Llama-3.1-70B, 4 | yes | 512/512 | 0.872 | 1.35 | no |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Operator | Stock | SGLang-Det | vLLM-Det |
|---|---|---|---|
| FA3 | split count from a batch-wide heuristic | one split | one split; ahead-of-time scheduler and cascade attention off |
| All-reduce (TP 1) | custom, symmetric-memory or NCCL kernels by message size | NCCL tree with fixed channels | NCCL tree, one channel; custom and symmetric-memory all-reduce off |
| Fused MoE | tuned table, nearest token count | one hard-coded configuration | one hard-coded configuration; kernels without batch-invariant support refused |
| GEMM | cuBLAS, kernel picked by shape | batch-invariant mm , addmm on DeepGEMM | Triton persistent matmul for linear layers; cuBLAS without split-K |
| Other row-local operators | native kernels | batch-invariant log_softmax , mean , rms_norm | batch-invariant log_softmax , softmax , mean , rms_norm |
| Sampling | default backend | PyTorch backend | unchanged |
| Measure | Qwen3-30B-A3B | Llama-3.1-70B |
|---|---|---|
| Kernel swap: Detent ’s kernel over Stock’s | ||
| GEMM: DeepGEMM / cuBLAS | 0.99 (0.93–1.05) | 1.02 (0.99–1.15) |
| with the tuple’s padding | 1.00 (0.93–1.06) | 1.01 (0.99–1.06) |
| All-reduce: ours / Stock’s pick | 1.10 (0.97–1.29) | 1.16 (1.04–1.37) |
| Added to Stock’s GPU time | +0.6% ( 0.3 to +1.6) | +2.8% (+0.1 to +10.3) |
| with the tuple’s padding | +0.6% ( 0.3 to +1.3) | +2.1% (+0.5 to +5.5) |
LLM Serving
Large language model serving studies how to deliver model responses with low latency, high throughput, and efficient use of compute and memory.
See all 249 papers