Reasoning LLMs generate long outputs whose key--value (KV) caches occupy GPU memory for minutes. When memory fills, the server preempts requests, and a preempted request can repeatedly lose and recover its state while producing little new output. We call this behavior KV thrashing. In certain cases, existing systems leave more than a fifth of requests stalled for over a minute. Because preemption cannot be avoided without delaying admission, we ask how to make it productive. We present Stint, a serving system that preserves a preempted request's work, bounds how long it waits, and gives it fresh output after it returns. Stint stores fixed-length KV snapshots in a contiguity-first CPU pool with an exact-extent allocator, and moves them with fused K/V gather/scatter kernels and double buffering. Age-based readmission brings stalled requests back, and protection gives them a turn of fresh decoding before another preemption. Remaining-length estimates from decoding hidden states guide victim selection. On olympiad-math replays with Qwen3-32B on H200 GPUs, no request pauses for more than 50 seconds under Stint and p99 time per output token falls by 50.7% relative to recomputation in the same engine. The replay as a whole finishes 10.7% sooner (12.0% higher throughput), because the requests that baselines starve no longer finish last.
Figures & tables
Figure 1. Historical output-length profiles for Llama-3.1-8B, Qwen3-14B and Gemini 3 Pro, separate from our serving experiments. Numbers label means; error bars show 95% confidence intervals. Horizontal grouped bars on a linear axis compare output token lengths on ShareGPT, LMSys, GSM8K, and AIME. Green, orange, and purple bars represent the three models, with exact mean values printed beside each bar. Capped error bars retain the original confidence intervals, recovered from the source figure's vector coordinates. Three panels connect GPU KV usage, repeated interruption, and output pauses. The first overlays sampled GPU KV usage in blue on the left axis and cumulative preemptions in a dashed orange step line on the right axis. The count reaches 31 during a 772-second replica replay. The second follows three pauses of 6.8, 21.5 and 26.3 seconds in a request on the same replica, separated by 22 and 28 fresh tokens. The last groups requests by the number of times each was preempted: 0, 1, or 2 or more. Their median longest pauses are 0.09, 22.1 and 39.7 seconds, respectively.
Model
Per token
10,000 tokens
DeepSeek-R1 ( Guo et al., 2025 )
68.625 KiB
0.654 GiB
Llama-3.1-405B ( Grattafiori et al., 2024 )
504 KiB
4.807 GiB
Qwen3-32B ( Yang et al., 2025 )
256 KiB
2.441 GiB
Llama-3.1-8B ( Grattafiori et al., 2024 )
128 KiB
1.221 GiB
Table 1. BF16 KV payload per request, before sharding or replication across GPUs. Context includes prompt and output tokens. Values exclude allocation padding and metadata. DeepSeek-R1 uses its compressed MLA cache.
Figure 3. Modeled cost of rebuilding a full context relative to offloading and reloading its saved state. Red favors offloading; blue favors recomputation. Cell labels show the time ratio, with a shared logarithmic color scale centered at equal cost ( R=1 ).
Figure 4. One replica combines prediction-guided victim selection, a contiguity-first CPU KV pool, bulk KV transfer, and age-based protected recovery. Arrows connect prediction, state preservation and return to decoding. Five boxes show the model runner, predictor, scheduler, KV manager and transfer engine. Hidden states flow from the runner through prediction to the scheduler, which sends scheduled requests back. The scheduler requests preemption or restoration from the KV manager. Block maps reach the transfer engine, whose layer-ready events order attention in the model runner.
Figure 5. Age-based protected recovery: readmit the oldest eligible stalled request and protect KV restoration and 64 fresh tokens, ending early on completion. Unfinished requests return to normal scheduling. A horizontal sequence follows one request from waiting to KV restoration, fresh decoding and normal scheduling. The oldest stalled request becomes eligible after five seconds. An orange outline starts at selection, before restoration, and covers both restoration and fresh decoding. Preemption is disabled through 64 new tokens or completion. Completed requests exit; unfinished requests return to normal scheduling.
Figure 6. Allocation in the same 24-block pool: A, B and C each need six blocks; release B, then request eight blocks for D. (a) Uniform eight-block chunks admit D but retain four padding blocks (internal fragmentation). (b) The FIFO-reclaim ring holds B’s released gap behind A (external fragmentation under this policy), so D waits. (c) Exact extents admit D using 6+2 blocks and leave four free. All sizes are illustrative KV-block counts. Three vertically stacked panels show allocation before and after releasing snapshot B and requesting space for D. All use a 24-block pool and six-block snapshots A, B and C. Uniform eight-block chunks admit the eight-block D but reserve two unused blocks in each of A and C. A ring that reclaims in allocation order cannot reuse B while A remains live, leaving only six usable tail blocks, so D waits. Exact extents reuse six blocks from B and two from the tail, admitting D in two extents and leaving four blocks free.
Figure 7. KV reload (one layer; K or V shown): (a) HiCache’s indexed unified virtual addressing (UVA) kernel; (b) Stint’s bulk DMA followed by scatter; (c) double buffering overlaps copying and scattering. Arrows denote readiness and safe reuse, not durations. HiCache’s separate staged-DMA offload path is not shown. Three panels compare transfer methods and explain overlap. HiCache reloads indexed CPU pages directly with a UVA copy kernel. \namereloads a contiguous CPU extent with cudaMemcpyAsync H2D DMA, followed by a fused K/V scatter kernel into paged GPU KV. Two staging pairs alternate so copying B can overlap scattering A. Solid dependency arrows wait for data readiness; a dashed arrow waits until scattering has finished before a buffer is reused.
Figure 8. Long-output fixed replays on Qwen3-32B, H200 DP4/TP2, at 4 requests/s. Higher throughput and lower p99 TPOT are better. Dots are three runs; bars and whiskers show mean and sample SD. AIME and LiveCodeBench compare recompute with Stint; IMO-AnswerBench adds LMCache and HiCache. All systems complete the same prescribed output per workload. Six linear bar panels show throughput and p99 time per output token for AIME, LiveCodeBench and IMO-AnswerBench. \namehas lower p99 TPOT in all three workloads. Throughput is close to recompute on AIME, modestly higher on LiveCodeBench, and higher than all three baselines on IMO-AnswerBench. Every run is plotted.
Metric
Recomp.
LMCache
HiCache
Stint
Throughput (tok/s)
4,100
4,023
3,092
4,592
Median TPOT (ms)
38.2
39.5
49.7
40.9
p99 TPOT (ms)
104.5
107.1
93.8
51.5
Pooled p99 ITL (ms)
47.9
111.7
57.4
78.0
Mean TTFT (ms)
49.4
74.7
147.4
56.7
p99 TTFT (ms)
67.9
535.5
567.9
246.7
Table 2. IMO-AnswerBench serving metrics, mean of three matched runs (Qwen3-32B, H200 DP4/TP2, 4 requests/s). Lower is better except throughput. HiCache does not report preemption events. Host memory is the sampled node/cgroup peak, not process RSS.
Figure 9. Latency tradeoffs in the same runs as Figure 8 ; lower is better. TTFT starts at HTTP dispatch. Completion time starts at scheduled arrival for AIME and LiveCodeBench, and at HTTP dispatch for IMO-AnswerBench; comparisons are within each workload. Dots show three runs, with mean and sample SD. Six bar panels show mean TTFT and p99 completion time for AIME, LiveCodeBench and IMO-AnswerBench. \name's TPOT gains do not imply that every request starts or finishes sooner. Completion tails worsen on AIME and LiveCodeBench; IMO-AnswerBench completion tails improve while TTFT is worse than recompute.
Figure 10. Longest output pause per request in the IMO-AnswerBench replays. Each request contributes one maximum inter-chunk gap, excluding time before its first output. Curves retain the three runs separately. Stint removes the multi-minute stalls while making the typical request’s longest pause larger. Empirical distributions and a tail view show each request's maximum output gap. \nameremoves the very large pauses seen in the baselines, but its median request maximum is larger. The metric includes scheduling and transport effects and is not request completion time.
Recomp.
LMCache
HiCache
Stint
Median (s)
0.24
0.62
2.31
29.9
p99 (s)
788
813
613
47.3
Maximum (s)
808
840
658
49.3
Requests >60 s
23.2%
23.3%
21.2%
0%
Requests >300 s
15.9%
16.2%
15.5%
0%
Max. share of lifetime
72.4%
–
–
8.4%
Table 3. Longest output pause per IMO-AnswerBench request, pooled over three runs (1,197 requests per system). Share of lifetime divides the pause by the request’s dispatch-to-completion time.
Figure 11. Earlier end-to-end fixed-output load sweeps, one run per curve point. Stint curves use Q64; the Llama/AIME star at 8 requests/s uses Q1024 (mean ± population SD, two runs). arXiv and BurstGPT use 2,000-request sweeps. TTFT uses a logarithmic scale. Llama/HiCache on AIME and LCB is unmeasured. LMCache curves predate the resume-path repair. Thirty-two load-curve panels compare the system with maintained-build vLLM recompute, LMCache with an 800-GiB host pool, and SGLang HiCache with a 797-GiB host pool. Columns are AIME1000, LCB880, arXiv2000, and BurstGPT2000. Each model has rows for throughput, mean TTFT on a logarithmic scale, mean TPOT and p99 TPOT. The continuous Llama AIME curve includes the original Q64 result at 8 requests/s; a separate star shows the Q1024 result. Missing Llama HiCache AIME and LCB curves are unmeasured. Gains and regressions are retained.
Allocator
Tok/s
Fallbacks
Padding (%)
Uniform 8K
4603±123
521±12
68.3±1.6
Mixed sizes
4615±35
1093±82
77.8±0.2
Exact extents
4792 ± 61
0 ± 0
0 ± 0
Table 4. AIME1000 allocator comparison at 800 GiB: mean ± population SD across three runs.
Allocator
Tok/s
Fallbacks
Padding (%)
Uniform 8K
3301±43
0±0
48.0±1.4
Mixed sizes
3188±61
433±13
64.9±1.2
Exact extents
3293 ± 29
0 ± 0
0 ± 0
Table 5. LCB800 allocator comparison at 800 GiB: mean ± population SD across three runs.
MemCopy
Prepare+MemCopy
Method
D2H
H2D
D2H
H2D
HiCache cohort (host A)
Stint
3.680
3.231
4.243
3.961
HiCache layer-first
3.274
3.361
8.929
9.075
HiCache page-first
15.793
3.360
21.948
9.086
Local-adapter cohort (host B)
Table 6. Local KV transfer latency (ms), averaged over three repetitions. Each operation moves 160 MiB of K/V (one layer, 2560 scattered GPU pages). The two cohorts run on different hosts, each with its own Stint control; compare within a cohort. Mooncake adapters are local; SparDA uses 16 CTAs. Standard deviations are in Appendix D .
K/V kernels
Staging pairs
D2H
H2D
Separate
1
60.438±0.121
53.065±0.050
Fused
1
48.434±0.136
44.170±0.128
Separate
2
47.792±0.079
43.140±1.091
Fused
2
46.425±0.022
42.503±0.122
Table 7. K/V kernel and buffering ablation: transfer latency (ms), mean ± population SD of ten operations, not serving runs. D2H offloads KV; H2D reloads it.
Figure 12. Output-length predictions on common step-500 survivors: 300 AIME requests (top), and 199 of 200 LiveCodeBench requests (bottom). Stint predicts at step 500; LTR, S3 and PiA use prompts. Kendall’s τb measures ordering; R2 measures numerical fit. Higher is better for both; 1 is perfect. DSMN results are in Appendix E . Eight square scatter plots compare predicted and actual output lengths for AIME and LiveCodeBench across \name, LTR, S3, and PiA. Each panel reports Kendall tau-b for ranking quality and R-squared for numerical fit. Higher is better for both.
Mode
Tok/s
Wait (s)
Offloads
Gap >10 s
Disabled
475.43
45.11
71
22%
Age-only
484.45
5.20
494
0%
Protected(Q=64)
494.10
13.75
286
68%
Table 8. Qwen/LCB recovery controls, one run per mode. Wait is p99 eviction-to-next-fresh-output time. Gap >10 s is the fraction of requests whose longest client chunk gap exceeds 10 s.
Change
Workloads
Runs
Throughput (%)
Other effect
Exact extents vs. uniform 8K
AIME1000, LCB800
3
+4.1 , −0.3
AIME fallbacks 521→0
Fused K/V, two pairs vs. legacy
AIME300, LCB200
2
−1.0 , +0.3
p99 ITL −4% , −8%
Prediction enabled but unused
AIME1000, LCB880
3
−0.6 , −0.3
–
SRJF vs. physical-largest victims
AIME1000, LCB880
3
−7.9 , −7.4
p99 ITL −14% , −15%
Protected recovery (Q64) vs. none
LCB50, 2 pairs
1
+3.9 , +6.8
mean decode batch +5% , +9%
Protected vs. age-only readmission
LCB50
1
+2.0
offloads 494→286
Table 9. Controlled comparisons within Stint. Each row changes one mechanism and holds the others fixed. Throughput changes are relative to the second configuration; paired values follow the workload order. Runs are per arm.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 13. Natural-completion LCB175: serving performance and coding outcomes across three arrival seeds. Circles show individual runs; diamonds and whiskers show means and population standard deviations. The same 175 tasks repeat, but generated lengths differ across arms. Four panels compare recompute, physical-largest \nameand SRJF \nameon output throughput, p99 TPOT, mean arrival-to-completion time and the number of programs that pass their tests.
Arrival seed
Recompute
P0
SRJF
30
103
103
98
31
100
100
107
32
101
105
106
Appendix
Table 10. Programs passing the offline tests, out of 175 tasks per run. All evaluations have a valid outcome. Repeated seeds do not create new independent tasks.
Group
Mean TTFT
Mean TPOT
p99 TPOT
Mean dispatch- to-completion
p99 dispatch- to-completion
Training100
+7.42%
+1.34%
−36.47%
+3.85%
−3.52%
Training-unseen299
+17.37%
−8.45%
−53.56%
−4.10%
−4.16%
Appendix
Table 11. IMO-AnswerBench subgroup changes with Stint relative to recompute. Percentages average three within-block ratios; negative latency changes are better. Both groups share each full 399-request replay.
Figure 14. Three-run confirmation of the selected configurations. Each row retains its own workload and arrival rate. Points are runs; black ticks and whiskers show means and population standard deviations. First-output delay uses a logarithmic axis; the other axes are linear. Three rows compare recompute and selected \namesettings on Qwen/arXiv, Llama/AIME and Qwen/BurstGPT. Columns show throughput, mean arrival-to-first-output time and p99 TPOT.
Figure 15. Cachewise structural replay, three matched blocks per system. Bars show means with sample standard deviation; points retain individual runs. Eligible-to-completion includes waiting after a parent permits a call, while session makespan also includes the dependency chain. Six panels compare four systems on output throughput, mean and p99 TTFT, p99 TPOT, p99 eligible-to-completion time and p99 session makespan. HiCache has lower TTFT but much higher p99 TPOT.
Workload
System
Output tok/s
Mean arrival-to- first-output (s)
p99 TPOT (ms)
Memory guard peak (GiB)
AIME1000
Recompute
7,741.27
4.687
118.26
417.18
FIFO64 recompute
6,493.93
260.629
32.16
416.35
Stint
7,701.16
0.074
76.86
1,446.90
LCB880
Recompute
5,007.50
0.084
225.14
451.33
FIFO64 recompute
4,703.59
225.188
39.99
451.01
Stint
5,189.30
0.087
57.16
1,481.02
Appendix
Table 12. FIFO admission reduces post-admission TPOT but adds waiting before output begins. Values average three runs. Arrival-to-first-output includes the admission queue; the memory column is the sampled node/cgroup guard, not process RSS or CPU KV capacity.
Figure 16. Three repeated runs with continued arrivals followed by a drain. Each column shows one matched block: delivered output, unfinished requests, and the client admission queue. Output rates use 10-second windows. Earlier phases can leave requests active in later phases. Three columns follow arrival seeds 30, 31 and 32. Rows show delivered output per second, unfinished requests and requests queued before dispatch for recompute, FIFO64 recompute and \name.
Figure 17. Longest client-chunk gaps in the six LCB recovery controls. Each curve is one complete run of 50 requests; the x-axis is logarithmic. Protected recovery uses age 5 s and a 64-fresh-token quantum. A lower curve means fewer requests exceed that pause threshold. In these runs, protected recovery reduces the longest pauses while increasing the fraction of requests with moderate pauses. Three panels show queue-tail, physical-largest and learned SRJF victims. Dashed blue curves disable recovery and solid red curves enable protected recovery. In all three panels, the protected curve is higher at a ten-second threshold but reaches zero before thirty seconds.
Prepared
With preparation
Method
D2H
H2D
D2H
H2D
HiCache cohort: H200 host A, CUDA 12.8
Stint
3.680±0.001
3.231±0.001
4.243±0.009
3.961±0.007
HiCache layer-first
3.274±0.002
3.361±0.002
8.929±0.068
9.075±0.086
HiCache page-first
15.793±0.009
3.360±0.001
21.948±0.012
9.086±0.054
Local-adapter cohort: H200 host B, CUDA 12.8
Appendix
Table 13. Local KV transfer latency in milliseconds (mean ± population SD, three repetitions). Each operation moves one layer of 2,560 pages (160 MiB of K/V), with scattered GPU pages and up to eight CPU regions. The cohorts use separate hosts and separately measured Stint controls.
Model
Workload
n
MAE
RMSE
R2
τb
Stint
AIME300
300
3,471
4,604
-0.14
0.34
DSMN
AIME300
300
3,942
5,580
-0.68
0.24
LTR
AIME300
300
3,179
3,993
0.14
0.24
S3
AIME300
300
4,459
5,833
-0.83
0.21
PiA
AIME300
300
3,377
4,114
0.09
0.30
Stint
LCB200
199
4,744
7,286
0.15
0.56
Appendix
Table 14. Output-length prediction on the common step-500 populations. Stint and DSMN predict at step 500; the other methods predict at prefill time. These development populations combine validation and test requests.
Figure 18. Prediction-interval sensitivity with the current H200 transfer path. Points show the three arrival seeds; diamonds and error bars show means and sample standard deviations. Throughput and p99 TPOT change much less than predictor activity. Two panels compare K1, K64 and K128 on output throughput and p99 TPOT. All three run values remain visible around each mean.
Model
L
D
I
A
C
KiB/token
Qwen3-32B
64
5,120
25,600
8,192
1,024
256
Llama-3.1-8B
32
4,096
14,336
4,096
1,024
128
Appendix
Table 15. Dense-model dimensions and BF16 KV payload before TP sharding. The payload counts both K and V across all layers.
Shift
Scored scope
Frozen MAE
Updated MAE
Change
AIME → LCB
494 requests/order
4753.73±0.00
3847.98±66.39
−19.05%
LCB → AIME
494 requests/order
4887.84±0.00
3981.35±12.60
−18.55%
AIME → LCB
Final target phase
7477.74±1251.80
5618.72±1655.96
−24.86%
LCB → AIME
Final target phase
5378.83±475.76
3567.57±54.90
−33.67%
Appendix
Table 16. Causal feature-replay accuracy at decode token 1,000. Entries are means ± population SD over three orders of the same development population. MAE is measured in output tokens; changes compare arm means.
Shift
Mode
Output tok/s
Mean TTFT
Mean TPOT
p99 TPOT
AIME → LCB
Frozen
1712.12±40.97
120.65±1.21
88.80±0.37
197.06±7.02
AIME → LCB
Original
1605.02±63.42
124.55±1.18
86.85±0.24
164.26±3.97
AIME → LCB
Packed
1638.00±19.44
123.39±1.15
86.61±0.62
172.86±14.31
LCB → AIME
Frozen
1679.15±97.17
115.03±1.67
94.15±0.78
234.94±10.84
LCB → AIME
Original
1690.33±9.06
116.97±1.21
92.34±0.05
192.23±16.29
LCB → AIME
Packed
1650.75±47.99
116.31±1.34
94.26±1.64
214.94±13.04
Appendix
Table 17. Live serving with frozen weights, original updates, or packed updates. Values are mean ± population SD across three runs per direction/mode. Latencies are milliseconds; p99 TPOT averages the per-run p99 values. Repetitions reuse one development population.
Model
Mode
TPOT (ms)
Thpt
avg
p50
p99
(tok/s)
QWQ-32B
Recompute
67.28
61.73
110.80
702.6
Swap
64.78
58.14
103.13
712.3
Llama-8B
Recompute
55.93
57.42
142.91
3885.8
Swap
69.61
74.50
144.37
3161.1
Appendix
Table 18. Recompute vs. synchronous swap on A100. QWQ-32B uses AIME (100 requests, rate=2); Llama-3.1-8B uses random (300 requests, input=128, output=1024).
Batch (blocks)
GPU → CPU
CPU → GPU
1
1,068
1,103
4
2,625
2,590
16
3,681
3,623
64
4,112
3,490
256
4,237
3,496
512
4,252
3,496
Appendix
Table 19. KV block swap throughput (blocks/s) on Llama-3.1-8B, A100-80GB. Each block is 16 tokens.
Policy
# Prepts.
TPOT (ms)
Thpt
avg
p50
p99
(tok/s)
LIFO (last)
448
167.8
46.2
2,618.0
269.4
Largest-first
456
105.8
45.5
990.4
240.1
Earliest
446
126.9
45.9
2,024.7
242.5
Random
479
209.7
44.7
1,199.5
242.1
Appendix
Table 20. Victim selection policy comparison (Qwen2.5-14B, A100-40GB, 200 requests). TPOT reported as avg/p50/p99 in ms.
Mode
TPOT (ms)
Thpt
Pre-
Tokens
avg
p50
p99
(tok/s)
empt.
preempt.
Swap
128.0
116.9
249.5
45.9
191
126,509
Recompute
170.5
149.9
680.7
46.6
202
101,311
Appendix
Table 21. Sustained memory pressure (QWQ-32B, A100, 40 random requests, rate=0.5). TPOT in ms.
Workload
Observation
Output tok/s
p99 TPOT (ms)
p99 TTFT (ms)
Preemptions
AIME1000
Original
6,844.34
169.02
282.79
1,732
Repaired follow-up
7,378.17
159.10
483.08
1,801
LCB880
Original
5,159.31
173.51
277.42
1,103
Repaired follow-up
5,247.53
172.57
614.82
1,122
Appendix
Table 22. Historical LMCache observations and their repaired follow-ups. Each cell is one run; old-to-new changes do not isolate the repair’s causal effect. Original observations remain available.
Figure 19. Earlier GSM8K trace: prediction errors and accuracy within 200 tokens as decoding advances. Horizontal lines show the prompt-time baselines. Individual steps need not improve monotonically. Two line plots show mean absolute error decreasing and accuracy within 200 tokens increasing as decoding advances from 100 to 500 tokens. Horizontal lines mark fixed ahead-of-time baselines.