cs.DCNov 15, 2026

Making Preemption Productive in Long-Output LLM Serving

Authors: Ziji Shi, Xinyi Wan, Chaoyi Ruan, Guangxing Huang, Min Lin, Jialin Li

Organizations: National University of Singapore Singapore · Sea AI Lab Singapore

Abstract

Reasoning LLMs generate long outputs whose key--value (KV) caches occupy GPU memory for minutes. When memory fills, the server preempts requests, and a preempted request can repeatedly lose and recover its state while producing little new output. We call this behavior KV thrashing. In certain cases, existing systems leave more than a fifth of requests stalled for over a minute. Because preemption cannot be avoided without delaying admission, we ask how to make it productive. We present Stint, a serving system that preserves a preempted request's work, bounds how long it waits, and gives it fresh output after it returns. Stint stores fixed-length KV snapshots in a contiguity-first CPU pool with an exact-extent allocator, and moves them with fused K/V gather/scatter kernels and double buffering. Age-based readmission brings stalled requests back, and protection gives them a turn of fresh decoding before another preemption. Remaining-length estimates from decoding hidden states guide victim selection. On olympiad-math replays with Qwen3-32B on H200 GPUs, no request pauses for more than 50 seconds under Stint and p99 time per output token falls by 50.7% relative to recomputation in the same engine. The replay as a whole finishes 10.7% sooner (12.0% higher throughput), because the requests that baselines starve no longer finish last.

Figures & tables

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. InferScale: GPU-Native KV Injection for Personalized LLM Serving

    Jul 29, 2026Peter Li, Prashant PandeyKV CachingLLM Serving

  2. Stream2LLM: Overlap Context Streaming and Prefill for Reduced Time-to-First-Token (TTFT)

    Mar 29, 2026Rajveer Bachkaniwala, Chengqi Luo, Richard So +2LLM InferenceLLM Inference Serving