cs.AIOct 13, 2026

Stable Token-level Scheduling for Serverless LLM Inference

Authors: Hardi Ashishbhai Desai, Balaji N K, Sourav Kanti Addya

Organizations: Cloud & Smart System Services Lab, Department of Computer Science and Engineering, National Institute of Technology Karnataka, Surathkal, India

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Serverless inference is emerging as an efficient deployment model for Large Language Models (LLMs) due to its ability to elastically scale resources, improve GPU utilization through resource sharing, and reduce infrastructure costs. However, existing approaches primarily optimize time to first token (TTFT) while overlooking inter-token latency (ITL), a key determinant of user experience during response generation. We observe that LLM inference comprises a computation-intensive prefill stage and a bandwidth-bound decode stage, where admitting new requests can disrupt ongoing decoding and increase latency. To address this, we propose Token Window Pacing (TWP), a closed-loop scheduling framework that leverages runtime feedback from ITL, queue depth, and workload characteristics to regulate request admission at the token level instead of the batch level. TWP dynamically adjusts a selfregulating token window and incorporates chunk-based prefill to minimize interference from incoming requests. Experiments on realistic serverless workloads show that TWP delivers stable TTFT, significantly reduces latency spikes, lowers P99 tail latency, minimizes ITL variation, and improves throughput by up to 45% over ServerlessLLM and 58% over conventional schedulers. These results demonstrate that feedback-driven tokenlevel scheduling enables stable, high-throughput serverless LLM inference without modifying the underlying model.

Figures & tables

Fig. 1: LLM inference pipeline showing the two-phase execution model.
Fig. 2: TTFT comparison across different scheduling policies.
Fig. 3: System-level design of the Token Window Pacing (TWP) scheduler. The scheduler regulates token-level admission using feedback from decode execution.
SymbolMeaning
HtH_{t}Token window at step tt
H0H_{0}Initial token window
KtK_{t}Total tokens remaining across active requests
BtB_{t}Active batch size
ℓmin⁡\ell_{\min}Baseline decode step latency (bandwidth-bound)
B0B_{0}Saturation batch size
TABLE I: Notation used in TWP .
SchedulerPolicy
FCFS [ 27 ]Admit in arrival order; no concurrency bound.
Round-Robin [ 28 ]Route requests to containers in rotation.
Priority [ 29 ]Admit in shortest-prompt-first order.
DynamicBatch [ 30 ]Cap batch size when ITL exceeds a static target threshold; no token-level feedback.
ServerlessLLM [ 2 ]Implemented within our runtime using FCFS-style admission with locality-aware startup scheduling, fast checkpoint loading, and live migration support.
TABLE II: Baseline schedulers and their admission strategies.
Fig. 4: TTFT comparison across different scheduling policies.
Fig. 5: Comparison of TTFT percentiles (P50, P95, and P99) across different scheduling policies.
Fig. 6: Throughput comparison across different scheduling policies.
Fig. 7: ITL distribution across different scheduling policies.
Fig. 8: Coefficient of variation (CV) of inter-token latency across scheduling policies.
SchedulerThroughputP99 LatencyTTFTITLCV ITL
 (req/s)(ms)(ms)(ms) 
Low Load
FCFS363.092130.9352.9944.290.0141
DynamicBatch363.092130.9352.1743.260.0145
Priority433.872134.3652.2043.470.0142
RoundRobin334.482147.9852.1743.300.0147
TABLE III: Performance across workloads
More in this topic

LLM Inference Scheduling

Large language model inference scheduling allocates requests, batches, compute, and KV-cache memory to balance latency, throughput, and service-level objectives.

See all 160 papers

Explore similar work

CardsList
Sep 16, 2026

Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Serving

LLM serving is typically offered as a shared, multi-tenant service, where high-demand workloads from one client can cause latency SLO violations for others. Existing solutions for performance isolation equalize client throughput in the long run, for example through queueing and batching fairness. However, these approaches do not provide latency isolation guarantees; as a result, well-behaved clients can still experience significant degradation to their token-level latencies. In this paper, we present FairInference, which provides the novel δ-token fairness guarantee: for a well-behaved client, if a token is generated in d time units in isolation, it will be generated within d + δ time units in multi-tenant execution, providing strong latency isolation guarantees for LLM serving. To achieve this, FairInference addresses a key challenge of LLM serving: bounding delays from sharing GPU resources without support for fine-grained scheduling or resource allocation. In FairInference, the scheduler enforces per-token deadlines, while bounding the delays from GPU compute sharing and accounting for the additional delays introduced by the shared KV caching in GPU memory. We show that FairInference effectively bounds token-level latency spikes for well-behaved clients and improves overall throughput compared to state-of-the-art LLM serving systems.

Dev Bali, Soujanya Ponnapalli, Yichuan Wang +3

University of California, Berkeley

LLM ServingLLM Inference Scheduling
AskPDF
Jun 16, 2026

Beyond Prediction: Tail-Aware Scheduling for LLM Inference

LLM serving exhibits extreme length variability, making size-based scheduling difficult in practice. Recent LLM schedulers approximate SJF/SRPT using predicted decode lengths or ranks and primarily report mean-centric metrics such as TTFT and TBT. We show that these prediction-driven policies can be fragile under distribution shifts, bursty arrivals, and GPU memory pressure, while offering limited control over the tail latency (P90-P99) that dominates user experience, even with perfect decode-length knowledge. We introduce a distribution-aware, prediction-free scheduling framework that replaces explicit length prediction with soft priority boosting driven by lightweight statistical signals. Our design co-optimizes scheduling and cache-aware preemption to account for memory-coupled decode dynamics across workload mixes. Evaluated on production and open-source traces, our method reduces P99 TTLT by up to 35-50% relative to SRPT with perfect length knowledge and reduces TTFT by 34-47% across workloads, including reasoning-heavy and chat-heavy tasks. These results demonstrate a robust alternative for optimizing tail latency in online LLM serving.

Yueying Li, Yuanfan Chen, Jiayang Chen +6

Cornell University, Computer Science Department, NY, USA · Microsoft Azure System Research, WA, USA · Cornell University, Electrical and Computer Engineering Department, NY, USA +2

LLM ServingInference-Time OptimizationLLM Inference SchedulingLLM Request Scheduling
AskPDF
Sep 25, 2025

Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rank

Efficient scheduling of large language model (LLM) inference tasks is critical for achieving low latency and high throughput, a challenge that is becoming increasingly acute with the rise of reasoning-capable LLMs whose generation lengths are highly variable. Traditional strategies like First Come, First-Serve (FCFS) often suffer from Head-of-Line (HOL) blocking, where long-running tasks delay shorter ones queued behind them. In this paper, we introduce PARS, a prompt-aware LLM task scheduler that mitigates HOL blocking by approximating shortest-job-first (SJF) scheduling through pairwise ranking with a margin ranking loss. PARS effectively predicts response-length-based task ordering directly from prompts, thereby optimizing scheduling decisions with minimal overhead. In addition, it integrates seamlessly with vLLM, a state-of-the-art LLM serving system, for the research community. Extensive experiments across multiple LLM models and real-world inference use cases, including chat, math, and code generation, demonstrate that PARS significantly reduces latency by up to 15.7x compared to the vLLM default scheduler. Cross-model evaluations demonstrate that our design generalizes effectively, allowing effective scheduling across diverse LLMs without requiring model-specific retraining.

Yiheng Tao, Yihe Zhang, Matthew Dearing +4

University of Illinois Chicago, Chicago, IL, USA · Argonne National Laboratory, Lemont, IL, USA

LLM ServingLearning to RankLLM Inference SchedulingLLM Inference ServingLLM Request Scheduling
AskPDF