Stable Token-level Scheduling for Serverless LLM Inference

Hardi Ashishbhai Desai; Balaji N K; Sourav Kanti Addya

cs.AIOct 13, 2026

Stable Token-level Scheduling for Serverless LLM Inference

Authors: Hardi Ashishbhai Desai, Balaji N K, Sourav Kanti Addya

Organizations: Cloud & Smart System Services Lab, Department of Computer Science and Engineering, National Institute of Technology Karnataka, Surathkal, India

Abstract

Serverless inference is emerging as an efficient deployment model for Large Language Models (LLMs) due to its ability to elastically scale resources, improve GPU utilization through resource sharing, and reduce infrastructure costs. However, existing approaches primarily optimize time to first token (TTFT) while overlooking inter-token latency (ITL), a key determinant of user experience during response generation. We observe that LLM inference comprises a computation-intensive prefill stage and a bandwidth-bound decode stage, where admitting new requests can disrupt ongoing decoding and increase latency. To address this, we propose Token Window Pacing (TWP), a closed-loop scheduling framework that leverages runtime feedback from ITL, queue depth, and workload characteristics to regulate request admission at the token level instead of the batch level. TWP dynamically adjusts a selfregulating token window and incorporates chunk-based prefill to minimize interference from incoming requests. Experiments on realistic serverless workloads show that TWP delivers stable TTFT, significantly reduces latency spikes, lowers P99 tail latency, minimizes ITL variation, and improves throughput by up to 45% over ServerlessLLM and 58% over conventional schedulers. These results demonstrate that feedback-driven tokenlevel scheduling enables stable, high-throughput serverless LLM inference without modifying the underlying model.

Figures & tables

Fig. 1: LLM inference pipeline showing the two-phase execution model.

Fig. 2: TTFT comparison across different scheduling policies.

Fig. 3: System-level design of the Token Window Pacing (TWP) scheduler. The scheduler regulates token-level admission using feedback from decode execution.

Symbol	Meaning
$H_{t}$	Token window at step $t$
$H_{0}$	Initial token window
$K_{t}$	Total tokens remaining across active requests
$B_{t}$	Active batch size
$\ell_{\min}$	Baseline decode step latency (bandwidth-bound)
$B_{0}$	Saturation batch size

TABLE I: Notation used in TWP .

Scheduler	Policy
FCFS [ 27 ]	Admit in arrival order; no concurrency bound.
Round-Robin [ 28 ]	Route requests to containers in rotation.
Priority [ 29 ]	Admit in shortest-prompt-first order.
DynamicBatch [ 30 ]	Cap batch size when ITL exceeds a static target threshold; no token-level feedback.
ServerlessLLM [ 2 ]	Implemented within our runtime using FCFS-style admission with locality-aware startup scheduling, fast checkpoint loading, and live migration support.

TABLE II: Baseline schedulers and their admission strategies.

Fig. 4: TTFT comparison across different scheduling policies.

Fig. 5: Comparison of TTFT percentiles (P50, P95, and P99) across different scheduling policies.

Fig. 6: Throughput comparison across different scheduling policies.

Fig. 7: ITL distribution across different scheduling policies.

Fig. 8: Coefficient of variation (CV) of inter-token latency across scheduling policies.

Scheduler	Throughput	P99 Latency	TTFT	ITL	CV ITL
	(req/s)	(ms)	(ms)	(ms)
Low Load
FCFS	363.09	2130.93	52.99	44.29	0.0141
DynamicBatch	363.09	2130.93	52.17	43.26	0.0145
Priority	433.87	2134.36	52.20	43.47	0.0142
RoundRobin	334.48	2147.98	52.17	43.30	0.0147

TABLE III: Performance across workloads

LLM Inference Scheduling

Large language model inference scheduling allocates requests, batches, compute, and KV-cache memory to balance latency, throughput, and service-level objectives.

See all 160 papers

Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Serving
Sep 16, 2026Dev Bali, Soujanya Ponnapalli, Yichuan Wang +3LLM Serving LLM Inference Scheduling
University of California, Berkeley
Beyond Prediction: Tail-Aware Scheduling for LLM Inference
Jun 16, 2026Yueying Li, Yuanfan Chen, Jiayang Chen +6LLM Serving Inference-Time Optimization
Cornell University, Computer Science Department, NY, USA · Microsoft Azure System Research, WA, USA · Cornell University, Electrical and Computer Engineering Department, NY, USA +2
Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rank
Sep 25, 2025Yiheng Tao, Yihe Zhang, Matthew Dearing +4LLM Serving Learning to Rank
University of Illinois Chicago, Chicago, IL, USA · Argonne National Laboratory, Lemont, IL, USA

Sep 16, 2026J/K move · Enter open · S save

Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Serving

Dev Bali, Soujanya Ponnapalli, Yichuan Wang, Natacha Crooks+2

University of California, Berkeley

LLM Serving LLM Inference Scheduling

Open paper AskPDF arXiv

LLM serving is typically offered as a shared, multi-tenant service, where high-demand workloads from one client can cause latency SLO violations for others. Existing solutions for performance isolation equalize client throughput in the long run, for example through queueing and batching fairness. However, these approaches do not provide latency isolation guarantees; as a result, well-behaved clients can still experience significant degradation to their token-level latencies. In this paper, we present FairInference, which provides the novel δ-token fairness guarantee: for a well-behaved client, if a token is generated in d time units in isolation, it will be generated within d + δ time units in multi-tenant execution, providing strong latency isolation guarantees for LLM serving. To achieve this, FairInference addresses a key challenge of LLM serving: bounding delays from sharing GPU resources without support for fine-grained scheduling or resource allocation. In FairInference, the scheduler enforces per-token deadlines, while bounding the delays from GPU compute sharing and accounting for the additional delays introduced by the shared KV caching in GPU memory. We show that FairInference effectively bounds token-level latency spikes for well-behaved clients and improves overall throughput compared to state-of-the-art LLM serving systems.

Abstract

LLM Inference Scheduling

Explore similar work

Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Serving

Beyond Prediction: Tail-Aware Scheduling for LLM Inference

Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rank