Lightweight Scheduling Policies for Continuously Batched LLM Serving Across GPU Architectures

Ashman Mehra; Akshan Agarwal; Aryeh Carmi

cs.DCOct 11, 2026

Lightweight Scheduling Policies for Continuously Batched LLM Serving Across GPU Architectures

Authors: Ashman Mehra, Akshan Agarwal, Aryeh Carmi

Organizations: Carnegie Mellon University, School of Computer Science

Abstract

Large language model serving is increasingly limited not only by model size and GPU speed, but also by scheduling during autoregressive decoding, where continuous batching interleaves requests at token granularity as sequences arrive and finish. We study whether lightweight scheduling policies can improve latency-throughput tradeoffs without changing model architectures, kernels, or deploying learned schedulers. We implement a Python/PyTorch continuous-batching testbench with real Hugging Face execution and evaluate eight policies across 1,152 GPU-model-scheduler-workload runs using ShareGPT-derived traffic, six Qwen, Gemma, and Mistral models, and Modal L40S, A100, and H100 GPUs. Scheduler choice changes throughput, TTFT, completion latency, and tail behavior under fixed model, GPU, workload, batch size, and prefill budget. Aging shortest-prompt-first is the strongest practical baseline in many L40S and mixed-workload settings, while oracle remaining-length scheduling shows that job-size knowledge alone is insufficient because batching and hardware effects also shape performance. The custom dataset has been uploaded to Hugging Face.

Figures & tables

Figure 1: System overview of the replay-based continuous-batching LLM serving testbench.

POLICY	PRIORITY SIGNAL
First-in, first-out (FIFO)	Arrival order.
Service-tier priority (Tier)	Service tier, then arrival order.
Shortest prompt first (SPF)	Visible prompt length.
Longest prompt first (LPF)	Negative prompt length.
Shortest remaining tokens first (SRTF)	GPT-3 output length.
Aging shortest-prompt-first (Aging)	Prompt waiting age.

Table 1: Scheduler policies and priority signals. Policy abbreviations used in the results appear in parentheses.

Figure 2: Latency/throughput frontier across GPU, model, and scheduler runs, faceted by GPU with independent axis ranges. Each point is one GPU-model-scheduler average across workload categories; color encodes scheduler and marker shape encodes model. Lower p95 latency and higher throughput are preferred. SPF denotes shortest prompt first, ESTF estimated shortest tokens first, SRTF shortest remaining tokens first, LPF longest prompt first, and AF-LCB adaptive fair LCB.

Figure 3: TTFT vs. completion latency. Left: each point is a scheduler-workload-GPU average across models; circles, squares, and triangles denote L40S, A100, and H100. Right: labeled scheduler means across all runs. Lower-left points give better interactivity and completion time.

Figure 4: Tail-latency comparison across GPUs, with p95 and p99 normalized to FIFO. Values below 1.0 improve on FIFO.

Figure 5: Batch efficiency breakdown across schedulers and GPUs: average batch size, decode-step time, and steps per request.

LLM Inference Scheduling

Large language model inference scheduling allocates requests, batches, compute, and KV-cache memory to balance latency, throughput, and service-level objectives.

See all 162 papers

Threshold-Based Exclusive Batching for LLM Inference
May 30, 2026Weifang Zhang, Yuzhou Nie, Bowen Pang +2GPU Acceleration LLM Inference
The Hong Kong Polytechnic University · UC Santa Barbara · Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences +1
LLM Serving Optimization with Variable Prefill and Decode Lengths
Aug 8, 2025Meixuan Wang, Yinyu Ye, Zijie ZhouLLM Serving LLM Inference Scheduling
Department of Computer Science and Technology, Tsinghua University · Department of Management Science and Engineering, Stanford University & Department of Industrial Engineering and Decision Analytics, HKUST · Department of Industrial Engineering and Decision Analytics, HKUST
Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
May 19, 2026Mert Yildiz, Pietro Spadaccino, Alexey Rolich +2LLM Inference Efficiency LLM Serving
Sapienza University of Rome, Rome, Italy

May 30, 2026J/K move · Enter open · S save

Threshold-Based Exclusive Batching for LLM Inference

Weifang Zhang, Yuzhou Nie, Bowen Pang, Guangrui Ma+1

The Hong Kong Polytechnic University · UC Santa Barbara · Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences+1

GPU Acceleration LLM Inference LLM Inference Scheduling LLM Inference Acceleration

Open paper AskPDF arXiv

Mixed batching (MB)--interleaving prefill and decode in a single batch--has become the standard scheduling strategy for large language model (LLM) inference due to its efficiency in maximizing compute and memory utilization. However, through controlled experiments, we find that prefill-decode interference inflates MB's per-step marginal cost above that of pure decode. On the high-bandwidth H200 (4.8 TB/s), this occurs only when decode tokens exceed 80% of the batch; however, on the bandwidth-constrained RTX PRO 6000 (1.792 TB/s), this threshold plummets to just 20%. Consequently, the optimal choice between MB and exclusive batching (EB) fundamentally depends on GPU memory bandwidth, model size, and workload composition. We derive a closed-form condition for this EB-MB performance crossover, along with asymptotically optimal phase-switching thresholds and memory-safe batch sizing for EB. Optimized EB achieves up to 41.9% higher throughput on bandwidth-constrained GPUs, while MB retains its advantage on high-bandwidth hardware with larger models. Our hybrid scheduler EB+ applies this condition online to dynamically switch between EB and MB without manual intervention. Under non-stationary traffic with distribution or concurrency shifts, EB+ attains the highest or near-highest throughput in every setting, outperforming MB by up to 36.4%.

Abstract

LLM Inference Scheduling

Explore similar work

Threshold-Based Exclusive Batching for LLM Inference

LLM Serving Optimization with Variable Prefill and Decode Lengths

Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption