The growing demand for large language models (LLMs) requires serving systems to handle many concurrent requests with diverse service level objectives (SLOs). This exacerbates head-of-line (HoL) blocking during the compute-intensive prefill phase, where long-running requests monopolize resources and delay higher-priority ones, leading to widespread time-to-first-token (TTFT) SLO violations. While chunked prefill enables interruptibility, it introduces an inherent trade-off between responsiveness and throughput: reducing chunk size improves response latency but degrades computational efficiency, whereas increasing chunk size maximizes throughput but exacerbates blocking. This necessitates an adaptive preemption mechanism. However, dynamically balancing execution granularity against scheduling overheads remains a key challenge. In this paper, we propose FlowPrefill, a TTFT-goodput-optimized serving system that resolves this conflict by decoupling preemption granularity from scheduling frequency. To achieve adaptive prefill scheduling, FlowPrefill introduces two key innovations: 1) Operator-Level Preemption, which leverages operator boundaries to enable fine-grained execution interruption without the efficiency loss associated with fixed small chunking; and 2) Event-Driven Scheduling, which triggers scheduling decisions only upon request arrival or completion events, thereby supporting efficient preemption responsiveness while minimizing control-plane overhead. Evaluation on real-world production traces shows that FlowPrefill improves maximum goodput by up to 5.6× compared to state-of-the-art systems while satisfying heterogeneous SLOs.
Figures & tables
Figure 1. Request length distributions and arrival patterns for different task types in QwenTrace ( Wang et al., 2025 ) . We explore the possibility of scheduling requests with diverse SLO guarantees.
Figure 2. Using different prefill granularities to meet the SLO of time-to-first-token.
Figure 3. Throughput and latency of chunked prefill for a 32K-token input under different chunk sizes when serving an Llama3-8B. Alternating colors within each bar demarcate individual chunks.
Figure 4. Throughput and normalized TTFT during the prefill phase for different input lengths when serving an Llama3-8B.
Figure 5. Overview of FlowPrefill.
Figure 6. Execution logic is hierarchical, and preemption checks can be placed at different execution boundaries. Preemption checks at the chunk or layer level result in coarser preemption granularity, whereas FlowPrefill places them at core operator boundaries for the finest-grained preemption. (Blue circles denote preemption checks.)
Figure 7. Cooperative preemption process.
Figure 8. Example of serving two requests with different priorities in FlowPrefill.
Model
Text
Image
Search
File
Llama3-8B
0.25s
0.5s
4.0s
6.0s
Qwen2.5-14B
0.4s
0.8s
6.5s
9.0s
Llama3-70B
1.0s
2.0s
15.0s
18.0s
Table 2. Settings of different TTFT SLO requirements.
Figure 9. End-to-end performance of three models under varying request rates and SLO requirements on QwenTrace.
Figure 10. Performance comparison under three scheduling policies.
Figure 11. Performance comparison under varying batch token budgets and no batching.
Figure 12. Average preemption blocking time under operator- and layer-level preemption boundaries. Different preemption granularities are realized by placing preemption checks at different execution boundaries.
Figure 13. Real and predicted TTFT for three models serving a trace segment from QwenTrace at a given request rate.
Figure 14. Performance comparison under a single-SLO workload. FlowPrefill matches baseline throughput, indicating negligible operator-level preemption (§ 5.1 ) overhead but delivers better SLO attainment as request rates scale.
Figure 15. Performance of FlowPrefill with chunked prefill under different chunk sizes.
Figure 16. Performance comparison under the PD-colocation setting across varying request rates, reporting TTFT and TBT SLO attainment.
Figure 17. Performance comparison for serving Qwen3-30B-A3B.
Continuous batching improves large language model (LLM) serving throughput, but long prompt prefills can delay decode iterations and violate inter-token latency objectives. Chunked prefill mitigates this interference, yet its chunk size is normally fixed: small chunks protect decode latency but repeatedly pay launch overhead, while large chunks improve prefill efficiency but create latency spikes. We introduce SLOWeave, an online scheduling method that selects the largest prefill chunk predicted to finish before the earliest active decode deadline. The decision requires no workload-specific chunk-size tuning and is computed by a logarithmic-time search over a monotone iteration-cost model. We prove that, whenever a decode-only iteration is feasible and the cost predictor is accurate, SLOWeave maximizes immediate prefill progress among decisions that preserve every active request's next-token deadline. We evaluate the method in a reproducible event-driven simulator and an iteration-level GPU runtime across chat, mixed-context, long-context, and bursty workloads. Under a 25ms time-per-output-token objective, SLOWeave improves goodput over the strongest fixed-chunk baseline by 39% on mixed requests and 38% on long-context requests. Under a stricter 10ms objective, the gains rise to 3.3× and 2.4×, respectively. These results isolate adaptive chunk sizing as a useful serving primitive and provide an implementation-ready controller for integration with iteration-level LLM runtimes.
Siyu Song, Qi Bai, Jinbo Hao +3
School of Computer Science and Technology, Beijing Institute of Technology · School of Computer Science and Engineering, Sun Yat-sen University · School of Computer Engineering, Jiangsu Ocean University
Disaggregated LLM serving runs prefill and decode on separate GPU pools to keep the two phases from interfering. In practice, this creates a new asymmetry: under bursty, heavy-tailed workloads prefill nodes saturate while decode nodes have compute underutilized, and on a production-style A100 cluster with 2 prefill and 2 decode nodes (2P2D), we find that prefill execution accounts for only 2-23% of P95 Time-to-First-Token (TTFT). Queuing and inter-node GPU-GPU KV-cache transfer account for the rest. We present a proactive prefill-deflecting scheduler that lets decode nodes serve prefill phase of requests as chunked-prefill steps interleaved with their in-flight decode batches. For each queued request, we estimate the TTFT it would see on the prefill node, and on every decode node, search for the largest chunk schedule that keeps in-flight decodes within their Time-Between-Tokens (TBT) SLO and deflect when the decode path helps tail latency. Because the prefill phase of deflected requests runs in place on the decode node, the inter-node KV transfer is eliminated. Implemented on vLLM and evaluated on production-style traces with DeepSeek-V2-Lite, our approach reduces P95 TTFT by upto 81% and raises SLO attainment by upto 79% over state-of-the-art disaggregated schedulers, at sub-millisecond per-request routing cost.
As Large Language Models (LLMs) become essential in privacy-sensitive sectors like hospitals and government agencies, the on-premise LLM servers offer a cost-effective and secure alternative to public cloud services. However, these resource-constrained servers struggle to guarantee heterogeneous Service Level Objectives (SLOs) when serving multiple LoRA-adapted services simultaneously. Existing serving frameworks suffer from severe SLO violations due to the computational overhead of LoRA layers and the rigid nature of batch scheduling. To address this, we propose HALO, a scheduling method tailored for LoRA-assisted on-premise LLM deployment. HALO introduces two key innovations: a spatial multiplexing strategy that overlaps Base and LoRA computations by partitioning GPU Streaming Multiprocessors (SMs), and an SLO-aware scheduler that decouples request execution based on "request-level slack." By prioritizing urgent tasks and utilizing idle budget for traffic shaping, HALO significantly mitigates resource contention. Our evaluation demonstrates that HALO minimizes SLO violations while improving throughput compared to state-of-the-art baselines.
Zeshen Zhang, Han Zhao, Weihao Cui +9
Shanghai Jiao Tong University, China · Ant Group, China