LEAP: Efficient LLM Inference Scheduling with Generalizable Length Prediction

Kwan Yan; Jialun Zhong; Yanzeng Li

cs.AIOct 30, 2026

LEAP: Efficient LLM Inference Scheduling with Generalizable Length Prediction

Authors: Kwan Yan, Jialun Zhong, Yanzeng Li

Organizations: Beijing Normal University · Peking University

Abstract

Length-aware scheduling reduces request latency in LLM serving by prioritizing shorter requests. However, despite their large model sizes, length predictors can suffer degraded ranking performance under domain and task shifts, and adapting them can require costly retraining. Their prediction overhead can further offset the benefits of scheduling. To improve the generalization and efficiency of length-aware scheduling, we propose LEAP, a length prediction system with fast online adaptation and asynchronous CPU execution. LEAP reuses the serving model's prefill hidden states, which contain information about the output length, and extracts this signal through a linear readout. To adapt to substantial domain and task shifts, it updates the readout at test time through recursive least squares, using completed requests as supervision without gradient computation or historical-data replay. Prediction and updates run asynchronously on CPU, while prediction and initial batch ordering overlap the remaining GPU prefill. Across the evaluated cross-domain transfers, LEAP retains strong ranking performance, and online adaptation improves Spearman correlation by 64.7% under a larger domain shift. Serving experiments show that LEAP reduces average per-token latency by 15.9--40.8% compared with the evaluated length predictors, while achieving 1.9% higher throughput with $30{,}569\times$ fewer predictor parameters than the strongest evaluated baseline.

Figures & tables

Figure 1: LEAP : CPU-asynchronous prediction, scheduling, and completion-driven adaptation.

Figure 2: Frozen linear readout with square-root length mapping.

Figure 3: Source-initialized RLS and asynchronous weight publication. Lengths are illustrative.

Qwen3-8B
	Method
Dataset	LEAP (ours)		TIE		TRAIL		LTR	SSJF		EGTP
	MAE $\downarrow$	$\rho\uparrow$	MAE $\downarrow$	$\rho\uparrow$	MAE $\downarrow$	$\rho\uparrow$	$\rho\uparrow$	MAE $\downarrow$	$\rho\uparrow$	MAE $\downarrow$	$\rho\uparrow$
Alpaca	71.5	0.955	161.8	0.875	114.0	0.887	0.863	139.6	0.847	241.9	0.482
ShareGPT	171.6	0.900	316.1	0.675	278.8	0.719	0.733	280.4	0.726	428.7	0.142
LMSYS	127.5	0.932	193.7	0.848	213.1	0.791	0.817	210.2	0.811	395.5	0.448

Table 1: Initial length prediction across models and datasets. MAE is in tokens;

\rho

is Spearman correlation. Bold marks the best value.

Figure 4: CPU feature transfer, prediction, and publication overlap the remaining GPU prefill. Timing is schematic.

	ShareGPT		LMSYS
Predictor	MAE	$\Delta$ Resp. (%)	MAE	$\Delta$ Resp. (%)
TRAIL-S	244.67	—	170.69	—
TRAIL-D	167.01	+12.89	108.84	+9.16
LEAP-S	154.60	—	111.92	—
LEAP-D	141.96	+9.42	103.82	+8.37

Table 2: Prediction accuracy and serving cost. Green rows use static countdown (-S); orange rows refresh hidden states during decoding (-D).

\Delta

Resp. is relative to the static version.

Figure 5: Ranking after LMSYS transfer on Qwen3-8B. Crosshatched extensions show RLS changes to LEAP .

Method	Mean PTL	P90 PTL	Mean TTFT	P90 TTFT	Tok/s
FCFS	250.97	359.96	55.10	74.85	1094.03
TRAIL	107.42	189.32	67.18	90.28	944.71
TIE	78.48	152.33	66.50	85.22	1089.84
SSJF	76.07	154.50	62.27	80.34	1108.46
LTR	75.62	151.95	60.40	80.17	1120.88
LEAP (ours)	63.58	127.47	55.27	73.43	1141.93

Table 3: Serving performance on ShareGPT under a shared selector. PTL: ms/token; TTFT: ms. Rows are ordered from higher to lower mean PTL. Our results are bold.

Figure 6: LMSYS serving latency versus offered rate under shared post-prefill scheduling. All four methods serve the same 1,024 requests and token trajectories.

	Aya		MATH
Method	$1.2$	$1.5$	$1.2$	$1.5$
SSJF	154.83	253.29	80.30	126.83
EGTP	136.62	223.09	96.54	157.83
FCFS	138.40	226.71	91.66	148.90
TRAIL	125.02	201.67	77.52	123.67
LTR	123.32	197.28	75.12	118.93

Table 4: Simulated mean response time after domain transfer with matched Aging (s; lower is better). Bold: ours.

Size	ShareGPT		LMSYS
	MAE	$\rho$	MAE	$\rho$
0.6B	151.38	0.836	94.81	0.910
1.7B	206.26	0.856	144.07	0.921
4B	184.84	0.879	124.81	0.932
8B	173.26	0.901	129.14	0.936
14B	160.31	0.909	116.13	0.936

Table 5: Length prediction across Qwen3 model sizes.

Map	Short	Long	All
$\log L$	50.7 / 0.7907	310.4 / 0.5832	155.9 / 0.9122
$\sqrt{L}$	68.5 / 0.7179	279.3 / 0.6009	149.6 / 0.9161
$L$	105.4 / 0.4881	266.3 / 0.6097	162.7 / 0.8984

Table 6: Effect of the target map on length prediction (mean MAE / Spearman

\rho

Input	Mean MAE	Mean Spearman $\rho$
Block 30	149.53	0.9160
All states	144.63	0.9204

Table 7: Single-layer versus all-layer length prediction on Qwen3-8B.

Predictor	Parameters	FP32 weights
TIE	185,211,394	0.741 GB
LTR	125,240,064	0.501 GB
SSJF	109,597,313	0.438 GB
EGTP	8,435,732	33.74 MB
TRAIL	2,118,184	8.47 MB
LEAP (ours)	4,097	16.39 KB

Table 8: Predictor parameter counts and FP32 weight storage.

Figure 7: CPU score delivery under matched core budgets (batch eight).

Figure 8: Share of trials in which prediction and initial batch order are ready before the final RMSNorm on Qwen3-8B (24 trials per bar).

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

		Raw targets		Standardized
Backbone	Dataset	MAE $\downarrow$	$\rho\uparrow$	MAE $\downarrow$	$\rho\uparrow$
Qwen3-8B	ShareGPT	779.39	0.0845	308.89	0.6716
Qwen3-8B	LMSYS	517.79	0.1658	240.90	0.7844

Appendix

Table 9: SSJF target standardization at the early 2,880-request training budget.

LLM Output Length Prediction

See all 8 papers

Beyond Prediction: Tail-Aware Scheduling for LLM Inference
Jun 16, 2026Yueying Li, Yuanfan Chen, Jiayang Chen +6LLM Serving Inference-Time Optimization
Cornell University, Computer Science Department, NY, USA · Microsoft Azure System Research, WA, USA · Cornell University, Electrical and Computer Engineering Department, NY, USA +2
OUTLETS: Output-Length Prediction from Speculative Decoding Backbones
Sep 1, 2026Weihuang Wen, Yingying Liu, Yichuan Liu +5LLM Serving LLM Output Length Prediction
The Chinese University of Hong Kong, Shenzhen · The University of Hong Kong · Huawei Technologies Co., Ltd.
Jumping the Line: Exploiting Length Predictions in LLM Scheduling
Oct 2, 2026Yuyang Dai, Rana Shahout, Mahmood SharifLLM Inference Scheduling LLM Serving
Florida State University · MIT · Tel Aviv University

Jun 16, 2026J/K move · Enter open · S save

Beyond Prediction: Tail-Aware Scheduling for LLM Inference

Yueying Li, Yuanfan Chen, Jiayang Chen, Esha Choukse+5

Cornell University, Computer Science Department, NY, USA · Microsoft Azure System Research, WA, USA · Cornell University, Electrical and Computer Engineering Department, NY, USA+2

LLM Serving Inference-Time Optimization LLM Inference Scheduling LLM Request Scheduling

Open paper AskPDF arXiv

LLM serving exhibits extreme length variability, making size-based scheduling difficult in practice. Recent LLM schedulers approximate SJF/SRPT using predicted decode lengths or ranks and primarily report mean-centric metrics such as TTFT and TBT. We show that these prediction-driven policies can be fragile under distribution shifts, bursty arrivals, and GPU memory pressure, while offering limited control over the tail latency (P90-P99) that dominates user experience, even with perfect decode-length knowledge. We introduce a distribution-aware, prediction-free scheduling framework that replaces explicit length prediction with soft priority boosting driven by lightweight statistical signals. Our design co-optimizes scheduling and cache-aware preemption to account for memory-coupled decode dynamics across workload mixes. Evaluated on production and open-source traces, our method reduces P99 TTLT by up to 35-50% relative to SRPT with perfect length knowledge and reduces TTFT by 34-47% across workloads, including reasoning-heavy and chat-heavy tasks. These results demonstrate a robust alternative for optimizing tail latency in online LLM serving.

Abstract

LLM Output Length Prediction

Explore similar work

Beyond Prediction: Tail-Aware Scheduling for LLM Inference

OUTLETS: Output-Length Prediction from Speculative Decoding Backbones

Jumping the Line: Exploiting Length Predictions in LLM Scheduling