LEAP: Efficient LLM Inference Scheduling with Generalizable Length Prediction

Kwan Yan; Jialun Zhong; Yanzeng Li

cs.AIOct 30, 2026

LEAP: Efficient LLM Inference Scheduling with Generalizable Length Prediction

Authors: Kwan Yan, Jialun Zhong, Yanzeng Li

Organizations: Beijing Normal University · Peking University

Abstract

Length-aware scheduling reduces request latency in LLM serving by prioritizing shorter requests. However, despite their large model sizes, length predictors can suffer degraded ranking performance under domain and task shifts, and adapting them can require costly retraining. Their prediction overhead can further offset the benefits of scheduling. To improve the generalization and efficiency of length-aware scheduling, we propose LEAP, a length prediction system with fast online adaptation and asynchronous CPU execution. LEAP reuses the serving model's prefill hidden states, which contain information about the output length, and extracts this signal through a linear readout. To adapt to substantial domain and task shifts, it updates the readout at test time through recursive least squares, using completed requests as supervision without gradient computation or historical-data replay. Prediction and updates run asynchronously on CPU, while prediction and initial batch ordering overlap the remaining GPU prefill. Across the evaluated cross-domain transfers, LEAP retains strong ranking performance, and online adaptation improves Spearman correlation by 64.7% under a larger domain shift. Serving experiments show that LEAP reduces average per-token latency by 15.9--40.8% compared with the evaluated length predictors, while achieving 1.9% higher throughput with $30{,}569\times$ fewer predictor parameters than the strongest evaluated baseline.

Figures & tables

Figure 1: LEAP : CPU-asynchronous prediction, scheduling, and completion-driven adaptation.

Figure 2: Frozen linear readout with square-root length mapping.

Figure 3: Source-initialized RLS and asynchronous weight publication. Lengths are illustrative.

Qwen3-8B
	Method
Dataset	LEAP (ours)		TIE		TRAIL		LTR	SSJF		EGTP
	MAE $\downarrow$	$\rho\uparrow$	MAE $\downarrow$	$\rho\uparrow$	MAE $\downarrow$	$\rho\uparrow$	$\rho\uparrow$	MAE $\downarrow$	$\rho\uparrow$	MAE $\downarrow$	$\rho\uparrow$
Alpaca	71.5	0.955	161.8	0.875	114.0	0.887	0.863	139.6	0.847	241.9	0.482
ShareGPT	171.6	0.900	316.1	0.675	278.8	0.719	0.733	280.4	0.726	428.7	0.142
LMSYS	127.5	0.932	193.7	0.848	213.1	0.791	0.817	210.2	0.811	395.5	0.448

Table 1: Initial length prediction across models and datasets. MAE is in tokens;

\rho

is Spearman correlation. Bold marks the best value.

Figure 4: CPU feature transfer, prediction, and publication overlap the remaining GPU prefill. Timing is schematic.

	ShareGPT		LMSYS
Predictor	MAE	$\Delta$ Resp. (%)	MAE	$\Delta$ Resp. (%)
TRAIL-S	244.67	—	170.69	—
TRAIL-D	167.01	+12.89	108.84	+9.16
LEAP-S	154.60	—	111.92	—
LEAP-D	141.96	+9.42	103.82	+8.37

Table 2: Prediction accuracy and serving cost. Green rows use static countdown (-S); orange rows refresh hidden states during decoding (-D).

\Delta

Resp. is relative to the static version.

Figure 5: Ranking after LMSYS transfer on Qwen3-8B. Crosshatched extensions show RLS changes to LEAP .

Method	Mean PTL	P90 PTL	Mean TTFT	P90 TTFT	Tok/s
FCFS	250.97	359.96	55.10	74.85	1094.03
TRAIL	107.42	189.32	67.18	90.28	944.71
TIE	78.48	152.33	66.50	85.22	1089.84
SSJF	76.07	154.50	62.27	80.34	1108.46
LTR	75.62	151.95	60.40	80.17	1120.88
LEAP (ours)	63.58	127.47	55.27	73.43	1141.93

Table 3: Serving performance on ShareGPT under a shared selector. PTL: ms/token; TTFT: ms. Rows are ordered from higher to lower mean PTL. Our results are bold.

Figure 6: LMSYS serving latency versus offered rate under shared post-prefill scheduling. All four methods serve the same 1,024 requests and token trajectories.

	Aya		MATH
Method	$1.2$	$1.5$	$1.2$	$1.5$
SSJF	154.83	253.29	80.30	126.83
EGTP	136.62	223.09	96.54	157.83
FCFS	138.40	226.71	91.66	148.90
TRAIL	125.02	201.67	77.52	123.67
LTR	123.32	197.28	75.12	118.93

Table 4: Simulated mean response time after domain transfer with matched Aging (s; lower is better). Bold: ours.

Size	ShareGPT		LMSYS
	MAE	$\rho$	MAE	$\rho$
0.6B	151.38	0.836	94.81	0.910
1.7B	206.26	0.856	144.07	0.921
4B	184.84	0.879	124.81	0.932
8B	173.26	0.901	129.14	0.936
14B	160.31	0.909	116.13	0.936

Table 5: Length prediction across Qwen3 model sizes.

Map	Short	Long	All
$\log L$	50.7 / 0.7907	310.4 / 0.5832	155.9 / 0.9122
$\sqrt{L}$	68.5 / 0.7179	279.3 / 0.6009	149.6 / 0.9161
$L$	105.4 / 0.4881	266.3 / 0.6097	162.7 / 0.8984

Table 6: Effect of the target map on length prediction (mean MAE / Spearman

\rho

Input	Mean MAE	Mean Spearman $\rho$
Block 30	149.53	0.9160
All states	144.63	0.9204

Table 7: Single-layer versus all-layer length prediction on Qwen3-8B.

Predictor	Parameters	FP32 weights
TIE	185,211,394	0.741 GB
LTR	125,240,064	0.501 GB
SSJF	109,597,313	0.438 GB
EGTP	8,435,732	33.74 MB
TRAIL	2,118,184	8.47 MB
LEAP (ours)	4,097	16.39 KB

Table 8: Predictor parameter counts and FP32 weight storage.

Figure 7: CPU score delivery under matched core budgets (batch eight).

Figure 8: Share of trials in which prediction and initial batch order are ready before the final RMSNorm on Qwen3-8B (24 trials per bar).

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

		Raw targets		Standardized
Backbone	Dataset	MAE $\downarrow$	$\rho\uparrow$	MAE $\downarrow$	$\rho\uparrow$
Qwen3-8B	ShareGPT	779.39	0.0845	308.89	0.6716
Qwen3-8B	LMSYS	517.79	0.1658	240.90	0.7844

Appendix

Table 9: SSJF target standardization at the early 2,880-request training budget.

LLM Output Length Prediction

See all 8 papers

Jun 16, 2026

Beyond Prediction: Tail-Aware Scheduling for LLM Inference

LLM serving exhibits extreme length variability, making size-based scheduling difficult in practice. Recent LLM schedulers approximate SJF/SRPT using predicted decode lengths or ranks and primarily report mean-centric metrics such as TTFT and TBT. We show that these prediction-driven policies can be fragile under distribution shifts, bursty arrivals, and GPU memory pressure, while offering limited control over the tail latency (P90-P99) that dominates user experience, even with perfect decode-length knowledge. We introduce a distribution-aware, prediction-free scheduling framework that replaces explicit length prediction with soft priority boosting driven by lightweight statistical signals. Our design co-optimizes scheduling and cache-aware preemption to account for memory-coupled decode dynamics across workload mixes. Evaluated on production and open-source traces, our method reduces P99 TTLT by up to 35-50% relative to SRPT with perfect length knowledge and reduces TTFT by 34-47% across workloads, including reasoning-heavy and chat-heavy tasks. These results demonstrate a robust alternative for optimizing tail latency in online LLM serving.

Yueying Li, Yuanfan Chen, Jiayang Chen +6

Cornell University, Computer Science Department, NY, USA · Microsoft Azure System Research, WA, USA · Cornell University, Electrical and Computer Engineering Department, NY, USA +2

LLM Serving Inference-Time Optimization LLM Inference Scheduling LLM Request Scheduling

Ask PDF

Sep 1, 2026

OUTLETS: Output-Length Prediction from Speculative Decoding Backbones

The heavy-tailed distribution of output lengths in Large Language Model (LLM) serving poses major challenges for resource provisioning and cluster scheduling. Although output-length prediction can mitigate these issues, existing approaches have key drawbacks: external proxy models add substantial latency and often have limited fidelity, whereas internal state-based methods are efficient but rely on shallow probes of current model states. We identify a structural connection between speculative decoding (SD) and length prediction: latent representations produced by the draft decoder in advanced frameworks (e.g., EAGLE-3) encode signals that are predictive of generation length. Building on this insight, we introduce OUTLETS (Output-Length Prediction from Speculative Decoding Backbones), which repurposes the speculative backbone as a trajectory-aware length predictor. When its draft representations are already computed for speculative decoding, OUTLETS adds only a lightweight regression head and achieves lower MAE than the evaluated methods. Under saturated disaggregated serving, OUTLETS predictions enable standard scheduling policies to prioritize shorter requests and distribute requests more evenly across decoding instances, reducing short-request P99 latency by 34.8%.

Weihuang Wen, Yingying Liu, Yichuan Liu +5

The Chinese University of Hong Kong, Shenzhen · The University of Hong Kong · Huawei Technologies Co., Ltd.

LLM Serving LLM Output Length Prediction LLM Inference Scheduling

Ask PDF

Oct 2, 2026

Jumping the Line: Exploiting Length Predictions in LLM Scheduling

Efficient request scheduling is increasingly important for reducing completion time in large language model (LLM) serving. Size-based policies such as Shortest Job First prioritize shorter requests, but output lengths are unknown before generation, so practical schedulers rely on predicted lengths. We introduce JIL, an attack on prediction-based LLM schedulers that manipulates the scheduling signal to obtain higher priority and reduce completion time. Using TRAIL as a case study, JIL optimizes an adversarial suffix that causes a lightweight output-length probe to underestimate a request's length. We evaluate JIL on two datasets and four LLMs across varied request profiles and deployment configurations. JIL reduces predicted output lengths by up to 83.4 percent, and adversarial requests complete up to 1.53 times faster on average in end-to-end serving experiments. The reduction in predicted length is substantially larger than the change in actual output length, revealing a mismatch between the scheduler's estimate and the request's realized size. Response utility varies across models and tasks, exposing a trade-off between scheduling advantage and response quality. We also evaluate scheduler-side defenses and find that grouping length predictions into coarse intervals reduces JIL's scheduling advantage and mitigates delays to benign requests.

Yuyang Dai, Rana Shahout, Mahmood Sharif

Florida State University · MIT · Tel Aviv University

LLM Inference Scheduling LLM Serving LLM Request Scheduling Adversarial Attacks on LLMs

Ask PDF

Abstract

LLM Output Length Prediction

Explore similar work

Beyond Prediction: Tail-Aware Scheduling for LLM Inference

OUTLETS: Output-Length Prediction from Speculative Decoding Backbones

Jumping the Line: Exploiting Length Predictions in LLM Scheduling