Jumping the Line: Exploiting Length Predictions in LLM Scheduling

Yuyang Dai; Rana Shahout; Mahmood Sharif

cs.AIOct 2, 2026

Jumping the Line: Exploiting Length Predictions in LLM Scheduling

Authors: Yuyang Dai, Rana Shahout, Mahmood Sharif

Organizations: Florida State University · MIT · Tel Aviv University

Abstract

Efficient request scheduling is increasingly important for reducing completion time in large language model (LLM) serving. Size-based policies such as Shortest Job First prioritize shorter requests, but output lengths are unknown before generation, so practical schedulers rely on predicted lengths. We introduce JIL, an attack on prediction-based LLM schedulers that manipulates the scheduling signal to obtain higher priority and reduce completion time. Using TRAIL as a case study, JIL optimizes an adversarial suffix that causes a lightweight output-length probe to underestimate a request's length. We evaluate JIL on two datasets and four LLMs across varied request profiles and deployment configurations. JIL reduces predicted output lengths by up to 83.4 percent, and adversarial requests complete up to 1.53 times faster on average in end-to-end serving experiments. The reduction in predicted length is substantially larger than the change in actual output length, revealing a mismatch between the scheduler's estimate and the request's realized size. Response utility varies across models and tasks, exposing a trade-off between scheduling advantage and response quality. We also evaluate scheduler-side defenses and find that grouping length predictions into coarse intervals reduces JIL's scheduling advantage and mitigates delays to benign requests.

Figures & tables

Panel of Figure 1 — Figure 1: Per-request relative reduction in predicted response length ( $n=200$ per model). Boxes show medians and quartiles; whiskers extend to the most extreme observations within 1.5 interquartile ranges, and points show the remaining observations. JIL produces large reductions on Alpaca, while UltraChat shows stronger dependence on the model.

Panel of Figure 2 — Figure 2: Relative scheduling effects across five paired workloads. Tar refers to Target, Ben refers to Benign. Each point represents one workload; boxes show medians and quartiles, and whiskers extend to observations within 1.5 interquartile ranges.

Model	Alpaca (%)	UltraChat	MC (%)
Llama-3-8B	$84.0\rightarrow 23.0$	$3.42\rightarrow 3.09$	$72.0\rightarrow 68.5$
Qwen2.5-7B	$93.5\rightarrow 63.0$	$3.49\rightarrow 3.23$	$82.5\rightarrow 83.0$
Mistral-7B	$89.0\rightarrow 74.0$	$3.35\rightarrow 3.17$	$71.5\rightarrow 62.5$
Llama-3.1-70B	$80.5\rightarrow 55.5$	$3.23\rightarrow 2.43$	$88.5\rightarrow 86.5$

Table 1: Task utility before and after JIL. Each entry reports clean

\rightarrow

attacked. Alpaca reports the win rate against the matching dataset answer, UltraChat reports the mean judge score on a 1–5 scale, and MC reports accuracy averaged over ARC-Challenge and HellaSwag.

Panel of Figure 3 — Figure 3: UltraChat utility over 200 paired requests per model. Panel (a) shows the distribution of attacked-minus-clean judge scores. Boxes span the interquartile range, and orange lines mark medians. The median equals $-1$ for all four models and coincides with the lower quartile for Llama-8B, Qwen-7B, and Mistral-7B. Panel (b) shows the fractions of comparisons assigned to the clean response, a tie, or the attacked response.

Panel of Figure 4 — Figure 4: Completion-time speedups for the full 30-step attack and three early-stop variants at 16 slots. Each point is the clean group-mean time divided by the attacked group-mean time for one paired workload. Boxes show medians and quartiles; whiskers extend to observations within 1.5 interquartile ranges.

Measure	Llama $\rightarrow$ Mistral	Mistral $\rightarrow$ Llama
Mean reduction (%)	58.8	25.2
Median reduction (%)	60.7	21.4
Clean output tokens	396.3	426.2
Transfer output tokens	345.2	377.6

Table 2: Cross-model transfer on 200 paired requests. Arrows indicate source

\rightarrow

receiving model; Llama and Mistral denote Llama-3-8B and Mistral-7B. Prediction reductions are computed per request using the receiving model’s probe. Output lengths are means in tokens.

Setting	Target speedup	Position gain	Benign tail	Slower (%)
Floor baseline	$1.43\times$	14.24	$1.35\times$	37.0
Floor 80	$1.36\times$	12.96	$1.26\times$	27.0
Floor 120	$1.20\times$	10.88	$1.22\times$	59.0
Grouping baseline	$1.42\times$	15.25	$1.43\times$	55.3
Interval 32	$1.42\times$	15.68	$1.38\times$	50.7
Interval 64	$1.35\times$	12.96	$1.21\times$	40.3

Table 3: Defenses at 16 sequence slots. Target speedup uses clean/attacked completion time; position gains are normalized by 25 requests and expressed in percentage points. Benign tail is the mean within-workload 95th percentile of attacked/clean request latency. Slower is the percentage of benign requests with increased latency. Each block uses its own batch’s original-prediction baseline.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Measure	Definition
Alpaca win rate	Fraction of comparisons won by the generated answer against the matching dataset answer; ties remain in the denominator.
UltraChat score	Mean judge score, with endpoints 1 (unusable) and 5 (highly useful).
UltraChat preference	Fractions assigned clean win, attacked win, and tie in direct paired comparisons.
MC accuracy	Fraction of parsed generated answers matching the correct option.
Output length	Number of generated model-token IDs after the input.
Length-cap rate	Fraction satisfying the generation batch’s recorded cap indicator.

Appendix

Table 4: Response measures and their aggregation. Each response receives equal weight within its model–dataset condition.

Dataset	Model	Mean	Median	$\geq 20\%$	$\geq 40\%$	$\geq 60\%$
Alpaca	Llama-8B	83.4	88.3	99.0	98.5	94.0
Alpaca	Llama-70B	79.3	83.2	100.0	97.0	89.5
Alpaca	Mistral-7B	76.6	83.5	99.0	95.5	78.5
Alpaca	Qwen2.5-7B	72.6	89.2	94.0	87.0	76.0
UltraChat	Llama-8B	25.3	18.2	47.5	18.0	6.5
UltraChat	Llama-70B	32.7	32.9	78.0	27.0	3.0

Appendix

Table 5: Per-request relative prediction reductions, with 200 pairs per model and dataset. All entries are percentages.

Direction	Pred. $c$	Pred. $a$	Reduction	Length $c$	Length $a$
Llama to Mistral	87.05	34.80	58.8%	396.33	345.16
Mistral to Llama	104.64	75.49	25.2%	426.24	377.60

Appendix

Table 6: Bidirectional suffix transfer on 200 paired requests. Prediction and length columns report receiving-model means in tokens; reduction is the mean per-request percentage.

Dataset	Model	MAE	Bias	Spearman	At 512 (%)
Alpaca	Llama-8B	350.46	-350.08	0.153	70.0
Alpaca	Qwen2.5-7B	339.65	-338.49	0.338	49.0
Alpaca	Mistral-7B	309.70	-309.13	0.477	39.5
Alpaca	Llama-70B	328.79	-328.41	0.279	49.0
UltraChat	Llama-8B	282.51	-277.50	0.718	61.5
UltraChat	Qwen2.5-7B	347.80	-347.20	0.320	65.5

Appendix

Table 7: Clean predictions compared with generated token counts under the 512-token limit. Each row contains 200 requests. Bias is predicted minus generated length; the final column gives the percentage with 512 output tokens.

Measurement	Clean	Attacked
Mean prediction	415.65	47.83
Mean generated length	426.24	346.01

Appendix

Table 8: Llama-8B predictions and generated lengths on the 200 attack requests, using the probe trained on generated response lengths. All entries are token counts.

Panel of Figure 5 — Figure 5: Frozen-probe prediction gaps for the fixed group of 141 pairs whose two responses reach 256 tokens. Positive values indicate a lower attacked prediction. Panel (a) follows raw and chat-formatted inputs; panel (b) shows medians, quartiles, 1.5-IQR whiskers, and outlying values for raw inputs at selected positions.

Slots	Role	Completion [95% CI]	TTFT	Position	Tail	Slower
64	Target	1.33 [1.03, 1.62]	5.22	-17.12	0.97	4.0
64	Benign	1.07 [0.97, 1.17]	3.82	+4.28	0.97	6.0
16	Target	1.53 [1.13, 1.93]	3.66	-17.92	0.97	8.0
16	Benign	1.05 [0.86, 1.25]	1.10	+4.48	1.45	43.0

Appendix

Table 9: Relative service in the primary paired workloads. Completion speedup includes a 95% workload-level interval. TTFT is a speedup factor; position is attacked minus clean in percentage points. Tail is the mean within-workload 95th-percentile slowdown; slower is a percentage.

Configuration	Target text	Target prediction	Output
Natural attack	With suffix	Attacked	Natural ending
Signal control	Clean	Attacked	Natural ending
Text control	With suffix	Clean	Natural ending
Fixed output	With suffix	Attacked	512 tokens
Fixed-output signal	Clean	Attacked	512 tokens

Appendix

Table 10: Controlled scheduling configurations. Entries describe the target in the attacked condition; its clean counterpart uses the clean text and prediction. All configurations retain 20 benign and five target requests per workload.

Slots	Configuration	Target TTFT	Target completion [95% CI]	Benign
16	Natural, prediction	3.25	1.43 [0.99, 1.88]	1.00
16	Natural, FCFS	1.34	1.22 [0.99, 1.44]	1.01
16	Fixed, prediction	6.05	1.05 [0.92, 1.18]	1.00
16	Fixed, FCFS	1.03	1.00 [0.98, 1.03]	1.00
16	Signal control	1.97	1.10 [1.01, 1.19]	0.99
16	Text control	0.97	1.18 [0.93, 1.43]	1.04

Appendix

Table 11: Relative service across controlled configurations. Times are reported as clean/attacked group-mean ratios. Target completion includes a 95% interval across five workloads; benign is completion speedup.

Slots	Role	Relative TTFT	Relative completion [95% CI]	Position contrast
16	Target	1.20	0.97 [0.95, 1.00]	-3.84
16	Benign	0.97	0.98 [0.96, 1.00]	+0.96
64	Target	1.28	1.01 [0.99, 1.03]	-3.47
64	Benign	1.07	1.01 [0.99, 1.03]	+0.87

Appendix

Table 12: Policy comparisons with clean request text and 512 generated tokens. Relative speedup divides the prediction-policy speedup by the FCFS speedup after averaging three repeats within each workload. Position contrast subtracts the FCFS normalized position change in percentage points.

Panel of Figure 6 — Figure 6: Policy comparisons for targets with clean request text and 512 generated tokens. Each point summarizes three repeats of one workload. Panel (a) subtracts the FCFS normalized position change from the prediction-policy change; negative values indicate greater forward movement. Panel (b) divides the prediction-policy completion speedup by the FCFS speedup; values above one indicate a larger speedup. Boxes show medians and quartiles, with 1.5-IQR whiskers.

Dataset	Model	Clean length	Attacked length	Malformed
Alpaca	Llama-8B	454.72	448.14	5.5%
Alpaca	Qwen2.5-7B	407.95	270.76	2.0%
Alpaca	Mistral-7B	395.56	367.68	0.5%
Alpaca	Llama-70B	409.77	361.52	4.5%
UltraChat	Llama-8B	395.66	388.95	0.0%
UltraChat	Qwen2.5-7B	417.18	403.87	11.0%

Appendix

Table 13: Generated output measurements for 200 pairs per model and dataset. Lengths are mean token counts; malformed is the attacked-condition flag rate.

Measure	Model	Change	95% interval
Alpaca win (pp)	Llama-8B	-61.00	[-68.00, -54.00]
UltraChat score	Llama-8B	-0.33	[-0.54, -0.13]
Alpaca win (pp)	Qwen2.5-7B	-30.50	[-37.50, -23.50]
UltraChat score	Qwen2.5-7B	-0.26	[-0.45, -0.07]
Alpaca win (pp)	Mistral-7B	-15.00	[-22.00, -8.50]
UltraChat score	Mistral-7B	-0.18	[-0.37, 0.01]

Appendix

Table 14: Paired utility changes with 95% bootstrap intervals from 10,000 resamples of 200 request pairs. Changes are attacked minus clean.

Model	Benchmark	Pred. $c$	Pred. $a$	Acc. $c$	Acc. $a$
Llama-8B	ARC-Challenge	10.87	6.83	84%	79%
Llama-8B	HellaSwag	10.49	7.53	60%	58%
Qwen2.5-7B	ARC-Challenge	54.78	28.16	89%	90%
Qwen2.5-7B	HellaSwag	61.66	33.36	76%	76%
Mistral-7B	ARC-Challenge	14.92	6.25	76%	66%
Mistral-7B	HellaSwag	8.65	4.72	67%	59%

Appendix

Table 15: Benchmark-level results, with 100 paired examples per row. Prediction columns are means in tokens; accuracy columns are percentages.

Condition	Wins / 200	Win rate	Mean tokens
Clean	168	84.0%	454.72
JIL	46	23.0%	448.14
Concise, high-quality	181	90.5%	344.99
Brief, complete, useful	181	90.5%	314.72

Appendix

Table 16: Llama-8B prompt baselines on the same 200 Alpaca requests. Wins compare each answer with the matching dataset reference.

Endpoint	Reduction	Win rate	Speedup	Position gain	Benign tail
20 tokens	28.8%	10.5%	1.34	13.28	1.42
40 tokens	46.3%	14.0%	1.50	17.28	1.30
60 tokens	62.2%	13.5%	1.46	16.00	1.36
30 steps	83.4%	23.0%	1.53	17.92	1.45

Appendix

Table 17: Early-stopping outcomes. Prediction reduction and win rate use 200 requests. Scheduling uses five 25-request workloads at 16 slots. Speedup is clean/attacked target completion time, position gain is in percentage points, and benign tail is a request-level slowdown factor.

$\lambda$	Prediction	Reduction	Output
0	14.52	85.8%	331.50
0.05	14.59	85.8%	329.89
0.5	153.21	-74.9%	415.91
5	153.21	-74.9%	415.91

Appendix

Table 18: Supplementary objective runs on 200 Llama-8B requests per configuration. Prediction and output columns are means in tokens; reduction is the mean per-request percentage. Clean predictions average 104.64 tokens and clean outputs average 426.24 tokens.

Slots	$\lambda$	Target completion	Target position	Benign completion	Benign tail
16	0	1.53	-13.60	1.04	1.21
16	0.05	1.42	-12.64	0.99	1.36
16	0.5	0.90	+3.84	1.00	1.44
16	5	0.98	+1.28	1.02	1.35
64	0	1.45	-22.56	1.04	1.00
64	0.05	1.28	-17.44	1.02	1.02

Appendix

Table 19: Relative scheduling outcomes. Completion is clean/attacked group-mean time; target position is attacked minus clean in percentage points. Benign tail is the 95th-percentile request slowdown. Entries average five workloads.

Slots	$\lambda$	Target TTFT	Benign TTFT	Target slower (%)	Benign slower (%)
16	0	2.48	1.06	4.0	27.0
16	0.05	2.32	0.98	16.0	53.0
16	0.5	0.63	1.14	64.0	55.0
16	5	0.92	1.16	68.0	32.0
64	0	1.25	1.31	4.0	17.0
64	0.05	2.08	1.50	12.0	27.0

Appendix

Table 20: First-token speedups and individual delays. TTFT is clean/attacked group-mean time; slower is the percentage of requests with attacked/clean completion latency above one. Entries average five workloads.

Slots	Floor	Target TTFT	Target completion [95% CI]	Position gain	Benign tail
16	Original	3.25	1.43 [0.99, 1.88]	14.24	1.35
16	80 tokens	5.65	1.36 [1.20, 1.53]	12.96	1.26
16	120 tokens	1.11	1.20 [0.97, 1.44]	10.88	1.22
64	Original	3.96	1.31 [1.00, 1.62]	17.92	1.00
64	80 tokens	1.35	1.25 [0.99, 1.51]	16.96	1.02
64	120 tokens	1.32	1.27 [0.98, 1.57]	16.32	0.99

Appendix

Table 21: Relative service under prediction floors. Target TTFT and completion use clean/attacked time; position gain is in percentage points. Benign tail is the mean within-workload 95th-percentile slowdown. Intervals summarize five workload speedups.

Slots	Floor	Condition	Original/floor time [95% CI]
16	80	Clean	1.02 [0.99, 1.06]
16	80	Attacked	1.07 [1.01, 1.13]
16	120	Clean	1.03 [1.02, 1.04]
16	120	Attacked	1.02 [0.95, 1.08]
64	80	Clean	0.98 [0.94, 1.03]
64	80	Attacked	0.96 [0.94, 0.98]

Appendix

Table 22: Benign completion speedup after applying a floor. Each workload compares its original-prediction mean time with its floor-setting mean time in the same condition. Values above one indicate faster completion with the floor.

Slots	$q$	Target completion	Target position	Benign completion	Benign tail
16	0	1.42	-15.25	1.01	1.43
16	32	1.42	-15.68	0.98	1.38
16	64	1.35	-12.96	1.01	1.21
16	128	1.25	-8.80	1.02	1.13
64	0	1.27	-18.24	1.02	1.04
64	32	1.27	-17.60	1.01	1.06

Appendix

Table 23: Relative scheduling outcomes. Completion is clean/attacked group-mean time; target position is attacked minus clean in percentage points. Benign tail is the 95th-percentile request slowdown. Entries average three repeats within each workload and then five workloads.

Slots	$q$	Target TTFT	Benign TTFT	Target slower (%)	Benign slower (%)
16	0	3.49	1.01	12.0	55.3
16	32	3.84	0.96	17.3	50.7
16	64	2.05	1.05	13.3	40.3
16	128	1.28	1.08	16.0	44.0
64	0	1.35	1.21	13.3	31.0
64	32	1.65	1.52	12.0	37.3

Appendix

Table 24: First-token speedups and individual delays. TTFT is clean/attacked group-mean time; slower is the percentage of requests with attacked/clean completion latency above one. Entries average three repeats within each workload and then five workloads.

LLM Inference Scheduling

Large language model inference scheduling allocates requests, batches, compute, and KV-cache memory to balance latency, throughput, and service-level objectives.

See all 160 papers

Beyond Prediction: Tail-Aware Scheduling for LLM Inference
Jun 16, 2026Yueying Li, Yuanfan Chen, Jiayang Chen +6LLM Serving Inference-Time Optimization
Cornell University, Computer Science Department, NY, USA · Microsoft Azure System Research, WA, USA · Cornell University, Electrical and Computer Engineering Department, NY, USA +2
Scheduling LLM Inference with Uncertainty-Aware Output Length Predictions
Apr 1, 2026Haoyu Zheng, Yongqiang Zhang, Fangcheng Fu +7LLM Inference Scheduling LLM Uncertainty Estimation
School of Computer Science, Wuhan University, Wuhan, China · Dameng Database Co., Ltd., Wuhan, China · School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China +2
OUTLETS: Output-Length Prediction from Speculative Decoding Backbones
Sep 1, 2026Weihuang Wen, Yingying Liu, Yichuan Liu +5LLM Serving LLM Inference Scheduling
The Chinese University of Hong Kong, Shenzhen · The University of Hong Kong · Huawei Technologies Co., Ltd.

Jun 16, 2026J/K move · Enter open · S save

Beyond Prediction: Tail-Aware Scheduling for LLM Inference

Yueying Li, Yuanfan Chen, Jiayang Chen, Esha Choukse+5

Cornell University, Computer Science Department, NY, USA · Microsoft Azure System Research, WA, USA · Cornell University, Electrical and Computer Engineering Department, NY, USA+2

LLM Serving Inference-Time Optimization LLM Inference Scheduling LLM Request Scheduling

Open paper AskPDF arXiv

LLM serving exhibits extreme length variability, making size-based scheduling difficult in practice. Recent LLM schedulers approximate SJF/SRPT using predicted decode lengths or ranks and primarily report mean-centric metrics such as TTFT and TBT. We show that these prediction-driven policies can be fragile under distribution shifts, bursty arrivals, and GPU memory pressure, while offering limited control over the tail latency (P90-P99) that dominates user experience, even with perfect decode-length knowledge. We introduce a distribution-aware, prediction-free scheduling framework that replaces explicit length prediction with soft priority boosting driven by lightweight statistical signals. Our design co-optimizes scheduling and cache-aware preemption to account for memory-coupled decode dynamics across workload mixes. Evaluated on production and open-source traces, our method reduces P99 TTLT by up to 35-50% relative to SRPT with perfect length knowledge and reduces TTFT by 34-47% across workloads, including reasoning-heavy and chat-heavy tasks. These results demonstrate a robust alternative for optimizing tail latency in online LLM serving.

Abstract

LLM Inference Scheduling

Explore similar work

Beyond Prediction: Tail-Aware Scheduling for LLM Inference

Scheduling LLM Inference with Uncertainty-Aware Output Length Predictions

OUTLETS: Output-Length Prediction from Speculative Decoding Backbones