Duration Aware Scheduling for ASR Serving Under Workload Drift

Darshan Makwana; Yash Jogi; Harsh Kotta; Aayush Kubba

cs.LGMar 11, 2026

Duration Aware Scheduling for ASR Serving Under Workload Drift

Authors: Darshan Makwana, Yash Jogi, Harsh Kotta, Aayush Kubba

Organizations: Sprinklr Gurugram, India

Abstract

Scheduling policies in large-scale Automatic Speech Recognition (ASR) serving pipelines play a key role in determining end-to-end (E2E) latency. Yet, widely used serving engines rely on first-come-first-served (FCFS) scheduling, which ignores variability in request duration and leads to head-of-line blocking under workload drift. We show that audio duration is an accurate proxy for job processing time in ASR models such as Whisper, and use this insight to enable duration-aware scheduling. We integrate two classical algorithms, Shortest Job First (SJF) and Highest Response Ratio Next (HRRN), into vLLM and evaluate them under realistic and drifted workloads. On LibriSpeech test-clean, compared to baseline, SJF reduces median E2E latency by up to $73\%$ at high load, but increases $90$ th-percentile tail latency by up to $97\%$ due to starvation of long requests. HRRN addresses this trade-off: it reduces median E2E latency by up to $28\%$ while bounding tail-latency degradation to at most $24\%$ . These gains persist under workload drift, with no throughput penalty and $<0.1$ ,ms scheduling overhead per request.

Figures & tables

Figure 1: Toy example illustrating head-of-line blocking under FCFS and the benefit of duration-aware scheduling. Three requests arrive in order $R_{1},R_{2},R_{3}$ with audio durations $8$ s, $4$ s, and $2$ s. We assume a constant encoder cost of $1$ s per request and a decoding rate of $5$ output tokens/s, yielding $40$ , $20$ , and $10$ output tokens, respectively. Under FCFS, serving $R_{1}$ first delays the two shorter requests and yields an average end-to-end latency of $(5{+}8{+}10)/3=7.66$ s. Reordering by shortest job first (SJF) reduces the average to $(2{+}5{+}10)/3=5.66$ s (a $1.4\times$ improvement). All numbers are illustrative and not intended to match measured system timings.

Figure 2: Scatter plots showing the relationship between audio duration and ASR output token count. (a) On the LibriSpeech English test set, token count increases linearly with audio duration, indicating a strong correlation. (b) On the FLEURS test sets for Spanish, Hindi, and Arabic, the linear duration–token relationship is maintained, demonstrating that this correlation generalizes across typologically diverse languages.

Rate	P50 TTFT (ms) $\downarrow$			P50 E2EL (ms) $\downarrow$			P90 E2EL (ms) $\downarrow$
Rate	FCFS	SJF	HRRN	FCFS	SJF	HRRN	FCFS	SJF	HRRN
1	64.6	61.5 ( $-$ 5%)	73.3 (+13%)	88.2	84.9 ( $-$ 4%)	96.4 (+9%)	143.8	141.6 ( $-$ 2%)	155.7 (+8%)
5	64.7	62.8 ( $-$ 3%)	64.4	112.4	108.9 ( $-$ 3%)	112.5	276.7	261.3 ( $-$ 6%)	275.9
10	118.6	115.5 ( $-$ 3%)	115.1 ( $-$ 3%)	213.0	200.8 ( $-$ 6%)	195.6 ( $-$ 8%)	742.7	682.5 ( $-$ 8%)	681.7 ( $-$ 8%)
15	192.0	144.2 ( $-$ 25%)	198.3 (+3%)	547.9	518.3 ( $-$ 5%)	598.3 (+9%)	1561.6	1467.9 ( $-$ 6%)	1615.4 (+3%)
20	1467.8	193.3 ( $-$ 87%)	1173.9 ( $-$ 20%)	2728.7	1325.8 ( $-$ 51%)	2231.2 ( $-$ 18%)	3954.8	7094.6 (+79%)	4996.6 (+26%)

Table 1: LibriSpeech test-clean: Latency comparison across request rates (ms). Parenthesized values show relative change versus FCFS. Bold values indicate the highest value among the three policies. Green :

>

3% improvement; red :

>

3% degradation.

Figure 3: LibriSpeech test-clean: End-to-end latency scaling ( $1$ – $25$ req/s). (a) SJF reduces $P50$ E2EL by up to $73\%$ at $25$ req/s, (b) while $P90$ E2EL reveals the starvation trade-off at high load.

Figure 4: LibriSpeech test-clean: Percentage change in E2EL versus FCFS ( $1$ – $25$ req/s). Negative values indicate improvement. SJF’s $P50$ gains deepen monotonically with load, while $P90$ degradation emerges beyond $15$ req/s.

Rate	P50 TTFT (ms) $\downarrow$			P50 E2EL (ms) $\downarrow$			P90 E2EL (ms) $\downarrow$
Rate	FCFS	SJF	HRRN	FCFS	SJF	HRRN	FCFS	SJF	HRRN
1	64.0	67.1 (+5%)	68.6 (+7%)	84.8	88.3 (+4%)	89.9 (+6%)	158.0	162.8 (+3%)	164.0 (+4%)
5	66.0	68.4 (+4%)	68.8 (+4%)	117.7	121.1 (+3%)	122.1 (+4%)	242.2	262.9 (+9%)	260.3 (+7%)
10	120.3	126.8 (+5%)	124.3 (+3%)	214.8	229.3 (+7%)	239.5 (+11%)	741.1	849.0 (+15%)	881.6 (+19%)
15	2654.0	267.1 ( $-$ 90%)	2263.5 ( $-$ 15%)	4788.6	2287.3 ( $-$ 52%)	4520.1 ( $-$ 6%)	7529.9	8902.4 (+18%)	9635.5 (+28%)
20	6661.2	906.7 ( $-$ 86%)	4702.7 ( $-$ 29%)	8686.9	3079.7 ( $-$ 65%)	6939.6 ( $-$ 20%)	14325.2	22478.8 (+57%)	16664.3 (+16%)

Table 2: Synthetic split: Latency comparison across request rates (ms). Parenthesized values show relative change versus FCFS. Bold values indicate the highest value among the three policies. Green :

>3\%

improvement; red :

>3\%

degradation.

Figure 5: Synthetic split: End-to-end latency scaling ( $1$ – $30$ req/s). (a) SJF’s $P50$ E2EL advantage ( $-67\%$ at $25$ req/s) persists under a uniform duration distribution, (b) while $P90$ E2EL degradation is moderated compared to LibriSpeech’s right-skewed workload.

Figure 6: Synthetic split: Percentage change in E2EL versus FCFS ( $1$ – $30$ req/s). SJF’s E2EL gains match LibriSpeech, confirming reordering benefits are not an artifact of the natural duration skew.

Figure 7: Request throughput for both workloads. All scheduling policies achieve identical throughput, saturating at $\sim 18$ req/s on LibriSpeech and $\sim 13$ req/s on the synthetic split, confirming negligible scheduling overhead.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: LibriSpeech TTFT scaling (1–25 req/s). SJF keeps $P50$ TTFT below $300$ ms at all rates while FCFS exceeds $4$ s.

Figure 9: LibriSpeech: percentage change in TTFT versus FCFS ( $1$ – $25$ req/s). SJF’s $P50$ TTFT gains deepen monotonically, reaching $-93\%$ at $25$ req/s.

Figure 10: Synthetic split TTFT scaling ( $1$ - $30$ req/s). SJF achieves $-84\%$ $P50$ TTFT at $25$ req/s, confirming that reordering benefits are independent of duration variance.

Figure 11: Synthetic split: percentage change in TTFT versus FCFS ( $1$ - $30$ req/s). The pattern closely mirrors LibriSpeech, confirming robustness to duration distribution.

Figure 12: Burst workload results (rate $=\infty$ , $500$ simultaneous requests). Bar heights show absolute metric values; annotations indicate percentage change versus FCFS. HRRN provides the most consistent improvements across all latency metrics.

Figure 13: Whisper-Medium E2EL scaling ( $15$ – $25$ req/s). $P25$ latency (a) shows that SJF benefits the bottom quartile of requests across all rates, while $P90$ latency (b) reveals the tail cost at high load, consistent with Large-v3 findings.

Real-Time Scheduling

Real-time scheduling allocates processors, accelerators, and network resources so tasks meet timing deadlines under workload and resource constraints.

See all 30 papers

Beyond Prediction: Tail-Aware Scheduling for LLM Inference
Jun 16, 2026Yueying Li, Yuanfan Chen, Jiayang Chen +6LLM Serving Inference-Time Optimization
Cornell University, Computer Science Department, NY, USA · Microsoft Azure System Research, WA, USA · Cornell University, Electrical and Computer Engineering Department, NY, USA +2
Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rank
Sep 25, 2025Yiheng Tao, Yihe Zhang, Matthew Dearing +4LLM Serving Learning to Rank
University of Illinois Chicago, Chicago, IL, USA · Argonne National Laboratory, Lemont, IL, USA
LLM Serving Optimization with Variable Prefill and Decode Lengths
Aug 8, 2025Meixuan Wang, Yinyu Ye, Zijie ZhouLLM Serving LLM Inference Scheduling
Department of Computer Science and Technology, Tsinghua University · Department of Management Science and Engineering, Stanford University & Department of Industrial Engineering and Decision Analytics, HKUST · Department of Industrial Engineering and Decision Analytics, HKUST

Jun 16, 2026J/K move · Enter open · S save

Beyond Prediction: Tail-Aware Scheduling for LLM Inference

Yueying Li, Yuanfan Chen, Jiayang Chen, Esha Choukse+5

Cornell University, Computer Science Department, NY, USA · Microsoft Azure System Research, WA, USA · Cornell University, Electrical and Computer Engineering Department, NY, USA+2

LLM Serving Inference-Time Optimization LLM Inference Scheduling LLM Request Scheduling

Open paper AskPDF arXiv

LLM serving exhibits extreme length variability, making size-based scheduling difficult in practice. Recent LLM schedulers approximate SJF/SRPT using predicted decode lengths or ranks and primarily report mean-centric metrics such as TTFT and TBT. We show that these prediction-driven policies can be fragile under distribution shifts, bursty arrivals, and GPU memory pressure, while offering limited control over the tail latency (P90-P99) that dominates user experience, even with perfect decode-length knowledge. We introduce a distribution-aware, prediction-free scheduling framework that replaces explicit length prediction with soft priority boosting driven by lightweight statistical signals. Our design co-optimizes scheduling and cache-aware preemption to account for memory-coupled decode dynamics across workload mixes. Evaluated on production and open-source traces, our method reduces P99 TTLT by up to 35-50% relative to SRPT with perfect length knowledge and reduces TTFT by 34-47% across workloads, including reasoning-heavy and chat-heavy tasks. These results demonstrate a robust alternative for optimizing tail latency in online LLM serving.

Abstract

Real-Time Scheduling

Explore similar work

Beyond Prediction: Tail-Aware Scheduling for LLM Inference

Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rank

LLM Serving Optimization with Variable Prefill and Decode Lengths