LLM-Based Exogenous World Modeling for Dynamic Scheduling

In dynamic flexible job shops, order arrivals, machine breakdowns, and processing-time deviations continually reshape the scheduling state and the priority trade-offs behind dispatching decisions. Dispatching rules are well suited to this setting because they are fast, interpretable, and easy to deploy, and recent LLM-assisted automatic heuristic design further expands their expressiveness by evolving composite priority functions. The key challenge is to make these evolved rule behaviors state-adaptive without losing the rapid response needed for online rescheduling. This paper proposes a dynamic self-evolutionary framework DSevolve, which separates offline rule-library construction from online state-conditioned rule selection. Offline, an LLM-guided quality-diversity search combines multi-persona seeding, a MAP-Elites behavioral archive, and behavior-guided variation to evolve a library of complementary rules rather than a single elite, and event-level simulation then trains a neural selector to rank the rules by state. Online, after each rescheduling event, a neural selector maps a 22-dimensional state fingerprint to rule scores and dispatches the top-ranked rule within about a second, meeting the response-time requirement after each disruption. Experiments on dynamic instances derived from standard flexible job shop benchmarks show that DSevolve achieves lower mean makespan than individual LLM-evolved rules, classical dispatching rules, and learning-based baselines under a one-active-rule deployment protocol. Trained only on small instances, the selector transfers zero-shot to substantially larger dynamic shops. These results show that state-conditioned selection preserves the speed and interpretability of dispatching rules while improving adaptability.

LLM serving exhibits extreme length variability, making size-based scheduling difficult in practice. Recent LLM schedulers approximate SJF/SRPT using predicted decode lengths or ranks and primarily report mean-centric metrics such as TTFT and TBT. We show that these prediction-driven policies can be fragile under distribution shifts, bursty arrivals, and GPU memory pressure, while offering limited control over the tail latency (P90-P99) that dominates user experience, even with perfect decode-length knowledge. We introduce a distribution-aware, prediction-free scheduling framework that replaces explicit length prediction with soft priority boosting driven by lightweight statistical signals. Our design co-optimizes scheduling and cache-aware preemption to account for memory-coupled decode dynamics across workload mixes. Evaluated on production and open-source traces, our method reduces P99 TTLT by up to 35-50% relative to SRPT with perfect length knowledge and reduces TTFT by 34-47% across workloads, including reasoning-heavy and chat-heavy tasks. These results demonstrate a robust alternative for optimizing tail latency in online LLM serving.

The surge of large language model (LLM) applications on personal devices imposes massive, bursty workloads on cloud serving infrastructure. While prefill-decode disaggregation improves throughput and scalability, memory-bound decode instances often suffer from persistent load imbalance, as output lengths are unknown when requests arrive at the cloud. To address this, we propose MAPS, a Memory-Aware Predictive Scheduling framework tailored for disaggregated LLM serving. MAPS performs device-assisted speculative output length prediction overlapped with cloud-side prefilling, incurring negligible latency overhead. To handle generation uncertainty, MAPS applies uncertainty-aware calibration to derive output-length upper bounds with target coverage, enabling safe scheduling decisions. Building on these bounds, MAPS employs a hierarchical global-local scheduling strategy to mitigate inter-decoder queue buildup and intra-decoder head-of-line blocking. Extensive experiments on two real-world workloads and two LLMs show that MAPS significantly outperforms three state-of-the-art systems, reducing average end-to-end latency by 42.6 and tail latency by up to 84.8.