cs.LGNov 18, 2026

Cap-and-Track: Optimizing LLM Inference Request Dispatch Across Shifting Capacity

Authors: Benjamin Novotny, Doug Szeto, Chang Wang, Shicheng Chen, George Zhihua Yuan, Bob Kamma, Michael Ge, John Xin Zhou, 

Organizations: Meta.

Abstract

A fixed dispatch rate for LLM inference becomes stale as serving fleets resize and request sizes change. Generative output length is unknown at admission, and shared provider pools can lose capacity. We present Cap-and-Track, a design for asynchronous inference across provider capacity and internal GPUs. Meta's traffic shaper implements Cap-and-Track for post and live-video understanding. Refusals signal the backend's current admission ceiling. Durable priority queues hold excess work outside execution, including refused requests for retry. Cap is a fail-fast admission check at the saturated resource that refuses work before a long wait. Using only those refusals, the Track dispatcher probes upward and backs off. It follows the ceiling without output-length prediction or a fleet-level rate to retune. In a concurrent same-traffic production A/B, the shaper ran per-replica Cap plus Track. It served a classifier on 43% fewer H100s than its queued, fixed-rate precursor, though lower-priority work waited longer at backfill peaks. In steady-capacity real-engine replay, Track ran within 1% of the goodput of a fixed rate tuned above measured capacity, with about a quarter of that rate's refusal share. The shaper carries more than 300 million online requests and hundreds of terabytes of media a day. During a backfill, it handled 1.1 billion requests in its busiest 24 hours. Lessons include keeping queues inside inference engines just long enough to fill batches, steering dispatch by capacity refusals alone, keeping dispatch stateful, and sizing fleets by how long low-priority work can wait.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. PowerSlider: Exploiting Phase Asymmetry for LLM Serving under Demand Response

    Aug 22, 2026Yueying Li, Jiayang Chen, Yuanfan Chen +5LLM ServingLLM Inference Scheduling

  2. Scheduling LLM Inference with Uncertainty-Aware Output Length Predictions

    Apr 1, 2026Haoyu Zheng, Yongqiang Zhang, Fangcheng Fu +7LLM Output Length PredictionLLM Inference Scheduling

  3. Asymptotic Optimality of cμcμ-Rule for LLM Inference Scheduling under Memory Constraint

    Nov 3, 2026Zhenghao Li, Shengbo Wang, Nian SiLLM Inference Scheduling