Cap-and-Track: Optimizing LLM Inference Request Dispatch Across Shifting Capacity
Organizations: Meta.
Abstract
A fixed dispatch rate for LLM inference becomes stale as serving fleets resize and request sizes change. Generative output length is unknown at admission, and shared provider pools can lose capacity. We present Cap-and-Track, a design for asynchronous inference across provider capacity and internal GPUs. Meta's traffic shaper implements Cap-and-Track for post and live-video understanding. Refusals signal the backend's current admission ceiling. Durable priority queues hold excess work outside execution, including refused requests for retry. Cap is a fail-fast admission check at the saturated resource that refuses work before a long wait. Using only those refusals, the Track dispatcher probes upward and backs off. It follows the ceiling without output-length prediction or a fleet-level rate to retune. In a concurrent same-traffic production A/B, the shaper ran per-replica Cap plus Track. It served a classifier on 43% fewer H100s than its queued, fixed-rate precursor, though lower-priority work waited longer at backfill peaks. In steady-capacity real-engine replay, Track ran within 1% of the goodput of a fixed rate tuned above measured capacity, with about a quarter of that rate's refusal share. The shaper carries more than 300 million online requests and hundreds of terabytes of media a day. During a backfill, it handled 1.1 billion requests in its busiest 24 hours. Lessons include keeping queues inside inference engines just long enough to fill batches, steering dispatch by capacity refusals alone, keeping dispatch stateful, and sizing fleets by how long low-priority work can wait.
Figures & tables
| Setting | Compared (window) | Result | Caveat |
|---|---|---|---|
| On-demand capacity loss | 26 h of loss (three mornings) against median rates | Other three workloads at 98.0–99.2%. On-demand queue emptied 2–3 h after recovery | Hourly, observational |
| Reserved, launch | Shaper’s first 20 days, target refusal share raised 5% to 30% in 2 days, against static system’s prior 23 days | Utilization 94.8% (first 4 full days) against 88.2%. Refused 28.3% (first 4 days) and 27.6% (20 days), against 16% (24.4% in the busiest quarter of hours) | Observational. Ceiling near 95–97% |
| Reserved, steady state | Shaper only, 15 weeks after the provider updated its admission | Utilization 95.8% (six-hour periods: median 98.6%, top-quarter mean 99.6%). Refused 4.2% in periods with 429s, 4.4% in the busiest quarter of hours | No baseline after the update |
| Classifier A/B, fleet sizing | Shaper against precursor on full traffic. Arrivals over 3 months | Precursor 68 H100s, erroring at about 2/3 of its profiled per-card rate. Shaper 39 at equal coverage and scores, 45 in production | Serve-on-arrival fleets (Appendix A.3 ) are lower bounds |
| Zero-shot, equal fleets | Shaper against precursor, 23 days, about 600 MI300X each | Shaper drained all shared backlogs first, a median 3 h sooner. At both paths’ dispatch limits, its engines ran a median 124 of 128 sequences against the precursor’s 41, and served 9% more per GPU | Prefill-heavy model. Fewer-GPU estimate untested |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Full capacity | Cut | Restored | Recovery | Latency | |||||
| Cap | Arm | goodput | refusals | goodput | refusals | goodput | refusals | (s) | (s) |
| 40 | Track | 108.5 | 0.04 | 54.7 | 0.49 | 102.3 | 0.03 | 220 | 0.54 |
| 40 | Fixed 100% | 107.0 | 0.01 | 54.9 | 0.97 | 107.1 | 0.01 | 0 | 0.34 |
| 40 | Fixed 120% | 109.5 | 0.18 | 54.9 | 1.35 | 109.8 | 0.18 | 0 | 0.65 |
| 40 | Fixed at cut ∗ | 54.5 | 0.00 | 54.0 | 0.01 | 54.5 | 0.00 | – | 0.07 |
| 64 | Track | 108.9 | 0.05 | 55.2 | 0.48 | 102.7 | 0.04 | 209 | 0.92 |
LLM Inference Scheduling
Large language model inference scheduling allocates requests, batches, compute, and KV-cache memory to balance latency, throughput, and service-level objectives.
See all 197 papers