cs.AINov 2, 2026

Look Before You Launch: Learned Throughput Prediction for Hardware Selection in a Production Training Fleet

Authors: Hossein Askari, Wei Sun, Chenguang Zhu, Bin Zan, Sreen Tallam, Keyu Man

Organizations: Meta · Work done at Meta

Abstract

In mixed-vendor accelerator fleets, many training jobs can run on several hardware types, and the best choice depends on each job's throughput on each type. Routing a job needs a throughput estimate for every candidate type before the job runs. We present LEAP, a machine learning performance prediction service that predicts each training job's throughput on every candidate hardware type at submission time and ranks the candidates by predicted throughput per unit cost. It routes tens of thousands of training job attempts a day across hundreds of different model types and more than 20 hardware types in a production fleet. LEAP predicts throughput from submission-time training configuration, including each job's prior throughput. LEAP reaches an accuracy of 77% over 30 nightly retrains, higher than the lookup table it replaced and at least as high as every alternative model we tested, including fine-tuned language models. Over six months, the share of eligible training GPU-hours that received a ranking by LEAP grew from 58% to 95%, and we estimate that it saves about 3% of training computational cost. In this paper, we identify the drivers that actually move accuracy and those that do not. We offer these as lessons for teams building performance predictors at fleet scale.

Figures & tables

Explore similar work

Jul 6, 2026

When Words Predict Workload

Standard distributed \ac{llm} schedulers rely on static token counts or rolling latency averages, making them susceptible to failures on statutorily constrained text. On \ac{epo} claims governed by Article 84 \ac{epc}, linguistic rigidity makes human and machine authorship statistically indistinguishable. Resolving this ambiguity mid-flight forces dynamic multi-model ensemble expansion, triggering unpredictable KV-cache and weight-allocation spikes that saturate consumer-grade edge GPU VRAM and cause severe \ac{oom} crashes. To prevent hardware collapse, we propose a CPU-side Linguistic Resource Forecasting (LRF) gateway. The gateway extracts a 16-dimensional text-structure vector and applies an XGBoost predictor to forecast trap-band membership. The resulting escalation probability (\Pesc\Pesc) is evaluated against a dynamic, closed-form routing threshold (\Tauroute(t)\Tauroute(t)) computed via real-time latency telemetry. Requests are safely routed to either a local Qwen2.5-7B edge worker or a remote contrastive ensemble (Qwen2.5 7B + 32B) on an NVIDIA H100 before any edge GPU memory is allocated. In a 6,000-request live trial, the LRF gateway reduced the operational misroute fraction (RmisR_{\mathrm{mis}}) to 0.0870.087--0.0950.095, an order of magnitude below the token-count baseline (0.8490.849). Peak edge VRAM remained safely bounded at \SI4.82\gibi\byte\SI{4.82}{\gibi\byte} (under the \SI8\gibi\byte\SI{8}{\gibi\byte} ceiling) across a 27×27\times variation in \ac{wan} delay. The predictor achieved a live-trial AUROC of 0.840.84, and the dynamic \Tauroute(t)\Tauroute(t) controller yielded an 8.2%8.2\% relative reduction in misroutes compared to an equivalent static threshold.
Oct 30, 2026

LEAP: Efficient LLM Inference Scheduling with Generalizable Length Prediction

Length-aware scheduling reduces request latency in LLM serving by prioritizing shorter requests. However, despite their large model sizes, length predictors can suffer degraded ranking performance under domain and task shifts, and adapting them can require costly retraining. Their prediction overhead can further offset the benefits of scheduling. To improve the generalization and efficiency of length-aware scheduling, we propose LEAP, a length prediction system with fast online adaptation and asynchronous CPU execution. LEAP reuses the serving model's prefill hidden states, which contain information about the output length, and extracts this signal through a linear readout. To adapt to substantial domain and task shifts, it updates the readout at test time through recursive least squares, using completed requests as supervision without gradient computation or historical-data replay. Prediction and updates run asynchronously on CPU, while prediction and initial batch ordering overlap the remaining GPU prefill. Across the evaluated cross-domain transfers, LEAP retains strong ranking performance, and online adaptation improves Spearman correlation by 64.7% under a larger domain shift. Serving experiments show that LEAP reduces average per-token latency by 15.9--40.8% compared with the evaluated length predictors, while achieving 1.9% higher throughput with 30,569×30{,}569\times fewer predictor parameters than the strongest evaluated baseline.
Sep 12, 2026

Leakage-Safe and Scheduler-Aware Machine Learning for Grid Job Runtime Prediction

Accurate job runtime prediction can improve scheduling-aware resource management in grid and distributed computing environments, but prediction models must be evaluated under realistic deployment constraints. This paper revisits CPU burst time prediction on the GWA-T-4 AuverGrid workload trace and reformulates it as leakage-safe pre-execution job runtime prediction. We define the target as job-level runtime, use only submission-time attributes, exclude post-execution variables, and evaluate models under temporal and cold-start settings rather than relying only on random cross-validation. We compare standard regressors, chronological historical baselines, categorical encoding strategies, and CatBoost with native categorical handling. We further add temporal hyperparameter tuning, runtime predictability analysis, feature ablation, error analysis by job length, and a minimal scheduling simulation. After temporal-validation tuning, CatBoost achieves the strongest deployment-oriented result with R^2=0.239, MAE=27,019, RMSE=46,587, and LogMAE=2.646 on the held-out temporal test set. A single-server simulation over all 69,523 held-out temporal test jobs shows that prediction-informed SJF reduces average waiting time by 50.92% relative to FCFS. The results show that random-split evaluation overestimates performance, categorical-native boosting improves temporal generalization, and long-job underestimation remains a scheduler-relevant challenge.