Scalable Joint Resource Allocation for SLO-Constrained LLM Inference in Heterogeneous GPU Clouds

Jiaming Cheng; Duong Tung Nguyen

cs.LGApr 8, 2026

Scalable Joint Resource Allocation for SLO-Constrained LLM Inference in Heterogeneous GPU Clouds

Authors: Jiaming Cheng, Duong Tung Nguyen

Organizations: The Ira A. Fulton Schools of Engineering, Arizona State University, Tempe, AZ, USA.

Abstract

Serving large language model (LLM) inference in cloud environments requires jointly optimizing model selection, GPU provisioning, parallelism configuration, and workload routing under latency, accuracy, memory, and budget constraints. While mixed-integer linear programming (MILP) can model this problem, its computational cost limits frequent re-optimization under demand variability. Existing heuristics often optimize individual components separately and may become infeasible when system-wide constraints are enforced. This paper presents a scalable framework for SLO-constrained LLM inference. We formulate the problem as an MILP with a two-phase delay model capturing both prefill and autoregressive decoding under tensor and pipeline parallelism. To solve it efficiently, we develop two constraint-aware heuristics: a Greedy Heuristic (GH) and an Adaptive Greedy Heuristic (AGH). AGH extends GH through multi-start construction, local search, and GPU consolidation. Both methods maintain feasibility through parallelism-aware filtering, cost-based ranking, and adaptive parallelism scaling. Experiments based on the Azure LLM Inference Trace show that GH generates feasible solutions within one second, while AGH achieves near-optimal performance within three seconds and scales to large instances where exact solvers fail to converge. Under out-of-sample stress with up to 1.5x delay and accuracy inflation, AGH degrades gracefully through provisioned headroom, yielding substantially lower cost and SLO violations than cost-minimal MILP solutions. Across synthetic and real Azure workloads, AGH maintains SLO compliance at significantly lower cost than exact MILP solutions. These results demonstrate that high-quality allocations provide substantial robustness to demand variability while enabling rapid adaptation to workload changes.

Figures & tables

Fig. 1: System model. Users submit queries that are classified into a finite set of types (summarization, code generation, translation, math solving, image generation, video generation). The SP hosts a catalog of foundation models on a set of heterogeneous GPU tiers; for each model–tier pair it chooses a tensor- and pipeline-parallelism configuration. Routing fractions split each query type’s traffic across the deployed pairs. Latency, accuracy, memory, compute, storage, and budget constraints couple all four decisions.

Symbol	Description
Sets and indices
$i\in\mathcal{I}$	query types and their index
$j\in\mathcal{J}$	foundation models and their index
$k\in\mathcal{K}$	GPU resource tiers and their index
$n\in\mathcal{N}_{k}$	feasible TP degrees on tier $k$
$m\in\mathcal{M}_{k}$	feasible PP depths on tier $k$

TABLE I: Summary of notation used throughout the paper.

Scenario	Algo.	Stage-1 ($)	Cost ($)	Viol. (%)
S1: Default $\delta\!=\!\$ 100,,\phi_{\mathrm{v}}!=!1!\times$	GH	81.2	115.7	0.0
	AGH	72.1	106.4	0.0
	LPR	67.0	8749.8	49.3
	DVR	66.9	47240.6	33.3
	HF	53.6	4250.9	32.6
S2: Tight $\delta\!=\!\$ 75,,\phi_{\mathrm{v}}!=!1!\times$	GH	63.7	33786.1	16.7

TABLE II: Stage-2 evaluation across scenarios S1–S5. Each row averages

S\!=\!500

perturbed scenarios; bold marks the lowest cost per scenario.

Panel of Figure 4 — Fig. 2: Budget sensitivity analysis with $\phi_{\mathrm{v}}=1\times$

Panel of Figure 5 — Fig. 3: Uncertainty-robustness study over stress multiplier $\alpha$ on $\bar{d},\bar{e}$ .

Panel of Figure 6 — Fig. 4: Sensitivity to the unmet cap $u_{\mathrm{ub}}$ under (a) nominal and (b) $1.5\times$ -stressed demand.

Panel of Figure 7 — Fig. 5: Stress and sensitivity study (strict cap $u_{i}\!\leq\!0.02$ ). (a)–(c) GH, AGH, DM under $1.0/1.2/1.5\times$ delay–error inflation: the heuristics’ headroom keeps demand served, while the cost-minimal exact plan breaches the cap, so its unmet penalty ( $\mathcal{C}_{5}$ ) dominates cost (panel (c)). (d)–(f) AGH variations over $\Delta_{i}$ , $\epsilon_{i}$ , $\delta$ , $p_{k}^{c}$ .

Configuration	Feasible?	Cost ($)
AGH (all M1–M3)	Yes	89.88
w/o M1 (feasibility filter)	No (memory)	n/a
w/o M2 (cost-only ranking)	Yes	134.52 ( $+50\%$ )
w/o M3 (TP upgrade)	No (delay)	n/a

TABLE III: Ablation of the three constraint-aware mechanisms.

	$\sigma\!=\!0.01$	$\sigma\!=\!0.02$	$\sigma\!=\!0.03$	$\sigma\!=\!0.04$	$\sigma\!=\!0.05$
DM-24h	$4139$	$4144$	$4152$	$4164$	$4179$
GH-24h	$118$	$118$	$165$	$3192$	$9589$
GH-5min	$118$	$118$	$284$	$6277$	$10768$
AGH-24h	$108$	$109$	$109$	$109$	$1036$
AGH-5min	$108$	$109$	$109$	$109$	$1723$
AGH-5min vs AGH-24h	$0.0\%$	$0.0\%$	$0.0\%$	$0.0\%$	$\!+\!66.3\%$

TABLE IV: Rolling-horizon cost under a synthetic geometric-random-walk volatility study (

, mean over the 24 h horizon, 30 trials). AGH-24h is the cheapest method at every

\sigma$ .

Method	Mean cost/win ($)	Total 24 h ($)	Viol. (%)
AGH-5min	111.7	32,159.7	0.0
AGH-static	112.9	32,503.0	0.0
DM-5min	4,440.6	1,278,879.1	6.6
DM-static	4,800.6	1,382,565.0	6.6
GH-5min	46,357.1	13,350,852.8	36.1
GH-static	53,798.2	15,493,877.0	21.2

TABLE V: Rolling-horizon cost on the real Azure diurnal trace

Panel of Figure 11 — Fig. 6: Per-window cost on the Azure code trace, shown separately for (a) static method and (b) 5-minute rolling horizon.

Method	$(4,4,5)$	$(6,6,10)$	$(10,10,10)$	$(15,15,10)$	$(20,20,20)$
DM	0.39	4.2	13.04	601.12	$>600$
GH	$<0.01$	$<0.01$	0.3	0.5	0.9
AGH	0.0149	0.113	0.57	1.09	2.3

TABLE VI: Runtime scaling with problem size

(I,J,K)

, in seconds. DM exceeds the 600 s cap at

(15,15,10)

; GH stays under 1 s and AGH under 3 s on all tested instances (speedup lower bound

\geq\!260\times

(20,20,20)

LLM Serving

Large language model serving studies how to deliver model responses with low latency, high throughput, and efficient use of compute and memory.

See all 206 papers

End-to-End Dynamic Sparsity for Resource-Adaptive LLM Inference
Jun 26, 2026Yuhang Chen, Jinhao Duan, Ruichen Zhang +11LLM Inference Structured Sparsity
University of North Carolina at Chapel Hill · Meta AI
A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints
May 6, 2026Chengyi Nie, Nian Si, Zijie ZhouLLM Serving LLM Inference
Department of Electrical and Computer Engineering, Stony Brook University · Department of Industrial Engineering and Decision Analytics, HKUST
Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off
Oct 14, 2026Javad Mirzaei, Jeebak MitraParallel LLM Inference LLM Inference
Office of CTO Dell Technologies Inc. Canada

Jun 26, 2026J/K move · Enter open · S save

End-to-End Dynamic Sparsity for Resource-Adaptive LLM Inference

Yuhang Chen, Jinhao Duan, Ruichen Zhang, Mingfu Liang+10

University of North Carolina at Chapel Hill · Meta AI

LLM Inference Structured Sparsity LLM Inference Acceleration Inference-Time Compute Allocation Sparse Neural Networks Conditional Computation

Open paper AskPDF arXiv

Large Language Models (LLMs) inference is typically deployed under a static resource assumption, where models execute a fixed computational graph regardless of the runtime environment. However, real-world cloud infrastructure is inherently dynamic, characterized by fluctuating availability (e.g., spot instance preemption) and tiered Quality-of-Service requirements. In such volatile settings, static models are inflexible: they either crash under resource constraints or waste compute on redundant operations. To bridge this gap, we propose Learning to Allocate (L2A), an end-to-end framework for resource-adaptive inference. Unlike prior methods that condition only on input difficulty, we formulate inference as a constrained allocation problem conditioned on both the input and the runtime resource budget itself. We introduce lightweight, budget-conditioned and input-aware gating networks integrated into the LLM. These gates are trained via a unified objective that jointly optimizes task performance, logical consistency, and resource costs along three axes matching how real-world dynamics manifest: layer skipping for memory and depth pressure, head pruning for throughput contention, and reasoning-token reduction for latency tightening. This lets the model learn a budget-aware policy beyond input difficulty alone: it adaptively configures its computational footprint with respect to real-time resource dynamics, maximizing reasoning depth when resources permit while enforcing strict frugality when budgets tighten. A single L2A model traces the entire compute-accuracy Pareto frontier on Llama-3-8B and Qwen-3-4B: at up to 34% realized layer sparsity, it stays within 0.6% of the dense baseline on GSM8K, with the same gap holding zero-shot on out-of-distribution tasks, while every static or heuristic baseline requires a separately tuned model and still drops by 5-10% at comparable inference time.

Abstract

LLM Serving

Explore similar work

End-to-End Dynamic Sparsity for Resource-Adaptive LLM Inference

A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints

Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off