GPU spatial sharing co-locates instances with complementary demands, such as compute-bound prefill and memory-bound decode, promising cost-efficient multi-LLM serving. Prior systems partition compute and coordinate shared memory capacity, but do not model bandwidth contention among co-located instances of heterogeneous LLMs. This contention makes each instance's latency depend on the memory-access patterns of its co-runners, even when its own resource allocation remains unchanged. Drawing on extensive measurements, we find that the latency impact of bandwidth contention can be captured by a time-shared memory-access model with contended and uncontended periods, so the overall latency can be analytically estimated as a time-weighted average of the latency estimates for these periods. We propose JigSLO, a contention-aware multi-LLM serving system that manages memory resources and schedules each model's prefill and decode phases as separate instances, treating their co-location as a placement decision. Its contention-aware predictor estimates per-instance latency under any co-location from standalone profiles alone. Guided by these predictions, its placement algorithm jointly selects each instance's tensor parallelism, compute share, and batch configuration, pairing complementary memory-access patterns to maximize throughput within each model's latency targets. JigSLO's shared resource manager complements this on the capacity side, sharing weights, pooling KV cache, and transferring it between prefill and decode instances. Across three datasets covering chat, code generation, and long-context summarization, JigSLO outperforms state-of-the-art baselines by 2.3× in SLO attainment and 1.2× in throughput on average, matching their best attainment with half the GPUs on long-context workloads.
Figures & tables
Figure 1. Two placement candidates for 4 instances on 2 GPUs. Instances differ in resource demands and memory-access intensity (left). Both candidates satisfy the compute and memory-capacity budgets, but candidate 1 packs the two memory-intensive instances (A, B) on the same GPU while candidate 2 separates them. A bandwidth contention-oblivious (CO) estimate deems both candidates feasible. Contention-aware (CA) prediction instead marks candidate 1 as infeasible because it is predicted to violate its latency SLO, and selects candidate 2.
Figure 2. Contention severity and memory-subsystem pressure. (a) CDF of latency slowdown across the configuration sweep. (b) Slowdown of a target GEMM kernel against interferer.
Figure 3. Sensitivity of co-location tolerance to target configuration. For each target (prefill in (a), decode in (b)), we plot average slowdown over standalone execution at the same SM allocation, when co-located with a fixed compute-bound interferer (C. int.) (prefill of Llama-3.1-8B at ISL = 1024, BF16) or a fixed memory-bound interferer (M. int.) (decode of Llama-3.1-8B at BS = 2, BF16). SM allocations are averaged over 20–80%.
Figure 4. HBM bandwidth achieved (bytes read / time) by the streaming-copy kernel across SM allocations (20 – 100 %) under different buffer footprints (16 – 1024 MB). Bandwidth scales with SM share up to roughly 50% , then saturates at 87% of the 3.35 TB/s HBM peak bandwidth. We denote the saturation point as s * .
Figure 5. JigSLO architecture.
Figure 6. The KV slab manager enables two co-located models with heterogeneous block sizes (Model 1: 64 KB, Model 2: 32 KB) to share a single physical KV tensor managed as fixed-size 128 KB slabs. The slab table tracks each slab’s block size and status. At runtime, each instance sees block IDs composed from (slab ID, local block ID), allowing the same physical memory to back blocks of different sizes for different models.
Component
Specification
GPU
8 × H100 SXM5 80 GB
NVLink
4.0 / NVSwitch
NICs
Mellanox ConnectX (RDMA)
CPU
2 × Xeon 8468 (96 cores)
DRAM
2 TB DDR5-4800
Table 1. Per-node testbed configuration. The two-node cluster has 16 H100 GPUs in total, interconnected by 400 Gbps NDR InfiniBand.
Azure-Code
ShareGPT
ArXiv Sum.
P90 input length
5187
405
16153
P90 output length
55
665
420
TTFT/token
1.5 ms
3.0 ms
1.0 ms
TTFT SLO
7.8 s
1.2 s
16 s
Table 2. Per-dataset SLO targets. TTFT SLO is derived from P90 input length × TTFT/token. TPOT SLO is 50 ms.
Figure 7. End - to - end performance of all baselines across three datasets. Columns represent (from left to right): SLO attainment rate (%), request throughput (req/s), P90 TTFT (s), and P90 TPOT (ms) tail latency.
System
Placement
P/D
KV cache
vanilla-vLLM ( Kwon et al., 2023 )
contention-oblivious
✓
static
MuxServe ⋆ ( Duan et al., 2024 )
greedy (compute)
✓
kvslab
Prism ⋆ ( Yu et al., 2026 )
greedy (memory)
✗
kvcached
JigSLO
contention-aware
✓
kvslab
Table 3. Summary of each baseline. Each column (starting from left to right) denotes the placement strategy, whether prefill/decode disaggregation is applied, and the KV cache management scheme.
Figure 8. Per-request TTFT and TPOT for three models on Azure-Code at aggregate request rates of 5 req/s (a) and 15 req/s (b). At 15 req/s, the baselines exhibit TTFT or TPOT SLO violations, while JigSLO satisfies both SLOs. In (b), MuxServe ⋆ ’s TTFT points for Llama-3.1-8B and Llama-3.1-70B are omitted because all exceed the plotted limit of 16 s.
Figure 9. SLO attainments under GPU scaling (left) and model scaling (right) across three datasets. JigSLO sustains high SLO attainment in all settings, while MuxServe ⋆ and Prism ⋆ degrade at smaller GPU counts and larger model counts.
Figure 10. Ablation of JigSLO’s kvslab and contention - aware predictor across the three datasets. The transition vanilla-vLLM → JigSLO (CO) isolates kvslab . JigSLO (CO) → JigSLO isolates the contention-aware predictor.
Figure 11. kvslab versus vLLM static KV partitioning deploying Qwen3-8B (15 req/s) and Llama-3.1-8B (5 req/s) on a single H100 GPU serving synthetic workload (input 1024, output 512). KV cache pool usage (Top-left) and queue length (bottom-left) timeline. Right: aggregate goodput and average TTFT per model.
Figure 12. Comparison between our kvslab and kvcached deploying Qwen3-8B and Llama-3.1-8B on a single H100 GPU for serving ShareGPT dataset. kvcached exhibits higher TTFT and TPOT than kvslab as the request rate increases, while kvslab remains close to the vLLM baseline.
Standalone
Contended
Prefill
Decode
Prefill
Decode
Model
BF16
FP8
BF16
FP8
BF16
FP8
BF16
FP8
Llama-3.1-8B
8.43
9.31
5.40
6.53
14.79
14.46
12.56
12.45
Qwen3-32B
7.10
8.63
6.97
7.61
13.16
13.86
12.26
11.99
Llama-3.1-70B
5.75
8.36
6.82
7.67
11.57
12.42
11.04
11.52
Table 4. Latency prediction MAPE (%) for standalone and contended execution, across precisions and phases.
Serving large language model (LLM) inference in cloud environments requires jointly optimizing model selection, GPU provisioning, parallelism configuration, and workload routing under latency, accuracy, memory, and budget constraints. While mixed-integer linear programming (MILP) can model this problem, its computational cost limits frequent re-optimization under demand variability. Existing heuristics often optimize individual components separately and may become infeasible when system-wide constraints are enforced. This paper presents a scalable framework for SLO-constrained LLM inference. We formulate the problem as an MILP with a two-phase delay model capturing both prefill and autoregressive decoding under tensor and pipeline parallelism. To solve it efficiently, we develop two constraint-aware heuristics: a Greedy Heuristic (GH) and an Adaptive Greedy Heuristic (AGH). AGH extends GH through multi-start construction, local search, and GPU consolidation. Both methods maintain feasibility through parallelism-aware filtering, cost-based ranking, and adaptive parallelism scaling. Experiments based on the Azure LLM Inference Trace show that GH generates feasible solutions within one second, while AGH achieves near-optimal performance within three seconds and scales to large instances where exact solvers fail to converge. Under out-of-sample stress with up to 1.5x delay and accuracy inflation, AGH degrades gracefully through provisioned headroom, yielding substantially lower cost and SLO violations than cost-minimal MILP solutions. Across synthetic and real Azure workloads, AGH maintains SLO compliance at significantly lower cost than exact MILP solutions. These results demonstrate that high-quality allocations provide substantial robustness to demand variability while enabling rapid adaptation to workload changes.
Jiaming Cheng, Duong Tung Nguyen
The Ira A. Fulton Schools of Engineering, Arizona State University, Tempe, AZ, USA.
Large Language Model (LLM) serving systems must balance latency and throughput while efficiently utilizing distributed GPU resources. This challenge arises because inference consists of two fundamentally different phases: prefill, which determines time-to-first-token (TTFT), and decode, which determines inter-token latency (ITL). The use of tensor parallelism (TP), pipeline parallelism (PP), and hybrid configurations exposes different trade-offs between these latency dimensions, making the optimal parallelism configuration highly dependent on workload characteristics, latency requirements, and context length. Moreover, despite the widespread adoption of distributed inference, there is still limited understanding of how parallelism configurations should be selected under varying workload conditions and service-level objectives (SLO). In this paper, we develop a systematic framework for characterizing LLM inference parallelism via throughput-latency trade-offs, SLO-aware serving, and long-context scalability. We introduce the notion of parallelism operating regions by constructing TTFT-ITL feasible sets and Pareto frontiers across different workload compositions, thereby revealing a systematic transition from TP-dominant to PP-dominant configurations as workloads become increasingly prefill-heavy. We further formulate SLO-aware parallelism selection using goodput, showing that stringent TTFT requirements favor PP-dominated configurations, whereas relaxed latency constraints progressively shift the optimality toward TP-dominated configurations with higher serving capacity. Finally, we demonstrate that increasing context length fundamentally changes the relative advantages of parallelism configurations, with PP-dominant configurations exhibiting more favorable scalability and SLO attainment in the long-context regime.
Modern deployments of Large Language Models (LLMs) increasingly require serving multiple models with diverse architectures, sizes, and specialization on shared, heterogeneous hardware. This setting introduces new challenges for resource allocation, dispatching, and scheduling, particularly under GPU memory constraints where partial CPU-GPU offloading and preemption become necessary. While existing systems primarily optimize throughput for a single model, comparatively little work addresses multi-model scheduling under these conditions. In this paper, we present an empirical study of how different LLMs behave across hardware platforms, focusing on the performance implications of layer offloading and preemption. We show that offloading leads to strongly non-linear and model-dependent degradation in decode throughput, with smaller models exhibiting sharper sensitivity to reduced GPU residency. We further demonstrate that preemption incurs substantial overhead, largely dominated by model state reload rather than key-value cache transfer, and that this cost varies significantly across models and hardware platforms. Additionally, we highlight the role of sequence length and interconnect bandwidth in amplifying data movement and execution inefficiencies. Based on these findings, we identify a set of key features that future schedulers must consider, including model-specific offloading sensitivity, workload characteristics, and the cost structure of preemption and data transfer. These insights provide guidance for the design of next-generation LLM serving systems capable of efficiently managing heterogeneous, multi-model workloads with hybrid CPU-GPU execution.