cs.DCSep 8, 2025

JigSLO: Solving the Resource Puzzle for Efficient Multi-LLM Serving

Authors: Kyungmin Bin, Seungbeom Choi, Jimyoung Son, Jieun Choi, Daseul Bae, Daehyeon Baek, Kihyo Moon, Minsung Jang, 

Organizations: Cloud Research Team, Samsung SDS

Abstract

GPU spatial sharing co-locates instances with complementary demands, such as compute-bound prefill and memory-bound decode, promising cost-efficient multi-LLM serving. Prior systems partition compute and coordinate shared memory capacity, but do not model bandwidth contention among co-located instances of heterogeneous LLMs. This contention makes each instance's latency depend on the memory-access patterns of its co-runners, even when its own resource allocation remains unchanged. Drawing on extensive measurements, we find that the latency impact of bandwidth contention can be captured by a time-shared memory-access model with contended and uncontended periods, so the overall latency can be analytically estimated as a time-weighted average of the latency estimates for these periods. We propose JigSLO, a contention-aware multi-LLM serving system that manages memory resources and schedules each model's prefill and decode phases as separate instances, treating their co-location as a placement decision. Its contention-aware predictor estimates per-instance latency under any co-location from standalone profiles alone. Guided by these predictions, its placement algorithm jointly selects each instance's tensor parallelism, compute share, and batch configuration, pairing complementary memory-access patterns to maximize throughput within each model's latency targets. JigSLO's shared resource manager complements this on the capacity side, sharing weights, pooling KV cache, and transferring it between prefill and decode instances. Across three datasets covering chat, code generation, and long-context summarization, JigSLO outperforms state-of-the-art baselines by 2.3×2.3\times in SLO attainment and 1.2×1.2\times in throughput on average, matching their best attainment with half the GPUs on long-context workloads.

Figures & tables

Explore similar work

CardsList
  1. Scalable Joint Resource Allocation for SLO-Constrained LLM Inference in Heterogeneous GPU Clouds

    Apr 8, 2026Jiaming Cheng, Duong Tung NguyenLLM ServingLLM Inference Scheduling

  2. Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off

    Oct 14, 2026Javad Mirzaei, Jeebak MitraParallel LLM InferenceLLM Inference

  3. Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption

    May 19, 2026Mert Yildiz, Pietro Spadaccino, Alexey Rolich +2LLM Inference EfficiencyLLM Serving