cs.DCFeb 26, 2026

LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructure

Authors: Jaehong Cho, Hyunmin Choi, Guseul Heo, Jongse Park

Organizations: School of Computing KAIST Daejeon, South Korea

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language model (LLM) serving infrastructures are undergoing a shift toward heterogeneity and disaggregation. Modern deployments increasingly integrate diverse accelerators and near-memory processing technologies, introducing significant hardware heterogeneity, while system software increasingly separates computation, memory, and model components across distributed resources to improve scalability and efficiency. As a result, LLM serving performance is no longer determined by hardware or software choices in isolation, but by their runtime interaction through scheduling, data movement, and interconnect behavior. However, understanding these interactions remains challenging, as existing simulators lack the ability to jointly model heterogeneous hardware and disaggregated serving techniques within a unified, runtime-driven framework. This paper presents LLMServingSim 2.0, a unified system-level simulator designed to make runtime-driven hardware-software interactions in heterogeneous and disaggregated LLM serving infrastructures explicit and analyzable. LLMServingSim 2.0 embeds serving decisions and hardware behavior into a single runtime loop, enabling interaction-aware modeling of batching, routing, offloading, memory, and power. The simulator supports extensible integration of emerging accelerators and memory systems through profile-based modeling, while capturing dynamic serving behavior and system-level effects. We validate LLMServingSim 2.0 against real deployments, showing that it reproduces key performance, memory, and power metrics with an average error of 0.95%, while maintaining simulation times of around 10 minutes even for complex configurations. These results demonstrate that LLMServingSim 2.0 provides a practical bridge between hardware innovation and serving-system design, enabling systematic exploration and co-design for next-generation LLM serving infrastructures.

Figures & tables

SimulatorDissagg.ParallelismModeling
 PDAFHTPP/TPDPEPPAPCEOPM
LLMServingSim [ 25 ]×\times✓\checkmark×\times✓\checkmark×\times×\times✓\checkmark×\times×\times×\times
Vidur [ 26 ]×\times×\times×\times✓\checkmark✓\checkmark×\times△\triangle×\times×\times×\times
APEX [ 27 ]×\times×\times×\times✓\checkmark✓\checkmark✓\checkmark×\times×\times×\times✓\checkmark
TokenSim [ 28 ]✓\checkmark×\times×\times✓\checkmark△\triangle×\times✓\checkmark△\triangle×\times×\times
Ours✓\checkmark✓\checkmark✓\checkmark✓\checkmark✓\checkmark✓\checkmark✓\checkmark✓\checkmark✓\checkmark✓\checkmark
TABLE I: Comparison of LLM serving simulators
Fig. 1: Overview of LLMServingSim 2.0.
Fig. 2: Model Serving Group (MSG) architecture. The MSG includes a customizable device pool enabling heterogeneous hardware composition and supports batching, operation mapping, and scheduling for LLM execution.
Fig. 3: Model serving group-level usage example.
Fig. 4: System-level usage example.
Fig. 5: Comparison of throughput over time between real GPU systems (RTX A6000, H100) and LLMServingSim 2.0 using the vLLM framework.
Fig. 6: Comparison of power consumption and energy breakdown between the real RTX A6000 system and LLMServingSim 2.0.
Fig. 7: Comparison of memory usage and prefix hit rate between the real RTX A6000 system and LLMServingSim 2.0.
Fig. 8: Comparison with existing LLM serving simulators in terms of throughput (TPS), TTFT, TPOT, and simulation time. (a) Normalized to vLLM results.
Fig. 9: Comparison of throughput over time and latency metrics between real TPU system and LLMServingSim 2.0 using vLLM framework.
Fig. 10: Performance comparison between a GPU-only system and a GPU+PIM system, including sub-batch interleaving (SBI).
More in this topic

LLM Serving

Large language model serving studies how to deliver model responses with low latency, high throughput, and efficient use of compute and memory.

See all 206 papers

Explore similar work

CardsList
May 20, 2026

Frontier: Towards Comprehensive and Accurate LLM Inference Simulation

Modern LLM serving is no longer homogeneous or monolithic. Production systems now combine disaggregated execution, complex parallelism, runtime optimizations, and stateful workloads such as reasoning, agents, and RL rollouts. Simulation is attractive for exploring this growing design space, yet existing simulators lack the architectural completeness and decision-grade fidelity it demands. Their monolithic-replica abstractions are ill-suited to disaggregated serving, while average-case analytical proxies can distort SLA predictions and even reverse optimization conclusions. We present Frontier, a discrete-event simulator for modern LLM inference serving. Frontier features a disaggregated abstraction. It captures the structure and dynamics of modern serving systems by modeling co-location, Prefill-Decode Disaggregation (PDD), and Attention-FFN Disaggregation (AFD) with role-specific cluster workers, incorporating key runtime optimizations (e.g., CUDA Graphs, speculative decoding) within the scheduler-batch-engine loop, and supporting stateful requests for emerging workloads. It further provides accurate and generalizable predictions of computation, communication, and memory costs across diverse serving scenarios with complex workload compositions. On 16-H800 GPU testbed, Frontier achieves an average throughput error below 4%. Compared with state-of-the-art simulators, it reduces end-to-end latency error from 44.9% to 6.4% under co-location and from 51.7% to 2.6% under disaggregation. It scales to over 1K GPUs on commodity CPUs and enables new use cases such as SLA-dependent Pareto frontier exploration, heterogeneous disaggregated allocation, agentic reasoning scheduling validation, and RL post-training reconfiguration. We release Frontier at https://github.com/NetX-lab/Frontier.

Yicheng Feng, Xin Tan, Yangtao Deng +3

The Chinese University of Hong Kong · Anuttacon · StepFun

LLM ServingDisaggregated LLM ServingLLM Inference SchedulingLLM Inference ServingDistributed Inference
AskPDF
Jun 8, 2026

AGENTSERVESIM: A Hardware-aware Simulator for Multi-Turn LLM Agent Serving

Multi-turn LLM agents interleave model calls with external tool invocations, shifting serving from stateless request processing to stateful program execution. Serving these workloads requires scheduling, KV-cache management, and routing policies that use program-level context, including turn dependencies, tool-induced gaps, and reusable KV state. Evaluating such policies directly on real systems is costly, since each design point may require dedicated accelerator time across arrival rates, model scales, serving-instance counts, and memory hierarchies. Simulation offers a scalable alternative, but existing LLM serving simulators target stateless request-level workloads and therefore omit the core dynamics of agent serving: multi-turn program execution, cross-turn cache locality, and KV-cache residency during tool gaps. We present AGENTSERVESIM, a hardware-aware simulator for multi-turn LLM agent serving. AGENTSERVESIM evaluates serving policies at program granularity through composable modules: a Program Orchestrator preserves program identity and turn order, a Tool Simulator materializes tool-induced gaps, a Session-Aware Router maintains program-to-instance affinity for cache-aware dispatch, and a KV Residency Model tracks policy-defined KV placement across HBM, host DRAM/CXL, and eviction. Across real serving deployments and hardware configurations, AGENTSERVESIM reproduces real-system behavior within 6% error across key performance metrics while running entirely on commodity CPUs. These results show that AGENTSERVESIM enables controlled, repeatable exploration of agent-serving policies without requiring exhaustive deployment on costly accelerators.

Rakibul Hasan Rajib, Mengxin Zheng, Qian Lou

University of Central Florida

LLM ServingKV-Cache ManagementLLM Inference ServingLLM Agent Serving
AskPDF
Oct 7, 2025

Systematic Exploration of Multi-core Architectures for Efficient LLM Serving using WaferAI-SIM

With the widespread adoption of Large Language Models (LLMs), the demand for high-performance LLM inference services continues to grow. Multi-core AI accelerators, such as Groq, Graphcore IPU, and Cerebras WSE, provide promising platforms for LLM serving, but their distributed memory systems require careful coordination between hardware configuration and serving policies. Otherwise, mismatched tensor partitioning, data placement, and memory management can substantially underutilize compute and communication resources. To address these challenges, we present WaferAI-SIM, a multi-level simulation framework that combines transaction-level simulation with an analytical performance model. WaferAI-SIM enables simulator-driven co-design of LLM serving strategies and multi-core accelerator architectures, targeting the early design stage in which emerging platforms are not yet broadly available for empirical serving studies. It captures how LLM serving policies interact with compute-core count, memory hierarchy, and interconnect topology, enabling architecture-aware exploration beyond GPU-centric assumptions. We evaluate representative LLMs across a range of chip configurations and serving scenarios. Across the evaluation, WaferAI-SIM reports 1.32×\times--6.03×\times latency improvements, where the lower endpoint comes from ring-based placement at TP=16 over the placement baselines, and the upper endpoint comes from K-dimension TP over MN-dimension TP for Qwen3_4B at TP=4 with sequence length 256. For LLM serving, our findings provide guidance for co-designing hardware architectures and serving strategies for multi-core AI accelerators across diverse LLM workloads.

Tianhao Zhu, Dahu Feng, Erhu Feng +1

Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University · Department of Precision Instrument, Tsinghua University

AI Accelerator InferenceLLM ServingTensor Parallelism
AskPDF