cs.ARAug 25, 2026

Simthesizer: An Agent-Driven Simulation Framework for LLM Serving Systems

Authors: Wonung Kim, Hyunmin Choi, Minsu Kim, Jaehong Cho, Yeongwook Kim, Jongse Park

Organizations: KAIST Daejeon, Republic of Korea

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

System-level simulation is an essential tool for exploring the rapidly expanding design space of LLM serving systems, where real deployments remain costly and often infeasible. However, modern LLM serving now evolves faster than human-driven simulator development can track, and emerging workloads and mechanisms, from agentic workflows to disaggregated serving, no longer fit the monolithic simulation pipeline that existing simulators assume. Each new mechanism therefore demands an invasive rewrite, leaving a widening development gap between deployed serving systems and the simulators that model them. To close this gap, we present Simthesizer, a framework that realizes agent-driven simulator development. Simthesizer introduces a composable simulator infrastructure that uniformly expresses the complete serving workflow, including the control decisions that coordinate it, and realizes it as a unified dynamic graph in Simthesizer simulator. Synthesizer agent, a harnessed coding agent, then lowers natural-language feature requests onto this abstraction under simulator-specific guardrails and fidelity validation, evolving one shared simulator instead of building a new one for every feature. Under the same coding agent and harnesses, extensions built on Simthesizer follow a vLLM-based real system with 2.51% average throughput error, versus 6.03% for extensions built on existing simulators. On identical workloads, Simthesizer also simulates up to 284.96x and 23.19x faster than two state-of-the-art simulators, LLMServingSim2.0 and Vidur, respectively.

Figures & tables

Fig. 1: (a) Existing LLM serving simulators require manually implementing new serving mechanisms as they emerge, leading to low-level modifications, while (b) Simthesizer framework instead composes the complete serving workflow from uniform elements of a unified dynamic DAG, which is automatically lowered into executable simulation through agent-driven development.
Fig. 2: Shifts in modern LLM serving: (a) fast evolution and (b) non-monolithic serving.
Fig. 3: Overview of the Simthesizer framework.
InterfaceDescription
add_logical_node (operation, state) -> node add_logical_node_at (operation, state, at) -> nodeInsert a logical node carrying a policy operation and its state. The _at variant seeds an explicit start time, e.g., for request arrival.
add_compute_node (layer, batch) -> nodeInsert a compute node tagged with a semantic layer descriptor and the scheduled batch state.
add_network_node (src, dst, bytes) -> nodeInsert a point-to-point transfer between two network devices. Its timing model resolves latency, including multi-hop routing when configured.
add_edge (parent, child)Declare a precedence constraint. A child becomes runnable only after all of its predecessors complete.
pop_edges (node) -> list[node]Detach the current node’s outgoing edges so the caller can splice a new subgraph before the original successor.
TABLE I: Dynamic DAG modification APIs.
Fig. 4: Architecture of Simthesizer simulator.
Fig. 6: Agent-driven simulator extension process.
BehaviorImplementation map
Target model verificationRoute WW through the existing request representation to the model runner and compute simulator.
Committed request stateAssign MM to scheduler-managed request and KV state, exposing only the committed prefix to the existing cache interface.
Speculation round executionReuse the common DAG scheduling and execution path without adding a mechanism-specific event loop.
TABLE II: Implementation map for speculative decoding.
Fig. 8: Comparison of throughput between unmodified Simthesizer (without any agent-synthesized extension) and a GPU-based serving system running vLLM under agentic workloads: (a) Llama3.1 dense model on mini-SWE-bench, (b) Llama3.1 dense model on tau-bench, (c) Qwen3 MoE model on mini-SWE-bench, and (d) Qwen3 MoE model on tau-bench.
Fig. 9: Throughput comparison of Simthesizer, LLMServingSim2.0, and Vidur each independently extended by the Synthesizer agent under a shared harness and coding agent across three feature-extension tasks: (a) KV cache quantization, (b) speculative decoding, and (c) hybrid Mamba model support. Each panel thus compares three distinct simulators extended with the same feature; the same convention applies to Fig. 10 and Fig. 11 .
TABLE III: Error rates of Simthesizer performance metrics relative to the real serving system across two workloads and two models.
TABLE IV: Error rates of Simthesizer with and without the Synthesizer agent harness, relative to the real vLLM-based serving system.
Fig. 10: Error rate comparison of (a) mean and (b) median performance metrics for Simthesizer, LLMServingSim2.0, and Vidur across three feature-extension tasks under a shared harness and coding agent.
Fig. 11: Comparison of simulation time for Simthesizer, LLMServingSim2.0, and Vidur across three feature-extension tasks.
Fig. 12: Generation throughput comparison between vLLM and the simulators produced by Codex and Claude Code for speculative decoding.

Explore similar work

CardsList

  1. AGENTSERVESIM: A Hardware-aware Simulator for Multi-Turn LLM Agent Serving

    Jun 8, 2026Rakibul Hasan Rajib, Mengxin Zheng, Qian LouLLM ServingKV-Cache Management

    University of Central Florida

  2. Frontier: Towards Comprehensive and Accurate LLM Inference Simulation

    May 20, 2026Yicheng Feng, Xin Tan, Yangtao Deng +3LLM ServingDisaggregated LLM Serving

    The Chinese University of Hong Kong · Anuttacon · StepFun

  3. VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?

    May 7, 2026Keisuke Kamahori, Shihang Li, Simon Peter +1Multi-Agent LLM SystemsLLM Serving

    University of Washington