Extracting actionable insights from complex value stream map simulations can be challenging, time-consuming, and error-prone. Recent advances in large language models offer new avenues to support users with this task. While existing approaches excel at processing raw data to gain information, they are structurally unfit to pick up on subtle situational differences needed to distinguish similar data sources in this domain. To address this issue, we propose a decoupled, two-step agentic architecture. By separating orchestration from data analysis, the system leverages progressive data discovery infused with domain expert knowledge. This architecture allows the orchestration to intelligently select data sources and perform multi-hop reasoning across data structures while maintaining a slim internal context. Results from multiple state-of-the-art large language models demonstrate the framework's viability: with top-tier models achieving accuracies of up to 86% and demonstrating high robustness across evaluation runs.
Figures & tables
Figure 1: A simple Example VSM.
Figure 2: Overview of the decoupled agentic architecture, showcasing four steps of the under-supplied supermarkets example. Red marks tool calls. Green marks the returned information.
Evaluator
Mean Rating
MAE ↓
Pearson r ↑
Expert Consensus
76.67
10.67
80.70
Qwen3-30B-A3B-Thinking-2507
72.67
15.67
69.92
GPT-4.1
73.33
7.33
89.99
GPT-5-mini
77.33
11
78.51
GPT-5.4
67.33
12.67
79.77
Table 1: LLM as a judge validation performed on a sub collection of the evaluation dataset. Ratings were multiplied by 100 to set their interval between 0 and 100. Ratings of the for evaluation runs of LLM judges were averaged. Mean rating should be as close as possible to the human base line. MAE should be as low as possible. Pearson r should be as high as possible. Bold marks best LLM metric.
Model
Ministral-3-14B-Reasoning-2512
Qwen3-30B-A3B-Thinking-2507
GLM-5
Claude-Opus 4.6
AVG ± SD Rating
57.34 ± 6.64
72.66 ± 2.1
83.88 ± 3.92
85.96 ± 0.86
Sample Token AVG
39,819
43,482
27,580
38,594
Table 2: Agent evaluation across 4 evaluation runs on the full evaluation dataset. The score in rating row represent the average rating across all samples followed by the standard divination (SD) across evaluation runs. Sample Token AVG denotes the average amount of tokens used to process a single triple.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
VSM name
question
answer
pharma
Which process is the bottleneck in the overall system?
The process bottling is with a utilization rate of 59.75% the bottleneck in the overall system.
automotive-01
Which process has the highest overall throughput?
The Outer_Stamping station has the highest total throughput with 27094 pieces.
spray-02
How long and how often does a storage tank/buffer run empty during the simulation time? (referring to the minimum storage level in parts)
The following warehouses/supermarkets ran out of stock during the simulation (10 days): Warehouse, Warehouse_02, and Warehouse_03 ran empty from the start and remained so. Warehouse_01 was filled on March 4, 2021, and ran empty again from March 8, 2021. Warehouse_04 Component Base_Lid_Skids ran out of stock on March 6, 2021, and then gradually refilled. Component KSWg ran empty from March 4, 2021, for the remainder of the simulation.
Appendix
Table 3: From German to English translated data examples used in this work.
Although large language models (LLMs) have demonstrated remarkable capabilities, their reliance on cloud-scale infrastructure poses fundamental challenges for deployment in agentic pipelines, including latency, privacy, connectivity, and substantial computational cost. Small language models (SLMs) offer a compelling alternative: recent studies suggest that many repetitive and narrowly scoped subtasks in agentic workloads may be better served by specialized SLMs than by monolithic LLMs. However, the limited capacity and context windows of SLMs can constrain long-horizon reasoning and interaction-heavy orchestration strategies such as iterative verification and debate. This motivates a complementary, non-interactive paradigm in which heterogeneous SLMs independently generate candidate solutions and a router orchestrates their cached samples without further model interaction. To further understand the mechanisms of such orchestration, we introduce OrchSLM, a routing framework that unifies existing non-interactive orchestration methods and exposes their underlying design choices as controllable parameters. Using OrchSLM as a systematic probe, we reveal how orchestration behavior emerges from diverse knobs, including the task structure, model-pool composition, and multi-agent consensus.
System-level simulation is an essential tool for exploring the rapidly expanding design space of LLM serving systems, where real deployments remain costly and often infeasible. However, modern LLM serving now evolves faster than human-driven simulator development can track, and emerging workloads and mechanisms, from agentic workflows to disaggregated serving, no longer fit the monolithic simulation pipeline that existing simulators assume. Each new mechanism therefore demands an invasive rewrite, leaving a widening development gap between deployed serving systems and the simulators that model them. To close this gap, we present Simthesizer, a framework that realizes agent-driven simulator development. Simthesizer introduces a composable simulator infrastructure that uniformly expresses the complete serving workflow, including the control decisions that coordinate it, and realizes it as a unified dynamic graph in Simthesizer simulator. Synthesizer agent, a harnessed coding agent, then lowers natural-language feature requests onto this abstraction under simulator-specific guardrails and fidelity validation, evolving one shared simulator instead of building a new one for every feature. Under the same coding agent and harnesses, extensions built on Simthesizer follow a vLLM-based real system with 2.51% average throughput error, versus 6.03% for extensions built on existing simulators. On identical workloads, Simthesizer also simulates up to 284.96x and 23.19x faster than two state-of-the-art simulators, LLMServingSim2.0 and Vidur, respectively.
The integration of Large Language Models (LLMs) with microscopic traffic simulation offers a promising path toward autonomous urban planning and intelligent transportation analysis. However, existing monolithic agent architectures often struggle with the complexity of end-to-end simulation workflows, leading to reasoning failures, parameter inconsistency, and a lack of systematic state management. This paper proposes a novel multi-agent collaborative framework designed to automate the entire lifecycle of traffic simulation in SUMO (Simulation of Urban Mobility). Our approach decouples the simulation pipeline into specialized roles, including Planner, Builder, Demand, Runner, and Analyst, coordinated by a high-level reasoning engine. We introduce a state-persistent Orchestrator leveraging the Model Context Protocol (MCP) to ensure seamless data handover and environmental consistency across distributed agent actions. This architecture enables a robust closed-loop refinement process, where simulation outcomes are iteratively analyzed and optimized to satisfy user-defined Key Performance Indicators (KPIs). Experimental results through role ablation studies demonstrate that the proposed multi-agent framework significantly enhances task success rates and parameter accuracy compared to single-agent baselines. Furthermore, case studies on real-world network extraction and traffic optimization highlight the system's capability to bridge the gap between high-level natural language intent and low-level simulation execution.
Shuyang Li, Ruimin Ke
Civil and Environmental Engineering, School of Engineering, Rensselaer Polytechnic Institute, Troy, United States.