cs.AINov 6, 2026

SIRIS: Can LLMs Tell When Simulation Errors Matter in AI-Driven Systems Research

Authors: Jinwoo Hwang, Sukmin Cho, Tao Zhang, Soheil Abbasloo, Jongse Park, Peng Cheng

Organizations: KAIST · Microsoft Research Asia

Abstract

LLM agents increasingly drive systems research and rely on simulators to evaluate candidate designs at scale. Simulators are imperfect by design: their modeling limitations can distort predictions yet leave the recommended design correct, or change which design should be selected. We introduce Siris to test whether LLMs can tell these cases apart, judging from their systems understanding when a limitation invalidates the simulator's recommendation. By perturbing a simulator, Siris constructs workload pairs with the same candidates, limitation, and simulator recommendation. The perturbation changes predictions on both workloads, yet a reference simulator supports the recommendation on only one. Evaluating thirteen frontier LLMs on LLM serving design problems, we find that identifying a limitation does not ensure a correct choice, and even the strongest models sometimes abandon valid recommendations. Self-correction through a review turn helps these models restore valid choices but introduces new errors in several others. These findings motivate evaluating systems understanding through design decisions under imperfect simulators.

Figures & tables

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Simthesizer: An Agent-Driven Simulation Framework for LLM Serving Systems

    Aug 25, 2026Wonung Kim, Hyunmin Choi, Minsu Kim +3Large Language Model-Based SimulationLLM Serving

  2. Simulate, Reason, Decide: Scientific Reasoning with LLMs for Simulation-Driven Decision Making

    Jun 3, 2026Yuhan Yang, Ruipu Li, Alexander RodríguezAI for ScienceNeuro-Symbolic Reasoning

  3. Mind the Sim2Real Gap in User Simulation for Agentic Tasks

    Mar 11, 2026Xuhui Zhou, Weiwei Sun, Qianou Ma +8User SimulationSim-to-Real Transfer