cs.AINov 10, 2026

LM-SimBench: A Unified Framework, Dataset, and Benchmark for Large Language Model Training Simulation

Authors: Yu Dong, Hong Li, Qiang Sun, Xu Zhou, Jinbiao Huo, Qixiu Cheng, Zhiyuan Liu

Organizations: Southeast University · The Hong Kong Polytechnic University

Abstract

Evaluating candidate parallelization strategies through distributed training runs makes strategy selection prohibitively expensive at scale. Performance simulation offers a lower cost alternative, yet controlled comparisons among existing simulators remain difficult because they differ in the training stages they model and the workloads on which they are evaluated. We introduce LM-SimBench, a unified framework, profiling dataset, and benchmark for evaluating LLM training simulators. The framework decomposes simulation into functional stages from model compilation to schedule level aggregation, connecting execution representations and cost estimators to measurable evaluation targets. Following this framework, we profile 23 model settings from 10 dense and Mixture-of-Experts (MoE) LLM families on an Ascend Neural Processing Unit (NPU) cluster. The resulting dataset contains 348 feasible configurations that jointly vary five parallelism dimensions and activation recomputation, with six views spanning configuration, rank, operator, event, and kernel granularity. Evaluations of representative training simulators demonstrate how LM-SimBench supports comparisons of prediction accuracy, assessment of configuration selection, and diagnosis of prediction errors. Matched GPU-NPU experiments validate the cross-platform applicability of the collection and evaluation protocol and demonstrate NPU-guided GPU configuration selection.

Figures & tables

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Charon: A Unified and Fine-Grained Simulator for Large-Scale LLM Training and Inference

    May 16, 2026Mengtian Yang, Zhekun Zhang, Mingheng Wu +3Language Model Performance PredictionLLM Inference

  2. A Few GPUs, A Whole Lotta Scale: Faithful LLM Training Emulation with PrismLLM

    May 15, 2026Shaoke Xi, ChonLam Lao, Boyi Jia +11High-Performance ComputingDistributed Training

  3. Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulation

    May 8, 2026Joon Ha Kim, Geon-Woo Kim, Anoop Rachakonda +1LLM Inference EfficiencyLLM Inference