LM-SimBench: A Unified Framework, Dataset, and Benchmark for Large Language Model Training Simulation
Organizations: Southeast University · The Hong Kong Polytechnic University
Abstract
Evaluating candidate parallelization strategies through distributed training runs makes strategy selection prohibitively expensive at scale. Performance simulation offers a lower cost alternative, yet controlled comparisons among existing simulators remain difficult because they differ in the training stages they model and the workloads on which they are evaluated. We introduce LM-SimBench, a unified framework, profiling dataset, and benchmark for evaluating LLM training simulators. The framework decomposes simulation into functional stages from model compilation to schedule level aggregation, connecting execution representations and cost estimators to measurable evaluation targets. Following this framework, we profile 23 model settings from 10 dense and Mixture-of-Experts (MoE) LLM families on an Ascend Neural Processing Unit (NPU) cluster. The resulting dataset contains 348 feasible configurations that jointly vary five parallelism dimensions and activation recomputation, with six views spanning configuration, rank, operator, event, and kernel granularity. Evaluations of representative training simulators demonstrate how LM-SimBench supports comparisons of prediction accuracy, assessment of configuration selection, and diagnosis of prediction errors. Matched GPU-NPU experiments validate the cross-platform applicability of the collection and evaluation protocol and demonstrate NPU-guided GPU configuration selection.
Figures & tables
| Simulator | Cost estimation | Time | Memory | Calibration tool |
| Calculon [ 18 ] | Analytical | |||
| SimuMax [ 26 ] | Analytical | |||
| Proteus [ 12 ] | Profiling-based | |||
| vTrain [ 6 ] | Profiling-based | |||
| Phantora [ 30 ] | Profiling-based | |||
| ASTRA-sim 2.0 [ 41 , 5 ] | Hybrid |
| Artifact | Type / workloads | Released signals or specifications | Parallelism coverage | Hardware |
|---|---|---|---|---|
| NeuSight [ 21 ] | Profiling dataset; 5 dense + 1 MoE Transformer | Operator and end-to-end latency measurements | DP/TP/PP sweeps | NVIDIA / AMD GPUs |
| AcmeTrace [ 17 ] | Cluster trace; production LLM jobs | Scheduler logs, resource monitoring, timing, and failures | Job-level observations | A100 GPUs |
| Alibaba GPU traces [ 1 ] | Cluster trace; production GPU jobs | Job-level resource and timing records | Job-level observations | GPUs |
| Chakra [ 36 ] | Execution-trace format | Graph schema and example traces; collection and replay tools | Representation schema | Hardware-agnostic |
| Lumos [ 23 ] | Modeling toolkit; GPT-3 family | Kineto and execution-trace tooling | Modeling methodology | H100 GPUs |
| AICB [ 2 ] | Workload generator; dense and MoE LLMs | Parameterized workload specifications | DP/TP/PP/SP/EP specifications | GPU-agnostic |
| Simulator | Execution | Computation | Communication |
|---|---|---|---|
| Calculon [ 18 ] | Analytical phases | Analytical | Mechanistic |
| SimAI [ 39 ] | Workload graph | Profiled kernels / analytical scaling | Mechanistic / packet-level |
| Optimus [ 20 ] | Operator graph | Analytical / structure-aware | Mechanistic |
| ASTRA-sim 2.0 [ 41 ] | Workload graph | Profiled / roofline | Mechanistic / packet-level |
| vTrain [ 6 ] | Execution graph | Profiled CUDA kernels | Profiled / analytical |
| LLMEmu [ 43 ] | Hybrid | Profiling-based | Trace-driven |
| View | Granularity | Evaluation use |
|---|---|---|
| Iteration time | Configuration | Runtime prediction and ranking |
| Peak memory | Rank | Memory prediction and feasibility |
| Time breakdown | Rank | Bottleneck diagnosis |
| Operator latency | Operator/rank | Computation cost modeling |
| Communication | Event/rank | Communication cost modeling |
| Matrix kernels | Kernel/rank | Matrix-operator prediction |
| Metric | Calculon | vTrain |
|---|---|---|
| Iteration-time MAE (s) | 1.059 | 0.207 |
| Iteration-time sMAPE (%) | 8.88 | 1.86 |
| Iteration-time | 0.630 | 0.990 |
| Ranking Spearman | 0.943 | 1.000 |
| Ranking Kendall | 0.867 | 1.000 |
| Top-1 hit / regret | 1 / 0% | 1 / 0% |
| Qwen3 / stack | Time | Memory |
|---|---|---|
| 0.6B / V100 | 10.28 / 0.861 | 15.99 / 0.993 |
| 0.6B / 910B3 | 10.19 / 0.950 | 13.20 / 0.986 |
| 1.7B / V100 | 11.22 / 0.853 | 12.56 / 0.993 |
| 1.7B / 910B3 | 7.97 / 0.935 | 12.55 / 0.994 |
| Level | Metric | Qwen3-0.6B | Qwen3-1.7B |
| Coverage | Paired complete configurations | 15 | 11 |
| Strategy | Iteration-time rank | 0.718 | 0.664 |
| NPU Top-1 selection: GPU regret | 0.0% | 6.9% | |
| GPU optimum in NPU Top-3 | Yes | Yes | |
| NPU Top-3 shortlist: GPU regret | 0.0% | 0.0% | |
| Memory | Peak-memory rank | 0.982 | 0.991 |
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
| Simulator | Workflow | Comp. | Comm. | Public artifact | Primary focus |
|---|---|---|---|---|---|
| Calculon [ 18 ] | A | An | Mech | ✓ | Analytical iteration-time and memory estimation for LLM training. |
| SimAI [ 39 ] | G | Pr/An | Mech/Pkt | ✓ | Full-stack large-scale training simulation across framework and network layers. |
| Optimus [ 20 ] | G | An/St | Mech | Operator-graph modeling for workload and accelerator microarchitecture exploration. | |
| ASTRA-sim 1.0 [ 31 ] | H | An | Mech/Pkt | ✓ | Network-topology and collective-communication simulation. |
| ASTRA-sim 2.0 [ 41 ] | G | Pr/An | Mech/Pkt | ✓ | Detailed workload graphs combined with network-level simulation. |
| vTrain [ 6 ] | G | Pr | Pr/An | ✓ | Graph-based training simulation calibrated by real-system measurements. |
| Input | Representative parameters | Role |
|---|---|---|
| Model and workload | Architecture, layer dimensions, attention heads, sequence length, global batch size | Defines the training workload. |
| Training configuration | DP/TP/PP/CP/EP, micro-batch size, recomputation, precision, optimizer-state partitioning, fusion, and overlap | Defines candidate execution strategies for a fixed workload and system. |
| System configuration | Device count, compute capability, memory capacity and bandwidth, interconnect, and topology | Defines resources and capacity constraints. |
| Item | Value |
|---|---|
| Model settings | 23 |
| Parallel configurations | 348 |
| Data format | CSV |
| Data modality | Tabular profiling records |
| Model coverage | Dense and MoE models from 10 LLM families |
| Configuration dimensions | DP, TP, PP, CP, EP, recomputation |
| Model-setting identifier | Configs | DP | TP | PP | CP | EP | Recompute |
|---|---|---|---|---|---|---|---|
| gemma2_27b_8k | 3 | 1/2 | 4/8 | 2 | 1/2 | – | RC |
| gemma2_9b_8k | 5 | 1/2 | 2/4 | 2 | 1/2 | – | RC/noRC |
| glm4_9b_8k | 6 | 1/2 | 2 | 2/4 | 1/2 | – | RC/noRC |
| internlm25_1point8b_32k | 16 | 1/2/4 | 1/2/4 | 1/2/4 | 2/4/8 | – | RC/noRC |
| internlm25_20b_32k_16p | 9 | 1/2 | 2/4/8 | 2/4/8 | 1/2/4 | – | RC |
| internlm25_7b_32k | 5 | 1/2 | 2/4 | 2/4 | 1/2 | – | RC/noRC |
| Category | Item | Value |
| Host processor | CPU architecture | aarch64 |
| CPU model | Kunpeng-920 | |
| CPU cores | 192 | |
| Accelerator | NPU model | Ascend 910B3 |
| HBM per NPU | 65536 MiB | |
| NPU interconnect | HCCS |
| File | Location | Granularity | Benchmark role |
|---|---|---|---|
| data_manifest.csv | Dataset root | Dataset / model level | Provides the global index of model directories, configuration counts, rank-directory counts, file coverage, and dataset-scale statistics. |
| step_time_summary.csv | Model directory | Configuration level | Provides configuration-level iteration-time labels derived from profiler time breakdowns for runtime prediction and configuration ranking. |
| peak_memory.csv | Model directory | Rank level | Provides rank-level peak memory labels for memory modeling, feasibility analysis, and imbalance analysis across ranks. |
| step_trace_time_summary.csv | Model directory | Rank level | Decomposes iteration time into computation, communication, idle time, and pipeline-related components for bottleneck diagnosis. |
| operator_latency_by_category.csv | Model directory | Operator-category / rank level | Aggregates profiler operators into semantic categories for operator-latency modeling and module-level performance analysis. |
| comm_summary.csv | Model directory | Communication-group / rank level | Provides aggregated communication statistics for comparing collective and point-to-point communication costs across configurations. |
| File | Representative fields |
|---|---|
| data_manifest.csv | model_name , parallel_config_count , rank_directory_count , size_bytes , size_human . |
| step_time_summary.csv | optimization_config , iteration_time_ms . |
| peak_memory.csv | optimization_config , rank_id , peak_memory_mb . |
| step_trace_time_summary.csv | optimization_config , rank_id , computing , communication , communication_not_overlapped , overlapped , free , stage , bubble . |
| operator_latency_by_category.csv | optimization_config , rank_id , op_category , op_type , core_type , count , total_time_us , avg_time_us , ratio_percent . |
| comm_summary.csv | comm_scope , comm_type , comm_group , op_count , elapsed_time_sum_ms , wait_time_sum_ms , link_bandwidth . |
| Topic | Summary |
|---|---|
| Intended use | Training-simulator evaluation and development: time and memory prediction, configuration ranking, operator and communication analysis, and calibration. |
| Coverage and evaluation scope | Measurements describe the sampled models, feasible configurations, and software/hardware stacks in Section C.1 and Appendix F . Coverage depends on model size, memory capacity, and platform implementations. Serving behavior and model-quality, safety, or fairness assessment require separate data. |
| Provenance and record type | Controlled distributed training runs produce the logs and profiler outputs; the processed tables are normalized derivatives of these measured records, without synthetic profiling samples or human annotations. |
| Personal or sensitive information | The release contains system telemetry, with no intentional personal records, user-generated text, or private user data. |
| Level | Task and metric | Calculon | vTrain |
|---|---|---|---|
| System | Iteration time, MAE / sMAPE / | 1.059 s / 8.88% / 0.630 | 0.207 s / 1.86% / 0.990 |
| Strategy | Iteration-time ranking, Spearman | 0.943 | 1.000 |
| Iteration-time ranking, Kendall | 0.867 | 1.000 | |
| Top-1 hit / selection regret | 1 / 0% | 1 / 0% | |
| Top-3 set overlap | 3/3 | 3/3 | |
| Efficiency | Median wall-clock per configuration | 1.78 ms | 25.46 ms |
| Task | Level | Metrics | Results |
|---|---|---|---|
| Iteration time prediction | System | MAE / sMAPE / | 1.97 s / 15.96% / 0.358 |
| Peak memory prediction | System | MAE / sMAPE / | 6.92 GiB / 39.34% / |
| Matrix-operator latency | Component | MAE / sMAPE / | 0.109 ms / 43.34% / 0.817 |
| Feasible-configuration retention | System | Retention rate | 65.0% (13/20) |
| Iteration time ranking | Strategy | Spearman / Top-1 hit | 0.753 / 1 |
| Target | MAE | sMAPE (%) | |
|---|---|---|---|
| Stage computation time | 716.10 ms | 11.60 | 0.462 |
| Recomputation-stage time | 37.99 ms | 2.75 | 0.801 |
| Non-overlapped communication time | 8.710 s | 136.20 |
| Parallel configuration | MAE (ms) | sMAPE (%) | Parallel configuration | MAE (ms) | sMAPE (%) | ||
|---|---|---|---|---|---|---|---|
| dp1-tp1-pp1-cp8-RC | 0.048 | 40.505 | 0.779 | dp2-tp1-pp1-cp4-RC | 0.072 | 36.826 | 0.864 |
| dp1-tp1-pp1-cp8-noRC | 0.053 | 42.190 | 0.778 | dp2-tp1-pp1-cp4-noRC | 0.076 | 38.311 | 0.873 |
| dp1-tp1-pp2-cp4-RC | 0.072 | 36.794 | 0.865 | dp2-tp1-pp2-cp2-RC | 0.120 | 30.403 | 0.890 |
| dp1-tp1-pp2-cp4-noRC | 0.076 | 38.126 | 0.874 | dp2-tp1-pp2-cp2-noRC | 0.127 | 31.581 | 0.897 |
| dp1-tp1-pp4-cp2-RC | 0.119 | 30.313 | 0.889 | dp2-tp1-pp4-cp1-RC | 0.252 | 31.580 | 0.874 |
| dp1-tp1-pp4-cp2-noRC | 0.128 | 31.596 | 0.895 | dp2-tp1-pp4-cp1-noRC | 0.268 | 32.662 | 0.880 |
| MAE (ms) | sMAPE (%) | MAE (ms) | sMAPE (%) | ||||||
|---|---|---|---|---|---|---|---|---|---|
| 1024 | 2048 | 2048 | 0.011 | 24.151 | 4096 | 3072 | 2048 | 0.078 | 38.374 |
| 1024 | 2048 | 5504 | 0.056 | 55.184 | 4096 | 5504 | 2048 | 0.114 | 32.488 |
| 1024 | 2048 | 6144 | 0.068 | 58.875 | 4096 | 6144 | 2048 | 0.108 | 28.405 |
| 1024 | 2048 | 11008 | 0.089 | 46.131 | 4096 | 11008 | 2048 | 0.193 | 29.637 |
| 1024 | 5504 | 2048 | 0.046 | 47.035 | 4096 | 37984 | 2048 | 1.271 | 48.828 |
| 1024 | 6144 | 2048 | 0.048 | 45.422 | 4096 | 75968 | 2048 | 1.350 | 30.161 |
| Aspect | Setting |
|---|---|
| Matched | Model architecture, layer counts, hidden size, attention heads, and grouped-query attention; sequence length 4096; micro/global batch 1/32; eight devices; CP ; candidate configuration names; optimizer hyperparameters and seed; profiled iteration index. |
| Stack-specific | Data type (FP16 on V100 vs. BF16 on 910B3); fused kernels (FlashAttention and fused RoPE/SwiGLU/RMSNorm disabled in the GPU sweep, enabled on the Ascend stack); sequence parallelism (enabled on the NPU side); profiler-visible hardware counters and communication-link fields; input data path. |
| Model | Configuration | GPU status | NPU status |
|---|---|---|---|
| Qwen3-0.6B | dp2-tp1-pp4-cp1-noRC | Missing/incomplete (empty directory) | Complete |
| Qwen3-0.6B | dp4-tp1-pp2-cp1-noRC | Missing/incomplete (empty directory) | Complete |
| Qwen3-0.6B | dp8-tp1-pp1-cp1-noRC | Missing/incomplete (empty directory) | Complete |
| Qwen3-1.7B | dp1-tp2-pp4-cp1-noRC | Missing/incomplete (empty directory) | Complete |
| Qwen3-1.7B | dp2-tp1-pp4-cp1-noRC | Missing/incomplete (empty directory) | Complete |
| Qwen3-1.7B | dp2-tp2-pp2-cp1-noRC | Missing/incomplete (empty directory) | Complete |
| Level | Metric | Qwen3-0.6B | Qwen3-1.7B |
|---|---|---|---|
| Coverage | Paired complete configurations | 15 | 11 |
| Strategy | Iteration rank / , pooled | 0.718 / 0.562 | 0.664 / 0.491 |
| Iteration rank , noRC / RC | 0.714 / 0.683 | 1.000 ∗ / 0.690 | |
| System | Peak-memory rank / | 0.982 / 0.924 | 0.991 / 0.964 |
| Lowest-memory Top-3 overlap | 3/3 | 3/3 | |
| RC time/memory direction agreement | 6 / 6 | 3 / 3 |
| Model | Platform | Time MAPE (%) | Time | Rank-mem. MAPE (%) | Config-mem. |
|---|---|---|---|---|---|
| Qwen3-0.6B | V100 | 10.28 | 0.861 | 15.99 | 0.993 |
| Qwen3-0.6B | 910B3 | 10.19 | 0.950 | 13.20 | 0.986 |
| Qwen3-1.7B | V100 | 11.22 | 0.853 | 12.56 | 0.993 |
| Qwen3-1.7B | 910B3 | 7.97 | 0.935 | 12.55 | 0.994 |
| Model | Paired candidates | Top-1 regret | GPU optimum in Top-3 | Top-3 regret |
|---|---|---|---|---|
| Qwen3-0.6B | 15 | 0.0% | Yes | 0.0% |
| Qwen3-1.7B | 11 | 6.9% | Yes | 0.0% |
| Model | Stratum | Top-3 | Top-1 agreement | |||
|---|---|---|---|---|---|---|
| Qwen3-0.6B | all | 15 | 0.718 | 0.562 | 2/3 | yes |
| Qwen3-0.6B | noRC | 6 | 0.714 | 0.600 | 2/3 | yes |
| Qwen3-0.6B | RC | 9 | 0.683 | 0.556 | 2/3 | yes |
| Qwen3-1.7B | all | 11 | 0.664 | 0.491 | 2/3 | no |
| Qwen3-1.7B | noRC | 3 | 1.000 | 1.000 | 3/3 | yes |
| Qwen3-1.7B | RC | 8 | 0.690 | 0.571 | 2/3 | yes |
| Configuration | GPU (s) | NPU (s) | GPU/NPU | GPU rank | NPU rank |
| Qwen3-0.6B | |||||
| dp4-tp2-pp1-cp1-noRC | 6.40 | 3.93 | 1.63 | 1 | 1 |
| dp2-tp4-pp1-cp1-noRC | 6.69 | 6.11 | 1.09 | 2 | 7 |
| dp2-tp2-pp2-cp1-noRC | 7.20 | 4.48 | 1.61 | 3 | 2 |
| dp1-tp4-pp2-cp1-noRC | 7.33 | 6.74 | 1.09 | 4 | 10 |
| dp8-tp1-pp1-cp1-RC | 7.53 | 4.69 | 1.61 | 5 | 3 |
| Configuration | GPU (GiB) | NPU (GiB) | GPU/NPU | GPU rank | NPU rank |
| Qwen3-0.6B | |||||
| dp1-tp8-pp1-cp1-RC | 1.47 | 2.18 | 0.67 | 1 | 1 |
| dp1-tp4-pp2-cp1-RC | 2.18 | 3.16 | 0.69 | 2 | 2 |
| dp2-tp4-pp1-cp1-RC | 2.37 | 3.25 | 0.73 | 3 | 3 |
| dp1-tp2-pp4-cp1-RC | 3.64 | 5.26 | 0.69 | 4 | 5 |
| dp2-tp2-pp2-cp1-RC | 3.67 | 5.09 | 0.72 | 5 | 4 |
| Model | Matrix problem | GPU (µs) | NPU (µs) | GPU/NPU |
|---|---|---|---|---|
| Qwen3-0.6B | b8: 4096x4096 @ 4096x128 | 654.5 | 388.4 | 1.69 |
| Qwen3-0.6B | b8: 4096x128 @ 128x4096 | 693.3 | 234.0 | 2.96 |
| Qwen3-0.6B | b16: 4096x4096 @ 4096x128 | 1187.3 | 697.2 | 1.70 |
| Qwen3-0.6B | b4: 4096x4096 @ 4096x128 | 278.4 | 133.5 | 2.09 |
| Qwen3-0.6B | b4: 4096x128 @ 128x4096 | 352.4 | 97.0 | 3.63 |
| Qwen3-0.6B | b16: 4096x128 @ 128x4096 | 1371.0 | 484.1 | 2.83 |
| Component | 0.6B GPU | 0.6B NPU | 1.7B GPU | 1.7B NPU |
|---|---|---|---|---|
| Operator categories | ||||
| matmul | 28.6% | 14.9% | 38.9% | 22.0% |
| memory_movement | 20.7% | 4.2% | 16.2% | 3.3% |
| elementwise | 18.6% | 17.2% | 19.1% | 19.9% |
| attention | 14.7% | 0.0% | 11.2% | 0.0% |
| memory_init | 13.0% | 0.2% | 10.1% | 0.2% |
LLM Training
Large language model training studies the data, objectives, optimizers, and systems used to build and adapt language models.
See all 377 papers