ModTrans: Translating Real-world Models for Distributed Training Simulator

Yi Lyu

cs.DCApr 2, 2026

ModTrans: Translating Real-world Models for Distributed Training Simulator

Authors: Yi Lyu

Organizations: University of Wisconsin-Madison Madison, WI, USA

Abstract

Large-scale distributed training has been a research hot spot in machine learning systems for industry and academia in recent years. However, conducting experiments without physical machines and corresponding resources is difficult. One solution is to leverage distributed training simulators, but current ones like ASTRA-sim do not support importing real-world developed models, which poses challenges for ML researchers seeking to use them. Based on this challenge, we developed ModTrans, a translator supporting format translation from any real-world model to the ASTRA-sim simulator's input, removing the barrier between machine learning experts and machine learning system researchers. The experiment results show that ModTrans's cost is negligible.

Figures & tables

Figure 1. Deep Learning Platform Software (SW) and Hardware (HW) Design Space ( Rashidi et al., 2020 )

Figure 2. Overview of ASTRAM-sim ( Rashidi et al., 2020 )

Figure 3. DNN description file ( Rashidi et al., 2020 )

Figure 6. Execution Time for ModTrans

Layer Name	Variables	Data Type	Model Size
vgg16-conv0-weight	1728	FLOAT	6912
vgg16-conv1-weight	36864	FLOAT	147456
vgg16-conv2-weight	73728	FLOAT	294912
vgg16-conv3-weight	147456	FLOAT	589824
vgg16-conv4-weight	294912	FLOAT	1179648
vgg16-conv5-weight	589824	FLOAT	2359296

Table 1. Layer-by-layer sizes extracted from VGG16 ONNX model

Layer Name	Variables	Data Type	Model Size
vgg19-conv0-weight	1728	FLOAT	6912
vgg19-conv1-weight	36864	FLOAT	147456
vgg19-conv2-weight	73728	FLOAT	294912
vgg19-conv3-weight	147456	FLOAT	589824
vgg19-conv4-weight	294912	FLOAT	1179648
vgg19-conv5-weight	589824	FLOAT	2359296

Table 2. Layer-by-layer sizes extracted from VGG19 ONNX model

Layer Name	Extracted Model	ASTRA-SIM Model
resnet-conv0	37632	37632
resnet-stage1-conv0	16384	16384
resnet-stage1-conv1	147456	147456
resnet-stage1-conv2	65536	65536
resnet-stage1-conv3	65536	65536
resnet-stage1-conv4	65536	65536

Table 3. Layer-by-layer sizes extracted from ResNet50 ONNX model

Distributed Training

Distributed training coordinates computation across multiple devices or machines to train models that exceed single-device limits.

See all 129 papers

ASTRA-sim 3.0: Next-Level Distributed Machine Learning Simulations via High-Fidelity GPU and Infrastructure Modeling
Jun 9, 2026William Won, Jinsun Yoo, Tuan Ta +16Distributed Training
AMD Research and Advanced Development · Georgia Institute of Technology · Keysight +1
Beyond Learning: A Training-Free Alternative to Model Adaptation
Feb 18, 2026Namkyung Yoon, Kyeonghyun Yoo, Wooyong Jung +2
TranslatePsy-AfriSLM: High-Quality Data Scaling For Low-Resource Machine Translation
Aug 19, 2026Milan Gritta, Patrik Lambert, Jihye Back +1Machine Translation Low-Resource MT
Tether AI Research

Jun 9, 2026J/K move · Enter open · S save

ASTRA-sim 3.0: Next-Level Distributed Machine Learning Simulations via High-Fidelity GPU and Infrastructure Modeling

William Won, Jinsun Yoo, Tuan Ta, Moumita Dey+15

AMD Research and Advanced Development · Georgia Institute of Technology · Keysight+1

Distributed Training

Open paper AskPDF arXiv

Distributed machine learning (ML) is a key paradigm for today's large-scale artificial intelligence applications. As model inference arises as an important use case, faithful modeling of latency-sensitive collective communication has never been more important. Capturing the device architecture and modeling control and data paths at high fidelity is therefore a necessity today. Having a common, detailed representation for distributed ML infrastructure is also crucial. We revisit the promising open-source, community-driven simulator: ASTRA-sim. In this work, we identify limitations of the current ASTRA-sim simulator and augment it with new features. To this end, we enable fine-grained, high-fidelity simulation with a standardized infrastructure representation, opening new design space exploration opportunities. We propose the simulation at cache-line-sized load-store granularity, with a detailed graphics processing unit (GPU) execution model, to balance simulation scalability and fidelity. We also introduce InfraGraph, a standardized representation to capture distributed ML network infrastructure in detail. Using the updated ASTRA-sim 3.0 simulator, we showcase interesting design space explorations for designing optimized collective algorithms, network requirements, and GPU architectures.

Abstract

Distributed Training

Explore similar work

ASTRA-sim 3.0: Next-Level Distributed Machine Learning Simulations via High-Fidelity GPU and Infrastructure Modeling

Beyond Learning: A Training-Free Alternative to Model Adaptation

TranslatePsy-AfriSLM: High-Quality Data Scaling For Low-Resource Machine Translation