Configuring distributed LLM training across hundreds of GPUs requires jointly optimizing tensor, pipeline, expert, context, and data parallelism alongside micro-batch size, recomputation, sharding, and overlap, a combinatorial space where memory legality can hinge on a single byte of activation accounting. Today, teams navigate it through expensive multi-node trial and error, burning GPU-hours on crashes and suboptimal recipes. We present Primus Projection, which closes this loop with two coupled components. First, a hybrid projection engine that measures what it can and simulates what it cannot: GPU microbenchmarks on as few as a single device capture compute kernels and any in-scope communication; a sub-node downscale-measure-upscale workflow analytically restores out-of-scope collectives, FSDP overlap, and the full pipeline schedule at the multi-node target. A pure simulation mode runs entirely on CPU, enabling pre-silicon planning without any GPU. Second, the Primus Tuning Agent treats the projection engine as a fast scoring oracle (seconds per candidate instead of tens of minutes) and uses it to drive an intelligent search over the configuration space. A deterministic seed planner sweeps single axes; an LLM investigation loop proposes cross-axis combinations the planner cannot reach, with an analytic memory pre-filter that rejects infeasible candidates before any GPU time is spent. Validated on Llama 3.1 (dense) and Mixtral 8x22B (MoE) across two GPU generations, all projections stay within ~10% of measured multi-node throughput, and fidelity holds at production scale: 4.7% error on DeepSeek V3 671B at 1,024 GPUs and 1.9% on Qwen3-235B-A22B at 256 GPUs. In a Mixtral 8x22B case study, the Primus Tuning Agent discovers a configuration delivering +27% throughput over the published 4-node BF16 reference at the same scale, in under 30 minutes of single-node exploration.
Figures & tables
Figure 1: Primus Projection architecture. Modeling components consume model card, hardware profile, and cluster spec; the projection oracle returns a memory and throughput estimate that drives both multi-node projection and the Primus Tuning Agent’s iterative search.
Model
Prec.
MBS
GBS
Seq
FSDP
Meas.
1 GPU
2 GPUs
4 GPUs
8 GPUs
Max Δ
Llama 8B
FP8
2
128
8192
No
16,186
17,619
(+8.9%)
17,628
(+8.9%)
17,604
(+8.8%)
17,607
(+8.8%)
0.1%
Llama 70B
FP8
4
256
8192
Yes
1,726
1,867
(+8.2%)
1,868
(+8.2%)
1,850
(+7.2%)
1,825
(+5.7%)
2.4%
Llama 70B
BF16
1
64
8192
Yes
1,174
1,087
( − 7.4%)
1,091
( − 7.1%)
1,087
( − 7.4%)
1,086
( − 7.5%)
0.5%
Table 1: Dense model sub-node benchmarking sweep (MI325X, target: 8 nodes / 64 GPUs, tok/s/GPU). Parenthetical percentages are error vs. measured. Max Δ = max deviation across benchmark GPU counts relative to the 8-GPU projection.
Figure 3Table 4
Model (FP8)
Scale
Projected
Measured
Error
DeepSeek V3 671B MoE
1,024 GPU (128 node)
2,808
2,946
4.7%
Qwen3-235B-A22B MoE
256 GPU (32 node)
4,860
4,952
1.9%
Table 7: Large-scale validation (tok/s/GPU). Both recipes are agent-selected end-to-end and confirmed on hardware; full search traces in Appendix D .
Figure 3: Primus Tuning Agent search on Mixtral 8 × 22B (4-node MI355X).
Workload
n
Spearman ρ
Kendall τ
Offset k (cv)
Regret
Mixtral-8 × 22B FP8 (32 GPU)
7
1.000
1.000
1.128 (2.4%)
0%
Qwen3-235B-A22B FP8 (32 GPU)
7
1.000
1.000
0.901 (11.3%)
0%
GPT-OSS-120B BF16 (64 GPU)
9
0.933
0.833
1.121 (5.4%)
0%
Table 10: Rank preservation across three MoE models spanning FP8/BF16 and 32–64 GPUs. Selection regret (agent pick vs. measured-optimal) is 0% throughout.
Method
Oracle
Proj. tok/s/GPU
vs. baseline
Primus Tuning Agent + Benchmark
Benchmark
4,698
+39.5%
Claude Code + Benchmark
Benchmark
4,275
+26.9%
Galvatron-style FP8 exhaustive
Simulate
3,656
+8.6%
Primus Tuning Agent + Simulate
Simulate
3,445
+2.3%
Galvatron-style budget=30
Simulate
2,155
− 36.0%
Table 11: Ablation and baseline comparison on Mixtral-8 × 22B (4-node MI355X, FP8; baseline 3,368 projected tok/s/GPU). Oracle contribution + 36%, agent-framework contribution + 10%; winning configs in Table 25 . Run on a different 4-node cluster from the case study in Section 4.7 (projected 4,908); cluster-to-cluster variation is < 5%. Details in Appendix G .
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
GPU
Arch
HBM (GB/s)
CUs
XCDs
MI300X
gfx942
5,300
304
8
MI325X
gfx942
6,000
304
8
MI355X
gfx950
8,000
256
4
Appendix
Table 12: AMD Instinct GPU architectures used in this paper.
Factor
Analytical Sim.
GPU Benchmark
Hardware
No GPU required
Requires target GPUs
Speed
Fast, deterministic
Slower, with variance
Pre-silicon
Primary use case
Not possible
GPU freq.
Estimated from model
Captures actual throttling
SW stack
Needs recorrelation
Reflects current kernels
Grouped GEMM
Depends on model fidelity
Measured directly
Appendix
Table 13: Benchmark vs. simulation trade-offs.
Algorithm
VPP
Description
1F1B
1
Standard one-forward-one-backward
Interleaved 1F1B [ 20 ]
> 1
Multiple virtual chunks per rank; reduces bubble by VPP factor
Zero-Bubble
1
Splits backward into B+W; F-B-W steady-state fills bubbles
ZBV Formatted
2
V-shape schedule with structured warm-up / stable / cool-down phases
ZBV Greedy
2
V-shape with greedy placement and memory-aware modes
Figure 4: Left: the agent’s batch-scaling trajectory on DeepSeek V3 671B at 1,024 GPUs—batching to the memory ceiling lifts throughput 50% (1,869 → 2,808 tok/s/GPU). Right: projected vs. measured throughput at production scale; error is 4.7% (DeepSeek V3, 1,024 GPU) and 1.9% (Qwen3-235B, 256 GPU).
Tag
Recomp. layers
MBS
Projected
Measured
k
r00_b1
0
1
4,957
4,213
1.177
r03_b1
3
1
4,620
4,059
1.138
r09_b2
9
2
4,397
3,884
1.132
r06_b1
6
1
4,353
3,860
1.128
r09_b1
9
1
4,095
3,709
1.104
r14_b2
14
2
4,029
3,558
1.132
Appendix
Table 18: Mixtral-8 × 22B FP8, 4-node/32-GPU recompute/MBS sweep: all 7 configs projected and measured ( ρ=τ=1.000 ). Tag rN_bM denotes N recomputed layers and MBS M ; all other settings are held fixed across the sweep. The offset k stays within 1.08–1.18 (mean 1.128, cv 2.4%), so no pair flips.
Tag
PP
VPP
Recomp. layers
MBS
GBS
Projected
Measured
k
qwen_i39 (winner)
4
3
1
2
2,048
4,418
4,380
1.009
qwen_i04
4
2
3
2
2,048
4,230
4,174
1.013
qwen_i06
4
2
6
2
2,048
3,839
4,156
0.924
qwen_i08
4
2
8
2
2,048
3,739
4,041
0.925
qwen_i21
4
2
23
2
2,048
3,481
3,984
0.874
qwen_i10
2
1
47
2
1,024
3,326
3,802
0.875
Appendix
Table 19: Qwen3-235B-A22B FP8, 4-node/32-GPU: all 7 configs projected and measured ( ρ=τ=1.000 ). The offset k drifts with configuration (1.01 at shallow recompute to 0.69 for the MBS=1, GBS=1,024 config) but stays small relative to the throughput gaps, so no pair flips.
Tag
Distinguishing knobs
Projected
Measured
k
winner (trial32)
mbs4, recompute none, sf1
13,206
12,168
1.085
vpp6 (trial54)
vpp6
12,917
11,838
1.091
pp4 (trial42)
pp4
12,693
11,593
1.095
rec9 (trial29)
mbs8, full recompute 9
12,403
11,029
1.125
mbs16 (trial20)
mbs16, full recompute 18
11,719
9,489
1.235
baseline (reference)
mbs8, full recompute 18
11,260
10,109
1.114
Appendix
Table 20: GPT-OSS-120B BF16, 8-node/64-GPU: 9 configs, no prior AMD anchor. Top-1 projected = top-1 measured ( + 20.4% over the reference baseline, tracking + 17.3% projected).
Figure 5: Projected vs. measured throughput per config (dashed line = identity). Left: Qwen3-235B is perfectly monotone ( ρ=τ=1.000 )—the multiplicative offset k shifts points off the identity line but never reorders them. Right: GPT-OSS-120B ( ρ=0.933 ); the only reversals involve one over-projected trio around mbs16 , while the top-1 pick and the extremes are exactly ordered (selection regret 0%).
Workload
Concordant
Discordant (meas. gap)
Median k
Max ∣k−k~∣
Conc. > 9%
Top-1/Top-3 regret
Mixtral-8 × 22B
21/21
none
1.132
0.048 ( r14_b1 )
100% (10/10)
0% / 0%
Qwen3-235B-A22B
21/21
none
0.924
0.236 ( i27 )
100% (10/10)
0% / 0%
GPT-OSS-120B
33/36
2.1%, 6.5%, 8.8%
1.095
0.140 ( mbs16 )
100% (27/27)
0% / 0%
Appendix
Table 21: Ranking-consistency metrics. Discordant pairs on GPT-OSS: baseline– mbs2 (2.1%), mbs16 –baseline (6.5%), mbs16 – mbs2 (8.8%). Mixtral tags as in Table 18 .
Table 19
Figure 6: Left: disabling load balancing inflates worst-case error to 53–71%, but the busiest-rank correction brings it back under 10% on all three models. Right: on the trained DeepSeek V3 checkpoint the per-layer imbalance grows with EP, yet the step-level pooled imbalance—the quantity that actually gates throughput—stays near 1.05–1.12 because the busiest device rotates layer-to-layer.
Config
Measured
Primus
Calculon
FP8 (GBS 256, MBS 4, FSDP)
1,726
1,825 ( + 5.7%)
2,006 ( + 16.2%)
BF16 (GBS 64, MBS 1, FSDP)
1,174
1,086 ( − 7.5%)
686 ( − 41.6%)
Mean ∣ error ∣
—
6.6%
28.9%
Appendix
Table 24: Head-to-head accuracy on Llama-3.1-70B, AMD MI325X, 8 nodes / 64 GPUs (tok/s/GPU). Measured values are real MI325X runs.
Figure 7: Left: on Mixtral-8 × 22B, Primus Tuning Agent+Benchmark (4,698) tops every simulate-oracle method—including the Galvatron-style exhaustive 530-config FP8 search (3,656)—because the benchmark oracle measures superposed kernel/collective effects. Right: signed prediction error on Llama-3.1-70B; Calculon’s large opposite-sign errors ( + 16.2% FP8, − 41.6% BF16) distort the cross-precision trade-off, whereas Primus stays within ± 8%.
Method
Oracle
Winning configuration
Proj.
vs. base
Primus Tuning Agent + Benchmark
Benchmark
TP=1 PP=4 EP=8 MBS=2 FP8 DeepEP, no recompute
4,698
+39.5%
Claude Code + Benchmark
Benchmark
TP=1 PP=8 EP=4 MBS=2 FP8 DeepEP, no recompute
4,275
+26.9%
Galvatron-style FP8 exhaustive
Simulate
TP=2 PP=2 EP=4 MBS=1 FP8, no recompute
3,656
+8.6%
Galvatron-style BF16 exhaustive
Simulate
TP=2 PP=4 EP=4 MBS=1 BF16, no recompute
3,576
+6.2%
Primus Tuning Agent + Simulate
Simulate
TP=1 PP=4 EP=8 MBS=1 FP8, full recompute
3,445
+2.3%
Galvatron-style budget=30
Simulate
TP=8 PP=1 EP=4 MBS=2 BF16, no recompute
2,155
− 36.0%
Appendix
Table 25: Winning configuration per method in the Mixtral-8 × 22B ablation (4-node MI355X; baseline 3,368 projected tok/s/GPU; Table 11 ). Only the benchmark oracle identifies the intra-node EP=8 DeepEP topology with TP=1.
Large language model (LLM) training today runs on clusters spanning thousands of GPUs. While this scale enables rapid model advances, developing, debugging, and performance-tuning the training framework inevitably becomes complex and costly. This is because engineers often need to reproduce production behaviors to diagnose failures or evaluate optimizations, thereby demanding frequent and even exclusive access to production-scale clusters -- which becomes increasingly hard given that the majority of GPUs are already committed to production workloads. Simulation relies on complex performance models that are difficult to maintain, and downscaled experiments often fail to capture scale-dependent behaviors. We present PrismLLM to decouple large-scale execution from the need to access large clusters, enabling engineers to run and observe ranks of interest under faithful large-scale behavior using only a few GPUs. PrismLLM constructs a high-fidelity execution graph via a slicing-based approach that captures computation, communication, and dependencies of the target scale. Then, PrismLLM performs hybrid emulation where selected ranks execute the original program while the remaining ranks are replayed as virtual participants. Experiments on large-scale LLM training workloads show that PrismLLM accurately reproduces performance and memory behavior, achieving only 0.58% average error in iteration time and less than 0.01% error in peak GPU memory usage. PrismLLM can emulate clusters of up to 8192 GPUs using fewer than 1% of the physical GPUs required by the original deployment.
Shaoke Xi, ChonLam Lao, Boyi Jia +11
Alibaba Group · Harvard University · Shanghai Jiao Tong University +1
The throughput of an LLM training job depends on its recipe: how the model is split across GPUs, how memory is traded for compute, and which kernels and communication overlaps it uses. The best recipe changes with every model, accelerator and framework version. Automated tuners search a fixed space of knobs and check only that a job runs, so they miss levers outside that space and can ship a configuration that silently changes training. ATTUNE is a coding agent that tunes any working recipe unattended within a GPU-hour budget. Guided by a skill of search rules, scripts and lessons from past tuning campaigns, it diagnoses each run and changes one lever at a time. A correctness gate outside the agent refuses any argument known to change the training objective. ATTUNE tunes Amazon's production post-training recipes on H100, H200 and B200 clusters, raising throughput by 1.7x on average. It beats the published throughput of four of NVIDIA's own benchmark recipes on NVIDIA's AutoModel framework by 6-17%. On models held out from its development, it improves the strongest public recipes most where they were written for another setting, by up to 2.6x on MiniMax-M2, and least where the vendor tuned them for the same hardware. Given the same GPU-hour budget, it outperforms existing automated tuners, including Bayesian optimization, random search and OptiCo, a recent LLM-agent tuner. Its gain on GLM-4.5 is more than four times the best alternative's. The same coding agent without ATTUNE's skill gains less and fails half its launches. We release ATTUNE at https://github.com/amazon-science/attune and OpenRecipes, a leaderboard of tuned and verified training recipes, at https://openrecipes.ai.
Selecting the optimal LLM inference configuration requires evaluation across hardware, serving engines, attention backends, and model architectures, since no single choice performs best across all workloads. Profile-based simulators are the standard tool, yet they hardcode their operation set to a specific configuration and re-profile every operation from scratch, making exploration prohibitively expensive. This cost stems from a missing structural understanding: every input dimension of each operation is fixed by the model configuration or determined by the incoming request. Many model-configuration values (e.g., head size, layer count) recur across models, so the same operation runs in many configurations; a single sweep over the request-dependent dimensions can serve them all. We present Dooly, which exploits this structure to achieve configuration-agnostic, redundancy-aware profiling. Dooly performs a single inference pass, labels each input dimension with its origin via taint propagation, and selectively profiles only operations absent from its latency database; stateful operations such as attention are isolated by reusing the serving engine's own initialization code, eliminating manual instrumentation. From this database, it builds latency regression models that serve as a drop-in backend for existing simulators. Across two GPU platforms, three attention backends, and diverse model architectures, Dooly achieves simulation accuracy within 5% MAPE for TTFT and 8% for TPOT while reducing profiling GPU-hours across 12 models by 56.4% compared to the existing profiling approach. We have open-sourced Dooly at https://dooly-project.github.io.