Data-Driven Prediction of Optimal MPI Rank Placement and Node Count for Memory-Bound CFD on AMD EPYC Clusters

Marcin Lawenda; Łukasz Szustak; Aleksandra Krasicka

cs.PFOct 16, 2026

Data-Driven Prediction of Optimal MPI Rank Placement and Node Count for Memory-Bound CFD on AMD EPYC Clusters

Authors: Marcin Lawenda, Łukasz Szustak, Aleksandra Krasicka

Organizations: Poznan Supercomputing and Networking Center, Jana Pawła II 10, 61-139 Poznań, Poland · Czestochowa University of Technology, Dąbrowskiego 69, 42-201 Częstochowa, Poland

Abstract

Users who run a memory-bound OpenFOAM case on an AMD EPYC cluster must decide how many nodes to use and how many MPI ranks to place on each core complex. The fastest choice is often not the cheapest one. This paper revisits our earlier analytical model of this performance and energy trade-off with 560 measured configurations of the motorBike case on five Zen 4 and Zen 5 EPYC processors, including socket power from processor telemetry. On up to four nodes, small meshes are both fastest and cheapest on a single node on most processors. For the largest mesh the largest node count is both the fastest and the cheapest option, because its speedup is superlinear. A genuine trade-off arises only for medium meshes, where full speed costs a median of 20 to 97% more node hours than the cheapest option. The L3 miss rate and the halo traffic per cell are consistent with this pattern. For a new mesh of the same case, the configuration that was optimal on the nearest measured mesh is near optimal in 75 to 85% of the decisions. Running the two configurations it recommends for speed and for cost raises this share to 85 to 90% for runtime and node hours and determines the trade-off with median errors of 3 to 5 percentage points. Neither gradient boosted trees trained on the same data nor the earlier polynomial model improves on this simple procedure.

Figures & tables

	Genoa	Genoa-X	Turin-96	Turin-128	Turin-c
Model	9654	9684X	9655	9755	9965
Microarchitecture	Zen 4	Zen 4	Zen 5	Zen 5	Zen 5c
Cores	96	96	96	128	192
Cores per CCX	8	8	8	8	16
L3 cache (MB)	384	1152	384	512	384
L3 per core (MB)	4	12	4	4	2

Table 1: Processors used in this study (one socket).

Figure 1: Example of the trade-off (a) and its two numbers for every processor and mesh (b, c).

Cells	Fastest	Cheapest	$\Delta C$ of full speed (%)		$\Delta T$ of cheapest (%)
(M)	$N$	$N$ (node h)	node hours	energy	node hours	energy
0.17	1	1	0	0	0	0
0.36	1–2	1	0	12	0	9
0.60	1–4	1	0	13	0	9
1.90	1–4	1	97	96	2	2
4.17	2–4	1–2	78	76	60	60

Table 2: Measured trade-off per mesh (medians over the five processors, node counts as ranges).

Figure 2: L3 miss rate of all runs (left) and parallel efficiency at the largest node count (right).

Cells	Recommended configuration			Runs	$\Delta C$	$\Delta T$
(M)	fastest	fewest node hours	balanced		node hours (%)	(%)
0.17	1N, 0.75	1N, 0.75	1N, 0.75	1	0	0
0.36	1N, 0.75	1N, 0.75	1N, 0.75	1	0	0
0.60	1N, 0.75	1N, 0.75	1N, 0.75	1	0	0
1.90	4N, 0.5	1N, 1	2N, 0.75	2	0	0
4.17	4N, 0.5	1N, 1	2N, 0.75	2	119	83

Table 3: Recommended configurations and estimated trade-off for Genoa as a new mesh.

Predictor	runs	Runtime	hours	Energy	Balanced
	New		Node
Nearest mesh	0	75	78	85	75
Nearest mesh, guided	2	85	90	85	75
Surrogate	0	43	25	40	30
Surrogate	3 random	57	49	60	45
Surrogate	2 guided	53	48	53	45

Table 4: Near optimal recommendations for a new mesh (% of the 40 decisions).

	New	$\Delta C$ (pp)		$\Delta T$ (pp)		Presence
Predictor	runs	median	mean	median	mean	correct (%)
Nearest mesh	0	59	65	38	47	78
Nearest mesh, guided	2	3	35	5	33	90
Surrogate	0	52	70	56	67	58
Surrogate	3 random	25	45	39	60	74
Surrogate	2 guided	11	33	20	54	80

Table 5: Error of the estimated trade-off for a new mesh (node hours).

Figure 3: Estimated and measured extra node hours of full speed for a new mesh.

High-Performance Computing

High-performance computing (HPC) uses parallel systems and optimized software to solve computationally demanding scientific and engineering problems.

See all 143 papers

One Communicator, Three Vendors: Extending the MoA Predictive Theory to a Real NVIDIA+AMD+CPU MPI Ensemble
Oct 16, 2026Lenore Mullin, Gaetan HainsHigh-Performance Computing Heterogeneous Computing
College of Nanotechnology, Science and Engineering, University at Albany, SUNY, Albany, NY, USA · LACL, Université Paris-Est Créteil, Créteil, France
TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes
Aug 13, 2026Jie Li, Chenxin Jia, Jinliang Shen +5Expert Parallelism Expert Load Balancing
KlingAI Research
HiDVFS: Hierarchical Multi-Agent DVFS for Real-Time OpenMP DAG Workloads
Jan 10, 2026Mohammad Pivezhandi, Abusayeed Saifullah, Ali JannesariReal-Time Scheduling Multi-Agent Reinforcement Learning
Department of Computer Science Wayne State University, Detroit, MI, USA · Department of Computer Science University of Texas at Dallas, Richardson, TX, USA · Department of Computer Science Iowa State University, Ames, IA, USA

Oct 16, 2026J/K move · Enter open · S save

One Communicator, Three Vendors: Extending the MoA Predictive Theory to a Real NVIDIA+AMD+CPU MPI Ensemble

Lenore Mullin, Gaetan Hains

College of Nanotechnology, Science and Engineering, University at Albany, SUNY, Albany, NY, USA · LACL, Université Paris-Est Créteil, Créteil, France

High-Performance Computing Heterogeneous Computing

Open paper AskPDF arXiv

The companion MPI-extension paper validated the first communication-bearing machine-shape ensemble: four NVIDIA A100 ranks and four CPU ranks on one NCSA Delta node under one Open MPI job, reassembled by MPI_Gatherv. This paper extends it to a multi-vendor communicator. A Slurm heterogeneous job spanning Delta's A100 and AMD MI100 partitions places A100, MI100 and CPU ranks (nine in all) under one communicator for the first time. We give the theory (a w_i proportional to 1/t_i weight split for three concurrent realizations; a het-job specification as an admissible directive) and report real results, including what the run exposed: the 1/t split assumes time linear in rows, and this fails for the MI100 rank (6 rows took 597 ms against a linear prediction of 1.6 ms). A launch-geometry model, with predictions recorded before measuring, succeeds for the A100 (time flat from 128 to 3456 rows, stepping at exactly 3520 and 6976) and places the MI100 floor in the key loop; changing its launch settings cut about 600 ms to 14.6 ms and the energy per call at 6 rows by 32x, though the MI100 remains about 95x slower than an A100 at 8192 rows. Controlled reruns show that the companion paper's 8.1x A100 result and 22 ms Gatherv were artifacts of four ranks sharing one GPU under Open MPI's default transport. A key/value-split partition whose message sizes are derived from shapes alone is then run: the ring Allreduce of the 8.5 MB message took 11.3, 16.0 and 15.3 ms against a predicted 11-12 ms, the small Allreduce missed its prediction in every run, and compute times reproduce to about 1% while communication times vary up to 4x between runs. Each idea is introduced with a figure and closed with worked examples.

Abstract

High-Performance Computing

Explore similar work

One Communicator, Three Vendors: Extending the MoA Predictive Theory to a Real NVIDIA+AMD+CPU MPI Ensemble

TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes

HiDVFS: Hierarchical Multi-Agent DVFS for Real-Time OpenMP DAG Workloads