Federated Learning of Car Aerodynamics Surrogates with Spectral Optimizers
Authors: Arman Bolatov, Nikolay Spitsyn, Daniil Medyakov, Vladimir Aletov, Dmitry Bylinkin, Martin Takáč, Aleksandr Beznosikov
Organizations: Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · Moscow Independent Research Institute of Artificial Intelligence (MIRAI) · Federated Learning Problems Laboratory · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Innopolis University
Neural surrogates predict the aerodynamic forces on a car in seconds, compared with hours for a CFD simulation, but training them requires thousands of CFD simulations of different car designs, which car manufacturers keep proprietary. Federated learning allows several manufacturers to train a shared surrogate without exchanging data. We study which federated optimizer brings such a surrogate closest to one trained on pooled data. On DrivAerNet++, we train a Transolver surrogate across six clients that each hold cars of a single body type, and we compare three spectral optimizers, which orthogonalize the update of each weight matrix (FedMuon and two methods we propose, FedMARS-M and FedOLion), with three non-spectral ones (FedAvg, SCAFFOLD-Adam and FedLion), all tuned with the same protocol. FedMuon and FedMARS-M reach a drag error within 2% of centralized training with Muon, using two to four local epochs per round. The non-spectral methods perform best with sixteen local epochs, the largest number we tested, and still have a 17 to 37% higher drag error than FedMARS-M. We relate this gap to the low stable rank of the weight gradients, for which an orthogonalized step yields about twice the first-order decrease of a gradient step of the same spectral norm. Code: https://github.com/armanbolatov/fedmars_drivaernet
Figures & tables
Figure 1: Problem setting: (a) the airflow exerts a pressure and a wall shear stress on the surface of the car (colors are schematic), (b) integrated over the surface, the two fields give the drag and lift forces, and (c) each manufacturer trains on its own cars while only model updates are exchanged.
Group
Method
E
cd error
cl error
Spectral
FedMARS-M
2
6.77±0.04
26.3±0.3
FedMuon
4
6.86±0.08
26.4±0.1
FedOLion
8
7.90±0.51
27.7±0.2
Non-spectral
SCAFFOLD-Adam
16
7.94
27.0
FedLion
16
8.94
33.3
FedAvg
16
9.26
37.1
Table 1: Single-class partition after 100 rounds: errors of the drag and lift coefficients in units of 10−3 , each method at its tuned number of local epochs E .
Figure 2: Errors on the four client partitions with one local epoch per round, after 100 rounds. Horizontal lines show the centralized models.
cd error
cl error
Method
single-class
α=0.5
α=10
α=1000
single-class
α=0.5
α=10
α=1000
FedMARS-M
7.02
6.86
6.82
6.78
27.1
27.1
26.8
26.8
FedMuon
6.99
6.98
6.90
6.86
27.2
27.2
27.0
27.0
FedOLion
7.47
7.23
7.13
7.07
28.1
27.6
27.1
27.4
SCAFFOLD-Adam
13.7
14.2
14.4
14.3
51.1
45.9
45.3
45.5
FedLion
16.4
11.2
12.8
13.2
51.2
41.0
43.7
45.2
Table 2: One local epoch per round on the four partitions after 100 rounds, errors in units of 10−3 . Values above 17.5 have the same color as 17.5.
Figure 3: Training loss and force errors of the global model over the rounds on the single-class partition, for the runs of Table 1 .
Figure 4: Errors after 30 rounds as a function of the number of local epochs per round, with drag in the top row and lift in the bottom row. On the Dirichlet partitions, only the spectral methods were run.
cd error
cl error
Method
E
K=1
2
3
6
K=1
2
3
6
FedMARS-M
2
7.21
6.65
6.69
6.73
27.1
26.2
26.3
26.0
FedMuon
4
7.30
7.07
6.85
6.93
34.3
26.1
25.9
26.4
FedOLion
8
8.59
7.47
7.02
8.45
43.0
27.5
27.8
27.7
SCAFFOLD-Adam
16
NaN
NaN
8.01
7.94
NaN
NaN
28.7
27.0
FedLion
16
12.6
8.32
9.27
8.94
44.3
30.9
34.6
33.3
Table 4: Single-class partition with K of the six clients per round, after 100 rounds, errors in units of 10−3 . NaN marks a diverged run.
Figure 5: Partial participation on the single-class partition: training loss and force errors over the rounds for K clients per round.
Figure 6: The runs of Figure 5 as a function of the number of client uploads.
Figure 7: Gradient spectra of the 42 weight matrices on the single-class partition: stable rank, gain g of an orthogonalized step, and relative spectral norm ρ of a sign step.
Movement per round
Kept by averaging (%)
Method
E=1
2
4
8
16
E=1
2
4
8
16
FedMARS-M
12.3
23.4
46.0
74.3
102
60
49
42
37
38
FedMuon
12.2
25.1
46.1
84.0
116
60
49
43
35
36
FedOLion
5.85
10.5
17.7
30.7
52.2
65
55
48
42
37
SCAFFOLD-Adam
0.53
0.88
1.28
1.45
2.08
83
91
90
87
90
FedLion
4.30
6.60
9.72
12.4
16.4
91
89
85
79
76
Table 5: Client drift at round 5 of the runs of Table 3 : the mean Frobenius norm of the client updates over one round (all parameters), and the norm of their average as a fraction of it.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Data
DrivAerNet++: 5 703 training and 1 154 test cars, about 414 thousand surface points per car (330 to 513 thousand), 125 GB
Input and output
point coordinates and surface normals in, pressure and wall shear stress at every point out
Model
Transolver: 5 blocks, width 256, 8 heads, 32 slices, MLP ratio 1, dropout 0.2, GELU. 1.87M parameters, of which 1.85M in 42 weight matrices
Loss
sum of the mean squared errors of the normalized pressure and wall shear stress
Points per car
50 thousand sampled per training pass, all points at test time
Clients
6, and all of them train in every round unless K is stated
Appendix
Table 6: Training setup shared by all methods.
Partition
Clients
Cars per client
Entropy
single-class
6
917 / 950.5 / 1074
0.00
Dirichlet α=0.5
6
554 / 950.5 / 1954
0.56
Dirichlet α=10
6
700 / 950.5 / 1231
0.81
Dirichlet α=1000
6
943 / 950.5 / 959
0.82
Appendix
Table 7: Client partitions of the 5 703 training cars: the smallest, mean and largest number of cars per client, and the mean normalized entropy of the car categories within a client (0.82 for the whole training set).
Method
E
LR at E=1
LR at tuned E
Adam LR
β1 / β2
Weight decay
FedMuon
4
3e-3
3e-3
3e-5
0.9 / 0.95
0.1 (matrices)
FedMARS-M
2
3e-3
3e-3
1e-5
0.9 / 0.95
0.1 (matrices)
FedOLion
8
1e-3
1e-3
3e-6
0.95 / 0.95
0.1 (matrices)
Centralized Muon
–
1e-3
–
1e-5
0.95 / 0.95
0.1 (matrices)
FedAvg (AdamW)
16
1e-4, 1e-3, 3e-3, 1e-4
3e-4
–
0.7 / 0.999
0.01
SCAFFOLD-Adam
16
3e-5
1e-4
–
0.8 / 0.95
–
Appendix
Table 8: Selected hyperparameters, with learning rates written a e b for a×10b . Rates that differ between partitions are listed for single-class, α=0.5 , 10 and 1000.
Methods
Searched values
Size
Searched
spectral
matrix LR {3e-4, 1e-3, 3e-3, 1e-2, 3e-2}, Adam LR {3e-6, 1e-5, 3e-5, 1e-4, 3e-4}, β1 {0.7, 0.8, 0.9, 0.95, 0.99}
125
19%
FedAvg, centralized AdamW
LR {1e-5, 3e-5, 1e-4, 3e-4, 1e-3}, β1 as above
25
68%, 72%
SCAFFOLD-Adam
LR {1e-6, 3e-6, 1e-5, 3e-5, 1e-4}, β1 as above
25
72%
FedLion
LR {1e-5, 3e-5, 1e-4, 3e-4, 1e-3}, β1 as above, β2 {0.9, 0.95, 0.99, 0.995}
100
24%
FedMARS-M, γ
MARS weight γ {0.01, 0.025, 0.04, 0.1, 1} at FedMARS-M’s selection. The value 0.04 scored 0.7% better than the reference 0.01, a difference within noise, so we kept 0.01
5
100%
Appendix
Table 9: Stage-1 search spaces. Size is the number of configurations in the space, and searched the share of them that was evaluated.
Figure 8: Stage-2 learning-rate walks: cd error after 30 rounds at one local epoch, one line per partition. Circles mark the selected rate, triangles a diverged run.
Figure 9: The comparison of Figure 2 after 30 rounds, each method at its selected rate.
Method
Schedule
E=1
2
4
8
16
FedAvg
constant
21.5 / 60
20.1 / 58
18.4 / 57
14.0 / 65
13.6 / 54
cosine
18.7 / 61
19.1 / 57
18.3 / 56
14.4 / 66
11.6 / 57
SCAFFOLD-Adam
constant
22.8 / 56
22.0 / 59
14.9 / 57
11.9 / 46
14.4 / 45
cosine
17.4 / 61
19.0 / 56
13.9 / 59
11.7 / 56
10.7 / 46
FedLion
constant
31.6 / 85
27.0 / 69
19.0 / 58
15.3 / 56
12.2 / 50
cosine
25.9 / 73
22.4 / 70
16.3 / 61
13.7 / 53
10.3 / 42
Appendix
Table 10: Constant learning rate against the cosine schedule for the non-spectral methods: cd / cl error after 30 rounds on the single-class partition, units of 10−3 . The constant-rate runs used A100 GPUs and the cosine runs A6000 GPUs.
Partition
Method
E
K
A100
RTX A6000
single-class
FedMARS-M
2
6
6.73 / 26.0
6.69 / 26.3
FedMuon
4
6
6.93 / 26.4
6.77 / 26.0
FedAvg
1
6
17.0 / 57.8
15.9 / 59.2
FedAvg
16
6
9.26 / 37.1
10.0 / 38.3
FedAvg
16
3
9.80 / 34.8
9.53 / 36.3
FedAvg
16
2
10.3 / 33.3
8.52 / 32.5
Appendix
Table 11: The same configurations on A100 GPUs (the results of the paper) and on RTX A6000 GPUs after 100 rounds: cd error / cl error in units of 10−3 , with K clients per round.
GPU
Runs
Time per round
Peak memory
A100 40 GB
non-spectral, E=1
130 s
31.4 GB
A100 40 GB
non-spectral, E=16
2 060 s (2.4 days for 100 rounds)
32.0 GB
A100 40 GB
non-spectral, E=16 , K=3
1 050 s
32.0 GB
RTX A6000 48 GB
non-spectral, E=1
233 s
31.4 GB
RTX A6000 48 GB
non-spectral, E=16
3 820 to 4 230 s
31.4 GB
A100 40 GB
spectral, E=1
140 s
31.4 GB
Appendix
Table 12: Wall-clock time per round of one run on one GPU. Times grow linearly with E and with the number of clients per round.