Conflict, Feasibility, Faithfulness, Displacement: Four Quantities That Come Apart in Federated Dual-Cone Aggregation
Authors: Grzegorz Gwardys, Karol Wojciechowski, Mateusz Gwardys, Bartosz {Ś}widerski
Organizations: 1*Faculty of Electronics and Information Technology, Warsaw University of Technology, Warsaw, Poland. · Department of Artificial Intelligence, Institute of Information Technology, Warsaw University of Life Sciences, Warsaw, Poland. · Institute of Computer Science, Cardinal Stefan Wyszyński University, Warsaw, Poland.
FedFV, FedGH and FedMGDA+ aggregate client updates with multi-objective gradient aggregators to remove conflict between them. We examine this rationale with UPGrad, which increases no client's loss to first order, in a comparison that changes only the direction of FedAvg's update. Four quantities the rationale treats as one -- conflict between client updates, feasibility of their weighted mean, faithfulness of the transmitted updates to the clients' gradients, and displacement of the update -- come apart. Under Dirichlet partitioning UPGrad improves CIFAR-10 accuracy by +0.78 pp (14/15 seeds, p < 0.001), only with full participation; on Oxford-IIIT Pet its +0.52 pp gain vanishes (+0.07 pp) once the data our protocol left unused are kept. Varying only the Dirichlet concentration, the gain grows with heterogeneity, yet the split with the most conflict shows none. Uniform client weights, blind to conflict, match the CIFAR-10 gain and exceed the Pets gain; a matched random rotation does neither. No fairness effect survives multiplicity correction, and on CIFAR-10 the guarantee does not reach the clients' gradients. We state what evaluations should report.
Figures & tables
accuracy (%)
p
Regime
n
mean
UPGrad
Δ (pp)
wins
t
sign
90% CI
pTOST
Disjoint
15
38.94
38.71
−0.23
6/15
0.362
0.607
[−0.66,+0.20]
0.141
Dirichlet
15
56.72
57.51
+0.78
14/15
< 0.001
< 0.001
[+0.55,+1.01]
0.974
Pets
10
80.66
81.18
+0.52
10/10
< 0.001
0.002
[+0.33,+0.71]
0.587
Table 1: Centralized test accuracy, weighted mean vs. UPGrad, paired by seed. Δ is UPGrad minus mean; wins counts seeds on which UPGrad scored higher; t and sign are two-sided paired t - and exact sign tests; the 90% CI and pTOST belong to the equivalence test against ±0.5 pp.
Figure 1: Paired differences (UPGrad minus mean) at the final round, with 95% confidence intervals. Positive favours UPGrad except for the gap and the spread, where negative does.
drop_last=True
drop_last=False
change
Metric
Δ
wins
p
Δ
wins
p
Δ
p
centralized accuracy
+0.52
10/10
< 0.001
+0.07
6/10
0.320
−0.46
< 0.001
client-mean accuracy
+0.68
9/10
0.007
+0.35
8/10
0.006
−0.33
0.219
sample-weighted accuracy
+0.63
9/10
0.004
+0.31
7/10
0.016
−0.32
0.177
worst-client accuracy
+1.18
5/10
0.124
−0.28
2/10
0.677
−1.46
0.136
worst-decile accuracy
+0.73
6/10
0.248
−0.22
4/10
0.645
−0.95
0.034
Table 2: Pets, UPGrad minus mean at round 40, under both drop_last settings on the same ten seeds; change is the paired difference of the two differences.
Dirichlet α
0.1
0.5
1.0
10.0
Δ accuracy (pp, wins)
+0.85 (7/10)
+0.92 (10/10)
+0.25 (9/10)
+0.06 (5/10)
p ( t / sign)
0.062 / 0.344
< 0.001 / 0.002
0.016 / 0.021
0.332 / 1.000
Δ client-mean (pp, wins)
+2.60 (10/10)
+0.93 (10/10)
+0.20 (7/10)
+0.12 (5/10)
p ( t )
< 0.001
0.009
0.097
0.182
Δ worst client (pp)
+2.91
+1.39
+0.01
+1.00
Δ gap (pp)
+0.64
−0.89
+0.16
−0.97
Table 3: Dirichlet CIFAR-10 at four concentrations ( K=20 , seeds 1–10; at α=0.5 the first ten seeds of the main sweep), UPGrad minus mean at round 60; lower α is more heterogeneous. % of floor : cˉ as a percentage of −1/(K−1) , negative when the clients agree on average. Infeasible : share of round-seeds in which the weighted mean of the transmitted updates would harm some client to first order, along each arm’s trajectory. Align : median cos(gk,hk) over clients (UPGrad arm); cos(d(g),d(h)) : UPGrad’s direction on the transmitted updates against its direction on the true gradients. Removed by equal norms : share of the rotation that setting all update norms equal removes (the Gramian counterfactual of Sect. 6.5 ); weight mass moved : median total-variation distance between UPGrad’s implied client weights and the sample-size weights.
mean pairwise cos
Regime
K
n
avg
peak
% floor
min pair
Disjoint
5
900
−0.151
−0.202
81%
−0.307
Dirichlet
20
900
+0.005
−0.032
61%
−0.650
Pets
20
400
+0.051
+0.011
—
−0.348
Table 4: Mean pairwise cosine between client updates, pooled over every round and seed of the UPGrad arm ( n = rounds × seeds). avg is the typical round, peak the most conflicted one, % floor the peak as a share of −1/(K−1) , and min pair the most opposed pair observed.
Figure 2: Seed-averaged, per round. (a) Mean pairwise cosine as a percentage of the floor −1/(K−1) ; negative values mean a net agreeing population. (b) Cosine between the UPGrad update and the weighted mean: the rotation that remains once the step norm is equalized.
Disjoint
Dirichlet
Pets
clients K
5
20
20
n (rounds × seeds)
900
900
400
infeasible: mean
57 (6%)
750 (83%)
0
infeasible: projected
0
0
0
max clients harmed
1/5
9/20
0/20
worst margin: mean
−0.099
−0.216
+0.032
Table 5: Feasibility of the weighted mean and of the UPGrad update, pooled over every round and seed. infeasible counts round-seeds in which at least one client has a negative margin; worst margin is the minimum cosine margin observed.
align , against the
harmed, mean → proj.
Regime
Round
eval function
train-mode function
transmitted
true
Disjoint
1
0.66±0.04
same function
0.0→0.0
2.1→2.5
20
0.63±0.06
same function
0.2→0.0
2.3→1.5
60
0.38±0.04
same function
0.0→0.0
2.1→1.9
Dirichlet
1
0.85±0.03
same function
0.1→0.0
7.3→6.4
20
0.40±0.03
same function
2.0→0.0
6.7→0.2
Table 6: Transmitted updates vs. true gradients at three probe rounds (mean ± sd over seeds). align is the median cos(gk,hk) over clients, against the eval -mode function and, for Pets, the train-mode function local training differentiates (the CIFAR-10 model has no BatchNorm). The last two columns count harmed clients for the mean → UPGrad, on the transmitted updates and on the true gradients.
step, as a fraction of ηs
Configuration
n
objective
1
1/4
1/16
1/64
Disjoint, 1 full-batch step
60
Lk
9
1
0
0
Lk
9
1
0
0
sign flips
0
0
0
0
Disjoint, 1 epoch
60
Lk
24
18
18
18
Lk
24
18
18
18
Table 7: Client-rounds in which UPGrad’s step raises the client’s loss, as the step shrinks from ηs to ηs/64 , counted on the criterion Lk and on the regularized objective Lk , with the number of client-rounds whose sign differs.
Regime
n
cos vs. mean
Norm ratio
Disjoint
900
0.986±0.011
2.15±0.45
[0.940,0.999]
[1.07,3.02]
Dirichlet
900
0.957±0.018
1.67±0.34
[0.906,0.999]
[1.05,2.35]
Pets
400
0.993±0.004
1.36±0.17
[0.980,1.000]
[1.01,1.75]
Table 8: Displacement of the UPGrad update from the weighted mean, pooled over every round and seed. Top: cosine to the mean and norm ratio (removed by our rescaling), mean ± sd and range. Bottom: mean rotation in degrees when one asymmetry of each round’s Gramian is removed—unequal weights p , unequal norms diag(G) , or dispersion of the pairwise cosines—with the share of the measured rotation the removal eliminates.
Figure 3: Paired difference per round, UPGrad minus weighted mean, with 95% confidence bands: (a) centralized test accuracy, (b) centralized test loss. The solid green line is Pets with every training sample kept (Sect. 5.2 ).
rotation
mean
difference from FedAvg, pp (wins, p )
arm
( ∘ )
infeasible (%)
centralized accuracy
client-mean accuracy
Dirichlet CIFAR-10
FedAvg
0
7.7
UPGrad
16.2
84
+0.92 (10/10, < 0.001)
+0.93 (10/10, 0.009)
uniform weights
22.0
79
+0.96 (9/10, 0.003)
+1.13 (9/10, 0.004)
matched rotation
9.9
22
+0.17 (6/10, 0.337)
+0.08 (6/10, 0.598)
Table 9: The two controls, seeds 1–10 paired with the main sweeps: Dirichlet CIFAR-10 ( K=20 , α=0.5 ) at round 60 and Pets under the original protocol ( drop_last=True ) at round 40. Rotation : mean angle between the arm’s direction and the weighted mean of the transmitted updates. Mean infeasible : share of round-seeds in which that weighted mean would harm some client to first order, along the arm’s own trajectory. Differences from FedAvg in pp, with the seeds in which the arm is ahead and the paired t -test p ; worst-client accuracy resolves for no arm.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Disjoint
Dirichlet
Pets
Metric
Δ
W
p
psign
Δ
W
p
psign
Δ
W
p
psign
centralized
−0.23
6/15
0.362
0.607
+0.78
14/15
< 0.001 ‡
< 0.001
+0.52
10/10
< 0.001 ‡
0.002
client mean
+0.003
7/15
0.841
1.000
+0.88
15/15
< 0.001 ‡
< 0.001
+0.68
9/10
0.007 †
0.021
sample-weighted
+0.003
7/15
0.841
1.000
+0.77
15/15
< 0.001 ‡
< 0.001
+0.63
9/10
0.004 †
0.021
worst client
+1.12
9/15
0.302
0.607
+1.64
12/15
0.047 †
0.035
+1.18
5/10
0.124
1.000
worst decile
+1.12
9/15
0.302
0.607
+1.54
11/15
0.038 †
0.118
+0.73
6/10
0.248
0.754
Appendix
Table 10: All 21 paired tests: seven metrics in three regimes. Δ in pp (UPGrad minus mean), W the seeds favouring UPGrad, p the paired t -test and psign the exact sign test. † : p<0.05 ; ‡ : below the Bonferroni threshold 0.05/21=0.0024 . Centralized, client-mean and sample-weighted accuracy are three readings of one quantity, and worst client and worst decile of another, so the tests are not independent; on the disjoint split, where shards are equal, client-mean and sample-weighted accuracy coincide.
rounds with
rel. error of predicted ratio
Regime
K
c<0
median
p90
max
Disjoint
5
897/900
2.1%
4.7%
10.5%
Dirichlet
20
636/900
5.6%
16.5%
33.2%
Pets
20
0/400
—
—
—
Appendix
Table 11: The symmetric model of Eq. ( 6 ) against every round with a negative mean cosine: relative error of the predicted norm ratio. On Pets the mean cosine is never negative.
prediction
step, as a fraction of ηs
Configuration
uses
1
1/4
1/16
1/64
Disjoint, 1 epoch
from update
502
652
644
644
interquartile
[-873, 1205]
[-336, 1901]
[-338, 1679]
[-339, 1676]
from gradient
0.99
1.01
1.00
1.00
Dirichlet, 1 epoch
from update
427
412
402
401
interquartile
[259, 792]
[235, 891]
[232, 857]
[233, 854]
Appendix
Table 12: The two error sources of Eq. ( 5 ), told apart by shrinking the step: median over clients and probe rounds of the predicted loss change divided by the measured one ( 1 means exact), predicted from the transmitted update or from the true gradient, with the interquartile range of the former. In the one-step control the ratio from the update tends to 1+λw⊤d/hk⊤d , not 1 , because there gk=hk+λw .
per client
after aggregation
Regime
Rd
∥hk∥∥gk∥
cos(gk,hk)
cos(d(g),d(h))
cos(d(s),d(h))
cos(d(g),d(s))
Disjoint, K=5
1
414
0.65
0.094
1.0000
0.094
10
45
0.71
0.077
0.9999
0.077
30
34
0.57
0.018
0.9998
0.018
60
41
0.40
0.032
0.9994
0.033
Dirichlet, K=20
1
315
0.85
0.249
0.9998
0.249
Appendix
Table 13: Scale and direction of the transmitted update (3 seeds). Per client: median norm ratio and alignment. After aggregation: cosines between UPGrad’s output on the updates d(g) , on the true gradients d(h) and on sk=(∥gk∥/∥hk∥)hk .
accuracy (%) at round
ηs
n
1
10
20
peak
beyond the scan
0.01
3
9.8
9.8
9.8
9.8
10.1 at round 150
0.1
1
9.4
10.0
10.7
10.7
not run further
0.5
1
9.6
12.3
19.0
21.2
used for the comparison
1.0
1
10.0
20.2
15.0
21.3
unstable: peaks, then falls back
Appendix
Table 14: Screening scan of the server rate for the one-step control: no-projection arm, disjoint split, accuracy (%) at three rounds and the peak over the first 20.
Federated learning aggregation combines client updates into shared or personalized models while balancing data heterogeneity, robustness, privacy, and communication.
Department of Mathematics, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany. · School of Computation, Information and Technology, Technical University of Munich, Garching, Germany. · Munich Center for Machine Learning, Munich, Germany.