Conflict, Feasibility, Faithfulness, Displacement: Four Quantities That Come Apart in Federated Dual-Cone Aggregation
Organizations: 1*Faculty of Electronics and Information Technology, Warsaw University of Technology, Warsaw, Poland. · Department of Artificial Intelligence, Institute of Information Technology, Warsaw University of Life Sciences, Warsaw, Poland. · Institute of Computer Science, Cardinal Stefan Wyszyński University, Warsaw, Poland.
Abstract
FedFV, FedGH and FedMGDA+ aggregate client updates with multi-objective gradient aggregators to remove conflict between them. We examine this rationale with UPGrad, which increases no client's loss to first order, in a comparison that changes only the direction of FedAvg's update. Four quantities the rationale treats as one -- conflict between client updates, feasibility of their weighted mean, faithfulness of the transmitted updates to the clients' gradients, and displacement of the update -- come apart. Under Dirichlet partitioning UPGrad improves CIFAR-10 accuracy by +0.78 pp (14/15 seeds, p < 0.001), only with full participation; on Oxford-IIIT Pet its +0.52 pp gain vanishes (+0.07 pp) once the data our protocol left unused are kept. Varying only the Dirichlet concentration, the gain grows with heterogeneity, yet the split with the most conflict shows none. Uniform client weights, blind to conflict, match the CIFAR-10 gain and exceed the Pets gain; a matched random rotation does neither. No fairness effect survives multiplicity correction, and on CIFAR-10 the guarantee does not reach the clients' gradients. We state what evaluations should report.
Figures & tables
| accuracy (%) | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Regime | mean | UPGrad | (pp) | wins | sign | CI | |||
| Disjoint | 15 | 38.94 | 38.71 | 6/15 | 0.362 | 0.607 | 0.141 | ||
| Dirichlet | 15 | 56.72 | 57.51 | 14/15 | 0.001 | 0.001 | 0.974 | ||
| Pets | 10 | 80.66 | 81.18 | 10/10 | 0.001 | 0.002 | 0.587 | ||
| drop_last=True | drop_last=False | change | ||||||
|---|---|---|---|---|---|---|---|---|
| Metric | wins | wins | ||||||
| centralized accuracy | 10/10 | 0.001 | 6/10 | 0.320 | 0.001 | |||
| client-mean accuracy | 9/10 | 0.007 | 8/10 | 0.006 | 0.219 | |||
| sample-weighted accuracy | 9/10 | 0.004 | 7/10 | 0.016 | 0.177 | |||
| worst-client accuracy | 5/10 | 0.124 | 2/10 | 0.677 | 0.136 | |||
| worst-decile accuracy | 6/10 | 0.248 | 4/10 | 0.645 | 0.034 | |||
| Dirichlet | 0.1 | 0.5 | 1.0 | 10.0 |
|---|---|---|---|---|
| accuracy (pp, wins) | (7/10) | (10/10) | (9/10) | (5/10) |
| ( / sign) | 0.062 / 0.344 | 0.001 / 0.002 | 0.016 / 0.021 | 0.332 / 1.000 |
| client-mean (pp, wins) | (10/10) | (10/10) | (7/10) | (5/10) |
| ( ) | 0.001 | 0.009 | 0.097 | 0.182 |
| worst client (pp) | ||||
| gap (pp) |
| mean pairwise | ||||||
|---|---|---|---|---|---|---|
| Regime | avg | peak | % floor | min pair | ||
| Disjoint | 5 | 900 | 81% | |||
| Dirichlet | 20 | 900 | 61% | |||
| Pets | 20 | 400 | — | |||
| Disjoint | Dirichlet | Pets | |
|---|---|---|---|
| clients | 5 | 20 | 20 |
| (rounds seeds) | 900 | 900 | 400 |
| infeasible: mean | 57 (6%) | 750 (83%) | 0 |
| infeasible: projected | 0 | 0 | 0 |
| max clients harmed | 1/5 | 9/20 | 0/20 |
| worst margin: mean |
| align , against the | harmed, mean proj. | ||||
|---|---|---|---|---|---|
| Regime | Round | eval function | train-mode function | transmitted | true |
| Disjoint | 1 | same function | |||
| 20 | same function | ||||
| 60 | same function | ||||
| Dirichlet | 1 | same function | |||
| 20 | same function | ||||
| step, as a fraction of | ||||||
|---|---|---|---|---|---|---|
| Configuration | objective | |||||
| Disjoint, 1 full-batch step | 60 | 9 | 1 | 0 | 0 | |
| 9 | 1 | 0 | 0 | |||
| sign flips | 0 | 0 | 0 | 0 | ||
| Disjoint, 1 epoch | 60 | 24 | 18 | 18 | 18 | |
| 24 | 18 | 18 | 18 | |||
| Regime | vs. mean | Norm ratio | |
|---|---|---|---|
| Disjoint | 900 | ||
| Dirichlet | 900 | ||
| Pets | 400 | ||
| rotation | mean | difference from FedAvg, pp (wins, ) | ||
|---|---|---|---|---|
| arm | ( ∘ ) | infeasible (%) | centralized accuracy | client-mean accuracy |
| Dirichlet CIFAR-10 | ||||
| FedAvg | 0 | 7.7 | ||
| UPGrad | 16.2 | 84 | (10/10, 0.001) | (10/10, 0.009) |
| uniform weights | 22.0 | 79 | (9/10, 0.003) | (9/10, 0.004) |
| matched rotation | 9.9 | 22 | (6/10, 0.337) | (6/10, 0.598) |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Disjoint | Dirichlet | Pets | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Metric | W | W | W | |||||||||
| centralized | 6/15 | 0.362 | 0.607 | 14/15 | 0.001 ‡ | 0.001 | 10/10 | 0.001 ‡ | 0.002 | |||
| client mean | 7/15 | 0.841 | 1.000 | 15/15 | 0.001 ‡ | 0.001 | 9/10 | 0.007 † | 0.021 | |||
| sample-weighted | 7/15 | 0.841 | 1.000 | 15/15 | 0.001 ‡ | 0.001 | 9/10 | 0.004 † | 0.021 | |||
| worst client | 9/15 | 0.302 | 0.607 | 12/15 | 0.047 † | 0.035 | 5/10 | 0.124 | 1.000 | |||
| worst decile | 9/15 | 0.302 | 0.607 | 11/15 | 0.038 † | 0.118 | 6/10 | 0.248 | 0.754 | |||
| rounds with | rel. error of predicted ratio | ||||
|---|---|---|---|---|---|
| Regime | median | p90 | max | ||
| Disjoint | 5 | 897/900 | 2.1% | 4.7% | 10.5% |
| Dirichlet | 20 | 636/900 | 5.6% | 16.5% | 33.2% |
| Pets | 20 | 0/400 | — | — | — |
| prediction | step, as a fraction of | ||||
|---|---|---|---|---|---|
| Configuration | uses | ||||
| Disjoint, 1 epoch | from update | 502 | 652 | 644 | 644 |
| interquartile | [-873, 1205] | [-336, 1901] | [-338, 1679] | [-339, 1676] | |
| from gradient | 0.99 | 1.01 | 1.00 | 1.00 | |
| Dirichlet, 1 epoch | from update | 427 | 412 | 402 | 401 |
| interquartile | [259, 792] | [235, 891] | [232, 857] | [233, 854] | |
| per client | after aggregation | |||||
|---|---|---|---|---|---|---|
| Regime | Rd | |||||
| Disjoint, | 1 | 414 | 0.65 | 0.094 | 1.0000 | 0.094 |
| 10 | 45 | 0.71 | 0.077 | 0.9999 | 0.077 | |
| 30 | 34 | 0.57 | 0.018 | 0.9998 | 0.018 | |
| 60 | 41 | 0.40 | 0.032 | 0.9994 | 0.033 | |
| Dirichlet, | 1 | 315 | 0.85 | 0.249 | 0.9998 | 0.249 |
| accuracy (%) at round | ||||||
|---|---|---|---|---|---|---|
| 1 | 10 | 20 | peak | beyond the scan | ||
| 0.01 | 3 | 9.8 | 9.8 | 9.8 | 9.8 | at round |
| 0.1 | 1 | 9.4 | 10.0 | 10.7 | 10.7 | not run further |
| 0.5 | 1 | 9.6 | 12.3 | 19.0 | 21.2 | used for the comparison |
| 1.0 | 1 | 10.0 | 20.2 | 15.0 | 21.3 | unstable: peaks, then falls back |
Federated Learning Aggregation
Federated learning aggregation combines client updates into shared or personalized models while balancing data heterogeneity, robustness, privacy, and communication.
See all 293 papers