cs.LGAug 24, 2026

How Much Regularization Survives Averaging? Update Masking in Federated Learning

Authors: Wenhao Yan, Fu Kuroda, Yucheng Jin, Zhenke Chen

Organizations: Faculty of Science & Technology Sophia University Tokyo, Japan · Technical R&D Department Shendian Energy Co., Ltd. Lishui, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Federated learning on non-IID data seeks flat minima to generalize across clients, and existing methods borrow sharpness-aware minimization from centralized training. There is a second way to reach flat minima, in which the regularization comes for free from noise added to the parameter updates, and it has never been carried over to the federated setting as an implicit regularizer. We show the reason. Masking charges the optimizer for moving in sharp directions. We prove that when each client draws its own mask, federated averaging weakens that charge by exactly the cohort size, and that giving every client the same mask brings it back by a factor equal to the inverse gradient diversity of the cohort. In our experiment setting on CIFAR-10, that factor is 1.19 out of a possible 10. Turning off minibatch sampling raises it to 8.96, while changing data heterogeneity a thousandfold leaves it between 1.17 and 1.50. The configurations keeping the regularization train far too poorly to use.

Figures & tables

   1/ΔS1/\Delta_{S}    
bsbsτ\tauTTunwtd.nkn_{k} -wtd.μ^2\hat{\mu}^{2}σ^2\hat{\sigma}^{2}accrds.<1%\begin{subarray}{c}\text{rds.}\\ <1\,\%\end{subarray}
full118.968.953.6×10−33.6\times 10^{-3}4.9×10−44.9\times 10^{-4}0.3140.0
full224.134.136.2×10−46.2\times 10^{-4}1.2×10−31.2\times 10^{-3}0.3830.0
full553.963.963.7×10−33.7\times 10^{-3}7.6×10−37.6\times 10^{-3}0.4930.0
128141.361.358.4×10−48.4\times 10^{-4}1.9×10−21.9\times 10^{-2}0.4494.5
128281.321.311.2×10−31.2\times 10^{-3}3.4×10−23.4\times 10^{-2}0.5341.7
TABLE I: FedAvg on a near-IID split ( α=100\alpha=100 ), Kc=10K_{c}=10 of 100 clients per round, 200 rounds, 3 seeds, sweeping local batch size bsbs and epochs τ\tau over T=⌈nˉk/bs⌉ τT=\lceil\bar{n}_{k}/bs\rceil\,\tau local steps per round ( nˉk≈500\bar{n}_{k}\approx 500 ). Entries are medians across seeds of each run’s median over the last 50 rounds; Kc=10K_{c}=10 is the ceiling on 1/ΔS1/\Delta_{S} .
Fig. 1: Recovery factor against final test accuracy for the nine configurations of Table I . Marker shape denotes local batch size, annotations denote τ\tau .
 1/ΔS1/\Delta_{S}    
α\alphaunwtd.nkn_{k} -wtd.μ^2\hat{\mu}^{2}σ^2\hat{\sigma}^{2}accrds.<1%\begin{subarray}{c}\text{rds.}\\ <1\,\%\end{subarray}
1001.191.194.1×10−34.1\times 10^{-3}1.9×10−11.9\times 10^{-1}0.6800.0
11.191.173.9×10−33.9\times 10^{-3}1.7×10−11.7\times 10^{-1}0.6260.2
0.11.501.186.5×10−36.5\times 10^{-3}1.1×10−11.1\times 10^{-1}0.4393.0
TABLE II: Varying the Dirichlet concentration α\alpha with the local protocol fixed at bs=32bs=32 , τ=2\tau=2 . Shard sizes are far more uneven at α=0.1\alpha=0.1 , so the two aggregation conventions differ there.

Explore similar work

CardsList

  1. FedeRage: Provably Convergent Agnostic Federated Learning under General Client Drift

    Sep 17, 2026Herlock Rahimi, Dionysis KalogeriasFederated LearningFederated Learning Aggregation

  2. Beyond Conventional Federated Learning via High-Order Regularization

    Sep 9, 2026Alireza Kabgani, Masoud AhookhoshFederated Learning

    Department of Mathematics, University of Antwerp, Antwerp, Belgium

  3. Beyond Non-IID: Learner--Client Distribution Mismatch in Federated Learning

    Aug 27, 2026Yiming Xie, Lili Su, Ningfang MiClient Selection in Federated LearningNon-IID Federated Learning

    Department of Electrical and Computer Engineering Northeastern University, Boston, MA, USA