Whether a federated predictor benefits an individual client depends on both population heterogeneity and estimation uncertainty. We study this question for Nadaraya--Watson regression in a triangular array with potentially different fitting and evaluation distributions. Each client transmits local kernel summaries, allowing the server to construct a covariate-density-weighted federated estimator without centralizing individual observations. A client combines its fitted local predictor with the federated predictor through a prescribed weight, and gain is the paired reduction in squared prediction loss on an independent audit observation. Exact identities characterize both the gain of the realized fitted pair and its repeated-fitting mean. We also establish pointwise inference for the federated estimator. A two-stage analysis of client gains separates fitting variability from audit variability, retains the covariance created when a client's observations enter both its local estimator and the common server estimator, and yields fixed-block simultaneous inference across clients. Robustness is examined under two perturbation models. A Kullback--Leibler neighborhood provides a variance-sensitive lower certificate in a regular positive-gain-variance regime, while an L2 neighborhood yields an exact least-favorable regression direction and robust value. Exact-model, vanishing-variance, and exact-zero-direction boundaries are kept separate because their scalar inputs are nonregular. The framework thus distinguishes decisions about a realized fitted pair from claims about the learning procedure under repeated fitting.
Figures & tables
Gain
95% coverage
Vfit/VfitMC
η
Regime
GP
Γ
Gfit
Correct
Indep.
Eval.
Correct
Indep.
0.00
Negative
−0.255
−0.250
0.950
0.938
0.901
0.756
0.980
0.626
0.25
Near-zero
0.000
0.005
0.953
0.947
0.925
0.729
0.967
0.740
1.00
Positive
0.766
0.769
0.951
0.951
0.977
0.591
0.938
1.354
Table 1: Two-client verification of realized-fit and procedure-level gain inference. “Correct” retains fitting uncertainty and local–server covariance; “Indep.” drops the shared-data covariance term; and “Eval.” uses evaluation variance only. Variance ratios compare the mean estimated fitting variance with its Monte Carlo benchmark.
bL
bF
Γ
Δbias
Δvar
D
Vfit/VfitMC
2-stage
Eval.
0.25
0.00
0.0639
0.0560
0.0079
−0.753
0.979
0.947
0.745
0.25
0.10
0.0462
0.0383
0.0079
−0.658
0.962
0.939
0.657
0.25
0.25
0.0089
0.0011
0.0079
0.765
0.917
0.926
0.521
0.50
0.00
0.2308
0.2228
0.0079
−0.931
1.025
0.946
0.773
0.50
0.10
0.1979
0.1899
0.0079
−0.920
1.025
0.945
0.738
0.50
0.25
0.1379
0.1299
0.0079
−0.885
1.023
0.950
0.641
Table 2: Complete grid for Experiment 1. Gain components are reported on the original scale. The mechanism score D is negative under bias-correction dominance and positive under variance-rescue dominance. The final two columns report empirical 95% coverage for the correct two-stage interval and for an interval using evaluation variance only.
Figure 1: Regular-gain diagnostics for Experiment 1. (a) Phase diagram of bias correction and variance rescue over the (bL,bF) grid. (b) Fitting, evaluation, and total variance calibration at (bL,bF)=(0.25,0) .
Figure 2: Decomposition of the marginal gain in Experiment 1. (a) Bias-correction contribution Δbias . (b) Variance-rescue contribution Δvar .
Figure 3: Exact-model boundary diagnostics. (a) Monte Carlo fitting variance compared with the quadratic-boundary and persistent-linear variance estimates. (b) Empirical 95% procedure-level coverage using quadratic plus evaluation variance, persistent-linear plus evaluation variance, and evaluation variance alone.
Client
Γj
Mean (Zj)
SD (Zj)
Coverage
Vj/VjMC
1
0.0491
−0.081
0.972
0.961
0.945
2
0.0280
−0.026
0.967
0.955
0.983
3
0.0891
−0.009
0.983
0.955
0.995
4
0.0196
−0.024
1.007
0.942
0.942
5
0.0658
0.007
1.012
0.952
0.983
Table 3: Marginal calibration for the five-client block in Experiment 3. The last column is the ratio of the mean estimated total variance to its Monte Carlo benchmark.
Figure 4: Client-indexed correlation calibration in Experiment 3. The left panel gives the Monte Carlo reference correlation matrix and the right panel gives the oracle-nuisance influence-based estimate. Diagonal entries equal one; the color scale summarizes the off-diagonal correlations generated by the shared federated estimator.
Event
Empirical
Gaussian MC
Gaussian est.
Independent
Gj>0,∀j
0.921
0.889
0.896
0.877
Gj>Γj−0.5sj,∀j
0.315
0.317
0.327
0.158
Table 4: Fixed-block orthant probabilities in Experiment 3. Gaussian approximations are computed using the Monte Carlo covariance, the estimated covariance, or a diagonal covariance that incorrectly treats clients as independent.
Figure 5: Oracle grid calibration of the sub-gamma scale used in Experiment 4. The calibrated envelope dominates the centered log-MGF over the displayed admissible grid.
Figure 6: KL robustness and inference diagnostics for Experiment 4. (a) Exact empirical KL-robust gain and the variance-sensitive certificate. (b) Empirical probability that the estimated certificate is positive. (c) Empirical coverage of the one-sided 95% lower confidence bound for the procedure-level marginal certificate.
cρ
ρ
Pr(C>0)
LCB coverage
0.00
0.0000
1.000
0.933
0.25
0.0051
1.000
0.934
0.50
0.0101
1.000
0.936
1.00
0.0203
1.000
0.937
2.00
0.0405
0.986
0.937
Table 5: KL robustness and inference diagnostics for Experiment 4.
Figure 7: L2 robustness in the persistent-direction regime. (a) Monte Carlo mean fitted robust value and its audit estimate at n=200 over the radius grid. (b) Convergence of the estimated least-favorable regression function toward the population oracle at cτ=1 .
n
Mean L2 error
100
0.1680
200
0.1202
400
0.0910
800
0.0699
Table 6: Convergence of the estimated least-favorable regression function in the persistent-direction regime.
Figure 8: L2 robustness at the vanishing-direction boundary. (a) Fitting-variance effect of the stage-specific coefficient κB=τ/ν , reported relative to the nominal gain-boundary variance. (b) Variance of the fitted robust value standardized by the full vanishing-direction boundary scale.
Federated learning (FL) is intrinsically heterogeneous: honest clients may have different data-generating models. On top of that, adversarial clients can make heterogeneity even more pronounced by sharing arbitrary updates. Existing analyses typically control the interaction between statistical heterogeneity and adversarial behavior through gradient-dissimilarity conditions. However, the underlying bound is imposed a priori and may yield conservative guarantees even for least-squares regression. We instead derive the gradient heterogeneity from the statistical model of linear and nonlinear regression with fresh data samples at every round. Our bounds separate heterogeneity among the honest clients' ground-truth model parameters, finite-sample label noise, and initialization. We then demonstrate that, for any (f,κ)-robust aggregator with coefficient κ=O(f/n), where f is the number of adversarial clients and n the total number of clients (with f/n<1/2), convergence holds after an explicit sample burn-in.
Leonardo F. Toso, James Anderson, Nirupam Gupta +1
Columbia University, USA · University of Copenhagen, Denmark · Sorbonne Université and Université Paris Cité, CNRS, LPSM, France
In federated averaging, local objectives can admit multiple optimal heads, making the aggregate depend on which heads clients return. We study this ambiguity in federated multivariate regression with private backbones and a shared linear head, using an unconstrained feature model (UFM) that treats training-sample features as free variables. We introduce a natural selection rule: each client returns the optimal head closest to the broadcast head. We show that global minimization with a vanishing proximal penalty on the head realizes this rule. When the clients' optimal Gram matrices and the initial shared Gram matrix are positive definite, the shared Gram matrix follows a closed recursion and converges to the unique Bures-Wasserstein barycenter of the clients' optimal Gram matrices. Even with this alignment, the limit generally differs from the centralized optimal Gram matrix. We decompose this gap into three positive-semidefinite terms arising from differences in client target means, covariance heterogeneity, and averaging the aligned heads. A correction based on a one-time exchange of target means and covariances recovers the centralized optimal Gram matrix in one round under exact local optimization and the same selection rule. We verify these results numerically in the UFM and test its predictions on five tabular and five image regression datasets using deep networks with feature regularization and long local training. In these experiments, ordinary training approaches the predicted barycenter, while a weak proximal penalty improves endpoint agreement and yields trajectories that closely follow the predicted Gram dynamics. The correction moves the final Gram matrices close to the centralized UFM prediction.
This paper introduces range regularization for federated learning with linear systematic components to enhance statistical accuracy and induce cross-client regularity conducive to quantization, coding, and resource efficiency. Our approach identifies features with shared weights across different clients and adaptively clusters the weights of personalized features at extreme values, a process we refer to as polar clustering. Theoretical analysis of the associated estimators poses significant challenges due to the seminorm nature and non-decomposability of the regularizer. We develop new proof techniques for the nonasymptotic analysis of statistical accuracy and faithful pattern recovery. Moreover, a fast optimization algorithm that leverages varying degrees of local strong convexity is proposed to reduce iteration complexity. Experiments support the efficacy and efficiency of the proposed approach.