Federated Nonparametric Regression with Heterogeneous Clients

Xiaoran Jiang; Anand N. Vidyashankar

stat.MLOct 23, 2026

Federated Nonparametric Regression with Heterogeneous Clients

Authors: Xiaoran Jiang, Anand N. Vidyashankar

Abstract

Whether a federated predictor benefits an individual client depends on both population heterogeneity and estimation uncertainty. We study this question for Nadaraya--Watson regression in a triangular array with potentially different fitting and evaluation distributions. Each client transmits local kernel summaries, allowing the server to construct a covariate-density-weighted federated estimator without centralizing individual observations. A client combines its fitted local predictor with the federated predictor through a prescribed weight, and gain is the paired reduction in squared prediction loss on an independent audit observation. Exact identities characterize both the gain of the realized fitted pair and its repeated-fitting mean. We also establish pointwise inference for the federated estimator. A two-stage analysis of client gains separates fitting variability from audit variability, retains the covariance created when a client's observations enter both its local estimator and the common server estimator, and yields fixed-block simultaneous inference across clients. Robustness is examined under two perturbation models. A Kullback--Leibler neighborhood provides a variance-sensitive lower certificate in a regular positive-gain-variance regime, while an $L_2$ neighborhood yields an exact least-favorable regression direction and robust value. Exact-model, vanishing-variance, and exact-zero-direction boundaries are kept separate because their scalar inputs are nonregular. The framework thus distinguishes decisions about a realized fitted pair from claims about the learning procedure under repeated fitting.

Figures & tables

		Gain		95% coverage				$\widehat{V}_{\rm fit}/V_{\rm fit}^{\rm MC}$
$\eta$	Regime	$G^{P}$	$\Gamma$	$G^{\rm fit}$	Correct	Indep.	Eval.	Correct	Indep.
0.00	Negative	$-0.255$	$-0.250$	0.950	0.938	0.901	0.756	0.980	0.626
0.25	Near-zero	$0.000$	$0.005$	0.953	0.947	0.925	0.729	0.967	0.740
1.00	Positive	$0.766$	$0.769$	0.951	0.951	0.977	0.591	0.938	1.354

Table 1: Two-client verification of realized-fit and procedure-level gain inference. “Correct” retains fitting uncertainty and local–server covariance; “Indep.” drops the shared-data covariance term; and “Eval.” uses evaluation variance only. Variance ratios compare the mean estimated fitting variance with its Monte Carlo benchmark.

$b_{L}$	$b_{F}$	$\Gamma$	$\Delta_{\rm bias}$	$\Delta_{\rm var}$	$D$	$\widehat{V}_{\rm fit}/V_{\rm fit}^{\rm MC}$	2-stage	Eval.
0.25	0.00	0.0639	0.0560	0.0079	$-0.753$	0.979	0.947	0.745
0.25	0.10	0.0462	0.0383	0.0079	$-0.658$	0.962	0.939	0.657
0.25	0.25	0.0089	0.0011	0.0079	$0.765$	0.917	0.926	0.521
0.50	0.00	0.2308	0.2228	0.0079	$-0.931$	1.025	0.946	0.773
0.50	0.10	0.1979	0.1899	0.0079	$-0.920$	1.025	0.945	0.738
0.50	0.25	0.1379	0.1299	0.0079	$-0.885$	1.023	0.950	0.641

Table 2: Complete grid for Experiment 1. Gain components are reported on the original scale. The mechanism score

D

is negative under bias-correction dominance and positive under variance-rescue dominance. The final two columns report empirical

95\%

coverage for the correct two-stage interval and for an interval using evaluation variance only.

Panel of Figure 1 — Figure 1: Regular-gain diagnostics for Experiment 1. (a) Phase diagram of bias correction and variance rescue over the $(b_{L},b_{F})$ grid. (b) Fitting, evaluation, and total variance calibration at $(b_{L},b_{F})=(0.25,0)$ .

Panel of Figure 2 — Figure 2: Decomposition of the marginal gain in Experiment 1. (a) Bias-correction contribution $\Delta_{\rm bias}$ . (b) Variance-rescue contribution $\Delta_{\rm var}$ .

Panel of Figure 3 — Figure 3: Exact-model boundary diagnostics. (a) Monte Carlo fitting variance compared with the quadratic-boundary and persistent-linear variance estimates. (b) Empirical $95\%$ procedure-level coverage using quadratic plus evaluation variance, persistent-linear plus evaluation variance, and evaluation variance alone.

Client	$\Gamma_{j}$	Mean $(Z_{j})$	SD $(Z_{j})$	Coverage	$\widehat{V}_{j}/V_{j}^{\rm MC}$
1	0.0491	$-0.081$	0.972	0.961	0.945
2	0.0280	$-0.026$	0.967	0.955	0.983
3	0.0891	$-0.009$	0.983	0.955	0.995
4	0.0196	$-0.024$	1.007	0.942	0.942
5	0.0658	$0.007$	1.012	0.952	0.983

Table 3: Marginal calibration for the five-client block in Experiment 3. The last column is the ratio of the mean estimated total variance to its Monte Carlo benchmark.

Figure 4: Client-indexed correlation calibration in Experiment 3. The left panel gives the Monte Carlo reference correlation matrix and the right panel gives the oracle-nuisance influence-based estimate. Diagonal entries equal one; the color scale summarizes the off-diagonal correlations generated by the shared federated estimator.

Event	Empirical	Gaussian MC	Gaussian est.	Independent
$\widehat{G}_{j}>0,\ \forall j$	0.921	0.889	0.896	0.877
$\widehat{G}_{j}>\Gamma_{j}-0.5s_{j},\ \forall j$	0.315	0.317	0.327	0.158

Table 4: Fixed-block orthant probabilities in Experiment 3. Gaussian approximations are computed using the Monte Carlo covariance, the estimated covariance, or a diagonal covariance that incorrectly treats clients as independent.

Figure 5: Oracle grid calibration of the sub-gamma scale used in Experiment 4. The calibrated envelope dominates the centered log-MGF over the displayed admissible grid.

Panel of Figure 6 — Figure 6: KL robustness and inference diagnostics for Experiment 4. (a) Exact empirical KL-robust gain and the variance-sensitive certificate. (b) Empirical probability that the estimated certificate is positive. (c) Empirical coverage of the one-sided $95\%$ lower confidence bound for the procedure-level marginal certificate.

$c_{\rho}$	$\rho$	$\Pr(\widehat{C}>0)$	LCB coverage
0.00	0.0000	1.000	0.933
0.25	0.0051	1.000	0.934
0.50	0.0101	1.000	0.936
1.00	0.0203	1.000	0.937
2.00	0.0405	0.986	0.937

Table 5: KL robustness and inference diagnostics for Experiment 4.

Panel of Figure 7 — Figure 7: $L_{2}$ robustness in the persistent-direction regime. (a) Monte Carlo mean fitted robust value and its audit estimate at $n=200$ over the radius grid. (b) Convergence of the estimated least-favorable regression function toward the population oracle at $c_{\tau}=1$ .

$n$	Mean $L_{2}$ error
100	0.1680
200	0.1202
400	0.0910
800	0.0699

Table 6: Convergence of the estimated least-favorable regression function in the persistent-direction regime.

Panel of Figure 8 — Figure 8: $L_{2}$ robustness at the vanishing-direction boundary. (a) Fitting-variance effect of the stage-specific coefficient $\kappa^{B}=\tau/\nu$ , reported relative to the nominal gain-boundary variance. (b) Variance of the fitted robust value standardized by the full vanishing-direction boundary scale.

Federated Learning

Federated learning trains shared models across data holders without requiring them to pool their raw data.

See all 951 papers

On the Gradient Heterogeneity Dynamics of Adversarially Robust Federated Regression
Sep 22, 2026Leonardo F. Toso, James Anderson, Nirupam Gupta +1Byzantine-Robust Federated Learning Non-IID Federated Learning
Columbia University, USA · University of Copenhagen, Denmark · Sorbonne Université and Université Paris Cité, CNRS, LPSM, France
Understanding Head Geometry and Dynamics in Federated Regression through a Natural Solution Selection Rule: An Unconstrained Feature Model Analysis
Sep 30, 2026Chuang Ma, Tomoyuki ObuchiFederated Learning Aggregation Federated Learning
Kyoto University · NII LLMC · RIKEN AIP
Range Penalization: Theoretical Insights with Applications in Federated Learning
Jun 9, 2026Yiyuan She, Zhaojun Hu, Yifan SunStatistical Learning Theory Clustered Federated Learning

Sep 22, 2026J/K move · Enter open · S save

On the Gradient Heterogeneity Dynamics of Adversarially Robust Federated Regression

Leonardo F. Toso, James Anderson, Nirupam Gupta, Rafael Pinot

Columbia University, USA · University of Copenhagen, Denmark · Sorbonne Université and Université Paris Cité, CNRS, LPSM, France

Byzantine-Robust Federated Learning Non-IID Federated Learning Statistical Learning Theory Federated Learning

Open paper AskPDF arXiv

Federated learning (FL) is intrinsically heterogeneous: honest clients may have different data-generating models. On top of that, adversarial clients can make heterogeneity even more pronounced by sharing arbitrary updates. Existing analyses typically control the interaction between statistical heterogeneity and adversarial behavior through gradient-dissimilarity conditions. However, the underlying bound is imposed a priori and may yield conservative guarantees even for least-squares regression. We instead derive the gradient heterogeneity from the statistical model of linear and nonlinear regression with fresh data samples at every round. Our bounds separate heterogeneity among the honest clients' ground-truth model parameters, finite-sample label noise, and initialization. We then demonstrate that, for any

(f,κ)

-robust aggregator with coefficient

κ= O(f/n)

, where

f

is the number of adversarial clients and

n

the total number of clients (with

f/n < 1/2

), convergence holds after an explicit sample burn-in.

Abstract

Federated Learning

Explore similar work

On the Gradient Heterogeneity Dynamics of Adversarially Robust Federated Regression

Understanding Head Geometry and Dynamics in Federated Regression through a Natural Solution Selection Rule: An Unconstrained Feature Model Analysis

Range Penalization: Theoretical Insights with Applications in Federated Learning