Asking the Crowd the Right Question: Bias-Cancelling Weights for Federated Learning

Ilya Kuruzov; Dmitrii Vishovan; Kirill Novoselov; Yuriy Dorn; Darina Dvinskikh; Alexander Gasnikov

math.OCOct 3, 2026

Asking the Crowd the Right Question: Bias-Cancelling Weights for Federated Learning

Authors: Ilya Kuruzov, Dmitrii Vishovan, Kirill Novoselov, Yuriy Dorn, Darina Dvinskikh, Alexander Gasnikov

Organizations: Innopolis University · Moscow Independent Research Institute of Artificial Intelligence · Lomonosov Moscow State University · Higher School of Economics · Trusted AI Research Center, RAS · Steklov Mathematical Institute, RAS

Abstract

A federated objective is a weighted sum of client risks, and the weights are almost always fixed in advance. We treat them instead as the only instrument of a wisdom-of-crowds mechanism: clients are noisy views of one truth, each seeing it through an independent distortion that is unbiased across the crowd. That the optimal weights are inversely proportional to the clients' error energies is classical; we begin at the question that answer presupposes, which energies belong there and whether a crowd can recover them from itself. Excess risk on the truth is of the exact order of the aggregate bias energy, so no optimizer can repair a bad weight vector; the truth itself is identifiable only up to a linear tilt, so subtracting estimated client biases provably reproduces uniform weighting. The expected per-client second moments, however, are exactly identified from the law of the crowd's disagreement by a well-conditioned linear inversion, a step random-effects meta-analysis cannot take because a source reports once; their realized counterparts are estimable up to an incoherence floor the algorithm can measure. This yields CROWD, which reads the disagreement off the optimization trajectory at no extra cost and matches a Bayesian minimax lower bound in the same constant: per instance as the horizon grows, and unconditionally as the prior becomes diffuse. For arbitrary distortions it stays competitive with the optimal weights, at a ratio governed by a geometric incoherence the algorithm can measure. On real scans split into sites with their own miscalibrated detectors it attains the oracle excess risk; on a companion federation that pulls bias and noise apart, weighting by noise variance is worse than not weighting at all, and CROWD is not.

Figures & tables

Panel of Figure 1 — Figure 1: One round of Crowd . The unknown $\nabla f_{0}(x^{k-1})$ cancels in the residual, so the disagreement is measured at any query point at no extra cost.

Figure 2

Model, trained end to end	clients	$\kappa$	$\widehat{\kappa}$	inv- $\sigma$	Crowd	oracle	Crowd $<$ unif.
two-layer ReLU, raw FEMNIST pixels	$10$ writers	$1.09$	$1.05$	$0.815$	$0.933$	$0.816$	$8/8$
	$20$ writers	$2.13$	$2.52$	$0.971$	$\mathbf{0.891}$	$0.970$	$7/8$
	$40$ writers, ampl.	$3.88$	$4.39$	$1.434$	$\mathbf{0.950}$	$1.331$	$4/6$
convolutional net, raw digit scans	$8$ sensor sites	$1.63$	$1.18$	$1.141$	$1.057$	$1.218$	$1/6$

Table 1: Neural networks trained end to end: excess risk relative to uniform weighting, medians (Appendix E.7 ). Crowd is unguarded;

\widehat{\kappa}

is the estimate the guard uses.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Defined in	Meaning
$n,d,b,K$	Sec. 3	clients, dimension, batch, samples per client
$f_{0},\ x_{0}^{\star},\ \mu,L$	Assum. A2	the truth, its optimum, its curvature bounds
$h_{i},\ f_{i}=f_{0}+h_{i}$	Assum. A1	client $i$ ’s perturbation and risk
$\ell_{i},\ \ell,\ \lambda_{w},\ L^{\prime}$	Assum. A1	curvature bound of $h_{i}$ ; $\ell=\max_{i}\ell_{i}$ ; $\lambda_{w}=\sup\lVert\nabla^{2}H_{w}\rVert$ ; $L^{\prime}=L+\lambda$
$\ell_{T}$	Thm. 5.11	logarithmic factor in the tuned stepsize, unrelated to $\ell$
$\beta_{i},\ q_{i},\ \zeta_{i}^{2}$	Assum. A1	realised tilt, realised energy $\lVert\beta_{i}\rVert^{2}$ , expected energy

Appendix

Table 5

Figure 3: Realised bias energy

\lVert b(w^{k})\rVert^{2}

against the round index on the default instance (

n=20

d=150

, quadratic

f_{0}

with spectrum geometric in

[0.5,5]

\log\zeta_{i}

and

\log\sigma_{i}

uniform with

\zeta_{i}\in[0.2,3]

and

\sigma_{i}\in[0.5,3]

b=1

; medians and interquartile bands over 12 seeds), all rules taking identical stochastic gradients and differing only in

w

. Crowd reaches the oracle level within

3.4\%

in a few tens of rounds; the oracle-selected best single client is

4.6\times

worse, as Proposition 3.4 (ii) implies. The companion excess-risk trajectories, with the exact floors

\tfrac{1}{2}b(w)^{\top}A^{-1}b(w)

, are Figure 2(a) .

Figure 4: Estimator accuracy on the default instance (12 seeds), error against the number of probing rounds

K_{0}

on log–log axes. (a) On the identifiable target

\lVert\widetilde{\beta}_{i}\rVert^{2}

the error decays at the predicted rate: measured log–log slope

-0.46

against the

-1/2

of Lemma 5.7 (i). (b) On

q_{i}

the error stalls at the incoherence floor of Lemma 5.4 , which does not decay with

K

at all; the floor falls by

4.9\times

when

d

grows by

16\times

, against the

4.0\times

that the

d^{-1/2}

of Lemma 5.6 predicts. The two individual

4\times

steps give

1.6\times

and

3.0\times

against

2.0\times

each, so the floor is of the predicted size and direction but not cleanly power-law at these dimensions.

Figure 5: Crowd applied outside its class:

n=20

K=3000

, 6 seeds, all rules taking identical gradients. (a) Non-linear crowds

\nabla h_{i}(x)=\beta_{i}\pm\tfrac{\lambda}{\omega}\sin(\omega(x-x^{\star}_{0}))

: the gain over uniform weights falls only from

6.7\times

5.4\times

\lambda

grows to

0.8\mu

, and Crowd stays within

1.00

–

1.03\times

of the oracle – including where Theorem 5.19 does not apply. (b) Correlated crowds: under pairwise correlation the gain stays between

5.7\times

and

6.1\times

, even where

\kappa

exceeds one, and the competitive ratio stays in

[1.07,1.42]

, while a common factor – the invisible kernel of Proposition 5.2 – erodes it smoothly.

Figure 6: The contribution of each of Crowd ’s two estimation devices, on the default instance with the crowd size

n

swept over

\{3,\dots,40\}

(12 seeds; medians). (a) Realised bias energy relative to the oracle. Fitting a single heterogeneity variance in the manner of random-effects meta-analysis is indistinguishable from uniform weighting at every

n

(

7.80

versus

7.80

n=3

): the gain comes from the per-client second moments. Dropping the inversion

T^{-1}

and using the raw disagreement energy costs

3.1\times

n=3

and

1.04\times

n=40

, decreasing with

n

C_{n}

does. (b) Estimated versus true

q_{i}

n=5

: the naive statistic is shrunk towards the crowd mean with measured slope

0.634

against the

1-2/n+1/n^{2}=0.64

that Lemma 5.4 predicts for a fit pooled over seeds, while Crowd ’s slope is

0.995

Figure 7: Optimisers with local steps against weights (

n=20

E=5

8

seeds, medians). Dotted lines are the exact floors of uniform (upper) and oracle (lower) weights. (a) Linear tilts, decaying step. (b) Client Hessians differ, constant local step.

Figure 8: Excess risk on the truth for different refresh sets

\mathcal{R}

, synthetic instance, medians over

12

seeds. All schedules of one seed see the same gradient noise.

Figure 9: Diagnostics. (a) Incoherence

\kappa

of the realised tilts under three mechanisms of heterogeneity, at fixed model dimension

d=240

(

5

seeds, medians and interquartile bands; dashed lines are the estimate

\widehat{\kappa}

of Proposition 5.23 , computed from residuals alone). (b) Crowd with the noise bound

\bar{\sigma}_{i}^{2}

inflated by a factor

\rho

, bias energy relative to the oracle,

12

seeds. (c) Median error of the estimated energies under uniform partial participation, inverting the rescaled map of Proposition 5.26 or the per-round map,

16

seeds.

Figure 10: The applicability window. (a) The largest crowd for which the realised

\kappa

still lies below one, against the number of independent causes of heterogeneity, for tilts drawn from a random subspace of that dimension (

7

seeds); the two markers are the label-channel and site-instrument mechanisms of Section E.2 , placed at their measured effective rank. (b) Estimated

\widehat{\kappa}

against true

\kappa

541

instances with

n=20

d=240

and

2500

probing rounds, in the linear regime. (c) UCI optical digits with

n=8

sites and per-site instrument distortion of varying strength (

4

seeds): true and estimated incoherence, and the bias energy of Crowd and of the oracle weights relative to uniform weighting.

Figure 11: UCI optical digits under two constructed federations (

n=8

8

seeds, medians). (a) Realised bias energy relative to the oracle weights, log scale. (b) Excess risk on

f_{0}

after

K=2500

rounds, relative to uniform weighting; below one is a gain. (c) The same ratio as the sensor federation grows (

4

seeds per

n

), with the measured and estimated incoherence on the right axis.

Mechanism	$\kappa$	$\widehat{\kappa}$	$\mathrm{AM}/\mathrm{HM}$	size	inv- $\sigma$	valid.	Crowd
Few pixels $+$ label channel	$1.34$ – $2.16$	$0.34$ – $0.45$	$1.4$ – $11.9$	—	—	—	$\geq 1^{\dagger}$
Whole sensor arrays	$0.48$	$0.52$	$86.9$	$1.069$	$0.778$	$0.918$	$\mathbf{0.787}$
Annotation conventions	$0.48$	$0.53$	$2.2$	$0.985$	$1.069$	$1.000$	$\mathbf{0.875}$

Appendix

Table 2: The three federations built on the same

1797

real scans. The last four columns are the excess risk on

f_{0}

after

K=2500

rounds, relative to uniform weighting: below one is a gain, and the oracle weights reach

0.789

and

0.934

in the second and third rows. Row one is the failure of Section E.3 (c); the other two are described below. † Bias energy relative to uniform weighting,

1.00

–

1.90

; those runs do not record the risk.

Figure 12: Sweeping the two knobs of the convention federation (

n=8

6

seeds, medians). (a) The common detector severity controls incoherence, the shaded band marking the region

\kappa<1

. (b) Excess risk relative to uniform weighting along that axis. (c) The same against the cohort-size contrast, which controls how far bias and sampling noise are pulled apart. Below one is a gain.

Writers	$\alpha$	$\kappa$	$\widehat{\kappa}$	uniform excess	inv- $\sigma$	Crowd	oracle	Crowd ${}<{}$ unif.
$10$	$1$	$0.52$	$0.50$	$0.70$	$0.904$	$\mathbf{0.897}$	$0.897$	$12/12$
$20$	$1$	$1.09$	$0.92$	$0.38$	$0.946$	$\mathbf{0.922}$	$0.909$	$10/12$
$40$	$1$	$2.07$	$1.87$	$0.19$	$0.973$	$0.991$	$0.980$	$6/12$
$80$	$1$	$4.08$	$3.54$	$0.10$	$1.106$	$1.112$	$1.102$	$0/12$
$20$	$1.41$	$1.10$	$0.93$	$0.51$	$0.925$	$\mathbf{0.922}$	$0.891$	$11/12$
$40$	$2$	$1.97$	$1.75$	$0.38$	$0.930$	$\mathbf{0.929}$	$0.920$	$10/12$

Appendix

Table 3: FEMNIST, medians over

12

seeds. The last four columns are the excess risk on

f_{0}

relative to uniform weighting and the number of seeds in which Crowd beats uniform weighting; “uniform excess” is the absolute excess risk of uniform weighting. Size-proportional weighting, validation tuning and the common-

\zeta^{2}

fit stay within

0.01

of uniform weighting in every row.

Explore similar work

May 12, 2026cs.LG

FedOUI: OUI-Guided Client Weighting for Federated Aggregation

Federated learning usually aggregates client updates using dataset size or gradient-level criteria, while overlooking internal signals about how each client model is organizing its input space during training. We introduce FedOUI, a simple aggregation rule based on the Overfitting-Underfitting Indicator (OUI), an activation-based and label-free metric. Each participating client sends its local update together with a OUI value computed on a fixed probe batch, and the server estimates the round-wise OUI distribution to assign lower weights to structurally atypical clients through a smooth reweighting rule. We evaluate FedOUI on CIFAR-10 under strong non-IID partitioning and noisy-client conditions, comparing it with FedAvg, FedProx, and a gradient-alignment baseline. The clearest gains appear under strong heterogeneity, where OUI-based weighting improves aggregation quality while remaining lightweight and interpretable. These results show that internal activation structure can provide useful information for federated aggregation beyond client size and gradient geometry.

Alberto Fernández-Hernández, Jose I. Mestre, Cristian Pérez-Corral +3

Universitat Politècnica de València, Valencia, Spain · Universitat Jaume I, Castelló de la Plana, Spain · Openchip & Software Technologies S.L., Spain

Federated Learning Aggregation Non-IID Federated Learning Federated Learning Heterogeneous Federated Learning

Ask PDF

Apr 29, 2026cs.LG

Who Trains Matters: Federated Learning under Enrollment and Participation Selection Biases

Federated learning (FL) trains a shared model from updates contributed by distributed clients, often implicitly assuming that contributing clients are representative of the target population. In practice, this representativeness assumption can fail at two distinct stages, inducing selection bias. First, eligibility rules such as device constraints, software requirements, or user consent determine which clients are ever enrolled and reachable for training, inducing \emph{enrollment bias}. Second, among enrolled clients, user and system factors such as battery state, network status, and local time determine which clients participate in each communication round, inducing \emph{participation bias}. Although existing work has largely addressed round-level participation bias, it has paid far less attention to population-level enrollment bias, which can induce a persistent mismatch between the training objective and the target-population objective. We formalize FL under a two-stage selection model and derive \textsc{FedIPW}, an inverse-probability-weighted aggregation scheme that recovers the target-population mean update under standard ignorability and positivity assumptions. Because client-level covariates are often unavailable for non-enrolled clients, we also introduce a limited-information aggregate-calibration extension that uses known target-population summaries to reweight the enrolled sample, partially correcting enrollment bias. We further provide an algorithm-agnostic optimization analysis under residual weighting error and show that incomplete selection correction can induce a non-vanishing bias floor. Finally, experiments on synthetic federated logistic regression validate the predicted objective mismatch and show that enrollment correction reduces target-population error under two-stage selection.

Gota Morishita

Centre for Brain, Mind and Markets University of Melbourne Parkville, Victoria, Australia

Inverse Probability Weighting Federated Learning Aggregation Client Selection in Federated Learning Selection Bias Federated Learning

Ask PDF

Sep 14, 2026cs.LG

SWB-DM: A Calibrated Sliced-Wasserstein-Barycenter Aggregator with Delayed-Momentum Caching for Byzantine-Robust Federated Learning under Partial Participation

Robust aggregation methods for federated learning quietly rest on a fragile assumption: that whoever shows up in a given round is a fair sample of the full population. In practice, they rarely are. When only a handful of clients participate per round, even a modest fraction of adversaries can dominate that sample and silently invalidate the finite-sample guarantees that coordinate-wise median, Krum, Bulyan, and trimmed mean all depend on. We introduce SWB-DM to address this directly. SWB treats each slice of a client update as a one-dimensional distribution, computes a trimmed Wasserstein barycenter across clients, and recovers coordinate identity via a medoid-based gauge-fixing step -- a heuristic we developed and do not claim it belongs to standard optimal-transport theory. DeMoA-style delayed momentum then caches updates across the full client population each round, decoupling robustness from whoever happened to be sampled. Trim ratio calibration is not cosmetic: under-trimming causes collapse at corruption levels a properly calibrated model survives. Across 448 CIFAR-10 configurations, plus CIFAR-100, FEMNIST, and a 500-client scalability run, we find several mechanistically distinct failure modes. Even-sample coordinate-wise median degrades to a deterministic wrong answer. Krum silently violates its own n greater than 2f+2 precondition and diverges without warning. Bulyan's n greater than or equal to 4f+3 threshold produces a sharp pass/fail boundary. On attacks, IPM defeats order-statistic defenses -- including SWB -- more reliably than ALIE, confirmed through delta-space measurements against a convergence bound. SWB-DM's cache carries a real warm-up cost, but extending all baselines to the same round budget shows its CIFAR-10 gains are disproportionately large. On CIFAR-100, FLTrust benefits more -- for reasons entirely unrelated to caching.

Saranraj S, Saranya M S, Alex David S +1

Department of AIML Vel Tech Rangarajan Dr. Sagunthala R&D Institute of Science and Technology Chennai, India · Assistant Professor, Department of AIML Vel Tech Rangarajan Dr. Sagunthala R&D Institute of Science and Technology Chennai, India · Professor, Department of AIML Vel Tech Rangarajan Dr. Sagunthala R&D Institute of Science and Technology Chennai, India

Federated Learning Aggregation Byzantine-Robust Federated Learning Wasserstein Barycenters Federated Learning

Ask PDF