Loss Gap Parity for Fairness in Heterogeneous Federated Learning

Brahim Erraji; Michaël Perrot; Aurélien Bellet

cs.LGMar 31, 2026

Loss Gap Parity for Fairness in Heterogeneous Federated Learning

Authors: Brahim Erraji, Michaël Perrot, Aurélien Bellet

Organizations: Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189 - CRIStAL, F-59000 Lille, France · Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189 - CRIStALF-59000 Lille, France · PreMeDICaL team, Inria, Idesp, Inserm, Université de Montpellier

Abstract

While clients may join federated learning to improve performance on data they rarely observe locally, they often remain self-interested, expecting the global model to perform well on their own data. This motivates an objective that ensures all clients achieve a similar loss gap -the difference in performance between the global model and the best model they could train using only their local data-. To this end, we propose EAGLE, a novel federated learning algorithm that explicitly regularizes the global model to minimize disparities in loss gaps across clients. Our approach is particularly effective in heterogeneous settings, where the optimal local models of the clients may be misaligned. Unlike existing methods that encourage loss parity, potentially degrading performance for many clients, EAGLE targets fairness in relative improvements. We provide theoretical convergence guarantees for EAGLE under non-convex loss functions, and characterize how its iterates perform relative to the standard federated learning objective using a novel heterogeneity measure. Empirically, we demonstrate that EAGLE reduces the disparity in loss gaps among clients by prioritizing those furthest from their local optimal loss, while maintaining competitive utility in both convex and non-convex cases compared to strong baselines.

Figures & tables

Figure 1: In this example, we use the AFL algorithm to enforce loss parity between the two clients, and our approach, EAGLE , to enforce loss gap parity. Enforcing loss parity favors

\text{client}~1

, which has noisier data and a more complex prediction task, resulting in poorer performance for

\text{client}~0

. In contrast, optimizing for loss gap parity handles this imbalance more effectively.

Panel of Figure 2 — Figure 2: EMNIST data: $\min\{L^{*}_{k}\}_{k\in[10]}$ (red) and $\max\{L^{*}_{k}\}_{k\in[10]}$ (blue) for a heterogeneous data split ( $\alpha=0.1$ ) and a homogeneous split ( $\alpha=2000$ ) using a linear model. As expected, the range of optimal local losses is larger for non-iid data.

Figure 3: Local losses and local gaps for EAGLE(

\lambda=2.0

), q-FFL (

q=5.0

) and AFL. The optimal local loss of

\text{client}~0

is nearly

0

by design and thus only its loss gap is visible in the plot. By balancing the losses across clients, q-FFL and AFL prioritize learning a good classifier for clients 1 and 2, which harms the performance of

\text{client}~0

. In contrast, by balancing the loss gaps across clients, EAGLE allows all clients to benefit equally from federated learning.

Figure 4: The distribution of loss gaps for the

10

clients for a heterogeneous split (

\alpha=0.5)

with CNN model. We notice that in the case of high non-iid data, EAGLE reduces the gap of the worst performing client in terms of loss and achieves less variance then other baselines.

Algorithm	$\alpha=0.1$
	$\max\limits_{k\in[K]}r_{k}(\theta)\downarrow$	$\min\limits_{k\in[K]}r_{k}(\theta)\downarrow$	accuracy $\uparrow$	$\mathbb{V}_{k\in[K]}r_{k}(\theta)\downarrow$
AFL	0.152 ( $\pm$ 0.207)	-0.477 ( $\pm$ 0.155)	0.687 ( $\pm$ 0.003)	0.037 ( $\pm$ 0.026)
FedAvg	0.106 ( $\pm$ 0.170)	-0.459 ( $\pm$ 0.211)	0.692 ( $\pm$ 0.004)	0.033 ( $\pm$ 0.021)
q-FFL $q=1.0$	0.104 ( $\pm$ 0.178)	-0.464 ( $\pm$ 0.193)	0.692 ( $\pm$ 0.002)	0.032 ( $\pm$ 0.021)
q-FFL $q=3.0$	0.109 ( $\pm$ 0.186)	-0.471 ( $\pm$ 0.178)	0.691 ( $\pm$ 0.004)	0.032 ( $\pm$ 0.022)
q-FFL $q=5.0$	0.118 ( $\pm$ 0.189)	-0.471 ( $\pm$ 0.169)	0.689 ( $\pm$ 0.004)	0.033 ( $\pm$ 0.023)

Table 1: Performance of different algorithms on EMNIST with a highly heterogeneous split (

\alpha=0.1

) and a linear model. EAGLE achieves the best performance for the worst-performing client in terms of loss gaps, while maintaining accuracy comparable to the other baselines.

Algorithm	$\max\limits_{k\in[K]}r_{k}(\theta)\downarrow$	$\min\limits_{k\in[K]}r_{k}(\theta)\downarrow$	accuracy $\uparrow$	$\mathbb{V}_{k\in[K]}r_{k}(\theta)\downarrow$
AFL	-0.016 $\pm$ 0.013	-0.822 $\pm$ 0.017	0.882 $\pm$ 0.003	0.083 $\pm$ 0.000
FedAvg	-0.083 $\pm$ 0.004	-0.877 $\pm$ 0.046	0.889 $\pm$ 0.007	0.081 $\pm$ 0.010
q-FFL $q=1.0$	-0.084 $\pm$ 0.004	-0.886 $\pm$ 0.022	0.889 $\pm$ 0.001	0.084 $\pm$ 0.003
q-FFL $q=3.0$	-0.062 $\pm$ 0.013	-0.883 $\pm$ 0.020	0.883 $\pm$ 0.005	0.088 $\pm$ 0.006
q-FFL $q=5.0$	-0.070 $\pm$ 0.003	-0.876 $\pm$ 0.018	0.879 $\pm$ 0.006	0.083 $\pm$ 0.005
EAGLE $\lambda=0.1$	-0.095 $\pm$ 0.003	-0.944 $\pm$ 0.018	0.897 $\pm$ 0.001	0.093 $\pm$ 0.005

Table 2: Performance of different algorithms on DirtyMNIST with a CNN model. EAGLE achieves the best performance for the worst-performing client in terms of loss gaps, while slightly improving accuracy comparable to the other baselines. Note that all approaches generally achieve good accuracy because clients with ambiguous examples provide gradients that are also useful for the clean data.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Notation	Description
$\lambda$	Regularization parameter to be controlled by the user
$\theta$	An arbitrary model in the space
$t$	counter of synchronization rounds
$\tau$	counter of local steps
$\theta^{(t,\tau)}_{k}$	Local model inside each client $k$ , after $t$ syncs and $\tau$ local steps.
$\bar{\theta}^{(t,\tau)}$	$\frac{1}{K}\sum_{k=1}^{K}\theta^{(t,\tau)}_{k}$

Appendix

Table 3: Table of notations and their descriptions

Figure 5: Synthetic data distributions of the three clients. Rotating data resulted in

\text{client}~1

and

\text{client}~2

sharing the same optimal separator which is different than the one for

\text{client}~0

. This difference resuls in conflicting gradient directions between local optimizers for different clients.

Figure 6: Training behavior on the synthetic dataset. AFL and q-FFL increase the training loss of

\text{client}~0

to match that of

\text{client}~2

, but fail to balance the losses of

\text{client}~1

and

\text{client}~2

, which share the same optimal model despite differing in data separability.

Algorithm	$\alpha=0.1$
	$\max\limits_{k\in[K]}r_{k}(\theta)\downarrow$	$\min\limits_{k\in[K]}r_{k}(\theta)\downarrow$	accuracy $\uparrow$	$\mathbb{V}_{k\in[K]}r_{k}(\theta)\downarrow$
AFL	0.152 ( $\pm$ 0.207)	-0.477 ( $\pm$ 0.155)	0.687 ( $\pm$ 0.003)	0.037 ( $\pm$ 0.026)
FedAvg	0.106 ( $\pm$ 0.170)	-0.459 ( $\pm$ 0.211)	0.692 ( $\pm$ 0.004)	0.033 ( $\pm$ 0.021)
q-FFL $q=1.0$	0.104 ( $\pm$ 0.178)	-0.464 ( $\pm$ 0.193)	0.692 ( $\pm$ 0.005)	0.032 ( $\pm$ 0.021)
q-FFL $q=3.0$	0.109 ( $\pm$ 0.186)	-0.471 ( $\pm$ 0.178)	0.691 ( $\pm$ 0.004)	0.032 ( $\pm$ 0.022)
q-FFL $q=5.0$	0.118 ( $\pm$ 0.189)	-0.471 ( $\pm$ 0.169)	0.689 ( $\pm$ 0.004)	0.033 ( $\pm$ 0.023)

Appendix

Table 4: Maximum, minimum and variance of loss gaps and accuracy different baselines with a linear model reported on the test split for a heterogeneous split

dir(\alpha=0.1)

with EMNIST dataset, results are aggregated over four independent repetitions.

Algorithm	$\alpha=0.5$
	$\max\limits_{k\in[K]}r_{k}(\theta)\downarrow$	$\min\limits_{k\in[K]}r_{k}(\theta)\downarrow$	accuracy $\uparrow$	$\mathbb{V}_{k\in[K]}r_{k}(\theta)\downarrow$
AFL	-0.949 ( $\pm$ 0.106)	-1.710 ( $\pm$ 0.054)	0.686 ( $\pm$ 0.004)	0.053 ( $\pm$ 0.012)
FedAvg	-0.961 ( $\pm$ 0.094)	-1.715 ( $\pm$ 0.044)	0.691 ( $\pm$ 0.004)	0.053 ( $\pm$ 0.014)
q-FFL $q=1$	-0.963 ( $\pm$ 0.096)	-1.717 ( $\pm$ 0.048)	0.690 ( $\pm$ 0.004)	0.053 ( $\pm$ 0.013)
q-FFL $q=3$	-0.965 ( $\pm$ 0.097)	-1.720 ( $\pm$ 0.051)	0.690 ( $\pm$ 0.004)	0.053 ( $\pm$ 0.013)
q-FFL $q=5.0$	-0.965 ( $\pm$ 0.098)	-1.723 ( $\pm$ 0.051)	0.690 ( $\pm$ 0.003)	0.053 ( $\pm$ 0.012)

Appendix

Table 5: Maximum, minimum and variance of loss gaps and accuracy different baselines with a linear model reported on the test split for a less heterogeneous split

dir(\alpha=0.5)

with EMNIST dataset, results are aggregated over four independent repetitions.

Algorithm	$\alpha=1.0$
	$\max\limits_{k\in[K]}r_{k}(\theta)\downarrow$	$\min\limits_{k\in[K]}r_{k}(\theta)\downarrow$	accuracy $\uparrow$	$\mathbb{V}_{k\in[K]}r_{k}(\theta)\downarrow$
AFL	-1.495 ( $\pm$ 0.141)	-2.011 ( $\pm$ 0.067)	0.688 ( $\pm$ 0.002)	0.028 ( $\pm$ 0.016)
FedAvg	-1.503 ( $\pm$ 0.132)	-2.015 ( $\pm$ 0.074)	0.693 ( $\pm$ 0.002)	0.028 ( $\pm$ 0.018)
q-FFL $q=1$	-1.505 ( $\pm$ 0.134)	-2.015 ( $\pm$ 0.072)	0.693 ( $\pm$ 0.002)	0.028 ( $\pm$ 0.017)
q-FFL $q=3$	-1.507 ( $\pm$ 0.135)	-2.016 ( $\pm$ 0.071)	0.692 ( $\pm$ 0.002)	0.028 ( $\pm$ 0.017)
q-FFL $q=5.0$	-1.508 ( $\pm$ 0.136)	-2.017 ( $\pm$ 0.070)	0.692 ( $\pm$ 0.002)	0.028 ( $\pm$ 0.017)

Appendix

Table 6: Maximum, minimum and variance of loss gaps and accuracy different baselines with a linear model reported on the test split for a moderately heterogeneous split

dir(\alpha=1.0)

with EMNIST dataset, results are aggregated over four independent repetitions.

Algorithm	$\alpha=2000.0$
	$\max\limits_{k\in[K]}r_{k}(\theta)\downarrow$	$\min\limits_{k\in[K]}r_{k}(\theta)\downarrow$	accuracy $\uparrow$	$\mathbb{V}_{k\in[K]}r_{k}(\theta)\downarrow$
AFL	-2.075 ( $\pm$ 0.126)	-2.571 ( $\pm$ 0.087)	0.692 ( $\pm$ 0.001)	0.017 ( $\pm$ 0.010)
FedAvg	-2.076 ( $\pm$ 0.126)	-2.570 ( $\pm$ 0.087)	0.693 ( $\pm$ 0.002)	0.017 ( $\pm$ 0.010)
q-FFL $q=1.0$	-2.072 ( $\pm$ 0.126)	-2.565 ( $\pm$ 0.087)	0.692 ( $\pm$ 0.002)	0.017 ( $\pm$ 0.010)
q-FFL $q=3.0$	-2.072 ( $\pm$ 0.126)	-2.566 ( $\pm$ 0.086)	0.692 ( $\pm$ 0.002)	0.017 ( $\pm$ 0.010)
q-FFL $q=5.0$	-2.072 ( $\pm$ 0.126)	-2.566 ( $\pm$ 0.086)	0.692 ( $\pm$ 0.001)	0.017 ( $\pm$ 0.010)

Appendix

Table 7: Maximum, minimum and variance of loss gaps and accuracy different baselines with a linear model reported on the test split for homogeneous split

dir(\alpha=2000.0)

(IID data) with EMNIST dataset, results are aggregated over four independent repetitions.

Algorithm	$\alpha=0.1$
	$\max\limits_{k\in[K]}r_{k}(\theta)\downarrow$	$\min\limits_{k\in[K]}r_{k}(\theta)\downarrow$	accuracy $\uparrow$	$\mathbb{V}_{k\in[K]}r_{k}(\theta)\downarrow$
AFL	0.144 ( $\pm$ 0.096)	-0.497 ( $\pm$ 0.159)	0.843 ( $\pm$ 0.001)	0.036 ( $\pm$ 0.018)
FedAvg	0.294 ( $\pm$ 0.196)	-0.512 ( $\pm$ 0.131)	0.844 ( $\pm$ 0.005)	0.047 ( $\pm$ 0.020)
q-FFL $q=1.0$	0.097 ( $\pm$ 0.067)	-0.473 ( $\pm$ 0.120)	0.835 ( $\pm$ 0.003)	0.028 ( $\pm$ 0.013)
q-FFL $q=3.0$	0.244 ( $\pm$ 0.089)	-0.330 ( $\pm$ 0.147)	0.800 ( $\pm$ 0.009)	0.030 ( $\pm$ 0.015)
q-FFL $q=5.0$	0.386 ( $\pm$ 0.089)	-0.201 ( $\pm$ 0.159)	0.765 ( $\pm$ 0.011)	0.032 ( $\pm$ 0.017)

Appendix

Table 8: Maximum, minimum and variance of loss gaps and accuracy different baselines with a CNN model reported on the test split for a heterogeneous split

dir(\alpha=0.1)

with EMNIST dataset, results are aggregated over four independent repetitions.

Algorithm	$\alpha=0.5$
	$\max\limits_{k\in[K]}r_{k}(\theta)\downarrow$	$\min\limits_{k\in[K]}r_{k}(\theta)\downarrow$	accuracy $\uparrow$	$\mathbb{V}_{k\in[K]}r_{k}(\theta)\downarrow$
AFL	-0.416 ( $\pm$ 0.132)	-1.480 ( $\pm$ 0.674)	0.841 ( $\pm$ 0.004)	0.110 ( $\pm$ 0.137)
FedAvg	-0.409 ( $\pm$ 0.130)	-1.477 ( $\pm$ 0.675)	0.840 ( $\pm$ 0.003)	0.112 ( $\pm$ 0.140)
q-FFL $q=1.0$	-0.534 ( $\pm$ 0.111)	-1.558 ( $\pm$ 0.670)	0.843 ( $\pm$ 0.002)	0.109 ( $\pm$ 0.141)
q-FFL $q=3.0$	-0.476 ( $\pm$ 0.115)	-1.524 ( $\pm$ 0.681)	0.833 ( $\pm$ 0.002)	0.114 ( $\pm$ 0.145)
q-FFL $q=5.0$	-0.416 ( $\pm$ 0.108)	-1.470 ( $\pm$ 0.679)	0.820 ( $\pm$ 0.005)	0.115 ( $\pm$ 0.144)

Appendix

Table 9: Maximum, minimum and variance of loss gaps and accuracy different baselines with a CNN model reported on the test split for a less heterogeneous split

dir(\alpha=0.5)

with EMNIST dataset, results are aggregated over four independent repetitions.

Algorithm	$\alpha=1$
	$\max\limits_{k\in[K]}r_{k}(\theta)\downarrow$	$\min\limits_{k\in[K]}r_{k}(\theta)\downarrow$	accuracy $\uparrow$	$\mathbb{V}_{k\in[K]}r_{k}(\theta)\downarrow$
AFL	-0.726 ( $\pm$ 0.078)	-1.367 ( $\pm$ 0.082)	0.845 ( $\pm$ 0.003)	0.038 ( $\pm$ 0.012)
FedAvg	-0.742 ( $\pm$ 0.052)	-1.400 ( $\pm$ 0.065)	0.843 ( $\pm$ 0.003)	0.039 ( $\pm$ 0.014)
q-FFL $q=1.0$	-0.760 ( $\pm$ 0.077)	-1.431 ( $\pm$ 0.096)	0.846 ( $\pm$ 0.004)	0.041 ( $\pm$ 0.016)
q-FFL $q=3.0$	-0.764 ( $\pm$ 0.079)	-1.437 ( $\pm$ 0.100)	0.842 ( $\pm$ 0.003)	0.042 ( $\pm$ 0.018)
q-FFL $q=5.0$	-0.745 ( $\pm$ 0.081)	-1.432 ( $\pm$ 0.102)	0.839 ( $\pm$ 0.004)	0.043 ( $\pm$ 0.018)

Appendix

Table 10: Maximum, minimum and variance of loss gaps and accuracy different baselines with CNN model reported on the test split for a moderately heterogeneous split

dir(\alpha=1.0)

with EMNIST dataset, results are aggregated over four independent repetitions

Algorithm	$\alpha=2000$
	$\max\limits_{k\in[K]}r_{k}(\theta)\downarrow$	$\min\limits_{k\in[K]}r_{k}(\theta)\downarrow$	accuracy $\uparrow$	$\mathbb{V}_{k\in[K]}r_{k}(\theta)\downarrow$
AFL	-1.140 ( $\pm$ 0.063)	-1.591 ( $\pm$ 0.060)	0.845 ( $\pm$ 0.002)	0.015 ( $\pm$ 0.002)
FedAvg	-1.140 ( $\pm$ 0.063)	-1.591 ( $\pm$ 0.060)	0.845 ( $\pm$ 0.002)	0.015 ( $\pm$ 0.002)
q-FFL $q=1.0$	-1.170 ( $\pm$ 0.053)	-1.615 ( $\pm$ 0.053)	0.849 ( $\pm$ 0.002)	0.015 ( $\pm$ 0.001)
q-FFL $q=3.0$	-1.180 ( $\pm$ 0.062)	-1.605 ( $\pm$ 0.069)	0.845 ( $\pm$ 0.004)	0.014 ( $\pm$ 0.002)
q-FFL $q=5.0$	-1.193 ( $\pm$ 0.040)	-1.618 ( $\pm$ 0.054)	0.846 ( $\pm$ 0.003)	0.014 ( $\pm$ 0.002)

Appendix

Table 11: Maximum, minimum and variance of loss gaps and accuracy different baselines with a CNN model reported on the test split for homogeneous split

dir(\alpha=2000.0)

(IID data) with EMNIST dataset, results are aggregated over four independent repetitions.

Federated Learning

Federated learning trains shared models across data holders without requiring them to pool their raw data.

See all 943 papers

FAIRVAR: Fair Federated Learning via Variance Regularization
Aug 16, 2025Zahra Kharaghani, Ali Dadras, Tommy LöfstedtAlgorithmic Fairness Federated Learning
Department of Computing Science, Umeå University, Umeå, Sweden · Department of Mathematics and Mathematical Statistics, Umeå University, Umeå, Sweden
FedPCD: Priority-Constrained Federated Aggregation
Oct 9, 2026Yousef Irshaid, Dara Varam, Mohamed I. AlHajriFederated Learning Federated Learning Aggregation
Department of Computer Science & Engineering, American University of Sharjah · Senseable City Laboratory, Massachusetts Institute of Technology
Heterogeneity-Aware Client Sampling for Optimal and Efficient Federated Learning
May 16, 2025Shudi Weng, Chao Ren, Ming Xiao +1Federated Learning Heterogeneous Federated Learning
Division of Information Science and Engineering, School of Electrical Engineering and Computer Science (EECS), KTH Royal Institute of Technology, 11428 Stockholm, Sweden

Aug 16, 2025J/K move · Enter open · S save

FAIRVAR: Fair Federated Learning via Variance Regularization

Zahra Kharaghani, Ali Dadras, Tommy Löfstedt

Department of Computing Science, Umeå University, Umeå, Sweden · Department of Mathematics and Mathematical Statistics, Umeå University, Umeå, Sweden

Algorithmic Fairness Federated Learning Heterogeneous Federated Learning

Open paper AskPDF arXiv

Federated learning (FL) allows collaborative training of machine learning models across multiple parties without sharing raw data. However, heterogeneous data can cause some clients to have disproportionate influence on the global model, leading to disparities in their performance. Fairness, understood as reducing these disparities, is therefore a crucial concern in FL and has been addressed in various ways. We studied performance equitable fairness in FL, where the goal is to minimize performance disparities across clients. We evaluated several existing fairness-aware methods and introduce here a new gradient-variance-regularized method, implemented in two variants: FairGrad (approximate) and FairGrad* (exact). We theoretically characterize the connections between these methods and, empirically, on heterogeneous benchmarks, show that FairGrad and FairGrad* consistently improve fairness by reducing variance in client accuracies, while maintaining competitive or improved mean performance compared to existing fairness-aware baselines.

Abstract

Federated Learning

Explore similar work

FAIRVAR: Fair Federated Learning via Variance Regularization

FedPCD: Priority-Constrained Federated Aggregation

Heterogeneity-Aware Client Sampling for Optimal and Efficient Federated Learning