Beyond performance-wise Contribution Evaluation in Federated Learning

Balazs Pejo

cs.LGFeb 25, 2026

Beyond performance-wise Contribution Evaluation in Federated Learning

Authors: Balazs Pejo

Organizations: CrySyS Lab, Hungary

Abstract

Federated learning offers a privacy-friendly collaborative learning framework, yet its success, like any joint venture, hinges on the contributions of its participants. Existing client evaluation methods predominantly focus on model performance, such as accuracy or loss, which represents only one dimension of a machine learning model's overall utility. In contrast, this work investigates the critical, yet overlooked, issue of client contributions towards a model's trustworthiness -- specifically, its reliability (tolerance to noisy data), resilience (resistance to adversarial examples), and fairness (measured via demographic parity). To quantify these multifaceted contributions, we employ the state-of-the-art approximation of the Shapley value, a principled method for value attribution. Our results reveal that no single client excels across all dimensions, which are largely independent from each other, highlighting a critical flaw in current evaluation scheme: no single metric is adequate for comprehensive evaluation and equitable rewarding allocation.

Figures & tables

Symbol	Description
$k\in[K]$	FL clients
$M_{k}^{t}$ & $M^{t}$	The local and global models after round $t$
$D_{k}$ & $D^{\prime}$	Training & Testing datasets
$x$ & $y$	Data sample & label
$n$ & $a$	Gaussian & malicious noise
$v(\cdot)$	Evaluation function for scoring

Table 1: Notations use in the paper.

Figure 1: Example to illustrate what the different trustworthiness scores capture based on $D^{\prime}=\{(1,G),(2,R),(3,G),(4,R),(5,R),(6,G)\}$ . While the model is $0.67$ accurate, its trustworthiness scores are widely different, e.g., $\mathtt{fair}(M)=0.33$ when Demographic Parity is used where the square numbers (1,2,4) are protected and ’Red’ is the target class. Regarding reliability, the model might misclassify three (1,3,4) samples with various probabilities, resulting in $\mathtt{rel}(M)=0.09$ overall chance. Regarding resilience, the attacker is successful once (1) out of the four (1,2,5,6) correctly classified samples, i.e., $\mathtt{res}(M)=0.25$ . Note, in the paper we transform these trustworthy metrics (i.e., $m\rightarrow(1-m)$ ) to be similar to accuracy: higher values mean better model.

Panel of Figure 2 — Figure 2: Score evolution of the model as the training progresses for the non-IID scenario.

Panel of Figure 3 — Figure 3: Computed scores for the 4 non-IID clients.

Panel of Figure 4 — Figure 4: Score distributions of the 20 non-IID clients.

Setting			$\mathtt{fair}$		$\mathtt{rel}$		$\mathtt{res}$
Setting			$\Phi$	$L_{2}$	$\Phi$	$L_{2}$	$\Phi$	$L_{2}$
ADULT	4	I	-0.145	0.029	0.056	0.026	-0.021	0.025
	4	N	-0.162	0.053	-0.152	0.036	-0.127	0.050
	20	I	-0.000	0.004	0.080	0.003	-0.021	0.004
	20	N	-0.040	0.014	-0.042	0.006	-0.036	0.011
CIFAR	4	I	-0.007	0.037	-0.166	0.048	-0.250	0.058

Table 2: Comparing the

\mathtt{GTG}

-based performance score to the trustworthy scores where I and N stands for IID and non-IID, respectively.

Panel of Figure 5 — Figure 5: Heatmap of the pair-wise score correlations for the 4 non-IID client setting. .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Property	ADULT	CIFAR-10	IMDB
Train / Test	40K / 10K	30K / 8K	20K / 5K
Input shape	108 features	(32, 32, 3)	512 (USE)
Model	MLP	CNN	NN
Act. func.	ReLU + sigmoid	ReLU + softmax	ReLU + softmax
Optimizer	Adam	Adam	Adam
LR / DO	$1\times 10^{-3}$ /0.1/0.05	$1\times 10^{-3}$ /0.25/0.35/0.5	$3\times 10^{-5}$ /0.5/0.5

Appendix

Table 3: Datasets and model configurations used in the experiments.

Setting			$\mathtt{perf}$	$\mathtt{fair}$	$\mathtt{rel}$	$\mathtt{res}$
ADULT	4	IID	9.9 $e$ -7±0.0	2.4 $e$ -5±0.0	3.0 $e$ -7±0.0	2.3 $e$ -5±0.0
	4	non-IID	1.1 $e$ -4±0.0	1.5 $e$ -4±0.0	2.7 $e$ -6±0.0	1.0 $e$ -4±0.0
	20	IID	2.8 $e$ -7±0.0	1.6 $e$ -6±0.0	0.0±0.0	2.2 $e$ -6±0.0
	20	non-IID	2.0 $e$ -5±0.0	2.0 $e$ -4±0.0	1.4 $e$ -6±0.0	2.1 $e$ -4±0.0
CIFAR	4	IID	2.9 $e$ -4±0.0	5.4 $e$ -5±0.0	8.4 $e$ -4±0.0	8.8 $e$ -4±0.0
	4	non-IID	1.4 $e$ -3±0.0	8.9 $e$ -5±0.0	3.6 $e$ -3±5.0 $e$ -5	6.5 $e$ -3±1.1 $e$ -4

Appendix

Table 4: Scores fluctuations across training rounds.

Panel of Figure 6 — Figure 6: Score evolution of the model as the training progresses with IID distribution.

Panel of Figure 7 — Figure 7: Scores of the 4 IID clients.

Panel of Figure 8 — Figure 8: Scores of the 20 clients.

Panel of Figure 9 — Figure 9: Score distributions of the 20 IID clients.

Panel of Figure 10 — Figure 10: Score distributions of the 4 clients.

Setting			$\mathtt{fair}$		$\mathtt{rel}$		$\mathtt{res}$
Setting			$\Phi$	$L_{2}$	$\Phi$	$L_{2}$	$\Phi$	$L_{2}$
ADULT	4	I	0.0006	0.0099	0.0912	0.0017	-0.0982	0.0079
	4	N	-0.0451	0.1099	-0.2477	0.0481	-0.0309	0.0602
	20	I	-0.0655	0.0024	-0.0845	0.0006	-0.2608	0.0023
	20	N	-0.2091	0.0959	-0.3382	0.0238	0.0364	0.0745
CIFAR	4	I	0.0595	0.0269	-0.0190	0.0465	-0.1259	0.0401

Appendix

Table 5: Comparing the

\mathtt{L1O}

-based performance score to the trustworthy scores where I and N stands for IID and non-IID, respectively.

Panel of Figure 11 — Figure 11: Heatmap of the pair-wise score correlations for the 4 IID client setting. .

Panel of Figure 12 — Figure 12: Heatmap of the pair-wise csore correlations for the 20 client setting.

Federated Learning

Federated learning trains shared models across data holders without requiring them to pool their raw data.

See all 921 papers

Private and Robust Contribution Evaluation in Federated Learning
Feb 25, 2026Delio Jaramillo Velez, Gergely Biczok, Alexandre Graell i Amat +2Federated Learning Privacy-Preserving ML
University of La Laguna Tenerife, Spain · HUN-REN Hungarian Research Network Budapest, Hungary · Chalmers University Gotheburg, Sweden +2
Fairness-Aware Federated Learning with Trajectory Shapley Value
May 28, 2026Daniel Kuznetsov, Ziqi WangFederated Learning Aggregation Algorithmic Fairness
Faculty of Mathematics, Ecole Normale Supérieure Paris-Saclay, 4 Av. des Sciences, 91190 Gif-sur-Yvette, France. · Chair for Dynamics, Control, Machine Learning and Numerics – Alexander von Humboldt Professorship, Department of Mathematics, Friedrich-Alexander-Universität Erlangen-Nürnberg, Cauerstrasse 11, 91058 Erlangen, Germany.
Assessing the Impacts of Imperfect Datasets on Client Selections in Federated Learning
Aug 3, 2026Yuan-Heng Tsai, Li-Hsing Yen, Yan-Wei ChenAlgorithmic Fairness Client Selection in Federated Learning
Department of Computer Science, National Yang Ming Chiao Tung University, Hsinchu, Taiwan.

Feb 25, 2026J/K move · Enter open · S save

Private and Robust Contribution Evaluation in Federated Learning

Delio Jaramillo Velez, Gergely Biczok, Alexandre Graell i Amat, Johan Ostman+1

University of La Laguna Tenerife, Spain · HUN-REN Hungarian Research Network Budapest, Hungary · Chalmers University Gotheburg, Sweden+2

Federated Learning Privacy-Preserving ML Privacy-Preserving Federated Learning Algorithmic Fairness

Open paper AskPDF arXiv

Cross-silo federated learning allows multiple organizations to collaboratively train machine learning models without sharing raw data, but client updates can still leak sensitive information through inference attacks. Secure aggregation protects privacy by hiding individual updates, yet it complicates contribution evaluation, which is critical for fair rewards and detecting low-quality or malicious participants. Existing marginal-contribution methods, such as the Shapley value, are incompatible with secure aggregation, and practical alternatives, such as Leave-One-Out, are crude and rely on self-evaluation. We introduce two marginal-difference contribution scores compatible with secure aggregation. Fair-Private satisfies standard fairness axioms, while Everybody-Else eliminates self-evaluation and provides resistance to manipulation, addressing a largely overlooked vulnerability. We provide theoretical guarantees for fairness, privacy, robustness, and computational efficiency, and evaluate our methods on multiple medical image datasets and CIFAR10 in cross-silo settings. Our scores consistently outperform existing baselines, better approximate Shapley-induced client rankings, and improve downstream model performance as well as misbehavior detection. These results demonstrate that fairness, privacy, robustness, and practical utility can be achieved jointly in federated contribution evaluation, offering a principled solution for real-world cross-silo deployments.

Abstract

Federated Learning

Explore similar work

Private and Robust Contribution Evaluation in Federated Learning

Fairness-Aware Federated Learning with Trajectory Shapley Value

Assessing the Impacts of Imperfect Datasets on Client Selections in Federated Learning