Task2vec Readiness: Diagnostics for Federated Learning from Pre-Training Embeddings

Cristiano Mafuz; Rodrigo Silva

cs.LGApr 12, 2026

Task2vec Readiness: Diagnostics for Federated Learning from Pre-Training Embeddings

Authors: Cristiano Mafuz, Rodrigo Silva

Abstract

Federated learning (FL) performance is highly sensitive to heterogeneity across clients, yet practitioners lack reliable methods to anticipate how a federation will behave before training. We propose readiness indices, derived from Task2Vec embeddings, that quantifies the alignment of a federation prior to training and correlates with its eventual performance. Our approach computes unsupervised metrics -- such as cohesion, dispersion, and density -- directly from client embeddings. We evaluate these indices across diverse datasets (CIFAR-10, FEMNIST, PathMNIST, BloodMNIST) and client counts (10--20), under Dirichlet heterogeneity levels spanning $α\in \{0.05,\dots,5.0\}$ and FedAVG aggregation strategy. Correlation analyses show consistent and significant Pearson and Spearman coefficients between some of the Task2Vec-based readiness and final performance, with values often exceeding 0.9 across dataset $\times$ client configurations, validating this approach as a robust proxy for FL outcomes. These findings establish Task2Vec-based readiness as a principled, pre-training diagnostic for FL that may offer both predictive insight and actionable guidance for client selection in heterogeneous federations.

Figures & tables

Dataset	K	Cohesion	-Dispersion	Density	CDI	Average Entropy
bloodmnist	10	0.994 ( 0.000 )	0.794 ( 0.019 )	-0.605(0.112)	0.940 ( 0.001 )	0.825 ( 0.012 )
bloodmnist	20	0.958 ( 0.000 )	0.868 ( 0.005 )	0.227(0.589)	0.940 ( 0.001 )	0.881 ( 0.004 )
cifar10	10	0.964 ( 0.000 )	0.966 ( 0.000 )	0.157(0.710)	0.982 ( 0.000 )	0.950 ( 0.000 )
cifar10	20	0.845 ( 0.008 )	0.985 ( 0.000 )	-0.217(0.606)	0.952 ( 0.000 )	0.981 ( 0.000 )
femnist	10	0.953 ( 0.000 )	0.795 ( 0.018 )	-0.152(0.720)	0.852 ( 0.007 )	0.857 ( 0.007 )
femnist	20	0.989 ( 0.000 )	0.988 ( 0.000 )	-0.386(0.345)	0.991 ( 0.000 )	0.899 ( 0.002 )

Table 1: Pearson correlation between final performance and readiness metrics by Dataset×K

Dataset	K	Cohesion	-Dispersion	Density	CDI	Average Entropy
bloodmnist	10	0.976 ( 0.000 )	0.929 ( 0.001 )	-0.119(0.779)	0.976 ( 0.000 )	0.976 ( 0.000 )
bloodmnist	20	0.976 ( 0.000 )	0.976 ( 0.000 )	-0.238(0.570)	0.976 ( 0.000 )	0.976 ( 0.000 )
cifar10	10	0.952 ( 0.000 )	0.976 ( 0.000 )	-0.095(0.823)	0.976 ( 0.000 )	0.952 ( 0.000 )
cifar10	20	0.970 ( 0.000 )	0.970 ( 0.000 )	-0.323(0.435)	0.970 ( 0.000 )	0.970 ( 0.000 )
femnist	10	0.905 ( 0.002 )	0.786 ( 0.021 )	-0.119(0.779)	0.881 ( 0.004 )	0.952 ( 0.000 )
femnist	20	1.000 ( 0.000 )	0.905 ( 0.002 )	-0.429(0.289)	0.976 ( 0.000 )	1.000 ( 0.000 )

Table 2: Spearman correlation between final performance and readiness metrics by Dataset×K

Federated Learning

Federated learning trains shared models across data holders without requiring them to pool their raw data.

See all 943 papers

May 8, 2026

FLAM: Evaluating Model Performance with Aggregatable Measures in Federated Learning

Performance evaluation is essential for assessing the quality of machine learning (ML) models and guiding deployment decisions. In federated learning (FL), assessing the performance is challenging because data are distributed across participants. Consequently, the coordinator must rely on locally computed evaluation metrics and aggregate them to assess the global model. A key challenge is that common aggregation strategies, such as weighted averaging based on the local samples per participant, do not always produce the same results as centralized evaluation. Existing definitions of performance evaluation are largely tailored to accuracy and do not generalize to other metrics, leading to inconsistencies between participant-based and centralized evaluation. However, such discrepancies are inconsistent with the FL objective and lead to a wrong calculation of the metric. To address this issue, we examine the underlying reasons for these discrepancies and propose FLAM, a performance evaluation method based on aggregatable measures that yields the same results as centralized evaluation without the need for a global test dataset.

Fabian Stricker, Jose A. Peregrina, David Bermbach +1

Institute of Data-Centric Software Systems (IDSS), Hochschule Karlsruhe, Karlsruhe, Germany · Scalable Software Systems Research Group, TU Berlin, Berlin, Germany

Federated Learning Aggregation Federated Learning

Ask PDF

Feb 25, 2026

Beyond performance-wise Contribution Evaluation in Federated Learning

Federated learning offers a privacy-friendly collaborative learning framework, yet its success, like any joint venture, hinges on the contributions of its participants. Existing client evaluation methods predominantly focus on model performance, such as accuracy or loss, which represents only one dimension of a machine learning model's overall utility. In contrast, this work investigates the critical, yet overlooked, issue of client contributions towards a model's trustworthiness -- specifically, its reliability (tolerance to noisy data), resilience (resistance to adversarial examples), and fairness (measured via demographic parity). To quantify these multifaceted contributions, we employ the state-of-the-art approximation of the Shapley value, a principled method for value attribution. Our results reveal that no single client excels across all dimensions, which are largely independent from each other, highlighting a critical flaw in current evaluation scheme: no single metric is adequate for comprehensive evaluation and equitable rewarding allocation.

Balazs Pejo

CrySyS Lab, Hungary

Federated Learning

Ask PDF

Aug 3, 2026

Assessing the Impacts of Imperfect Datasets on Client Selections in Federated Learning

Federated learning (FL) is a popular distributed learning framework where multiple clients perform local training and a server aggregates the locally updated models. FL enables decentralized training while preserving the privacy of clients' datasets. However, non-independent and identically distributed (non-IID) or noisy datasets can lead to low model accuracy or high convergence latency. Precluding these clients through client selection may mitigate the problem, but heavily biased client selections may also degrade the learning performance. In this study, we first experimentally measure the impact of non-IID data (including skews in data quantity and label distribution), noisy data, and fairness in client selection on model accuracy and convergence. We then propose a privacy-preserving scoring method to assess each client's contribution in FL, with experiments conducted to demonstrate the effectiveness of the proposed assessment.

Yuan-Heng Tsai, Li-Hsing Yen, Yan-Wei Chen

Department of Computer Science, National Yang Ming Chiao Tung University, Hsinchu, Taiwan.

Algorithmic Fairness Client Selection in Federated Learning Non-IID Federated Learning Privacy-Preserving ML Federated Learning

Ask PDF

Abstract

Federated Learning

Explore similar work

FLAM: Evaluating Model Performance with Aggregatable Measures in Federated Learning

Beyond performance-wise Contribution Evaluation in Federated Learning

Assessing the Impacts of Imperfect Datasets on Client Selections in Federated Learning