SCOPE: Scalable Training-Free Semantic Coreset Selection for Long-Tailed Federated Learning

Md Anwar Hossen; Nathan R. Tallent; Luanzheng Guo; Ali Jannesary

cs.LGMar 13, 2026

SCOPE: Scalable Training-Free Semantic Coreset Selection for Long-Tailed Federated Learning

Authors: Md Anwar Hossen, Nathan R. Tallent, Luanzheng Guo, Ali Jannesary

Organizations: Iowa State University, Ames, Iowa, USA · Pacific Northwest National Laboratory, Richland, WA, USA

Abstract

Scientific discovery increasingly relies on federated learning over data from high-resolution instruments, and these data are strongly non-IID and long-tailed. Coreset selection could shorten training, but existing federated coreset methods either prune with local heuristics that ignore the global class distribution, or first train a warm-up model at every client and upload class centroids of its features. This makes coreset selection a distributed bottleneck that consumes GPU time at every client, grows with the model and the number of clients, and must be repeated for every architecture. We introduce SCOPE, named for Semantic Coreset using Orthogonal Projection Embeddings, a scalable, training-free federated coreset framework for long-tailed data. Each client scores its samples once with a frozen, compact vision-language model and uploads scalar statistics per class. The server merges these into a global class profile, which lets each client remove anomalies and prune redundant samples only from classes that are locally abundant but globally common. SCOPE therefore requires no warm-up training, needs a single exchange with the server, and uploads a payload that depends only on the number of classes. On GPU nodes of an HPC cluster, with four datasets, including scientific micrographs, and six baselines, SCOPE selects its coreset up to 7.7x faster and uploads up to 293x less data than the feature-based baseline. As the number of clients grows from 10 to 500, SCOPE's time before training grows by fewer seconds than any baseline's, and at 500 clients the baselines need 1.7 to 3.6x SCOPE's time. Trained on the coreset, the global model reaches 98% of the full-data accuracy in 18 to 42% fewer global rounds than full-data training, with 8 to 21% less client computation over 200 rounds.

Figures & tables

Figure 1: Scaling with the number of clients

K

on Tiny-ImageNet, total data fixed. (a) Time before training of all clients: selection, warm-up and one scorer load per client included, plus transfer at 200 Gb/s; labels give the ratio to SCOPE at

K=500

. FedCore uses the initial model and is faster only at 10 clients. (b) Data exchanged before training: SCOPE sends a few statistics per class, the baselines full models; FedCore exchanges nothing.

Method	Warm-up	Time	Scaling, $K{=}10\to 500$	Reuse
EL2N	5	797 s	962 $\to$ 2,736 s	No
GraND	5	840 s	963 $\to$ 2,355 s	No
Forgetting	1	920 s	966 $\to$ 3,721 s	No
FedCore	5	1,078 s	105 $\to$ 1,736 s	No
FedCS	1	583 s	355 $\to$ 3,729 s	No
SCOPE	0	321 s	233 $\to$ 1,049 s	Yes

Table 1: Coreset selection before training on Tiny-ImageNet with ResNet-50. Warm-up is the number of FedAvg rounds before selection, and Time is the GPU time of selection, warm-up included, of 100 clients with imbalance ratio

IR=5

. Scaling is the time before training, transfer included, for 10 and 500 clients with the total data fixed at

IR=2

(Fig. 1 a). SCOPE’s times include one scorer load per client. FedCore’s Scaling uses the initial model, its Warm-up and Time the model after five warm-up rounds. Reuse means that one coreset serves every training model.

Figure 2: Overview of SCOPE. (a) Each client computes the scores

RS

DS

, and

S_{neg}

of its samples with a frozen MobileCLIP-S2 model and uploads per-class statistics. (b) The server merges them into a global profile. (c) The Consensus Filter removes semantic anomalies. (d) The Balancing Filter removes redundant samples of classes that are locally abundant and globally common. (e) The remaining samples form the local coreset used in federated training.

Figure 3: Samples removed by the two filters on one client (CIFAR-100,

p_{f}=0.5

p_{l}=0.1

), plotted by representation score (RS) against boundary proximity (

S_{\text{neg}}

). In the shaded Anomaly Zone above the dashed line

RS=S_{\text{neg}}

, a sample is closer to another class than to its own. The axes show raw cosine similarities; the Consensus Filter ranks samples by the standardized score of Eq. ( 4 ).

Method	Ep	Compute (s)	Energy (kJ)	Exch.	Upload	Download	Transfer (s)	Rounds	Compute (s)	Upload (GB)	Transfer (s)
	Coreset selection							Training to 95 / 98% of Full DB
Full DB	–	–	–	–	–	–	–	51 / 109	639 / 1,366	48.9 / 104.5	3.9 / 8.4
Random	0	67	–	0	0	0	0	83 / 147	900 / 1,542	79.6 / 140.9	6.4 / 11.3
EL2N	25	797	102.6	5	47.9 GB	47.9 GB	3.84	58.5 / 108	1,376 / 1,866	104.0 / 151.5	8.3 / 12.1
GraND	25	840	106.1	5	47.9 GB	47.9 GB	3.84	75.5 / 119	1,565 / 1,982	120.3 / 162.0	9.6 / 13.0
FedCS	5	583	52.0	2	9.6 GB	9.6 GB	0.77	57 / 106	1,145 / 1,627	64.3 / 111.2	5.1 / 8.9

Table 2: Cost of coreset selection and of training to a target accuracy on Tiny-ImageNet (

IR=5

p_{f}=0.5

), with 100 clients and 10 sampled per round. The left part gives the selection cost of all clients before training. The right part gives the rounds, GPU time, upload, and transfer time needed to reach 95 and 98% of the Full DB accuracy, with selection included. Transfer times use the 200 Gb/s line rate of the Delta network without latency, so they are lower bounds. SCOPE’s compute includes one scorer load per client; its energy does not. Bold marks the best coreset method; Random scores nothing and is not ranked in the selection columns; energy is from one seed.

Figure 4: Coreset selection cost for different training models on CIFAR-100. FedCS selects with the model being trained, so it must warm the model up and pass every sample through it, while SCOPE runs one pass of the same frozen MobileCLIP-S2 encoder for every model. (a) Computation per sample; (b) GPU time of selection, all clients; (c) peak GPU memory. Labels give FedCS’s cost relative to SCOPE’s. Lower is better in all panels.

	CIFAR-10		CIFAR-10		UHCS		UHCS		TIN, $\alpha{=}1$		TIN		CIFAR-100		CIFAR-100
	$IR=2$		$IR=10$		$IR=2$		$IR=10$		$IR=2$		$IR=5$		$IR=2$		$IR=10$
$p_{f}$	0.1	0.9	0.1	0.9	0.1	0.5	0.1	0.5	0.5	0.9	0.5	0.9	0.1	0.5	0.1	0.5
Random	41.69	46.50	44.84	41.91	88.52	86.61	91.53	92.08	57.36	55.67	53.89	52.50	84.76	84.15	80.13	79.98
Forgetting	55.49	46.61	44.10	div.	84.43	86.34	91.26	89.89	58.15	56.57	54.21	54.04	84.51	83.76	80.55	80.30
EL2N	47.27	42.19	37.54	40.58	85.25	81.15	92.62	88.52	59.13	58.18	54.34	53.12	83.80	82.56	75.93	76.81
GraND	48.44	39.83	36.53	15.93	82.24	78.96	91.53	92.08	58.88	58.33	54.14	52.58	83.80	82.57	75.96	77.15

Table 3: Accuracy (%) of the global model trained on each coreset with

p_{l}=0.1

, at two of the three pruning rates per dataset. SCOPE is the best coreset method in 19 of all 24 settings. CIFAR-10 and UHCS use 10 clients; Tiny-ImageNet (TIN) and CIFAR-100 use 100 clients with 10 per round;

\alpha=0.1

unless stated. Bold marks the best coreset method and div. a diverged run; Forgetting on CIFAR-10 and Random on CIFAR-100 use one seed. UHCS accuracy moves in steps of 0.27.

Figure 5: Communication time on the 200 Gb/s Delta network, up and down, on Tiny-ImageNet, a lower bound without latency. (a) Before training, for the data of Fig. 1 b. (b) Until 98% of the Full DB accuracy; dark: coreset selection, light: training rounds. Lower is better.

Figure 6: Accuracy of the global model trained on each coreset. (a) Accuracy relative to Full DB under large pruning,

p_{f}=0.9

; C10: CIFAR-10, TIN: Tiny-ImageNet, div.: diverged. (b) Mean accuracy of the 10% of clients with the lowest accuracy at

p_{f}=0.9

, one seed. (c) Test accuracy over global rounds on UHCS with

IR=10

and

p_{f}=0.5

; FedCore uses five warm-up rounds. Higher is better in all panels.

Figure 7: (a) Selection time, one seed, to provide coresets for three training models on Tiny-ImageNet; each baseline selects again for every model, while SCOPE selects once, including one scorer load per client, and labels give the ratio to SCOPE. (b) Accuracy and selection time of different scorer backbones on Tiny-ImageNet; marker area shows peak GPU memory. (c, d) Accuracy on CIFAR-10 and Tiny-ImageNet when the global profile, the Consensus Filter, or the Balancing Filter is removed.

Configuration	Acc. (%)	Rounds to 98%	Comp.
FedAvg, $E$ =5	54.4 / 54.5 / 54.0	109 / 106 / 83.5	$-8\%$
FedAvg, $E$ =20	47.9 / 45.7 / 47.6	95 / – / 78	$-21\%$
FedProx, $E$ =5	54.4 / 54.8 / 54.3	95 / 89 / 55	$-14\%$

Table 4: Robustness of the training result on Tiny-ImageNet at

p_{f}=0.5

under different local training. Accuracy and global rounds to 98% of the Full DB accuracy are given for Full DB / FedCS / SCOPE, where – means not reached; both come from a separate set of runs that log every round, so accuracies differ slightly from Table 3 . Comp. is SCOPE’s client computation over 200 rounds, selection included, relative to Full DB. The runs with

E=20

and FedProx use one seed.

Figure 8: Selection parameters and accuracy on a long-tailed federation. (a) Share of the 20% rarest classes of Tiny-ImageNet that SCOPE keeps at

p_{f}=0.9

, against the rarity exponent

\gamma

for three targeting margins

\beta

. (b) Accuracy on UHCS with

IR=10

as the pruning rate grows.

GPUs (nodes)	Clients per GPU	Wall (s)	Speedup	Efficiency
1 (1)	100	292.1	$1.0\times$	–
4 (1)	25	72.3	$4.0\times$	101%
8 (2)	12–13	46.5	$6.3\times$	79%

Table 5: Wall-clock time of SCOPE’s selection for all 100 TIN clients, 49,754 images, spread over 1, 4, or 8 GPUs; each GPU scores its clients one after another. The time covers loading the scorer, reading the images from the shared file system, scoring, exchanging the statistics, and pruning. Every configuration uploads 272 KB and keeps the same coreset.

$p_{f}$	$\beta$	$\gamma=0$	$\gamma=0.35$	$\gamma=0.7$	$\gamma=1.4$
0.5	0.3	80.6 / 88.5	80.9 / 89.0	81.2 / 89.0	81.9 / 89.0
	0.5	76.5 / 87.8	77.4 / 88.7	78.3 / 89.0	79.2 / 89.0
	0.9	57.0 / 69.6	58.7 / 75.7	60.6 / 80.9	63.4 / 87.7
0.9	0.3	72.9 / 88.1	73.6 / 89.0	74.1 / 89.0	75.4 / 89.0
	0.5	65.6 / 86.9	67.3 / 88.5	68.8 / 89.0	70.4 / 89.0
	0.9	32.0 / 56.3	34.7 / 65.8	37.8 / 74.7	42.6 / 86.8

Table 6: Data kept (%) by SCOPE’s selection on Tiny-ImageNet (

IR=5

p_{l}=0.1

): all classes / 20% rarest classes, for the pruning rate

p_{f}

, the targeting margin

\beta

, and the rarity exponent

\gamma

(

\gamma=0

: rarity weight off). Bold: defaults.

Method	ResNet-50	Swin-Tiny	ViT-Tiny	DeiT-Tiny	EffNet-B0
GraND	53.48	67.19	43.71	41.48	35.20
EL2N	53.35	66.76	44.22	43.79	35.48
FedCS	54.90	66.85	45.05	43.35	36.71
FedCore	55.06	66.47	44.52	45.04	38.69
SCOPE	55.68	67.17	45.27	46.48	38.72

Table 7: Accuracy (%) of different architectures trained on one SCOPE coreset, selected once with MobileCLIP-S2, on Tiny-ImageNet with

IR=5

and

p_{f}=0.7

. The baselines select their coresets with ResNet-50.

Setting	$\eta$	Prec.	Rec.	Acc.	$\Delta$	Med. $\rho_{k}$	Drift, raw $\to$ core
		Label noise				Transport and drift
CIFAR-10, $IR{=}10$	0.1	59.4	58.5	37.73 / 42.26	$+4.5$	0.95	15.38 $\to$ 7.53
CIFAR-10, $IR{=}10$	0.2	66.1	67.0	35.67 / 38.86	$+3.2$	0.95	15.38 $\to$ 7.53
UHCS, $IR{=}10$	0.1	18.0	12.9	87.43 / 88.25	$+0.8$	1.09	1.36 $\to$ 1.50
UHCS, $IR{=}10$	0.2	32.7	26.5	73.77 / 73.50	$-0.3$	1.09	1.36 $\to$ 1.50

Table 8: Diagnostics of the two filters at

p_{f}=0.5

and

p_{l}=\eta

, one seed. Left, Consensus Filter with a fraction

\eta

of labels flipped: precision and recall of its removals, and accuracy of FedCS / SCOPE. Right, Balancing Filter: median contraction

\rho_{k}

of the client-to-global transport distance, and client drift before and after selection.

Federated Learning

Federated learning trains shared models across data holders without requiring them to pool their raw data.

See all 938 papers

May 20, 2026

Choose Wisely and Privately: Proactive Client Selection for Fair and Efficient Federated Learning

Federated Learning enables collaborative model training across decentralized data sources without data transfer. Averaging-based FL is limited by the presence of non-IID data, which negatively impacts convergence speed and final model accuracy. Conventional alternatives suffer from significant inefficiency. Clients with noisy or highly heterogeneous data contribute expensive gradient computations that are either discarded or heavily down-weighted before aggregation. These reactive approaches waste computational resources, require more communication rounds and result in unnecessary privacy exposure. In this paper, we propose a proactive client selection framework that aims to find an optimal federation of clients whose combined data match utility and fairness requirements before training begins. Our method relies on mutual information computed from differentially private contingency tables to quantify the relevance of cross-feature correlations in the union dataset. We introduce a Potential Federation Loss (PFL) over the set of fixed-size federations, which balances two objectives. Maximizing collective data utility while ensuring fair cross-features correlations to prevent group unfairness. Client selection is expressed as an optimal subset search problem over the PFL objective, which we solve using simulated annealing under strong differential privacy guarantees for clients' local statistics. Experimental results on four benchmarks show faster, fairer, and more accurate models trained on optimally found federations, compared to uniform sampling, even when state-of-the-art adaptive aggregation or sampling strategies are employed.

Adda Akram Bendoukha, Heber Hwang Arcolezi, Nesrine Kaaniche +1

Samovar, T´el´ecom SudParis, Institut Polytechnique de Paris, France · ´ETS Montr´eal · CEA-LIST

Client Selection in Federated Learning Non-IID Federated Learning Federated Learning

Ask PDF

Jun 9, 2026

Accurate and Resource-Efficient Federated Continual Learning

Federated continual learning (FCL) must learn from distributed task streams under limited resources, such as communication, computation, memory, and label availability. Existing FCL methods often rely on repeated local optimization, replay, and full supervision. Analytic alternatives avoid iterative training and replay, but using high-dimensional random features to improve accuracy requires a second-order feature statistic, the Gram matrix, which has a quadratic communication cost in the random feature size

M

. We propose FedRAN, a resource-aware analytic FCL framework that replaces gradient-based updates with compact random feature statistics. Each client transmits a truncated-SVD summary of its Gram matrix, reducing the dominant second-order upload from quadratic to linear in

M

for fixed rank. The server performs a two-level QR-SVD subspace merge, spatially across clients and temporally across tasks, and solves a ridge classifier in closed form. FedRAN further supports label scarcity through prototype-based pseudo-labeling. Across CIFAR-100, ImageNet-R, and VTAB datasets, FedRAN improves average accuracy by up to 4.8 percentage points over the strongest baseline, uses 30.6-121.8

\times

less per-client communication than optimization-based FCL, and is 190.3

\times

faster on average than gradient-based baselines; with only 20% labels, pseudo-labeling improves average accuracy by up to 6.61 points. These results show that FedRAN enables accurate and resource-efficient FCL under communication, computation, and label constraints. The source code is available at https://github.com/JebacyrilArockiaraj/Fed-RAN-SSL.

Jebacyril Arockiaraj, Dhruv Parikh, Jayashree Adivarahan +2

University of Southern California · Los Angeles, USA · DEVCOM Army Research Office +1

Communication-Efficient Distributed Training Continual Learning Federated Continual Learning Random Feature Methods Federated Learning Data-Efficient Learning

Ask PDF

Aug 19, 2026

Coordination on a Budget: Federated Active Learning with Few Labels

Federated Active Learning (FAL) addresses the dual challenges of data privacy and label scarcity, where the absence of a global data view introduces additional hurdles for coordinated query selection. We study cross-silo FAL in the low-budget regime, where annotation decisions are most critical. We characterize, both theoretically and empirically, a heterogeneity reversal: in low-budget settings, homogeneous (IID) data requires stronger coordination to avoid redundant queries, whereas heterogeneous data naturally promotes diversity; this trend reverses at higher budgets. Thus, in contrast to the standard federated learning (FL) narrative where heterogeneity is a primary challenge, we show that IID settings are more challenging for query selection in FAL. Motivated by these findings, we propose a new FAL framework that utilizes federated representation learning to align client data in a shared embedding space. This enables the server to perform globally coordinated active selection over optionally obfuscated client embeddings, while annotation remains local to each client. Although our framework operates in the more challenging low-budget regime, it achieves performance that surpasses existing FAL methods even when they are given substantially larger annotation budgets, demonstrating the value of centralized coordination under privacy constraints.

Liam Mohr, Daphna Weinshall

School of Computer Science and Engineering The Hebrew University of Jerusalem, Jerusalem 91904, Israel

Active Learning Privacy-Preserving ML Federated Learning

Ask PDF

Abstract

Federated Learning

Explore similar work

Choose Wisely and Privately: Proactive Client Selection for Fair and Efficient Federated Learning

Accurate and Resource-Efficient Federated Continual Learning

Coordination on a Budget: Federated Active Learning with Few Labels